Collections Cache

The aim of the Collections Cache is to cache the current state of the resources so that they can be easily retrieved without having to query the sites every time. The cache will only be used by the Resource Manager when processing GET requests that return collections, e.g. /locations/be-ibbt/computes. GET requests to single resources will always be forwarded to the specific site, e.g. /locations/be-ibbt/computes/2434. The Collections Cache will only store and return limited information on the specific resources, this is called the condensed view. BonFIRE resource properties fall into three categories:

  • static: fixed and known at creation time (in the OCCI reply)
  • assign once: fixed when set but not set at creation time
  • dynamic: can change at any time

The Collections Cache will support almost all of the static properties and almost none of the assigned once properties. The only dynamic property this version will support is the resource state. In general only the information needed by the Portal is cached.

APIs provided

There is only one kind of API call that can be invoked on the Collections Cache:

GET /cache?userId={user_id}&{resource_id}(&{resource_id})*

The call will return a collection of condensed views of the resources identified by the URL query list ({resource_id}). The {user_id} is the user that requests the collection. All these query attributes are provided by the Resource Manager.

<?xml version="1.0" encoding="UTF-8"?>
<collection xmlns="http://api.bonfire-project.eu/doc/schemas/occi">
  <networks>
    ... condensed view ...
  </networks>
  <computes>
    ... condensed view ...
  </computes>
  <storages>
    ... condensed view ...
  </storages>
  <routers>
    ... condensed view ...
  </routers>
  <site_links>
    ... condensed view ...
  </site_links>
</collection>

Condensed View

Networks

Default Network
<network href="{id}" name="{name}">
  <name>{name}</name>
  <uname>{creator}</uname>

  <address>{address}</address>?
  (<netmask>{netmask}</netmask>
    |<size>{size}</size>)?
  <vlan>{vlan}</vlan>?

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</network>
Managed Network
<network href="{id}" name="{name}">
  <name>{name}</name>
  <uname>{creator}</uname>

  <address>{address}</address>?
  (<netmask>{netmask}</netmask>
    |<size>{size}</size>)?
  <vlan>{vlan}</vlan>?

  <state>{state}</state>
  <bandwidth>{bandwidth}</bandwidth>
  <latency>{latency}</latency>
  <lossrate>{lossrate}</lossrate>

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</network>
Active Network
<network href="{id}" name="{name}">
  <name>{name}</name>
  <uname>{creator}</uname>

  <address>{address}</address>
  (<netmask>{netmask}</netmask>
    |<size>{size}</size>)
  <vlan>{vlan}</vlan>?

  <state>{state}</state>
  <bandwidth>{bandwidth}</bandwidth>
  <latency>{latency}</latency>
  <lossrate>{lossrate}</lossrate>

  <throughput>{throughput}</throughput>
  <protocol>{protocol}</protocol>
  <packetsize>{packetsize}</packetsize>

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</network>
Federica Network
<network href="{id}" name="{name}">
  <name>{name}</name>
  <status>{state}</status>

  <network_link>
    <endpoint>
      <router href="{router_id}" name="{router_name}"/> <!-- reference view of router -->
      <router_interface>{router_interface}</router_interface>
    </endpoint>+
  </network_link>+

  <vlan>{vlan}</vlan>?

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</network>
Parameters
  • static: id, name, local_id, creator, address, netmask, size, protocol, location_id, experiment_id
  • assign once: vlan
  • dynamic: state

Storages

<storage href="{id}" name="{name}">
  <name>{name}</name>
  <user_id>{creator}</user_id>
  <state>{state}</state>
  <type>{type}</type>

  <size>{size}</size>?
  <fstype>{fstype}</fstype>?
  <persistent>{persistent}</persistent>?

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</storage>
Parameters
  • static: id, name, local_id, creator, type, size, fstype, persistent, location_id, experiment_id
  • dynamic: state

Computes

<compute href="{id}" name="{name}">
  <name>{name}</name>
  <uname>{creator}</uname>

  <cpu>{cpu}</cpu>
  <memory>{memory}</memory>
  <instance_type href="{instance_type_id}">{instance_type_name}</instance_type>
  <state>{state}</state>

  <disk id="{disk_id}">
    <storage href="{storage_id}" name="{storage_name}"/> <!-- reference view of storage -->
    <type>{disk_type}</type>
    <target>{disk_target}</target>
  </disk>+

  <nic>
    <network href="{network_id}" name="{network_name}"/> <!-- reference view of network -->
    <ip>{nic_ip}</ip>
    <mac>{nic_mac}</mac>
  </nic>+

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>

</compute>
Parameters
  • static: id, name, local_id, creator, cpu, memory, instance_type_id, instance_type_name, disk_id, storage_id, storage_name, disk_type, disk_target, network_id, network_name, location_id, experiment_id
  • assign once: nic_ip, nic_mac
  • dynamic: state

Routers

<router href="{id}" name="{name}">
  <name>{name}</name>
  <uname>{creator}</uname>

  <host>{host}</host>
  <status>{state}</status>

  <interface>
    <name>{if_name}</name>
    <physical_interface>{if_physical}</physical_interface>
    <ip>{if_ip}</ip>
    <netmask>{if_netmask}</netmask>
  </interface>+

  <link href="{location_id}" rel="location"/>
  <link href="{experiment_id}" rel="experiment" type="application/vnd.bonfire+xml"/>?
</router>
Parameters
  • static: id, name, local_id, creator, host, if_name, if_physical, if_ip, if_netmask, location_id, experiment_id
  • dynamic: state

Source code location

All code is maintained in the BonFIRE SVN.

Workflows

Collection API Requests

Retrieving the information of a single resource

For example, when an experimenter does a GET /locations/be-ibbt/computes/123.

The Resource Manager immediately forwards this call to the specific testbed (here be-ibbt) or adapter via the Enactor, i.e. the Collections Cache does not get involved in this. The returned message is the full description of the resource.

Retrieving the information of a user’s experiments

For example, when an experimenter does a GET /experiments.

The Resource Manager has all the information at hand to give a proper response, since it only has to provide a collection of experiments that use the reference view for the experiment’s resources. The reference view only holds the URIs of the Experiment and the individual resources.

Retrieving the information of a single experiment

For example, when an experimenter (USER) does a GET /experiments/123.

The Resource Manager has to respond with condensed views of the experiment’s resources. For this it will reach to the Collections Cache. It needs to know the IDs of all the resources involved in the Experiment. These IDs can be looked up in the internal RM data base. For each resource type the RM then asks the Collections Cache.

Example:

Say /experiments/123 has two Computes with the following IDs: /locations/be-ibbt/computes/1 and /locations/be-ibbt/computes/2. The Experiment also has a Network between these Computes; the Network ID is: /locations/be-ibbt/networks/2.

To complete the response, the RM has to do the following invoke:

GET {CS}/cache?userId=USER&/locations/be-ibbt/computes/1&/locations/be-ibbt/computes/2&/locations/be-ibbt/networks/2

Retrieving the information of a user’s resources on a particular testbed of a particular resource type

For example, when an experimenter does a GET /locations/be-ibbt/computes.

Again the Resource Manager responds with condensed view collected from the information coming from the Collections Cache. Different from the previous call, the RM now has to provide the IDs of a particular testbed to the CS. If the RM doesn’t know all IDs – for example with Storage collections, the base images are not in the RM DB – it can ask the specific site for all the IDs.

Surgical Synchronization/GET

This process kicks in, when the Collections Cache discovers that a resource requested by the Resource Manager is not in the Collection Cache’s database. If the eventing system works properly, this should not occur frequently.

For every resource that is not yet in the CC’s database, it will do a regular GET for that resource on the Resource Manager. This call should give the full description of the resource. The CC can add the necessary information to its database where it will be available for future calls.

APIs used

The Collections Cache only uses the Resource Manager’s API (more specifically the resource GET) when it has to do a surgical synchronization.

Message queue use

The collections Cache listens to all site specific events coming from the MMQ, with its own credentials.

The information it uses to fill in the Collections Cache DB, is only derived from the events that are provided by the sites. This means that for example with Storages and Networks, which has on most sites a limited support of state events, the collections Cache only can provide the states it knows of. This can cause discrepancies between the Collections GET and the individual GET.

Implementation details

Here a diagram of the architecture of the Collections Cache. It perfectly shows the dependencies between the different BonFIRE entities.

../_images/CC_arch.png