BonFIRE logo and link to main BonFIRE site

Table Of Contents

Previous topic

Command Line Interface Tools

Next topic

Overview of Compute

This Page

Restfully

What is Restfully

Restfully is a general-purpose client library for RESTful APIs. It is written in Ruby. Its goal is to abstract the nitty-gritty details of exchanging HTTP requests between the user-agent and the server. It also discovers resources at runtime, which means should the API change and add a new functionality, the client will automatically discover it.

In the next section we will describe how to use Restfully in the specific case of BonFIRE.

Installation

Please refer to that Restfully Wiki page for installation instructions for UNIX-based systems and Windows.

Note

On Ubuntu, you might have to install the ruby1.8-dev package, otherwise you may have issues installing Ruby gems later:

$ apt-get install ruby1.8-dev

Note

If you try to install Restfully at IBBT, you will encounter an error, because IBBT nodes don’t have direct internet access. HTTP and FTP internet access is available through the use of a proxy: http://proxy.atlantis.ugent.be:8080. Therefore you have to install gems with the -p flag:

$ gem install -p "http://proxy.atlantis.ugent.be:8080" restfully restfully-addons

Note

On Windows, when Restfully uses your private key to connect to SSH gateways etc it tries to ask your password at the command line but I am not able to type it in. In Unix systems ssh agents can be used eliminate this need to get the password. I have not yet investigated if there is a windows equivalent that will eliminate this need to obtain a password. In the meantime I cannot use the Restfully commands that require my private key. Using a private key with no password may be a way around this problem but that has its own risks.

Getting Started

Start the interactive Restfully shell (replace LOGIN and PASSWORD by your BonFIRE credentials, see the FAQ for an alternative way of passing parameters):

$ restfully --uri=https://api.bonfire-project.eu/ -u LOGIN -p PASSWORD -r ApplicationVndBonfireXml

You should get back a prompt like the following:

Restfully/0.8.1 - The root resource is available in the 'root' variable.
ruby-1.8.7-p249 >

As indicated, enter root and hit RETURN:

ruby-1.8.7-p249 > root
 => {"timestamp"=>"1305300530", "version"=>"0.5.1"}

You just fetched the root of the BonFIRE API. You can see the BonFIRE API version (0.5.1) and the server timestamp. If you want more information of that particular resource, enter pp (‘pretty print’) followed by your request:

ruby-1.8.7-p249 > pp root
#<Resource:0x813f065c uri=https://api.bonfire-project.eu/
  RELATIONSHIPS
    experiments, locations, self
  PROPERTIES
    "timestamp"=>"1305300530"
    "version"=>"0.5.1">
 => nil

The RELATIONSHIPS header contains links to other API resources (here: experiments, locations, self). You can follow any of these links and see what comes back. For instance, following locations:

ruby-1.8.7-p249 > pp root.locations
#<Collection:0x813dd534 uri=https://api.bonfire-project.eu/locations
  RELATIONSHIPS
    parent, self
  ITEMS (0..5)/5
    #<Resource:0x813c6b04 uri=https://api.bonfire-project.eu/locations/be-ibbt>
    #<Resource:0x813b5868 uri=https://api.bonfire-project.eu/locations/de-hlrs>
    #<Resource:0x813a45cc uri=https://api.bonfire-project.eu/locations/fr-inria>
    #<Resource:0x81393330 uri=https://api.bonfire-project.eu/locations/uk-epcc>>
 => nil

It returns the collection of locations available in BonFIRE. If you need to access a specific location, you can either use the find function of the Ruby Enumerable objects:

ruby-1.8.7-p249 > pp root.locations.find{|l| l['name'] == 'fr-inria'}
#<Resource:0x81337e68 uri=https://api.bonfire-project.eu/locations/fr-inria
  RELATIONSHIPS
    computes, networks, parent, self, storages
  PROPERTIES
    "name"=>"fr-inria"
    "url"=>"https://bonfire.grid5000.fr:443">
 => nil

Or use the shortcut method [], with a Symbol as key:

ruby-1.8.7-p249 > pp root.locations[:'fr-inria']
#<Resource:0x812e9240 uri=https://api.bonfire-project.eu/locations/fr-inria
  RELATIONSHIPS
    computes, networks, parent, self, storages
  PROPERTIES
    "name"=>"fr-inria"
    "url"=>"https://bonfire.grid5000.fr:443">
 => nil

Again, you see the RELATIONSHIPS, and you can look at any of them by following the links:

ruby-1.8.7-p249 > pp root.locations[:'fr-inria'].networks
#<Collection:0x813fff1c uri=https://api.bonfire-project.eu/locations/fr-inria/networks
  ITEMS (0..2)/2
    #<Resource:0x813f754c uri=https://api.bonfire-project.eu/locations/fr-inria/networks/1>
    #<Resource:0x813f1e6c uri=https://api.bonfire-project.eu/locations/fr-inria/networks/2>>
 => nil

A collection includes the Ruby Enumerable module, so you can use any of the functions defined in that module. For instance, pretty-printing each of the networks can be achieved with:

ruby-1.8.7-p249 > root.locations[:'fr-inria'].networks.each{|net| pp net}
#<Resource:0x80fed9d8 uri=https://api.bonfire-project.eu/locations/fr-inria/networks/1
  RELATIONSHIPS
    self
  PROPERTIES
    "name"=>"Public Network"
    "public"=>"YES"
    "id"=>"1">
#<Resource:0x80fdb15c uri=https://api.bonfire-project.eu/locations/fr-inria/networks/2
  RELATIONSHIPS
    self
  PROPERTIES
    "name"=>"WAN Network"
    "public"=>"YES"
    "id"=>"2">
 => [{"name"=>"Public Network", "public"=>"YES", "id"=>"1"}, {"name"=>"WAN Network", "public"=>"YES", "id"=>"2"}]

Take your time to follow every relationship you find to see what you get back.

You can also use additional features, like doing the save_as feature.

You must create an experiment with a compute, and the save-as feature can be achieved with:

ruby-1.8.7-p249 > pp root.locations[:"fr-inria"].computes[:"vm"].update(:disk => [{:save_as => {:name => "testing_save_as", :groups => "your_name_of_your_group"}}])
ruby-1.8.7-p249 > pp root.locations[:"fr-inria"].computes[:"vm"].update(:state => "shutdown")

At be-ibbt testbed, the status needs to be stopped, not shutdown contrary to others testbeds.

Then in the next section we’ll see how we can create things.

Creating a BonFIRE Experiment

You could do all the following using the interactive shell, but it’s probably easier to write everything in a script file ready to be executed as many times as you need. For this we’ll use Restfully in library mode, and we can write a short script that deploy the resources we need, and SSH into them to configure them. This script uses a configuration file for restfully authentification, see the FAQ entry for How do I avoid passing my password each time I want to use Restfully? in order to set it up.

require 'rubygems'
require 'restfully'
require 'restfully/addons/bonfire'

SERVER_IMAGE_NAME = "BonFIRE Debian Squeeze v6"
WAN_NAME          = "BonFIRE WAN"

logger       = Logger.new(STDOUT)
logger.level = Logger::INFO

session = Restfully::Session.new(
  :configuration_file => "~/.restfully/api.bonfire-project.eu.yml",
  :gateway            => "ssh.bonfire.grid5000.fr",
  :keys               => ["~/.ssh/id_rsa"],
  :cache              => false,
  :logger             => logger
)

experiment = nil

begin
  # Find an existing running experiment with the same name or submit a new
  # one. This allows re-using an experiment when developing a new script.
  experiment = session.root.experiments.find{|e|
    e['name'] == "Demo SSH" && e['status'] == "running"
  } || session.root.experiments.submit(
    :name        => "Demo SSH",
    :description => "SSH demo using Restfully - #{Time.now.to_s}",
    #define the group(s) you want to use if it differs from your username
    #:groups      => "mygroup",
    :walltime    => 8*3600 # 8 hours
  )

  # Create shortcuts for location resources:
  inria = session.root.locations[:'fr-inria']
  fail "Can't select the fr-inria location" if inria.nil?

  session.logger.info "Launching VM..."
  # Find an existing server in the experiment, or set up a new one:
  server = experiment.computes.find{|vm|
    vm['name'] == "VM-experiment#{experiment['id']}"
  } || experiment.computes.submit(
    :name          => "VM-experiment#{experiment['id']}",
    :instance_type => "small",
    :disk          => [{
      :storage => inria.storages.find{|s|
        s['name'] == SERVER_IMAGE_NAME
      },
      :type    => "OS"
    }],
    :nic           => [
      {:network => inria.networks.find{|n| n['name'] == WAN_NAME}}
    ],
    :location      => inria
  )
  server_ip = server['nic'][0]['ip']
  session.logger.info "SERVER IP=#{server_ip}"

  # Pass the experiment status to running.
  # If it was already running this has no effect.
  experiment.update(:status => "running")

  # Wait until all VMs are ACTIVE or RUNNING and ssh-able.
  # Fail if one of them has FAILED.
  until [server].all?{|vm|
    ['RUNNING', 'ACTIVE'].include?(vm.reload['state']) && vm.ssh.accessible?
  } do
    fail "One of the VM has failed" if [server].any?{|vm|
      vm['state'] == 'FAILED'
    }
    session.logger.info "One of the VMs is not ready. Waiting..."
    sleep 20
  end

  session.logger.info "VMs are now READY!"
  # Display VM IPs
  session.logger.info "*** Server IP: #{server['nic'][0]['ip']}"

  server.ssh do |ssh|
    session.logger.info "Uploading content..."
    # Here is how you would upload a file:
    # ssh.scp.upload!("/path/to/file", '/tmp/file.log')
    # Here is how you can upload some in-memory data:
    ssh.scp.upload!(StringIO.new('some data'), '/tmp/file.log')
    # See <http://net-ssh.github.com/scp/v1/api/index.html> for more details.

    session.logger.info "Content of uploaded file:"
    puts ssh.exec!("cat /tmp/file.log")

    session.logger.info "Installing things..."
    output = ssh.exec!("apt-get update && apt-get install curl -y")
    session.logger.debug output

    session.logger.info "Running query against API..."
    puts ssh.exec!("source /etc/default/bonfire && curl -k $BONFIRE_URI/locations/$BONFIRE_PROVIDER/computes/$BONFIRE_RESOURCE_ID -u $BONFIRE_CREDENTIALS")
  end

  session.logger.warn "Success! Will delete experiment in 10 seconds. Hit CTRL-C now to keep your VMs..."
  sleep 10
  experiment.delete

rescue Exception => e
  session.logger.error "#{e.class.name}: #{e.message}"
  session.logger.error e.backtrace.join("\n")
  session.logger.warn "Cleaning up in 30 seconds. Hit CTRL-C now to keep your VMs..."
  sleep 30
  experiment.delete unless experiment.nil?
end

A video Demonstrating Control and Observability on BonFIRE

The script demo_experiment_using_restfully.rb launches an experiment on BonFIRE using restfully. (The following files are also needed to run the demo: web_backend.tar.gz, web_frontend-sites-available.tar.gz, web_frontend-www.tar.gz, load_balancer.tar.gz)

4 Computes are defined, launched and configured:

  • A web backend hosting some video files (debian+nginx)
  • A web frontend hosting a web page and a js video player requesting the videos from the web backend (debian+nginx+medialelement)
  • A load balancer proxyfying the web frontend (debian+nginx)
  • An aggregator used to collect the metrics of the other computes and their hosts (when the computes are virtual machines) (debian+zabbix)

The load balancer, launched at fr-inria, is available on Public Network, and connects to the web frontend through BonFIRE WAN. The web frontend, launched at be-ibbt, connects to the web backend, launched at be-ibbt, through a VirtualWall network: backend network. All the computes connect to the aggregator, launched at fr-inria, through BonFIRE WAN. The aggregator is also available through the BonFIRE portal. The zabbix API is then used to measure the transmission rate (TX) of the different interfaces involved in serving the video files. A metric measuring the number of hits in the last five minutes is added to the aggregator through the BonFIRE API context.

You can download this demo video here.

FAQ

How do I avoid passing my password each time I want to use Restfully?

$ mkdir -p ~/.restfully && echo '
uri: https://api.bonfire-project.eu/
username: LOGIN
password: PASSWORD
require: ['ApplicationVndBonfireXml']
' > ~/.restfully/api.bonfire-project.eu.yml && chmod 600 ~/.restfully/api.bonfire-project.eu.yml

Then:

$ restfully -c ~/.restfully/api.bonfire-project.eu.yml

Where can I find examples?

Some BonFIRE-specific examples are maintained at https://github.com/crohr/restfully-addons/tree/master/examples/bonfire.