Test Infrastructure

The main objective of software testing is to detect software failures, so they may be corrected; taking into consideration that no software testing procedure can verify that the software is fully functional under all possible conditions. Software testing can only verify that the software under test will function properly under specific testbed conditions.

As it is detailed in the in the integration and validation procedures deliverable, D2.4.6, the testing of the BonFIRE software is divided into three different categories: unit, integration, and system testing. Unit testing is responsibility of the developers of each module, integration and system testing is responsibility of WP2.4. Also, there is a special kind of infrastructures tests, called “site tests”, designed with the objective of validate the OCCI servers of the different testbeds used in BonFIRE. All the previous types of tests are automatically executed in a daily basis by a continuous integration infrastructure .

In coordination with Activity 3, the BonFIRE facility was divided into three different flavours: integration, qualification, and production infrastructures. Production has installed a stable and fully tested version of the BonFIRE software stack, it is the main facility and the one used by the experimenters. Qualification it is a replica of production facility and it is used to validate new updates or bugfixes before sending them to production. Finally, the integration facility has the latest version of the software, developers can update any modules at any time, and verify if the new changes break compatibility with other modules or work as expected.

The Continuous Integration infrastructure runs integration and system tests in a daily basis in each one of those infrastructures, sending mails reports to the different developers if a problem is found.

The deliverables D2.4.6 and D2.4.10 present a detailed description of the already commented validation and testing workflow with a collection of the tests that are actually running to verify the actual and new releases of BonFIRE.

We are not going to enter in detailing each of the tests, but the following sections are going to detail how to execute the SITE-TESTs and System Tests locally (these instructions could be used also to program those tests into a Continuous Integration infrastructure like the Jenkins environment currently used in BonFIRE).


The BonFIRE testbeds infrastructure is composed of seven different sites: EPCC, HP, IBBT, INRIA, USTUTT, WT, and PSNC. Each one of these sites exposes an OCCI interface. Independently of the code of the BonFIRE Resource Manager those interfaces are verified on a daily basis to ensure that the service is working within normal parameters and under standard conditions.

Those tests are composed of simple OCCI commands that emulate the complete set of actions that the BonFIRE Enactor can perform over those sites. Those commands are executed in a form Bash11 scripts.

For example, a typical command executed in Bash will resemble the following

$ curl -vki --header x_bonfire_asserted_id:jenkins --cert path/to/cert:certpassword https://frontend.bonfire.grid5000.fr:443/network

The above command would return a complete list of networks provided by Inria frontend. Subsequently, the output is parsed to verify that everything works as expected.

As commented previously, the test code is executed as bash scripts that make use of cURL tool to talk with the different testbed OCCI servers. At the time of writing BonFIRE talked with four different types of Cloud testbeds that exposed an OCCI interface: OpenNebula, HP Cells, Virtual Wall, and Wellness Telecom. Since these OCCI server had minor differences in their behaviour, different test scripts were created to each one of them. Anyway, as you can see in the following command-line capture, the script folder strutucture for each one of them is identical.

$ tree
|-- hp
|   |-- certs
|   |   `-- eilidhkeyandcert.pem
|   |-- functions
|   |   |-- common.sh
|   |   |-- compute-functions.sh
|   |   |-- groups-test.sh
|   |   |-- network-functions.sh
|   |   `-- storage-functions.sh
|   `-- verify-hp.sh
|-- one
|   |-- certs
|   |   `-- eilidhkeyandcert.pem
|   |-- functions
|   |   |-- common.sh
|   |   |-- compute-functions.sh
|   |   |-- groups-test.sh
|   |   |-- network-functions.sh
|   |   |-- storage-functions.sh
|   |   `-- user-functions.sh
|   |-- README.TXT
|   `-- verify-one.sh
|-- vw
|   |-- certs
|   |   `-- eilidhkeyandcert.pem
|   |-- functions
|   |   |-- common.sh
|   |   |-- compute-functions.sh
|   |   |-- experiment-functions.sh
|   |   |-- groups-test.sh
|   |   |-- network-functions.sh
|   |   |-- storage-functions.sh
|   |   `-- user-functions.sh
|   `-- verify-vw.sh
`-- wt
    |-- certs
    |   `-- eilidhkeyandcert.pem
    |-- functions
    |   |-- common.sh
    |   |-- compute-functions.sh
    |   |-- groups-test.sh
    |   |-- network-functions.sh
    |   |-- storage-functions.sh
    |   `-- user-functions.sh
    |-- README.TXT
    `-- verify-wt.sh

We have a verify-NAME.sh script that it is the main one to be executed, each functions folder contains individual scripts for each type of resources exposed by the testbed, and the certs folder contains a certificate to be able to establish a secure connection with the OCCI server. Details of which OCCI commands accept each testbed can be seen in the Enactor Core section.

To execute the tests we have to write the following commands:

$ export INSTANCE_TYPE=lite
$ ./verify-one.sh ${URL} ${BONFIRE_WAN_ID} ${STORAGE_ID}

Where URL is the url of the OCCI server for the specific testbed, BONFIRE_WAN_ID is the id in the testbed of the BonFIRE WAN network, and STORAGE_ID is the id of the image that we want to use to build the computes for the different compute tests.

If everything goes well the test will exit with code status 0. If there is any kind of error the test will exit with any other number (following the typical UNIX standard to indicate an error in the execution of a command). It is necessary to look at the log so you can know exactly why a test file.


Each one of those scripts define fuctions that user cURL libraries to perform some action over the OCCI server of a testbed and verify the output is the expected one:

list_storage() {
        curl $CURL_OPTIONS $STORAGE_URL > output
        cat output
        STORAGE_LIST_OUTPUT=`more output | grep HTTP | head -n 1 | cut -d' ' -f2`

        if [ "$LIST_OK" == "$STORAGE_LIST_OUTPUT" ]; then

System Tests

System tests will verify the typical behaviour of the BonFIRE infrastructure. The system tests are implemented using Behaviour Driven Development tool Cucumber together with restfully/index.

Each test is described as a scenario, which is linked to a specific feature. The scenarios are described in (almost) plain English, and we use Cucumber to translate the requirements into proper code.

Have a look at the .feature files in the features/ folder to see what it looks like. For example, the Elasticity requirement is expressed as follows (only one scenario included for brevity):

Feature: Elasticity
  In order to interfere with my experiment at runtime
  As a BonFIRE user
  I want to be able to use the BonFIRE API to do crazy things at runtime

  Scenario Outline: Access the BonFIRE API
    Given I have reserved 1 small VM at <location>
    And the VMs are initialized
    And I have installed "curl"
    When I remotely run "source /etc/default/bonfire && curl -k -I $BONFIRE_URI -u $BONFIRE_CREDENTIALS"
    Then the output should look like "HTTP/1.1 200 OK"

    | location |
    | uk-epcc  |
    | fr-inria |

Each Given, When, Then corresponds to a step of the scenario. These steps do the hard work of translating a requirement to working code. You can find those step definitions in the features/step_definitions/ folder. They basically call Restfully to perform all the steps indicated there.

Lastly, helper functions are defined in Ruby files located in the features/support/ folder. Pay attention to the file “env.rb” that defines several configuration variables for the test, for example:

ENV['BONFIRE_CREDENTIALS'] ||= 'test:test'
ENV['INFRASTRUCTURE_DOMAIN'] ||= 'integration.bonfire.grid5000.fr'
ENV['BONFIRE_SSH_GATEWAY'] ||= "ssh.fr-inria.bonfire-project.eu"
ENV['BONFIRE_SSH_KEY'] ||= "~/.ssh/bonfire"

ENV['BONFIRE_AMQP_PASS'] ||= 'password'
ENV['BONFIRE_AMQP_VHOST'] ||= 'bonfire'
ENV['BONFIRE_AMQP_EXCHANGE'] ||= 'experiments'

ENV['BONFIRE_EM'] ||= "https://api.#{ENV['INFRASTRUCTURE_DOMAIN']}/managed_experiments"

The goal is that “normal” people can contribute to the tests by writing a scenario with only Given, When, and Then steps and plain English, and developers can instantiate those tests by writing the corresponding step definitions.

Also, it should be easier to have a look at the .feature files to see what is currently being tested and what is not.

How to run the tests

You must have installed into your system Ruby and Rubygems. After that, you need to install bundler gem:

$ gem install bundler

Inside the folder containing the system test code you need to execute the following command to isntall the required dependencies:

$ bundle install

Now you can luanch Cucumber to execute the tests

$ bundle exec cucumber

Check the file features/support/env.rb to see what environment variables can be given. E.g.:

$ BONFIRE_SSH_KEY=~/.ssh/my_key DEBUG=DEBUG BONFIRE_CREDENTIALS=me:pw bundle exec cucumber

You can run specific features with:

$ bundle exec cucumber features/network.feature

Or only run tests with a specific tag. For example if you want to run only the “light” tests:

$ bundle exec cucumber -t @light

Or run a specific scenario:

$ bundle exec cucumber -n "Access the BonFIRE API"

There are a particular set of tests called multiaxis tests (their feature filename ends with multi.feature), for that you need first to define a location variable, that will limit the scope of the tests to an specific testbed:

$ export location=fr-inria
$ DEBUG=DEBUG BONFIRE_SSH_KEY=~/.ssh/id_dsa BONFIRE_CREDENTIALS='user:password' INFRASTRUCTURE_DOMAIN=bonfire-project.eu bundle exec cucumber --color features/basic_multi.feature -f junit --out results --format pretty

In this case we are telling cucumber to create JUnit ouput format results that can be easily read by a Continuous Integration enviroment such as Jenkins to create automatic reports. Also pay attention the the variable “INFRASTRUCTURE_DOMAIN”, this variable indicates in the infrastructure against we want to run the tests.

See cucumber –help for more options.

Table Of Contents

Previous topic

Command Line Tools

This Page