BonFIRE logo and link to main BonFIRE site

Table Of Contents

Previous topic

Testbed Specificities

Next topic

Instance Types

This Page

Deploying resources on Amazon EC2

Overview

Since the release 3, BonFIRE supports the creation of resources in Amazon EC2. To do it, the Enactor’s Amazon connector transforms the OCCI data it receives to calls to the Amazon API.

Since the launch of the third release of BonFIRE, an Amazon connector to interact to EC2 services is available. BonFIRE includes a subset of the features of that service providing a way to manage VMs in the same way you could do with the other testbeds. It eases the task of managing various resources distributed on many services providing a single interface.

The set of features provided by the BonFIRE’s Amazon connector includes creating compute instances, create storages (Amazon volumes) to its later use in compute instances, save snapshots of running compute instances and view detailed information of compute and storage resources.

Previous steps

BonFIRE does not provide Amazon’s credential management. So, In order to be able to use the interconnector, you have to be registered in Amazon Web Services.

Then, you have to create a security group called “bonfire” from the AWS console:

../_images/01.png
../_images/02.png
../_images/03.png
../_images/04.png

The next step to get your Amazon Web Services account configured to get petitions from BonFIRE is to create a user with full access to EC2, write down the access keys (both public and secret) and append the following code to the policy document:

{
    "Effect": "Allow",
    "Action": "iam:GetUser",
    "Resource": "*"
}
../_images/05.png
../_images/06.png
../_images/07.png
../_images/08.png
../_images/09.png
../_images/10.png
../_images/11.png
../_images/12.png
../_images/13.png

With these steps, you will have your account set up to work with BonFIRE. Take in account that you will be asked for the user’s keys when creating an experiment in BonFIRE.

Interface

The Amazon connector uses the Java API to interact with the infrastructure and manage resources. Anyway, to fully fullfill the BonFIRE requirements, the interaction with the rest of the project’s infrastructure is handled by OCCI-compliant messages. Below some examples of the OCCI interaction with the connector to manage resources in Amazon are presented.

Storages

List storages

Storage resources in Amazon are persistent, and can be public or private. This has made that a lot of public storage resources are available, each one with its own customization. Within this large set of items the “official” Amazon storages can be found. The whole list of public storages is so huge (11813 objects at the moment of writing this), that it was decided to provide a filter to distinguish the Amazon images and the user’s ones.

The usual way of retrieving storages in all the sites:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/storages

...returns the whole list of storages:

<collection xmlns="http://api.bonfire-project.eu/doc/schemas/occi">
<items>
    <storage href="/locations/useast-aws/storages/ami-000af969" name="null" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-0011e069" name="clovr-standard-2011-01-07-16-01-33" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-0017b369" name="clovr-standard-2012-05-15-03-00-26" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-0022c769" name="null" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-002bf169" name="null" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-002dd269" name="xsd.web.server" platform="windows" owner="null" />
    ...
    <storage href="/locations/useast-aws/storages/ami-fff83296" name="Windows2003_Apply" platform="windows" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-fffd3796" name="inspection2" platform="null" owner="null" />
    <storage href="/locations/useast-aws/storages/ami-fffe2a96" name="vnoc-images-2012-01-01_04-48-52/new_name" platform="windows" owner="null" />
</items>
</collection>

In order to get only the Amazon images the following command is needed:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/storages/amazon

And, for the user’s own images:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/storages/user

And, of course, the information retrieving of a single one is performed as usual:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/storages/ami-fffe2a96

Retuning:

<storage xmlns="http://api.bonfire-project.eu/doc/schemas/occi" href="/locations/useast-aws/storages/ami-fffe2a96">
    <id>ami-fffe2a96</id>
    <name>vnoc-images-2012-01-01_04-48-52/new_name</name>
    <type>OS</type>
    <description>10-82-43-216//Daily</description>
    <public>YES</public>
    <persistent>YES</persistent>
</storage>

Create storages

The creation of storages in Amazon is managed by the Elastic Block Storage system. The tools Amazon provides, allows the specification of the size, the availability zone where it will be created, the possibility of using a previously saved snapshot (public images) and some other options.

In order to maximice simplicity, we decided to limit this creation options and make them match, as much as possible, the storage creation in the rest of BonFIRE sites. A curl command to the enactor for creating a storage looks as follows:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/experiments/<experiment id>/storages \
        -X POST -d '<storage xmlns="http://api.bonfire-project.eu/doc/schemas/occi">\
                        <name>AmazonStorage02</name>\
                        <groups>sgarcia</groups>\
                        <description>AmazonStorage description</description>\
                        <type>DATABLOCK</type>\
                        <size>2048</size>\
                        <fstype>ext3</fstype>\
                        <persistent>YES</persistent>\
                        <link href="/locations/useast-aws" rel="location"/>\
                    </storage>' \
        --header Content-Type:application/vnd.bonfire+xml

Where the units for the size are MB and the host is optional (if no host is provided, the first one available is chosen).

Computes

In the same way than the storage creation, the management of Amazon computes in BonFIRE tries to follow the most the management in other testbeds.

List configurations

The retrieving of the compute types can be achieved with the following command:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/configurations

Then, a OCCI payload as the following one is returned:

<collection xmlns="http://api.bonfire-project.eu/doc/schemas/occi">
    <configuration>
        <vcpu>1</vcpu>
        <vmem>1740.8</vmem>
        <name>m1.small</name>
    <configuration>
    <configuration>
        <vcpu>2</vcpu>
        <vmem>3840</vmem>
        <name>m1.medium</name>
    <configuration>
    <configuration>
        <vcpu>4</vcpu>
        <vmem>7680</vmem>
        <name>m1.large</name>
    <configuration>
    <configuration>
        <vcpu>8</vcpu>
        <vmem>15360</vmem>
        <name>m1.xlarge</name>
    <configuration>
    <configuration>
        <vcpu>1</vcpu>
        <vmem>613</vmem>
        <name>t1.micro</name>
    <configuration>
        ...
</collection>

List of computes

As well as the rest of BonFIRE site, a command like the following one allows the retrieving of the current computes at Amazon:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/computes

Then, a OCCI payload as the following one is returned:

<collection xmlns="http://api.bonfire-project.eu/doc/schemas/occi">
    <items>
        <compute href="/locations/useast-aws/computes/i-e9e1c592" name="Amazon instance"/>
    </items>
</collection>

If, moreover, we provide the id of an specific one:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/computes/i-e9e1c592

We get a payload with its information:

<compute xmlns="http://api.bonfire-project.eu/doc/schemas/occi" href="/locations/useast-aws/computes/i-e9e1c592">
    <id>i-e9e1c592</id>
    <name>Amazon instance</name>
    <type>m1.small</type>
    <state>RUNNING</state>
    <disk>
        <storage href="/locations/useast-aws/storages/ami-31814f58" />
        <type>OS</type>
        <target>/dev/sda1</target>
    </disk>
    <disk>
        <storage href="/locations/useast-aws/storages/vol-c60cecbd" />
        <type>DATABLOCK</type>
        <target>/dev/sdb</target>
    </disk>
    <nic>
        <network href="/locations/useast-aws/networks/dummy" name="Public Network" />
        <ip>ec2-67-202-15-241.compute-1.amazonaws.com</ip>
    </nic>
</compute>

Here we have to note an important point. Amazon provides automatically a public IP for each of the deployed machines, but no specific network is defined.

To match also as much as possible the rest of the infrastructure, the enactor maps the Amazon compute states to the following ONE ones: PENDING, RUNNING, SHUTDOWN, DONE, EPILOG-STOP and STOPPED.

Creation of computes

To create a compute the following commnad can be sent to the enactor:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/experiments/<experiment id>/computes \
        -X POST -d "<?xml version='1.0' encoding='UTF-8'?>\
                    <compute xmlns='http://api.bonfire-project.eu/doc/schemas/occi'>\
                        <name>Amazon instance</name>\
                        <description>Description</description>\
                        <instance_type>t1.micro</instance_type>\
                        <host>us-east-1a</host>\
                        <disk>\
                                <storage href='/locations/useast-aws/storages/ami-82fa58eb'/>\
                                <type>OS</type>\
                        </disk>\
                        <disk>\
                                <storage href='/locations/useast-aws/storages/vol-922bc9e8'/>\
                                <target>/dev/sdc1</target>\
                                <type>DATABLOCK</type>\
                        </disk>\
                        <disk>\
                                <storage href='/locations/useast-aws/storages/vol-2028ca5a'/>\
                                <target>/dev/sdd1</target>\
                                <type>DATABLOCK</type>\
                        </disk>\
                        <link href='/locations/useast-aws' rel='location'/>\
                    </compute>"\
        --header Content-Type:application/vnd.bonfire+xml

Noting that the DATABLOCK storages, the description and the host are optional. If no host is provided, the first one available is chosen.

Saving VMs images

As well as the other BonFIRE sites, the Amazon connector allows the user to save an image of a running VM to its future use. To request this, a command like the next one should be used:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/computes/<compute_id> \
        -X PUT -d "<compute xmlns='http://api.bonfire-project.eu/doc/schemas/occi'>\
                        <disk>\
                                <save_as name='TEST_SAVE_AS_OCCI'/>\
                        </disk>\
                   </compute>" \
        --header Content-Type:application/vnd.bonfire+xml

Shutdown a compute

To shutdown a compute in Amazon, the only action needed is changing its state with a PUT command:

curl -vki --user <user_id> http://api.integration.bonfire.grid5000.fr/locations/useast-aws/computes/<compute_id> \
        -X PUT -d "<compute xmlns='http://api.bonfire-project.eu/doc/schemas/occi'>\
                        <state>SHUTDOWN<state>\
                   </compute>" \
        --header Content-Type:application/vnd.bonfire+xml