Get Your Dockerized ECS on in AWS

EMC launched ECS 2.0 at DockerCon 2015 and a great big bonus was included – you can download a community edition for FREE.  The devs were an early adopter of containerization so this software defined object storage platform runs as, you guessed it, a Docker container.  Let’s deploy it in AWS.

This will walk you through the steps to run the free community supported, non-prod version of ECS in AWS as a Docker container using a CentOS instance.  You can start at section 2 to deploy the container to a bare metal or vSphere CentOS machine.  If you do, be sure to add a disk of at least 100GB to be used for ECS persistent storage.

Updated 7/13/15: Note- There are regular updates occurring to the installation scripts located at the EMCECS Github page.  Please review the documentation there to ensure you are using the latest commands.

Overview of Steps:

  1. Launch and Configure a CentOS Instance in AWS

  2. Execute ECS Deployment Script

  3. Configure ECS with Script

Launch and Configure a CentOS Instance in AWS

  1. Login to the AWS Console and navigate to EC2
    ecsaws_35
  2. Click Launch Instance.  Excited yet?ecsaws_34
  3. Choose AWS Marketplace, enter CentOS, and click Select.ecsaws_32
  4. Updated 6/29: Thanks @mcowger for pointing out a lower cost instance option. Select the instance M4.2xlarge R3.xlarge ($0.35/hr) as this meets the minimum requirements of 30GB of RAM and 4 CPUs.  Then, click Next. The cloud costs money.  This could add up if you leave it running!  NOTE: ECS can run with less than 30GB of RAM with some tweaks not covered here.ecsaws_31
  5. Select or create a new VPC.  Click Next.ecsaws_30
  6. Click Add New Volume, enter a size of at least 100GB, and select Magnetic for the volume type.  Again, this surely is not free captain cloud builder.ecsaws_28
  7. next…
    ecsaws_27
  8. Configure the security group to allow the ports necessary to access ECS.  I recommend restricting access to specific IP addresses.  The initial deployment will have a well known default password.
    ecsaws_26 ecsaws_25
  9. No need to boot from SSD unless you want to pay for it.ecsaws_24
  10. OMG Launch it already!  Ain’t the cloudz so simple?  SHAMELESS PLUG ALERT: you too can have a simple on-prem cloud > EMC Federation Hybrid Cloud gets you there quickly.  Tell them I sent you.  If only I got referral payments…ecsaws_23
  11. Create a new key pair unless you already have existing keys to use.  Do not lose this or give it to your neighbor.  They will hack your ECS instance.ecsaws_22
  12. Launch Status.  Houston, we have a new instance in the cloud.  Time to make it rain ECS swift or S3 objects.ecsaws_21
  13. Select your new instance and copy your public address.  You will need this to navigate to your fabulous new cloud instance.  Tell the CIO you are using the cloud.  RUN!
    ecsaws_19
  14. Open terminal on your Mac.  PC users go get a Mac or an ssh client.  Execute:
    ssh -i <yourkey>.pem centos@<youraddressfromabove>
  15. yesecsaws_15
  16. Updates.  Yummy ->
    sudo yum update
  17. Install required bits ->
    sudo yum install git tar wget

Execute ECS Deployment Script

You can perform these steps on any docker capable machine with enough CPU and RAM.

  1. Clone the ECS repository from github:
    git clone https://github.com/EMCECS/ECS-CommunityEdition

    ecsaws_11

  2. cd ECS-CommunityEdition/ecs-single-node/

    ecsaws_10

  3. Get the device name for the volume you added and the NIC name.
    sudo fdisk -l

    then

    ifconfig -a

    Also note the IP address of eth0 for later.ecsaws_8

  4. Run the configuration script substituting what you have from the previous commands.  Here is what it looks like in my super CentOS machine in the cloud.
    sudo python step1_ecs_singlenode_install.py --disks xvdb --ethadapter eth0 --hostname <yourhostname>
  5. If you failed to mess up these steps you will eventually see a prompt instructing you to navigate to the web interface. See what I did there?
    1. SUCCESS!!  You just deployed a cloud-scale object platform with geo-distribution capabilities.  Nice.
    2. Put it on your resume.ecsaws_6
  6. You can verify the container is running with a
    sudo docker ps

    command.ecsaws_5

  7. You can now navigate to the web interface but be sure to continue on to the next section to configure the object settings.


ecsaws_4

Configure ECS with Script

  1. This section is quick.  One script.  You can do it. The main thing you need to be sure to change is the ECSNodes flag.  Use your NIC IP from above.  The other options can be customized if you want. ecsaws_2
    sudo python step2_object_provisioning.py --ECSNodes=10.0.1.10 --Namespace=ns1 --ObjectVArray=ova1 --ObjectVPool=ovp1 --UserName=emccode --DataStoreName=ds1 --VDCName=vdc1 --MethodName
  2. Let the script complete.  Notice it is using the ECS REST API to configure the virtual data center, virtual array, and more.
    ecsaws_1

Now that you have a fully functional Elastic Cloud Storage platform deployed, go push all of your objects up there.  You can use it for HDFS too.  Check out the ECS 2.0 Product Documentation Index to really geek out.  Also check out the handiwork of the EMC {code} team.

Leave a Reply