This tutorial aims at guiding your first steps at controlling your EC2 instances from the command line. It is by no means even remotely complete but it will give you an impression of the basic structure and concepts, so you can quickly fill in the gaps for your personal use case. The tutorial starts with setting up your account and forges a bridge from requesting a Spot instance, over exchanging files with it, hooking up additional storage, to finally terminating it. I am not though explaining interaction with the AWS web console – we’ll only resort it for some initial configuration. As usual the target audience are Linux users but the AWS CLI tools are pretty much identical for Windows.

First of all …

… get yourself an AWS account at aws.amazon.com, enter the AWS console and switch to the “Security Credentials” section (see drop-down menu under your name on the right side in the bar at the top). There you create yourself a new user and you grant it full access to EC2 and S3 by attaching the policies “AmazonEC2FullAccess” and “AmazonS3FullAccess”. Also create for that user Access Keys and store its ID and the Secret Key somewhere (we’ll need it soon). Furthermore create yourself in the EC2 section under “Network & Security” / “Key pairs” a new key pair named “aws-test” and store the private key on your computer as ~/.ssh/aws-test.pem. Finally you’re ready to go.

Setting up AWS CLI

[*]: AWS regions and availability zones

Spot vs On-Demand Instances

At AWS you can either request an On-Demand instance or a Spot instance. For On-Demand you pay the regular price and for a Spot instance you enter a bid – if your bid exceeds the Spot price you get your instance – if the Spot price suddenly exceeds your bid then your instance gets automatically terminated. We’ll request a m3.medium Spot instance.

Let’s have a look at the bidding history for a m3.medium Linux/UNIX machine in the eu-west-1 region:

For a regular Irish m3.medium Linux On-Demand instance we would pay (at the time of writing) $0.077. The spot price history indicates that we can get away with about $0.02 for a stable and soon available m3.medium instance.

A Few Notes on the Command Above

The CLI command structure is aws [options] <command> <subcommand> [parameters] . If you need help on the aws  CLI type aws help , if you need guidance on aws ec2  go aws ec2 help  and if you are in need for instructions regarding aws ec2 describe-spot-price-history  then aws ec2 describe-spot-price-history help  is your friend. You get the idea.

--region and --output are available for most of the aws commands. --region  allows you to override the default region (which you set with aws configure). --output  may be text, json or table. In case you want to pipe the table into a less or a file then you might have to switch the coloring off ( --color off).

The commands in this tutorial will not work if just copy-pasted due to the newlines, which would have to be escaped by appending a backslash at the end. I chose not to do that for the sake of readability.

Creating a Security Group

Let’s assume we not just want to connect to the instance via SSH but also intend to run some HTTP server on port 5000. For that we need a custom security group which specifies which ports are open for what IPs.

Requesting a Spot Instance

For the Spot instance we are going to load a Ubuntu 14.04 64bit image (AMI).

To keep an eye on your pending Spot instance requests you can periodically check the status with watch (leave with Ctrl+C):

Connecting to the Instance

If you don’t know what tmux is, I highly recommend you check it out. It is indispensable for efficiently working via a terminal. It allows you to keep a session on a remote computer open and what’s even more important to segment one terminal window into windows and panes – so you can work comfortably in multiple terminal sessions.

Before we can connect we need the public domain name of the instance – not sure if this is the most beautiful method, but it works:

With the EC2 instance up and running, its domain at hand and the keys available – we are ready to connect:

As soon as you’re on the instance it is a good idea to upgrade the software.

Pump up the Volume

The instance comes by default with 8GB of disk space. Maybe that is not enough or you would like to have an external volume for your data which you can attach and detach to instances as needed. Especially b/c the data on your instance is gone if you terminate it (you could create a new image though – see below). For that purpose you can request an almost arbitrarily large EBS (Elastic Block Store) volume. Let’s request a 100 GB volume EBS volume of type “gp2”.

At this time the volume is likely to be available already, but will just be registered as an available storage device. To use and access it we’ll have to format and mount it first.

Some instance types come with additional available – yet ephemeral (i.e. it’s gone after termination) – storage. m3.medium for example comes along with (meager) 4GB of free space. This volume (named xvdb ) is currently mounted at /mnt  as you can see from the output of lsblk  above.

Exchanging Data between the Local Computer and the EC2 Instance

Now let’s assume we have a file on our local computer at ~/file-local and a file on the just attached volume of the remote instance at ~/volume/file-ec2.

Preserve State as Image

Maybe you invested considerable effort into the setup of your instance and you would like to start out at where you finished this time. Then you should create an instance capturing the state of your instance.

 Detach Volume and Terminate Instance

Let’s call it a day – so we’ll detach the volume and terminate the instance.

Enough for today – have fun with your rentable data center!

---

(original article published on www.joyofdata.de)