Amazon Web Services

Introduction

I need to get an AMI up and running to host this bliki. I am most proficient using Ubuntu and it is significantly easier to install planet venus and redland rdf using Ubuntu's package manager.

Launching an Instance

Amazon have a good page, Getting Started Guide. This focuses on Launching an Instance. Launching your 1st instance allows you to obtain an ssh key and declare some firewall rules. Make sure that ssh is enabled.

To login you'll have to move the key to somewhere convenient, chmod 400 *.pem and then use ssh.

ssh -i ${KEYFILE} ${EC2USER}@${INSTANCEDNSNAME}

The user and instance name are available from the aws console.

Saving to persistant storage

Amazon have a documentation page, Where to go from here? which says,

One of the first tasks you'll want to do after completing this guide is to create one or more custom AMIs. You can modify Amazon AMIs or public AMIs as needed (a process called rebundling). For general information about AMIs, go to AMIs and to Preparing and Creating AMIs in the Amazon Elastic Compute Cloud User Guide. See also Bundling an AMI.

You can choose between Amazon S3 or Amazon Elastic Block Store as the root device for your AMI (for a brief description of Amazon EBS, see Amazon Elastic Block Store later in this section). We recommend using instances backed by Amazon EBS, because they launch faster and use persistent storage. For more information, go to AMIs Backed by Amazon EBS in the Amazon Elastic Compute Cloud User Guide.

Console Access

I have used AWS Management Console and filtered on EBS storage and on the string 10.10. I have chosen to use ami-f46f5a80, which is an Ubuntu 10.10 Server i386. I created a new key pair. 12th Jan 2011

Since I am using PUTTY, I need to convert the key to PUTTY format, using PUTTYGEN.

The AWS instances disconnect from the network after periods of non-use. There is a flag in Putty,

PuTTY Configuration > Categories > Connection > Seconds between keepalives

and using a .ssh/config file and setting the following flags

ServerAliveInterval 60
ServerAliveCountMax 5

Fedora

Fedora have also created 'official' images, the are published on the EC2 Images page at http://fedoraproject.org

Ubuntu

The official Ubuntu AMI's are listed and found from http://cloud.ubuntu.com/ami/. They can be launched from this page if logged into the console.

Experimenting with Hadoop

Single Node Cluster

Importing a VM

I plan to upload a single node hadoop cluster to AWS. I plan to locate it on EBS so it can survive a system halt. I originally planned to create it on a Virtual Box image, and upload the image. 20th Sept 2011

I experimented with looking to build the AMI on my Laptop and uploading it to Amazon, my notes are held in the third comment on this snip. Having a further think, I am about to give up on this as it seems to be too much work when Greg Scher points me at

• http://aws.amazon.com/ec2/vmimport/

• http://forums.virtualbox.org/viewtopic.php?f=7&t=40292 which talks about encapsulating VB in an Amazon AMI, they suggest that uploading guest additions might muck things up and
• http://forums.virtualbox.org/viewtopic.php?f=6&t=39993 which asks about uploading a VM

Fedora's contribution to uploading a VM is at a page called EC2 Creation on their site.

Two Node Cluster

Michael Noll has written a tutorial for multi-node clusters, so I plan to make one of these.

The first thing is that the two nodes need to be able to talk to each other, this requires enabling in the firewall rules. It is not possible to change the ruleset name for an instance once created. It is possible to change the rules and so I have created a private AMI with a single node hadoop server configured with firewalls that permit ping, http and ssh. I later enmhanced these rules to include ports 54310-54311 and other ports mentioned on the Cloudera Hadoop Ports Quick Reference page. If I want to monitor my AMIs with Zabbix, I will need to add the Zabbix ports to the firewall rules.

Any way I have distributed the master public key to slave node. I now need to consider how dynamic to make the name discovery. I could use dyndns. Probably best to just get it done.

Important Amazon Support

• EC2 support Forums

Essential Services

1. SSH - don't forget to allow it in the firewall rules
2. DNS - I use DynDNS, whose home page is at http://www.dyndns.com/, who have a page called DNS Update Clients. I used ddclient on the late lamented Qube. It's download page and a how to page are on the site, and some aspects of my Cobalt Installation might be on the Dell or even elsewhere on this site.
3. apache - need to test that dyndns is working
4. git - planet venus now uses git, see also my planet planet page
5. scp/ftp - I need a file transfer program to up load stuff, it seems I have one

Installing ddclient

I use this to maintain the server's dns name. My notes have been transferred to my ddclient page on this site. They include the magic runes needed to obtain the public IP address, which are documented on both my ddclient page and on the internet at Amazon EC2 - What You May Not Have Known a blog article at codesta.com. Ubuntu users installing from the tarball will need to install the PERL SSL libraries.

Zabbix

I downloaded the source and created the zabbix agents having installed gcc.

1. Create the User
2. Create the Config directory, i.e. /etc/zabbix
3. Copy the conf file to the directory
4. Set the configuration options in the conf file to hold pid & log file in /var/tmp
5. Reserve the ports by editing /etc/services
6. Get my initialisation script from the Zabbix forums
7. Ensure that all local configurations required are are accuratly stated in the script e.g. /usr/local/sbin, /etc/zabbix and /var/tmp

Java

I need Java 5 for Snipsnap. This needs to be downloaded from the Oracle site.

• Oracle's Java 5 download page.

Running a remote desktop using VNC

I started my build from http://cloud.ubuntu.com/ami/, for various reasons I( used a 32 bit image, based on Natty hosted in the EU. This is a server.

sudo tasksel install ubuntu-desktop
sudo apt-get install update
sudo apt-get install upgrade

I then referred to

• https://help.ubuntu.com/community/VNC/Servers

• http://linux.die.net/man/1/x11vnc
• http://www.tightvnc.com/vncserver.1.php
• http://www.realvnc.com/products/free/4.1/man/Xvnc.html

This works quite nicely, with just one problem. In my VNC sessions, the 'D' key is bound to 'Hide All Windows', this thread, tightvnc Keyboard Oddity, documents the solution, using 'System -> Prefs -> Keyboard Short Cuts'. I ran into some dead ends while looking for a command line solution, and so to ensure that GNOME was fully upto date I ran,

sudo apt-get install gnome

tightvnc needs to initialise the user file system; I do not run it as root, but it does have a password function. I used ideas documented on

• https://help.ubuntu.com/community/VNC/Servers

ubuntu's vnc page

Links

I have started to use Second Brain to store some links in collections. I have an AWS collection.

One day

• Need to sort Rightscale vs Landscape, one day.
• Need to understand the billable activity of the AMI, its more expensive than I expect
• Get Zabbix working, fix ddclient's syslog interface and write zabbix monitors for that and mingle
• Get Snipsnap working.