James Gardner: Home > Blog > 2007 > Custom Debian EC2 AMIs From Xen Images

Custom Debian EC2 AMIs From Xen Images

Posted:2007-09-01 19:06
Tags:Virtulization, EC2

Now that you have seen how to create an AMI from another AMI in my last tutorial its time to try a different technique: creating an AMI from a Xen image.

The advantages of this approach are:

  1. You can install the operating system from scratch so you can be confident of the files you are adding
  2. You can quickly convert between AMI and Xen images which allows you to host your images on either system
  3. You can extensively test your images in an environment where you have total control over the whole set up

To begin with you need to setup Xen on Debian. Follow one of the many tutorials such as this one: http://jimmyg.org/2007/04/30/xen-on-thinkpad-r50e-debian-40-etch/

Once you have Xen working you can start thinking about using it for EC2 images.

First you need to install the ec2-ami-tools package on the host. Beause we are using Debian we need to first install alien to be able to handle RPM files so that we can use Amazon's AMI tools which are a series of useful Ruby programs:

sudo apt-get install alien curl
curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
sudo alien -i ec2-ami-tools.noarch.rpm
rm ec2-ami-tools.noarch.rpm

The tools require Ruby so you need to install them too:

sudo apt-get install ruby libopenssl-ruby1.8 rsync

We also need to do a bit of tweaking to make the compatible with Debian. First create a symlink:

sudo ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/

Then we need to patch-up the AMI tools to work on a debian-based system:

sudo vi /usr/lib/site_ruby/aes/amiutil/image.rb

and around line 149 change:

exec( 'for i in console null zero ; do /sbin/MAKEDEV -d ' + dev_dir + ' -x $i ; done' )

to:

exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")

Creating the Xen Image

Now you need to create the base image using the xen-create-image tool (you will have installed it if you followed the Xen tutorial linked to above). Obviously since the EC2 hardware is i686 hardware this needs to be done on a PC platform:

sudo mkdir /mnt/xen sudo xen-create-image --debootstrap --verbose
--dir=/mnt/xen --size=1Gb --memory=512Mb --fs=ext3 --cache=yes --dist=etch
--mirror=http://ftp.uk.debian.org/debian/ --hostname=debian_etch
--initrd=/boot/initrd.img-2.6.18-4-xen-686
--kernel=/boot/vmlinuz-2.6.18-4-xen-686 --dhcp

These settings will create an ext3 filesystem of a Debian Etch install in an image which will get a network address using DHCP.

This will spit out a 1.0GB Debian Etch minimal image in /mnt/xen/domain/debian/disk.img don't worry about the memory size being different from the amount Xen uses. Even the kernel version doesn't matter too much since EC2 will replace it with its own.

At this point you have two choices:

  1. Use xm create -c /etc/xen/debian.cfg to start the Xen virtual machine and configure it from there
  2. Mount the image as a filesystem and make changes directly

If you are used to using Xen you'll have no problem making the changes directly to the booted image so I'll show you how to do it the other way.

Incidentally, you can also create everything entirely from scrath as described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/public-ami-guidelines.html

To create an AMI from the Xen image first mount the image:

sudo mkdir /mnt/image
sudo mount -t ext3 -o loop /mnt/xen/domains/debian/disk.img /mnt/image

Then set up the fstab which EC2 will expect:

cd /mnt/image/etc
sudo mv fstab fstab.xen
sudo vi /etc/fstab.ec2

Add the following content:

/dev/sda1 /     ext3  defaults             1 1
/dev/sda2 /mnt  ext3  defaults,user_xattr  1 2
/dev/sda3 swap  swap  defaults             0 0
none      /proc proc  defaults             0 0
none      /sys  sysfs defaults             0 0

Then:

sudo ln -sf fstab.ec2 fstab

At this point you need to set a root password or follow the instructions in the next section to make a public AMI.

Extra Steps For Public AMIs

If you want to make your AMIs available to the publis there are some extra steps you need to take described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-sharingamis.html

Basically you need to perform three steps:

  1. Randomise the root password the first time an instance boots
  2. Change the default SSH setup
  3. Set up the public part of the SSH key pair so that the user can sign in

Here is how you can do this on Debian Etch.

First tweak the SSH config as recommended by Amazon:

sed -i -e 's/PermitRootLogin no/#PermitRootLogin no/g' /mnt/image/etc/ssh/sshd_config

cat <<EOL >> /mnt/image/etc/ssh/sshd_config
UseDNS  no
PermitRootLogin without-password
EOL

Then alter the boot process to set up the public key. Edit this file:

vim mnt/image/etc/rc.local

So it looks like this:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

if [ ! -d /root/.ssh ] ; then
        mkdir -p /root/.ssh
        chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
wget -q -P /tmp http://169.254.169.254/2007-03-01//meta-data/public-keys/0/openssh-key
cat /tmp/openssh-key > /tmp/my-key
rm /tmp/openssh-key
if [ $? -eq 0 ] ; then
        cat /tmp/my-key >> /root/.ssh/authorized_keys
        chmod 600 /root/.ssh/authorized_keys
        rm /tmp/my-key
fi
if [ -f "/.firstrun" ] ; then
  dd if=/dev/urandom count=50|md5sum > /tmp/p.out
  POUT=`cat /tmp/p.out | cut -f1 -d' '`
  rm -f /tmp/p.out
  /usr/sbin/usermod -p $POUT root
  rm -f /.firstrun
fi
exit 0

Then create the .firstrun file so that the password changing is triggered on boot:

touch /mnt/image/.firstrun

Appending the public key to the authorized keys file on every boot might seem overkill but this is how Amazon does it and it is unlikely you'll be rebooting your image too many times anyway.

Now you can unmount the image:

cd /
sudo umount /mnt/image

Caution!

When creating your own image for public it is important to ensure you don't leave any of the AWS identifiers, account number, certificates or keys anywhere on the image. There are some tips for how to avoid this at the bottom of this page.

Bundling

This is very similar to before but rather than using ec2-bundle-vol you use ec2-bundle-image. You'll need the X.509 certificate and private key to bundle the image as before.

The name given by the option -p determines the name of the manifest file. In this case we chose debian_etch:

ec2-bundle-image -p debian_etch -i /mnt/xen/domains/debian/disk.img -k ~/Desktop/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -c ~/Desktop/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132
Splitting /tmp/debian_etch.tar.gz.enc...
Created debian_etch.part.00
Created debian_etch.part.01
Created debian_etch.part.02
...
Created debian_etch.part.51
Created debian_etch.part.52
Generating digests for each part...
Digests generated.
Creating bundle manifest...
ec2-bundle-image complete.

By default images are written to /tmp unless you use the -d option. Let's check they are there:

james@dirac:/$  ls -lh /tmp/debian_etch.*
-rw-r--r-- 1 james james 9.0K 2007-08-27 21:48 /tmp/debian_etch.manifest.xml
-rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.00
-rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.01
-rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.02
...
-rw-r--r-- 1 james james  10M 2007-08-27 21:48 /tmp/debian_etch.part.51
-rw-r--r-- 1 james james 8.9M 2007-08-27 21:48 /tmp/debian_etch.part.52

As you can see, the Debian Etch install has 53 parts which is quite a lot more than the 23 my previous tutorial had.

You can now upload it to S3 but unless your network connection is as fast as the 250Mb/s each EC2 instance has this will take a lot longer than it took to to upload the parts from EC2 which is what we did in the last tutorial:

ec2-upload-bundle -b james-music -m /tmp/debian_etch.manifest.xml -a <AWS Access Key>  -s <AWS shared key>

Once the image is uploaded we need to register it. Load up a Python prompt again and use boto to create a connection before registering the image:

>>> from boto.ec2.connection import EC2Connection
>>> conn = EC2Connection('<AWS Access Key>', '<AWS shared key>')
>>> conn.register_image('james-music/debian_etch.manifest.xml')
u'ami-ddec09b4'

You should now be able to start the image:

>>> for i, image in enumerate(images):
...     if image.location.startswith('james-music'):
...         print "%s, %s, %s"%(i, image.id, image.location)

Once you have found the image number of the image (in this case 40) you can start it:

>>> image = images[40]
>>> reservation = image.run(key_name='gsg-keypair', security_groups=['web_test'])

And check progress:

>>> reservation.instances[0].update()
>>> reservation.instances[0].state
u'pending'

Note

The web_test security group was one we created in the last tutoiral. If you haven't set up the group you will need to set security permissions manually or you won't be able to access your EC2 instance once it has loaded.

Note

There is also an unbundle command so you can always extract an image from a set of AMI parts, make some modifications and re-bundle it again.

Custom Modules

When you boot an instance, EC2 replaces the kernel with its own which is a Xen 2.6.16 kernel. This means you can't use a modified kernel but you can use kernel modules. Although I haven't tried it yet I've read you do this by booting the Fedora Development image, compiling the modules you want and copying them to the /lib/modules/ directory of the Debian image. You need to use Fedora because the modules need compiling with GCC 4.0 but Etch uses 4.1

http://elastic8.com/?q=node/2 http://tim.dysinger.net/2007/07/28/compiling-fuse-kernel-modules-for-debian-40-on-ec2/ http://info.rightscale.com/2007/8/7/recompiling-kernel-modules-for-ec2-instances

You can also get the default ones directly from Amazon but they seem to get set up anyway so you don't need to:

wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz
tar zxfv modules-2.6.16-ec2.tgz
mv the files to /lib/modules
depmod -a

(view source)

James Gardner: Home > Blog > 2007 > Custom Debian EC2 AMIs From Xen Images