Custom Debian EC2 AMIs From Xen Images
+++++++++++++++++++++++++++++++++++++++++

:Posted: 2007-09-01 19:06
:Tags: Virtulization, EC2

Now that you have seen how to create an AMI from another AMI in my `last tutorial <http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/>`_ its time to try a
different technique: creating an AMI from a Xen image. 

The advantages of this approach are:

1. You can install the operating system from scratch so you can be confident of the files you are adding
2. You can quickly convert between AMI and Xen images which allows you to host your images on either system
3. You can extensively test your images in an environment where you have total control over the whole set up

To begin with you need to setup Xen on Debian. Follow one of the many tutorials
such as this one:
http://jimmyg.org/2007/04/30/xen-on-thinkpad-r50e-debian-40-etch/

Once you have Xen working you can start thinking about using it for EC2 images.

First you need to install the ``ec2-ami-tools`` package on the host. Beause we
are using Debian we need to first install ``alien`` to be able to handle RPM
files so that we can use Amazon's AMI tools which are a series of useful Ruby
programs::

    sudo apt-get install alien curl
    curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm
    sudo alien -i ec2-ami-tools.noarch.rpm
    rm ec2-ami-tools.noarch.rpm

The tools require Ruby so you need to install them too::

    sudo apt-get install ruby libopenssl-ruby1.8 rsync

We also need to do a bit of tweaking to make the compatible with Debian. First
create a symlink::

    sudo ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/

Then we need to patch-up the AMI tools to work on a debian-based system::

    sudo vi /usr/lib/site_ruby/aes/amiutil/image.rb

and around line 149 change::

    exec( 'for i in console null zero ; do /sbin/MAKEDEV -d ' + dev_dir + ' -x $i ; done' )

to::

    exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")

Creating the Xen Image
======================

Now you need to create the base image using the ``xen-create-image`` tool (you will
have installed it if you followed the Xen tutorial linked to above). Obviously
since the EC2 hardware is i686 hardware this needs to be done on a PC platform::

    sudo mkdir /mnt/xen sudo xen-create-image --debootstrap --verbose
    --dir=/mnt/xen --size=1Gb --memory=512Mb --fs=ext3 --cache=yes --dist=etch
    --mirror=http://ftp.uk.debian.org/debian/ --hostname=debian_etch
    --initrd=/boot/initrd.img-2.6.18-4-xen-686
    --kernel=/boot/vmlinuz-2.6.18-4-xen-686 --dhcp

These settings will create an ext3 filesystem of a Debian Etch install in an
image which will get a network address using DHCP.

This will spit out a 1.0GB Debian Etch minimal image in
``/mnt/xen/domain/debian/disk.img`` don't worry about the memory size being
different from the amount Xen uses. Even the kernel version doesn't matter too
much since EC2 will replace it with its own.

At this point you have two choices:

1. Use ``xm create -c /etc/xen/debian.cfg`` to start the Xen virtual machine and configure it from there
2. Mount the image as a filesystem and make changes directly

If you are used to using Xen you'll have no problem making the changes directly
to the booted image so I'll show you how to do it the other way. 

Incidentally, you can also create everything entirely from scrath as described here:
http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/public-ami-guidelines.html

To create an AMI from the Xen image first mount the image::

    sudo mkdir /mnt/image
    sudo mount -t ext3 -o loop /mnt/xen/domains/debian/disk.img /mnt/image

Then set up the ``fstab`` which EC2 will expect:: 

    cd /mnt/image/etc
    sudo mv fstab fstab.xen
    sudo vi /etc/fstab.ec2 

Add the following content::

    /dev/sda1 /     ext3  defaults             1 1
    /dev/sda2 /mnt  ext3  defaults,user_xattr  1 2
    /dev/sda3 swap  swap  defaults             0 0
    none      /proc proc  defaults             0 0
    none      /sys  sysfs defaults             0 0

Then::

    sudo ln -sf fstab.ec2 fstab

At this point you need to set a root password or follow the instructions in the
next section to make a public AMI.

Extra Steps For Public AMIs
===========================

If you want to make your AMIs available to the publis there are some extra steps you need to take described here:
http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-sharingamis.html

Basically you need to perform three steps:

#. Randomise the root password the first time an instance boots
#. Change the default SSH setup
#. Set up the public part of the SSH key pair so that the user can sign in


Here is how you can do this on Debian Etch.

First tweak the SSH config as recommended by Amazon::

    sed -i -e 's/PermitRootLogin no/#PermitRootLogin no/g' /mnt/image/etc/ssh/sshd_config
    
    cat <<EOL >> /mnt/image/etc/ssh/sshd_config
    UseDNS  no
    PermitRootLogin without-password
    EOL

Then alter the boot process to set up the public key. Edit this file::

    vim mnt/image/etc/rc.local

So it looks like this::
    
    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will "exit 0" on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    #
    # By default this script does nothing.
    
    if [ ! -d /root/.ssh ] ; then
            mkdir -p /root/.ssh
            chmod 700 /root/.ssh
    fi
    # Fetch public key using HTTP
    wget -q -P /tmp http://169.254.169.254/2007-03-01//meta-data/public-keys/0/openssh-key
    cat /tmp/openssh-key > /tmp/my-key
    rm /tmp/openssh-key
    if [ $? -eq 0 ] ; then
            cat /tmp/my-key >> /root/.ssh/authorized_keys
            chmod 600 /root/.ssh/authorized_keys
            rm /tmp/my-key
    fi
    if [ -f "/.firstrun" ] ; then
      dd if=/dev/urandom count=50|md5sum > /tmp/p.out
      POUT=`cat /tmp/p.out | cut -f1 -d' '`
      rm -f /tmp/p.out
      /usr/sbin/usermod -p $POUT root
      rm -f /.firstrun
    fi
    exit 0

Then create the ``.firstrun`` file so that the password changing is triggered on boot::

    touch /mnt/image/.firstrun

Appending the public key to the authorized keys file on every boot might seem
overkill but this is how Amazon does it and it is unlikely you'll be rebooting
your image too many times anyway.

Now you can unmount the image::

    cd /
    sudo umount /mnt/image

.. caution :: 

    When creating your own image for public it is important to ensure you don't leave any of the AWS identifiers, account number, certificates or keys anywhere on the image. `There are some tips for how to avoid this at the bottom of this page <http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/public-ami-guidelines.html>`_.

Bundling
========

This is very similar to before but rather than using ``ec2-bundle-vol`` you use
``ec2-bundle-image``. You'll need the X.509 certificate and private key to
bundle the image as before.

The name given by the option ``-p`` determines the name of the manifest file.
In this case we chose ``debian_etch``::

    ec2-bundle-image -p debian_etch -i /mnt/xen/domains/debian/disk.img -k ~/Desktop/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -c ~/Desktop/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132
    Splitting /tmp/debian_etch.tar.gz.enc...
    Created debian_etch.part.00
    Created debian_etch.part.01
    Created debian_etch.part.02
    ...
    Created debian_etch.part.51
    Created debian_etch.part.52
    Generating digests for each part...
    Digests generated.
    Creating bundle manifest...
    ec2-bundle-image complete.
    
By default images are written to ``/tmp`` unless you use the ``-d`` option. Let's check they are there::
    
    james@dirac:/$  ls -lh /tmp/debian_etch.*
    -rw-r--r-- 1 james james 9.0K 2007-08-27 21:48 /tmp/debian_etch.manifest.xml
    -rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.00
    -rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.01
    -rw-r--r-- 1 james james  10M 2007-08-27 21:47 /tmp/debian_etch.part.02
    ...
    -rw-r--r-- 1 james james  10M 2007-08-27 21:48 /tmp/debian_etch.part.51
    -rw-r--r-- 1 james james 8.9M 2007-08-27 21:48 /tmp/debian_etch.part.52

As you can see, the Debian Etch install has 53 parts which is quite a lot more
than the 23 my previous tutorial had. 

You can now upload it to S3 but unless your network connection is as fast as
the 250Mb/s each EC2 instance has this will take a lot longer than it took to
to upload the parts from EC2 which is what we did in the last tutorial::

    ec2-upload-bundle -b james-music -m /tmp/debian_etch.manifest.xml -a <AWS Access Key>  -s <AWS shared key> 

Once the image is uploaded we need to register it. Load up a Python prompt
again and use boto to create a connection before registering the image::

    >>> from boto.ec2.connection import EC2Connection
    >>> conn = EC2Connection('<AWS Access Key>', '<AWS shared key>')
    >>> conn.register_image('james-music/debian_etch.manifest.xml')
    u'ami-ddec09b4'

You should now be able to start the image::

   >>> for i, image in enumerate(images):
   ...     if image.location.startswith('james-music'):
   ...         print "%s, %s, %s"%(i, image.id, image.location)
  
Once you have found the image number of the image (in this case 40) you can start it::

   >>> image = images[40]
   >>> reservation = image.run(key_name='gsg-keypair', security_groups=['web_test'])

And check progress::

   >>> reservation.instances[0].update()
   >>> reservation.instances[0].state
   u'pending'

.. Note ::

    The ``web_test`` security group was one we created in the last tutoiral. If
    you haven't set up the group you will need to set security permissions
    manually or you won't be able to access your EC2 instance once it has
    loaded.

.. Note ::

    There is also an unbundle command so you can always extract an image from a
    set of AMI parts, make some modifications and re-bundle it again.

Custom Modules
==============

When you boot an instance, EC2 replaces the kernel with its own which is a Xen
2.6.16 kernel. This means you can't use a modified kernel but you can use
kernel modules. Although I haven't tried it yet I've read you do this by
booting the Fedora Development image, compiling the modules you want and
copying them to the ``/lib/modules/`` directory of the Debian image. You need
to use Fedora because the modules need compiling with GCC 4.0 but Etch uses 4.1 

http://elastic8.com/?q=node/2
http://tim.dysinger.net/2007/07/28/compiling-fuse-kernel-modules-for-debian-40-on-ec2/
http://info.rightscale.com/2007/8/7/recompiling-kernel-modules-for-ec2-instances

You can also get the default ones directly from Amazon but they seem to get set up anyway so you don't need to::

    wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz
    tar zxfv modules-2.6.16-ec2.tgz
    mv the files to /lib/modules
    depmod -a