1 Sep, 2007
Now that you have seen how to create an AMI from another AMI in my last tutorial its time to try a different technique: creating an AMI from a Xen image.
The advantages of this approach are:
You can install the operating system from scratch so you can be confident of the files you are adding
You can quickly convert between AMI and Xen images which allows you to host your images on either system
You can extensively test your images in an environment where you have total control over the whole set up
To begin with you need to setup Xen on Debian. Follow one of the many tutorials such as this one: http://jimmyg.org/2007/04/30/xen-on-thinkpad-r50e-debian-40-etch/
Once you have Xen working you can start thinking about using it for EC2 images.
First you need to install the ec2-ami-tools package on the host. Beause we are using Debian we need to first install alien to be able to handle RPM files so that we can use Amazon's AMI tools which are a series of useful Ruby programs:
sudo apt-get install alien curl curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm sudo alien -i ec2-ami-tools.noarch.rpm rm ec2-ami-tools.noarch.rpm
The tools require Ruby so you need to install them too:
sudo apt-get install ruby libopenssl-ruby1.8 rsync
We also need to do a bit of tweaking to make the compatible with Debian. First create a symlink:
sudo ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/
Then we need to patch-up the AMI tools to work on a debian-based system:
sudo vi /usr/lib/site_ruby/aes/amiutil/image.rb
and around line 149 change:
exec( 'for i in console null zero ; do /sbin/MAKEDEV -d ' + dev_dir + ' -x $i ; done' )
to:
exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")
Now you need to create the base image using the xen-create-image tool (you will have installed it if you followed the Xen tutorial linked to above). Obviously since the EC2 hardware is i686 hardware this needs to be done on a PC platform:
sudo mkdir /mnt/xen sudo xen-create-image --debootstrap --verbose --dir=/mnt/xen --size=1Gb --memory=512Mb --fs=ext3 --cache=yes --dist=etch --mirror=http://ftp.uk.debian.org/debian/ --hostname=debian_etch --initrd=/boot/initrd.img-2.6.18-4-xen-686 --kernel=/boot/vmlinuz-2.6.18-4-xen-686 --dhcp
These settings will create an ext3 filesystem of a Debian Etch install in an image which will get a network address using DHCP.
This will spit out a 1.0GB Debian Etch minimal image in /mnt/xen/domain/debian/disk.img don't worry about the memory size being different from the amount Xen uses. Even the kernel version doesn't matter too much since EC2 will replace it with its own.
At this point you have two choices:
Use xm create -c /etc/xen/debian.cfg to start the Xen virtual machine and configure it from there
Mount the image as a filesystem and make changes directly
If you are used to using Xen you'll have no problem making the changes directly to the booted image so I'll show you how to do it the other way.
Incidentally, you can also create everything entirely from scrath as described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/public-ami-guidelines.html
To create an AMI from the Xen image first mount the image:
sudo mkdir /mnt/image sudo mount -t ext3 -o loop /mnt/xen/domains/debian/disk.img /mnt/image
Then set up the fstab which EC2 will expect:
cd /mnt/image/etc sudo mv fstab fstab.xen sudo vi /etc/fstab.ec2
Add the following content:
/dev/sda1 / ext3 defaults 1 1 /dev/sda2 /mnt ext3 defaults,user_xattr 1 2 /dev/sda3 swap swap defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0
Then:
sudo ln -sf fstab.ec2 fstab
At this point you need to set a root password or follow the instructions in the next section to make a public AMI.
If you want to make your AMIs available to the publis there are some extra steps you need to take described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-sharingamis.html
Basically you need to perform three steps:
Randomise the root password the first time an instance boots
Change the default SSH setup
Set up the public part of the SSH key pair so that the user can sign in
Here is how you can do this on Debian Etch.
First tweak the SSH config as recommended by Amazon:
sed -i -e 's/PermitRootLogin no/#PermitRootLogin no/g' /mnt/image/etc/ssh/sshd_config cat <<EOL >> /mnt/image/etc/ssh/sshd_config UseDNS no PermitRootLogin without-password EOL
Then alter the boot process to set up the public key. Edit this file:
vim mnt/image/etc/rc.local
So it looks like this:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
if [ ! -d /root/.ssh ] ; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
wget -q -P /tmp http://169.254.169.254/2007-03-01//meta-data/public-keys/0/openssh-key
cat /tmp/openssh-key > /tmp/my-key
rm /tmp/openssh-key
if [ $? -eq 0 ] ; then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm /tmp/my-key
fi
if [ -f "/.firstrun" ] ; then
dd if=/dev/urandom count=50|md5sum > /tmp/p.out
POUT=`cat /tmp/p.out | cut -f1 -d' '`
rm -f /tmp/p.out
/usr/sbin/usermod -p $POUT root
rm -f /.firstrun
fi
exit 0
Then create the .firstrun file so that the password changing is triggered on boot:
touch /mnt/image/.firstrun
Appending the public key to the authorized keys file on every boot might seem overkill but this is how Amazon does it and it is unlikely you'll be rebooting your image too many times anyway.
Now you can unmount the image:
cd / sudo umount /mnt/image
Caution!
When creating your own image for public it is important to ensure you don't leave any of the AWS identifiers, account number, certificates or keys anywhere on the image. There are some tips for how to avoid this at the bottom of this page.
This is very similar to before but rather than using ec2-bundle-vol you use ec2-bundle-image. You'll need the X.509 certificate and private key to bundle the image as before.
The name given by the option -p determines the name of the manifest file. In this case we chose debian_etch:
ec2-bundle-image -p debian_etch -i /mnt/xen/domains/debian/disk.img -k ~/Desktop/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -c ~/Desktop/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 Splitting /tmp/debian_etch.tar.gz.enc... Created debian_etch.part.00 Created debian_etch.part.01 Created debian_etch.part.02 ... Created debian_etch.part.51 Created debian_etch.part.52 Generating digests for each part... Digests generated. Creating bundle manifest... ec2-bundle-image complete.
By default images are written to /tmp unless you use the -d option. Let's check they are there:
james@dirac:/$ ls -lh /tmp/debian_etch.* -rw-r--r-- 1 james james 9.0K 2007-08-27 21:48 /tmp/debian_etch.manifest.xml -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.00 -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.01 -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.02 ... -rw-r--r-- 1 james james 10M 2007-08-27 21:48 /tmp/debian_etch.part.51 -rw-r--r-- 1 james james 8.9M 2007-08-27 21:48 /tmp/debian_etch.part.52
As you can see, the Debian Etch install has 53 parts which is quite a lot more than the 23 my previous tutorial had.
You can now upload it to S3 but unless your network connection is as fast as the 250Mb/s each EC2 instance has this will take a lot longer than it took to to upload the parts from EC2 which is what we did in the last tutorial:
ec2-upload-bundle -b james-music -m /tmp/debian_etch.manifest.xml -a <AWS Access Key> -s <AWS shared key>
Once the image is uploaded we need to register it. Load up a Python prompt again and use boto to create a connection before registering the image:
>>> from boto.ec2.connection import EC2Connection
>>> conn = EC2Connection('<AWS Access Key>', '<AWS shared key>')
>>> conn.register_image('james-music/debian_etch.manifest.xml')
u'ami-ddec09b4'
You should now be able to start the image:
>>> for i, image in enumerate(images):
... if image.location.startswith('james-music'):
... print "%s, %s, %s"%(i, image.id, image.location)
Once you have found the image number of the image (in this case 40) you can start it:
>>> image = images[40] >>> reservation = image.run(key_name='gsg-keypair', security_groups=['web_test'])
And check progress:
>>> reservation.instances[0].update() >>> reservation.instances[0].state u'pending'
Note
The web_test security group was one we created in the last tutoiral. If you haven't set up the group you will need to set security permissions manually or you won't be able to access your EC2 instance once it has loaded.
Note
There is also an unbundle command so you can always extract an image from a set of AMI parts, make some modifications and re-bundle it again.
When you boot an instance, EC2 replaces the kernel with its own which is a Xen 2.6.16 kernel. This means you can't use a modified kernel but you can use kernel modules. Although I haven't tried it yet I've read you do this by booting the Fedora Development image, compiling the modules you want and copying them to the /lib/modules/ directory of the Debian image. You need to use Fedora because the modules need compiling with GCC 4.0 but Etch uses 4.1
http://elastic8.com/?q=node/2 http://tim.dysinger.net/2007/07/28/compiling-fuse-kernel-modules-for-debian-40-on-ec2/ http://info.rightscale.com/2007/8/7/recompiling-kernel-modules-for-ec2-instances
You can also get the default ones directly from Amazon but they seem to get set up anyway so you don't need to:
wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz tar zxfv modules-2.6.16-ec2.tgz mv the files to /lib/modules depmod -a
Copyright James Gardner 1996-2020 All Rights Reserved. Admin.