Migrating a 32bit server to a 64bit server as a Xen image
I’ve got a 32bit intel Mac Mini server running Ubuntu Dapper Drake and I wanted to package it up as a Xen image to run on an AMD 64 server running Debian. Here’s what I did.
First create the Xen image on the 64bit server:
sudo mkdir /var/xen sudo xen-create-image --debootstrap --dir=/var/xen --size=60Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm2 --ip 78.47.146.253 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb
Then mount it:
sudo mkdir /mount/vm2 sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2
Then sync the filesystems:
cd /mount/vm2/ sudo rsync -aHxvzgop --progress --delete --numeric-ids root@pauli.3aims.com:/ .
Now shutdown the old server because any changes will be lost.
Create a new fstab for the copy on the 64bit server after backing up the old one:
sudo cp -p /mount/vm2/etc/fstab /mount/vm2/etc/fstab.bak
Make the new one like this:
/dev/sda1 / ext3 errors=remount-ro 0 1 proc /proc proc rw,nodev,nosuid,noexec 0 0 /dev/sda2 none swap sw 0 0
Unmount the disk and boot the image:
# You can't unmount a directory you are still in so we move out. cd .. sudo umount /mount/vm2 # You might want to skip this, it takes ages. sudo cp /var/xen/domains/vm2/disk.img /var/xen/domains/vm2/disk.img.bak sudo xm create -c /etc/xen/vm2.cfg
Check the server is up and can ping and be pinged then change the DNS settings to point all the services to the new server.
You might need the lib64 modules:
sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2 sudo cp -r /lib/modules/2.6.18-4-xen-amd64 /mount/vm2/lib/modules/ sudo ln -s /mount/vm2/lib /mount/vm2/lib64 sudo umount /mount/vm2 sudo xm create -c /etc/xen/vm2.cfg
Also need to mess with any files containing the wrong IP. Some of these commands might be handy:
sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2 sudo cp -pr /mount/vm2/etc /mount/vm2/etc.bak cd /mount/vm2/etc sudo grep -rl "212.69.38.27" . 2> /dev/null sudo perl -pi -e 's/212.69.38.27/78.47.146.253/g' /apache2/vhosts/* sudo grep -rl "212.69.38.27" . 2> /dev/null cd / sudo umount /mount/vm2 sudo xm create -c /etc/xen/vm2.cfg
Finally you’ll need to update your networking configuration, here’s how mine look, /etc/network/interfaces:
auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 78.47.146.253 network 78.47.146.248 netmask 255.255.255.248 up route add 78.46.35.5 dev eth0 up route add default gw 78.46.35.5 down route del default gw 78.46.35.5 down route del 78.46.35.5 dev eth0
and /etc/resolve.conf:
### Hetzner Online AG installimage # nameserver config nameserver 213.133.99.99 nameserver 213.133.100.100 nameserver 213.133.98.98
I run ISPConfig too so I also need to update its database so changes it makes for bind are correct:
mysql> update dns_a set ip_adresse='78.47.146.253' where ip_adresse='212.69.38.27'; mysql> update dns_isp_dns set dns_soa_ip= '78.47.146.253' where dns_soa_ip='212.69.38.27';
Pylons 0.9.7-dev Nginx Deployment on a Two-Way Routed Xen Setup
On Dom0 set up port forwarding from port 80 to the DomU running NginX, in this case 10.0.0.1:
sudo -i iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 80 -j DNAT --to 10.0.0.1:80 exit
These get lost on each reboot unless you put them into a script in /etc/network/if-up.d/iptables but that’s OK for the moment.
First install Nginx:
sudo apt-get install nginx
If you visit the URL of the Dom0 server you should see the Welcome to nginx! message served from Nginx on the DomU.
Next you need to setup Python and Pylons. Install python:
sudo apt-get install python mercurial
Get vitualenv:
wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-0.9.1.tar.gz tar zxfv virtualenv-0.9.1.tar.gz cp virtualenv-0.9.1/virtualenv.py ./ rm virtualenv-0.9.1.tar.gz rm -r virtualenv-0.9.1
Decide where you want your Pylons applications. I like mine in /var/web so lets create a test project:
sudo mkdir /var/web sudo mkdir /var/web/test sudo chown james:james /var/web/test
Setup a virtual environment:
james@vm2:~$ python virtualenv.py /var/web/test New python executable in /var/web/test/bin/python Installing setuptools.............................done. james@vm2:~$ cd /var/web/test
Now you can install Pylons:
mkdir dep cd dep hg clone http://pylonshq.com/hg/pylons-dev cd pylons-dev ../../bin/python setup.py develop
Now create a project:
mkdir src cd src ../bin/paster create -t pylons Test
Now edit Test/test/config/middleware.py and set use_webop=True. Serve the app with:
cd Test/ ../../bin/python setup.py develop ../../bin/easy_install sqlalchemy <-- So that beaker doesn't break ../../bin/paster serve --reload development.ini
That’s it. Now you just need to setup Nginx to reverse proxy to the paste server running on port 5000.
Edit the /etc/nginx/nginx.conf file so that the location / part looks like this:
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_pass http://127.0.0.1:5000;
proxy_redirect default;
}
You’ll then need to restart nginx and the paster server:
sudo /etc/init.d/nginx restart ../../bin/paster serve --reload development
If you press refresh in your browser you should see the Pylons welcome page.
Debain 4.0 Etch AMD 64 X2 Server from Hetzner
I bought a new server from Hetzner (German) on Sunday 4th November and on the evening of the 5th I received my sign in details by email in both German and English for the server, already installed with the 64bit version of Debian Etch. This post documents my progress setting it up.
Exploring the Control Panel
The first email you get gives you a sign in to the Hetzner control panel (they call it Robot). Everything is in German but here are some translations of the interface (thanks to Google Translate):
Verwaltung Administration
Sicherheit Security
E-Mail-Adressen Emails
Support-Anfragen Support requests
Status-Benachrichtigung Status Notification
Newsletter Newsletter
RIPE-Registration RIPE-Registration
Rechnungen Bills
Offene Posten Open Item
Traffic-Statistik Traffic Statistics
Traffic-Limit-Reporting Traffic limit Reporting
Tagesbericht Daily Report
Monatsbericht Monthly Report
Jahresbericht Annual Report
Leistungsübersicht Performance Overview
Resetaufträge Reset orders
Rescuesystem Rescue System
VNC-Installation VNC-Installation
neuen Server bestellen New server order
Separater Admin-Zugang Separate Admin Access
Reverse-DNS-Einträge Reverse-DNS
Eintrag anlegen Entry
Eintrag löschen Delete Entry
Dokumentation Documentation
Daten-Export-Schnittstelle Data Export interface
Perhaps the two most useful entries are Resetaufträge and Rescuesystem. The Resetaufträge page gives you three main options:
``Automatischen Hardware-Reset auslösen`` - Perform an automatic hardware reset.
STRG+ALT+ENTF an den Server senden - Send a CTRL+ALT+DEL signal to the server
Manuellen Hardware-Reset beauftragen - Request a member of the Hetzner staff manually reboot your server.
There is a warning with this last one which translates as:
Please note that manual Hardware-Resets only during our business hours, Monday through Friday 6:30-22:45 pm, Saturday 10-17 hours delay Edited. Outside of business hours, you can Robot menu under "support requests" our 24-h-Rufbereitschaft contact. Also available in the Guide for support operations valuable information.
I’m not sure if there is a charge for the manual reset.
The Rescuesystem sets up a config file for 5 minutes so that if you server reboots it will boot over the network using DHCP into a Hetzner-specific rescue system as long as you have a modern machine (mine was bought in November 2007 and works perfectly). The message translates as:
When activating the rescue system is a DHCP server on our configuration file. When rebooted your server will be booted from the network grabs this configuration file for the rescue system and loads a minimal base system from our TFTP. You can rescue the system will use as long as you need it. The order for the rescue system remains 5 minutes activated. If you then reboot your server will return your usual system of hard.
Caution:
Whether your server about the rescue system can boot depends on whether the network card on the server network is set to boot. This is only the latest in servers as a default by us so. If it is still not set, the server after activating the rescue system at the next reboot with the existing system from the hard disks. Should your server will be converted, then send an e-mail to support@hetzner.de or if you know where your server is equal to the appropriate data center. Indicate whether the change immediately, or at any time any of us can be carried out or by appointment only. The conversion is associated with down-time of about 5 minutes. We ask for your understanding if it in the handling of the changeover contracts to small delays can occur.
You have to choose whether to use the 32bit or 64bit system and then when you clcik the Aktivieren button the rescue system config is set up and you are given a password you’ll need to login then if you reboot your server within 5 mins you will be booted in to the rescue system. Very handy.
It is well worth reading the help files (in this case run through Google translate) for more information about the Hetzner setup.
Checking the Stats
Here are some stats showing what a fresh setup looks like:
Debian-40-etch-64-minimal:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 365G 520M 346G 1% /
tmpfs 2.0G 0 2.0G 0% /lib/init/rw
udev 10M 36K 10M 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
Debian-40-etch-64-minimal:~# cat /etc/issue
Debian GNU/Linux 4.0 \n \l
Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
total used free shared buffers cached
Mem: 3926 73 3853 0 12 19
-/+ buffers/cache: 41 3885
Swap: 2055 0 2055
These commands show I’m using Debian 4.0 with a 364Gb hard disk on an AMD 64 machine and that I have 3853Mb of free RAM, using only 73Mb in total.
Re-Installing The Operating System
The first time I tried to setup the server with Xen it all went horribly wrong. This is because Xen expects Grub to be present but the default Hetzner Etch AMD64 image comes with Lilo. I tried removing lilo and setting up grub manually but to no avail although I did learn a lot about Grub as a result of my experiments and can recommend this excellent guide to grub.
Following the instructions in the previous section I initialed the rescue system, rebooted and then used SSH to connect. Since this isn’t the same install you get an error from your SSH client until you remove the existing host from the known hosts file on your local machine:
james@dirac:~$ ssh root@doppler.3aims.com @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 50:10:6b:5f:dc:e7:45:17:73:00:d0:50:f0:0e:48:be. Please contact your system administrator. Add correct host key in /home/james/.ssh/known_hosts to get rid of this message. Offending key in /home/james/.ssh/known_hosts:1 RSA host key for doppler.3aims.com has changed and you have requested strict checking. Host key verification failed.
Sign in as root with the password you got after clicking Aktivieren. Once you are in you see this:
Linux rescue 2.6.22.2 #2 SMP Tue Aug 28 09:28:15 CEST 2007 x86_64 ------------------------------------------------------------------ Welcome to the Hetzner Rescue System. This Rescue System is based on Debian 4.0 (etch) with a newer kernel. You can install software like in a normal system. To install a new operating system from one of our prebuilt images, run 'installimage' and follow the instructions. For more information take a look at http://wiki.hetzner.de ------------------------------------------------------------------ root@rescue ~ #
I ran the install script installimage choosing 64bit Debian 4.0 Etch and then editing the following settings in the config file:
- FORMATDRIVE2 = 1
- This sets up the second hard drive so you can actually use it.
- BOOTLOADER = grub
- This sets up the bootloader to use grub so that you can install Xen.
When you’re done the install sets to work:
Hetzner Online AG - installimage
#~ server will be installed now. this will take a few minutes.
#~ you can abort at any time with CTRL+C ..
#~ ( init) ~ reading vars... [ OK ]
#~ ( 1/11) ~ deleting partitions... [ OK ]
#~ ( 2/11) ~ creating partitions and fstab... [ OK ]
#~ ( 3/11) ~ formatting partitions... [ OK ]
#~ ( 4/11) ~ mounting partitions... [ OK ]
#~ ( 5/11) ~ extracting imagefile from local... [ OK ]
#~ ( 6/11) ~ setting up network config for eth0... [ OK ]
#~ ( 7/11) ~ chrooting some commands... [ OK ]
#~ ( 8/11) ~ clearing logfiles... [ OK ]
#~ ( 9/11) ~ setting up some files... [ OK ]
#~ (10/11) ~ setting up rootpassword... [ OK ]
#~ (11/11) ~ setting up bootloader grub... [ OK ]
#~~~ INSTALLATION COMPLETE ~~~#
you may now reboot into your new system
you can login to your new system with the same
password as you logged in into the rescue system
root@rescue ~ #
When you type reboot the server reboots back into a new Debian Etch install, this time using Grub and with the second hard disk available.
Installing Xen
Now that we have grub set up, installing Xen is as simple as entering two commands:
apt-get install xen-linux-system-2.6.18-4-xen-amd64 reboot
The OS will then reboot into the Xen DomU. You’ll probably want the 5 free IP addresses Hetzner offer so that each Xen virtual machine can have its own IP address. You can request them from the Support-Anfragen section of the control panel. Choose the Subnetze für DS2000/DS3000/DS5000/DS7000/DS8000/DS9000 beantragen option.
According to this article about Xen on Hetzner you cannot use bridging on Hetzner. Instead you have to use routing via DomU.
Edit /etc/xen/xend-config.sxp so that it only contains these lines:
# -*- sh -*- (network-script network-route) (vif-script vif-route)
Restart xend:
/etc/init.d/xend restart
According to this page you will also need to run this command:
echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
Xen is now setup ready for you to create some images.
Checking the Stats Again
Here are some stats showing what the new Xen setup looks like:
Debian-40-etch-64-minimal:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 365G 612M 346G 1% /
tmpfs 1.7G 0 1.7G 0% /lib/init/rw
udev 10M 36K 10M 1% /dev
tmpfs 1.7G 0 1.7G 0% /dev/shm
Debian-40-etch-64-minimal:~# cat /etc/issue
Debian GNU/Linux 4.0 \n \l
Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-4-xen-amd64 #1 SMP Fri May 4 02:40:51 UTC 2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
total used free shared buffers cached
Mem: 3366 200 3165 0 1 15
-/+ buffers/cache: 182 3183
Swap: 2055 0 2055
Debian-40-etch-64-minimal:~# fdisk -l
Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 262 2104514+ 82 Linux swap / Solaris
/dev/sda2 263 48641 388604317+ 83 Linux
Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
As you can see, using Xen takes a bit more memory but it doesn’t look like the second disk has any paritions or has been formatted by the Hetzner install after all.
Mounting the Other Hard Disk
Let’s create a partition:
Debian-40-etch-64-minimal:~# fdisk -u /dev/sdb The number of cylinders for this disk is set to 48641. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First sector (63-781422767, default 63): Using default value 63 Last sector or +size or +sizeM or +sizeK (63-781422767, default 781422767): Using default value 781422767 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
And then format it using an ext3 filesystem:
Debian-40-etch-64-minimal:~# /sbin/mkfs -t ext3 /dev/sdb1
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
48840704 inodes, 97677838 blocks
4883891 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2981 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Add this line to /etc/fstab:
/dev/sdb1 /mount/sdb1 ext3 defaults 0 0
Then mount the partition:
mkdir /mount mkdir /mount/sdb1 mount /dev/sdb1
Checking the stats again we have:
Debian-40-etch-64-minimal:~# fdisk -l Disk /dev/sda: 400.0 GB, 400088457216 bytes 255 heads, 63 sectors/track, 48641 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 262 2104514+ 82 Linux swap / Solaris /dev/sda2 263 48641 388604317+ 83 Linux Disk /dev/sdb: 400.0 GB, 400088457216 bytes 255 heads, 63 sectors/track, 48641 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 1 48642 390711352+ 83 Linux Debian-40-etch-64-minimal:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 365G 612M 346G 1% / tmpfs 1.7G 0 1.7G 0% /lib/init/rw udev 10M 40K 10M 1% /dev tmpfs 1.7G 0 1.7G 0% /dev/shm /dev/sdb1 367G 195M 348G 1% /mount/sdb1 Much better, 674Gb free space.
Setting up Domain and Host Names
All my machines have hostnames named after famous Physicists so this one is doppler.3aims.com named after Christian Doppler. I set up its DNS entries through the DynDNS control panel as follows:
doppler.3aims.com. 1440 A 78.46.35.5
Then I SSH’d into the machine, set a new root password and changed /etc/hostname to doppler and replaced the line after localhost with this in /etc/hosts:
78.46.35.5 doppler doppler.3aims.com
You then run hostname doppler to save yourself a reboot. More info on etch hostnames here:
Debian-40-etch-64-minimal:~# vim /etc/hostname Debian-40-etch-64-minimal:~# vim /etc/hosts Debian-40-etch-64-minimal:~# hostname doppler
All good so far.
Basic Configuration
Rather than doing everything as root it is best to add a normal user account and grant sudo privileges:
Debian-40-etch-64-minimal:~# apt-get install sudo
Debian-40-etch-64-minimal:~# adduser james
Adding user `james' ...
Adding new group `james' (1000) ...
Adding new user `james' (1000) with group `james' ...
Creating home directory `/home/james' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for james
Enter the new value, or press ENTER for the default
Full Name []: James Gardner
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [y/N] y
You can then grant privileges with the visudo command and at the end of the file add james ALL=(ALL) ALL then become james with su james.
Now you can become james with su james. Notice that the Bash prompt will now pick up the hostname change:
Debian-40-etch-64-minimal:~# su james james@doppler:/root$
Locales
The first thing to set up are locales before you install any extra packages:
$ sudo dpkg-reconfigure locales
I chose en_GB.UTF-8 UTF-8 and also left en_US.ISO-8859-15 ISO-8859-15 selected too, choosing en_GB.UTF-8 UTF-8 as the default.
Timezones
The default setup has the German timezone. Lets change that to UTC:
james@doppler:/root$ sudo tzconfig
Your current time zone is set to Europe/Berlin
Do you want to change that? [n]: y
Please enter the number of the geographic area in which you live:
1) Africa 7) Australia
2) America <img src='http://jimmyg.org/wp-includes/images/smilies/icon_cool.gif' alt='8)' class='wp-smiley' /> Europe
3) US time zones 9) Indian Ocean
4) Canada time zones 10) Pacific Ocean
5) Asia 11) Use System V style time zones
6) Atlantic Ocean 12) None of the above
Then you will be shown a list of cities which represent the time zone
in which they are located. You should choose a city in your time zone.
Number: 8
Amsterdam Andorra Athens Belfast Belgrade Berlin Bratislava Brussels
Bucharest Budapest Chisinau Copenhagen Dublin Gibraltar Guernsey Helsinki
Isle_of_Man Istanbul Jersey Kaliningrad Kiev Lisbon Ljubljana London
Luxembourg Madrid Malta Mariehamn Minsk Monaco Moscow Nicosia Oslo Paris
Podgorica Prague Riga Rome Samara San_Marino Sarajevo Simferopol Skopje
Sofia Stockholm Tallinn Tirane Tiraspol Uzhgorod Vaduz Vatican Vienna
Vilnius Volgograd Warsaw Zagreb Zaporozhye Zurich
Please enter the name of one of these cities or zones
You just need to type enough letters to resolve ambiguities
Press Enter to view all of them again
Name: [] London
Your default time zone is set to 'Europe/London'.
Local time is now: Fri Nov 9 16:53:52 GMT 2007.
Universal Time is now: Fri Nov 9 16:53:52 UTC 2007.
SSH Config
Next we’ll change the default SSH configuration to make it more secure:
Make a backup of /etc/ssh/sshd_config and then check or change the following:
Port 30000 <-- change to a port other than 22 PermitRootLogin no AllowUsers james
These are self-explainatory. This will disable root logins, allow only the user james to login, only on port 30000. If you like you can also set:
PasswordAuthentication no
This will mean you will only be able to login via an SSH private/public key pair which you will have needed to set up in advance. I didn’t do this.
Note that if you get them wrong you might not be able to login to the machine so be careful. Once you happy with the settings restart:
doppler:~# /etc/init.d/ssh restart Restarting OpenBSD Secure Shell server: sshd.
Don’t exit that shell though before you’ve loaded up another terminal and checked you can connect again. This time you’ll need to use this:
ssh james@doppler.3aims.com -p 30000
If you do make a mistake and can’t reconnect, boot into the rescue system and mount the drive with this command:
mount /dev/sda2 /mnt -t ext3
You will then be able to edit the file as /mnt/etc/ssh/sshd_config and then reboot back into your normal setup.
Also, publishing the settings you’ve chosen on a blog like this defeats the purposes of setting them so choose something different if you are going to publish them!
Firewall
You set up the firewall walls using iptables. This can’t be done using sudo so you have to become the root user:
james@doppler:~$ sudo -i Password: doppler:~# whoami root
You can see the firwall rules like this:
doppler:~# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
If there were any rule you could save them like this:
iptables-save > /etc/iptables.current.rules
Here’s a simple firewall configuration from the Slicehost Blog which you can use to block all access apart from on SSH, HTTP and HTTPS. Save this as /etc/iptables.test.rules. Feel free to modify it for your own use and be aware that if you have didn’t change the SSH port to 30000 or you chose another port, you’ll need to update the firewall config before you apply it.
Now load the rules with:
iptables-restore < /etc/iptables.test.rules
Again, test that you can still sign in using SSH before you exit the shell.
If you list the rules again you will see this:
doppler:~# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT 0 -- anywhere anywhere REJECT 0 -- anywhere loopback/8 reject-with icmp-port-unreachable ACCEPT 0 -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT tcp -- anywhere anywhere tcp dpt:www ACCEPT tcp -- anywhere anywhere tcp dpt:https ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:30000 ACCEPT icmp -- anywhere anywhere icmp echo-request LOG 0 -- anywhere anywhere limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: ' REJECT 0 -- anywhere anywhere reject-with icmp-port-unreachable Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT 0 -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination ACCEPT 0 -- anywhere anywhere
Once you are happy with the rules, save them permanently:
iptables-save > /etc/iptables.up.rules
Now we need to ensure that the iptables rules are applied when we reboot the server. At the moment, the changes will be lost and it will go back to allowing everything from everywhere.
Edit /etc/network/interfaces and add a pre-up line (shown below) just after iface lo inet loopback:
... auto lo iface lo inet loopback pre-up iptables-restore < /etc/iptables.up.rules ...
This line will restore the iptables rules from the /etc/iptables.up.rules file.
Updating
Next I installed any updates from the repositories listed in /etc/apt/sources.list:
sudo apt-get update sudo apt-get upgrade
perl-base was the only package upgraded. After I’d made all these changes I rebooted to ensure they were all applied.
That’s as far as I can get until I recieve the new IP addresses. Might be some updates to the above if I run into any problems.
Note
Now you can create your virtual machines. See my previous article for how to do this. Start reading from the xen-create-image line.
Custom Debian EC2 AMIs From Xen Images
Now that you have seen how to create an AMI from another AMI in my last tutorial its time to try a different technique: creating an AMI from a Xen image.
The advantages of this approach are:
- You can install the operating system from scratch so you can be confident of the files you are adding
- You can quickly convert between AMI and Xen images which allows you to host your images on either system
- You can extensively test your images in an environment where you have total control over the whole set up
To begin with you need to setup Xen on Debian. Follow one of the many tutorials such as this one: http://jimmyg.org/2007/04/30/xen-on-thinkpad-r50e-debian-40-etch/
Once you have Xen working you can start thinking about using it for EC2 images.
First you need to install the ec2-ami-tools package on the host. Beause we are using Debian we need to first install alien to be able to handle RPM files so that we can use Amazon’s AMI tools which are a series of useful Ruby programs:
sudo apt-get install alien curl curl -O http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm sudo alien -i ec2-ami-tools.noarch.rpm rm ec2-ami-tools.noarch.rpm
The tools require Ruby so you need to install them too:
sudo apt-get install ruby libopenssl-ruby1.8 rsync
We also need to do a bit of tweaking to make the compatible with Debian. First create a symlink:
sudo ln -s /usr/lib/site_ruby/aes /usr/lib/ruby/1.8/
Then we need to patch-up the AMI tools to work on a debian-based system:
sudo vi /usr/lib/site_ruby/aes/amiutil/image.rb
and around line 149 change:
exec( 'for i in console null zero ; do /sbin/MAKEDEV -d ' + dev_dir + ' -x $i ; done' )
to:
exec("cd #{dev_dir} && /sbin/MAKEDEV console && /sbin/MAKEDEV std && /sbin/MAKEDEV generic")
Creating the Xen Image
Now you need to create the base image using the xen-create-image tool (you will have installed it if you followed the Xen tutorial linked to above). Obviously since the EC2 hardware is i686 hardware this needs to be done on a PC platform:
sudo mkdir /mnt/xen sudo xen-create-image --debootstrap --verbose --dir=/mnt/xen --size=1Gb --memory=512Mb --fs=ext3 --cache=yes --dist=etch --mirror=http://ftp.uk.debian.org/debian/ --hostname=debian_etch --initrd=/boot/initrd.img-2.6.18-4-xen-686 --kernel=/boot/vmlinuz-2.6.18-4-xen-686 --dhcp
These settings will create an ext3 filesystem of a Debian Etch install in an image which will get a network address using DHCP.
This will spit out a 1.0GB Debian Etch minimal image in /mnt/xen/domain/debian/disk.img don’t worry about the memory size being different from the amount Xen uses. Even the kernel version doesn’t matter too much since EC2 will replace it with its own.
At this point you have two choices:
- Use xm create -c /etc/xen/debian.cfg to start the Xen virtual machine and configure it from there
- Mount the image as a filesystem and make changes directly
If you are used to using Xen you’ll have no problem making the changes directly to the booted image so I’ll show you how to do it the other way.
Incidentally, you can also create everything entirely from scrath as described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/public-ami-guidelines.html
To create an AMI from the Xen image first mount the image:
sudo mkdir /mnt/image sudo mount -t ext3 -o loop /mnt/xen/domains/debian/disk.img /mnt/image
Then set up the fstab which EC2 will expect:
cd /mnt/image/etc sudo mv fstab fstab.xen sudo vi /etc/fstab.ec2
Add the following content:
/dev/sda1 / ext3 defaults 1 1 /dev/sda2 /mnt ext3 defaults,user_xattr 1 2 /dev/sda3 swap swap defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0
Then:
sudo ln -sf fstab.ec2 fstab
At this point you need to set a root password or follow the instructions in the next section to make a public AMI.
Extra Steps For Public AMIs
If you want to make your AMIs available to the publis there are some extra steps you need to take described here: http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-sharingamis.html
Basically you need to perform three steps:
- Randomise the root password the first time an instance boots
- Change the default SSH setup
- Set up the public part of the SSH key pair so that the user can sign in
Here is how you can do this on Debian Etch.
First tweak the SSH config as recommended by Amazon:
sed -i -e 's/PermitRootLogin no/#PermitRootLogin no/g' /mnt/image/etc/ssh/sshd_config cat <<EOL >> /mnt/image/etc/ssh/sshd_config UseDNS no PermitRootLogin without-password EOL
Then alter the boot process to set up the public key. Edit this file:
vim mnt/image/etc/rc.local
So it looks like this:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
if [ ! -d /root/.ssh ] ; then
mkdir -p /root/.ssh
chmod 700 /root/.ssh
fi
# Fetch public key using HTTP
wget -q -P /tmp http://169.254.169.254/2007-03-01//meta-data/public-keys/0/openssh-key
cat /tmp/openssh-key > /tmp/my-key
rm /tmp/openssh-key
if [ $? -eq 0 ] ; then
cat /tmp/my-key >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
rm /tmp/my-key
fi
if [ -f "/.firstrun" ] ; then
dd if=/dev/urandom count=50|md5sum > /tmp/p.out
POUT=`cat /tmp/p.out | cut -f1 -d' '`
rm -f /tmp/p.out
/usr/sbin/usermod -p $POUT root
rm -f /.firstrun
fi
exit 0
Then create the .firstrun file so that the password changing is triggered on boot:
touch /mnt/image/.firstrun
Appending the public key to the authorized keys file on every boot might seem overkill but this is how Amazon does it and it is unlikely you’ll be rebooting your image too many times anyway.
Now you can unmount the image:
cd / sudo umount /mnt/image
Caution!
When creating your own image for public it is important to ensure you don’t leave any of the AWS identifiers, account number, certificates or keys anywhere on the image. There are some tips for how to avoid this at the bottom of this page.
Bundling
This is very similar to before but rather than using ec2-bundle-vol you use ec2-bundle-image. You’ll need the X.509 certificate and private key to bundle the image as before.
The name given by the option -p determines the name of the manifest file. In this case we chose debian_etch:
ec2-bundle-image -p debian_etch -i /mnt/xen/domains/debian/disk.img -k ~/Desktop/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -c ~/Desktop/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 Splitting /tmp/debian_etch.tar.gz.enc... Created debian_etch.part.00 Created debian_etch.part.01 Created debian_etch.part.02 ... Created debian_etch.part.51 Created debian_etch.part.52 Generating digests for each part... Digests generated. Creating bundle manifest... ec2-bundle-image complete.
By default images are written to /tmp unless you use the -d option. Let’s check they are there:
james@dirac:/$ ls -lh /tmp/debian_etch.* -rw-r--r-- 1 james james 9.0K 2007-08-27 21:48 /tmp/debian_etch.manifest.xml -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.00 -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.01 -rw-r--r-- 1 james james 10M 2007-08-27 21:47 /tmp/debian_etch.part.02 ... -rw-r--r-- 1 james james 10M 2007-08-27 21:48 /tmp/debian_etch.part.51 -rw-r--r-- 1 james james 8.9M 2007-08-27 21:48 /tmp/debian_etch.part.52
As you can see, the Debian Etch install has 53 parts which is quite a lot more than the 23 my previous tutorial had.
You can now upload it to S3 but unless your network connection is as fast as the 250Mb/s each EC2 instance has this will take a lot longer than it took to to upload the parts from EC2 which is what we did in the last tutorial:
ec2-upload-bundle -b james-music -m /tmp/debian_etch.manifest.xml -a <AWS Access Key> -s <AWS shared key>
Once the image is uploaded we need to register it. Load up a Python prompt again and use boto to create a connection before registering the image:
>>> from boto.ec2.connection import EC2Connection
>>> conn = EC2Connection('<AWS Access Key>', '<AWS shared key>')
>>> conn.register_image('james-music/debian_etch.manifest.xml')
u'ami-ddec09b4'
You should now be able to start the image:
>>> for i, image in enumerate(images):
... if image.location.startswith('james-music'):
... print "%s, %s, %s"%(i, image.id, image.location)
Once you have found the image number of the image (in this case 40) you can start it:
>>> image = images[40] >>> reservation = image.run(key_name='gsg-keypair', security_groups=['web_test'])
And check progress:
>>> reservation.instances[0].update() >>> reservation.instances[0].state u'pending'
Note
The web_test security group was one we created in the last tutoiral. If you haven’t set up the group you will need to set security permissions manually or you won’t be able to access your EC2 instance once it has loaded.
Note
There is also an unbundle command so you can always extract an image from a set of AMI parts, make some modifications and re-bundle it again.
Custom Modules
When you boot an instance, EC2 replaces the kernel with its own which is a Xen 2.6.16 kernel. This means you can’t use a modified kernel but you can use kernel modules. Although I haven’t tried it yet I’ve read you do this by booting the Fedora Development image, compiling the modules you want and copying them to the /lib/modules/ directory of the Debian image. You need to use Fedora because the modules need compiling with GCC 4.0 but Etch uses 4.1
http://elastic8.com/?q=node/2 http://tim.dysinger.net/2007/07/28/compiling-fuse-kernel-modules-for-debian-40-on-ec2/ http://info.rightscale.com/2007/8/7/recompiling-kernel-modules-for-ec2-instances
You can also get the default ones directly from Amazon but they seem to get set up anyway so you don’t need to:
wget http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz tar zxfv modules-2.6.16-ec2.tgz mv the files to /lib/modules depmod -a
Windows XP on Debian with VMWare
There are two really good tutorials on VMWare and Debain. The first describes how to install VMWare on Debian and the second describes how to create a Windows XP image.
You need a serial number but the basic commands to set it up are:
tar zxfv VMware-server-1.0.2-39867.tar.gz
cd vmware-server-distrib/
sudo apt-get install linux-headers-`uname -r` libx11-6 libx11-dev x-window-system-core
sudo apt-get install x-window-system xspecs libxtst6 psmisc build-essential
sudo ./vmware-install.pl
rm -fr vmware-server-distrib/
rm -f VMware-server*.tar.gz
A new icon will appear in the Applications->System Tools menu called VMWare Server Console.
You then need to do this to fix a bug which causes VMWare to crash when you browse for images:
cd /usr/lib/vmware/lib/libgcc_s.so.1/
sudo mv libgcc_s.so.1 libgcc_s.so.1.old
sudo ln -sf /lib/libgcc_s.so.1
cd /usr/lib/vmware/lib/libpng12.so.0
sudo mv libpng12.so.0 libpng12.so.0.old
sudo ln -sf /usr/lib/libpng12.so.0
If you want sound working in Windows XP you need to add a sound card to the virtual device. Start VMWare from the console with superuser privileges:
sudo vmware &
When you have browsed to open the Windows XP image, click the Edit virtual machine settings button then in the dialog which appears click the Add… button underneath the list of devices. Choose a sound adapter and finish the wizard which appears. When you next start the Windows XP image you should have sound.
Now, if only I could get VMware running at the same time as Xen…
