James Gardner

So, today’s been an interesting day. I finally decided on Sunday against building my future projects with Amazon web services on the basis that I’m more likely to be able to build a faster responding and more tailored solution to each of my needs with other tools. I decided to investigate MogileFS as an alternative to Amazon S3 and at the same time I decided to start work on a new set of Python tools to be named "good ingredients" which would basically be very low level components using a more genericised version of the WSGI spec which also allows services (something I’ve been thinking about recently). I mentioned some of my ideas to some of the Rails community including one of the core comitters but didn’t get a lot of interest. I’m not too suprised becuase I haven’t had a lot of interest from the Pylons community either and the Rails community are far more in favour of abstractions than the Pylons community and abstractions are precicely what good ingredients were not going to be about.

Anyway, the plan was to put together existing tools that provide the sorts of very scalable infrastructure which would otherwise require something like AWS and expose useful a useful low-level web framework stack with the good ingredients.

Then after all this planning Google go an release their App Engine which offers both a scalable infrastructure and a new set of Python modules which seem farily low-level and support WSGI. Still the infrastructure isn’t customisable and the modules don’t use anything akin to the modular WSGI services API I am planning so I think it makes sense for me to carry on with my ideas anyway. Interesting times though.

I’ve got a 32bit intel Mac Mini server running Ubuntu Dapper Drake and I wanted to package it up as a Xen image to run on an AMD 64 server running Debian. Here’s what I did.

First create the Xen image on the 64bit server:

sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=60Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm2 --ip 78.47.146.253 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

Then mount it:

sudo mkdir /mount/vm2
sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2

Then sync the filesystems:

cd /mount/vm2/
sudo rsync -aHxvzgop  --progress --delete --numeric-ids root@pauli.3aims.com:/ .

Now shutdown the old server because any changes will be lost.

Create a new fstab for the copy on the 64bit server after backing up the old one:

sudo cp -p /mount/vm2/etc/fstab /mount/vm2/etc/fstab.bak

Make the new one like this:

/dev/sda1     /     ext3     errors=remount-ro   0     1
proc          /proc proc     rw,nodev,nosuid,noexec              0     0
/dev/sda2     none  swap     sw                    0     0

Unmount the disk and boot the image:

# You can't unmount a directory you are still in so we move out.
cd ..
sudo umount /mount/vm2
# You might want to skip this, it takes ages.
sudo cp /var/xen/domains/vm2/disk.img /var/xen/domains/vm2/disk.img.bak
sudo xm create -c /etc/xen/vm2.cfg

Check the server is up and can ping and be pinged then change the DNS settings to point all the services to the new server.

You might need the lib64 modules:

sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2
sudo cp -r /lib/modules/2.6.18-4-xen-amd64 /mount/vm2/lib/modules/
sudo ln -s /mount/vm2/lib /mount/vm2/lib64
sudo umount /mount/vm2
sudo xm create -c /etc/xen/vm2.cfg

Also need to mess with any files containing the wrong IP. Some of these commands might be handy:

sudo mount -t ext3 -o loop /var/xen/domains/vm2/disk.img /mount/vm2
sudo cp -pr /mount/vm2/etc /mount/vm2/etc.bak
cd /mount/vm2/etc
sudo grep -rl "212.69.38.27" . 2> /dev/null
sudo perl -pi -e 's/212.69.38.27/78.47.146.253/g' /apache2/vhosts/*
sudo grep -rl "212.69.38.27" . 2> /dev/null
cd /
sudo umount /mount/vm2
sudo xm create -c /etc/xen/vm2.cfg

Finally you’ll need to update your networking configuration, here’s how mine look, /etc/network/interfaces:

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.253
 network 78.47.146.248
 netmask 255.255.255.248
 up route add 78.46.35.5 dev eth0
 up route add default gw 78.46.35.5
 down route del default gw 78.46.35.5
 down route del 78.46.35.5 dev eth0

and /etc/resolve.conf:

### Hetzner Online AG installimage
# nameserver config
nameserver 213.133.99.99
nameserver 213.133.100.100
nameserver 213.133.98.98

I run ISPConfig too so I also need to update its database so changes it makes for bind are correct:

mysql> update dns_a set ip_adresse='78.47.146.253' where ip_adresse='212.69.38.27';
mysql> update dns_isp_dns set dns_soa_ip= '78.47.146.253' where dns_soa_ip='212.69.38.27';

The easy way to install Nginx is like this:

sudo aptitude install nginx

The problem is that this installs an old version (0.4) and doesn’t have SSL support built in. Nginx is designed to be super fast so and modules need to be compiled in manually. This means you need remove the Nginx package if you want SSL support.

If you try sudo apt-get remove –purge nginx you get this error:

Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED
  nginx*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0B of archives.
After unpacking 524kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 26362 files and directories currently installed.)
Removing nginx ...
Stopping nginx: invoke-rc.d: initscript nginx, action "stop" failed.
dpkg: error processing nginx (--purge):
 subprocess pre-removal script returned error exit status 1
Starting nginx: 2007/12/07 14:45:03 [emerg] 29441#0: unknown directive "ssl" in /etc/nginx/nginx.conf:30
invoke-rc.d: initscript nginx, action "start" failed.
dpkg: error while cleaning up:
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

The solution is to edit /var/lib/dpkg/info/nginx.prerm and comment out the line with invoke-rc.d which gives the stop statement for nginx. If you run the command again it will remove successfully, purging your nginx.conf config file too. If this isn’t what you want make a backup first.

You can then remove your logs with:

sudo rm -rf /var/log/nginx/

Now to install Nginx 0.6 from source. Fisrt you will need some build tools:

sudo aptitude install build-essential

Then we need some libraries:

sudo aptitude install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev

Check what the latest version is at http://nginx.net. At the time of writing the latest stable version is 0.5.33 so download that:

wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar zxfv nginx-0.5.33.tar.gz
cd nginx-0.5.33

Now you are ready to compile Nginx. As mentioned earlier many of the options are set at compile time so have a look at the Compile Time Options page and decide which you need. We are going to set two options. The first is:

--with-http_ssl_module

This enables the SSL module. The second option is to customise where Nginx is installed to. The default is /usr/local/nginx but this means the Nginx binary will be /usr/local/nginx/sbin/nginx which isn’t on the PATH. A better place is /usr/local/sbin where it is easily accessible by root or by users with sudo access:

--sbin-path=/usr/local/sbin

Now we are ready to configure:

./configure --sbin-path=/usr/local/sbin --with-http_ssl_module

There is a useful configuration summary:

nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/sbin"
nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
nginx pid file: "/usr/local/nginx/logs/nginx.pid"
nginx error log file: "/usr/local/nginx/logs/error.log"
nginx http access log file: "/usr/local/nginx/logs/access.log"
nginx http client request body temporary files: "/usr/local/nginx/client_body_temp"
nginx http proxy temporary files: "/usr/local/nginx/proxy_temp"
nginx http fastcgi temporary files: "/usr/local/nginx/fastcgi_temp"

Then compile and install:

make
sudo make install

The first thing I do next is create a backup of the original file:

sudo cp /usr/local/nginx/conf/nginx.conf  /usr/local/nginx/conf/nginx.conf.bak

Then start the server:

sudo /usr/local/sbin/nginx

If you visit http://yourdomain.com you should see the Nginx welcome message Welcome to nginx!. If you don’t it might be because you have another server such as Apache running.

To kill the server you can use the pid file:

sudo kill `cat /usr/local/nginx/logs/nginx.pid`

Notice this uses the ` character (normally underneath Esc) not the ‘ character. Obviously it is nicer to have an init.d setup so you can start and stop Nginx the same way as Apache and other servers. We’ll use the script from Slicehost which is itself based on the Debian package:

wget http://articles.slicehost.com/assets/2007/10/19/nginx
sudo chmod +x nginx
sudo mv nginx /etc/init.d

Now we can add this script to the runlevels:

sudo /usr/sbin/update-rc.d -f nginx defaults

Now you can start, stop and restart Nginx with these commands as normal:

sudo /etc/init.d/nginx start
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx restart

Now let’s setup Nginx to proxy to a Pylons application. Replace these lines in /usr/local/nginx/conf/nginx.conf:

location / {
    root   html;
    index  index.html index.htm;
}

with these:

location / {
    proxy_redirect          off;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Port $server_port;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffer_size       4k;
    proxy_buffers           4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass  http://127.0.0.1:5000;
    proxy_redirect  default;
}

Update: Added X-Forwarded-Port and X-Forwarded-Host to the above since I use them both so much.

This will proxy all requests to another server running on port 5000, for example a running Pylons application. To test this you should start your Pylons application (making sure debug is set to false if this is a production setup):

paster serve development.ini

Now restart Nginx:

sudo /etc/init.d/nginx restart

and f you visit the site you should see your Pylons application.

Now for the SSL:

sudo aptitude install openssh-server

First create a the key, pem and certificate files:

openssl genrsa 1024 > host.key
chmod 400 host.key
openssl req -new -x509 -nodes -sha1 -days 365 -key host.key > host.cert
cat host.cert host.key > host.pem
chmod 400 host.pem

Make sure the Common Name is the same as the domain name the certificate is for.

Now we need to setup SSL on Nginx. This is described in detail here but it really just requires making the server section look like this:

worker_processes 1;
http {
    ...
    server {
        listen               443;
        ssl                  on;
        ssl_certificate      /path/to/host.pem;
        ssl_certificate_key  /path/to/host.key;
        keepalive_timeout    70;
    }
}

Restart Nginx again:

sudo /etc/init.d/nginx restart

Now if you visit http://yourdomain.com you’ll get no response but if you visit https://yourdomain.com you’ll get a certificate warning because the SSL certificate is not signed by a certificate authority. Click OK and you will have access to your application.

The final part of the setup is to setup a static file which redirects from the http version of the site to the https version. The easy way to do this is by setting up another server listening on port 80 and creating an error 404 page which redirects to the https version. Add this to the nginx config just before the existing server:

server {
    listen 80;
    server_name  yourdomain.com;
    error_page  404              /404.html;
    location = /404.html {
        root   /path/to/directory/containing/404/doc/;
    }
}

Then create a 404.html file which looks something like this but adjusted for your URL instead of yourdomain.com:

<html>
<head>
<meta http-equiv="refresh" content="2;URL=https://yourdomain.com/">

</head>
<body>
<h2>Redirecting</h2>
<p>You are being redirected to the <a href="https://yourdomain.com/">secure version of this site.</a></p>
</body>
</html>

Then restart Nginx again.

You should now have a secure Pylons app on the https port and a redirect page on the http port.

Further Reading:

http://articles.slicehost.com/2007/10/19/debian-etch-installing-nginx http://ubuntuforums.org/showthread.php?t=453053 http://sudhanshuraheja.com/2007/09/remove-nginx-from-ubuntu-fiesty-fawn.html http://www.rkblog.rk.edu.pl/w/p/pylons-and-nginx/

The traditional way of setting up subversion is via Apache but what if you don’t want to install Apache? In that case you can use svnserve but what if your host won’t let you run a persistant server? In that case you can create a linux system account for every user and give them induvidual SSH logins. You then give your users a URL starting svn+ssh:// instead of the usual http://. The Subversion client recognises this form of URL and invokes a local ssh process, connecting to the host, authenticating as the user, then spawning a private svnserve process on the remote machine running as that user. The svnserve command is being invoked in tunnel mode (-t) and its network protocol is being tunneled over the encrypted connection by ssh, the tunnel-agent. svnserve is aware that it’s running as the user, and if the client performs a commit, the authenticated username will be attributed as the author of the new revision.

This is all well and good but you might not wa nt to have to create user accounts for each of the users you want to give access to subversion. In this case you can create one system user account and use a different SSH private key for each user you want to grant access to. You then setup the ~/.ssh/authorized_keys file so that each user only has access to svnserve and that each svnserve process is started with their particular username so that all the commits are handled correctly. This is only possible with subversion 1.1.0 and above.

This is what we are going to do for two users, james and mike. james is a linux user, mike is a Windows user.

First let’s setup the server:

sudo apt-get install subversion openssh-server

Then create a subversion user svn, a home directory, an .ssh directory and finally set the correct permissions:

sudo useradd svn
sudo mkdir /home/svn
sudo mkdir /home/svn/.ssh
sudo chown -R svn:svn /home/svn

If you intend to be able to login as svn for testing purposes later you should set a password:

sudo passwd svn

Create the directory for the repositories:

sudo mkdir /var/svn

Setup a repo repository and commit the first import:

sudo mkdir /var/svn/repo
sudo mkdir /tmp/repo
sudo mkdir /tmp/repo/branches
sudo mkdir /tmp/repo/tags
sudo mkdir /tmp/repo/trunk
sudo svnadmin create /var/svn/repo
sudo svn import /tmp/repo file:///var/svn/repo -m "initial import"
sudo rm -rf /tmp/repo

Change the permissions so that the svn user and svn group have access:

sudo chown -R svn:svn /var/svn/repo

Set the permissions to 770 and the umask to 2 so that any files are created have the correct permissions:

sudo chmod 2770 -R /var/svn/repo

At this point you can test the setup manually. As a user other than svn try this:

svn co svn+ssh://svn@localhost/var/svn/repo .

This should prompt you for the password a couple of times and then checkout the repository.

We are half way there, next we need to set up two SSH public-private key pairs, one for Mike, one for James. As the svn user run:

ssh-keygen -t rsa -b 1024 -f mike.key
ssh-keygen -t rsa -b 1024 -f james.key

With each command you can just press <Enter> twice unless you want to set a password. You will now have the files mike.key, mike.key.pub, james.key and james.key.pub in the current directory. The .key files are the private keys. They should be given securely to Mike and James repsectively. Anyone who gets hold of the files will have the same permissions to the repository as James and Mike so they should be treated with the same respect as a password and not posted online!

Since Mike is on Windows he will need WinSCP, PuTTY, PuTTYgen and TortoiseSVN (the latter requires adminsitrator access to install).

Here’s what he does:

• Use WinSCP to copy the mike.key file to his desktop if he didn’t recieve it any other way

• Open PuTTYgen and from the Conversions menu, select Import, then browse to and select mike.key. Leave the passphrase boxes blank and select Save private key. Click Yes on the confirmation box, and save the key with a .ppk extension, like mike.ppk. This is now a key that PuTTY can use.

• Start PuTTY again. Fill out the hostname box, then scroll down to SSH > Auth. Browse for the mike.ppk file. Then go back up to Sesson, give the session a name (for example mike) and save it. Test by logging in with the username ssh. You should see something like this:

  login as: svn
  Authenticating with public key "imported-openssh-key"
  ( success ( 1 2 ( ANONYMOUS EXTERNAL ) ( edit-pipeline svndiff1 absent-entries ) ) )
  

• With this in place you can set up TortoiseSVN with the key, [I haven’t tested this next bit yet] First, create a new folder, then right-click it and select SVN Checkout. In the dialog box, enter this in the URL field svn+ssh://svn@mike/var/svn/repo. svn+ssh is how TortoiseSVN will access your server, svn@mike is your credentials for logging in (mike is the session we set up using the keys in PuTTY), and /var/svn/repo is where the repository is located on the server. Later on in this tutorial we’ll configure the server so this can be accessed as svn+ssh://svn@mike/repo.

From James’s point of view the process is a bit simpler becasue on Linux the .key file doesn’t need to be converted to a .ppk file:

• Create a ~/.ssh directory:

  mkdir ~/.ssh
  

• Copy the private key to ~/.ssh/id_dsa you don’t already have it:

  scp svn@example.com:/home/svn/james.key ~/.ssh/id_dsa
  

Now that the private keys are set up it is time to setup the public keys on the subversion server. Copy the contents of mike.key.pub and james.key.pub to /home/svn/.ssh/authorized_keys so that the each publick key is on a separate line:

touch .ssh/authorized_keys
cat mike.key.pub >> .ssh/authorized_keys
cat james.key.pub >> .ssh/authorized_keys

It should look something like this but with lots of characters in place of …:

ssh-rsa AAA...acFHU= svn@example
ssh-rsa AAA...acBDH= svn@example

The three columns are TYPE KEY COMMENT. Since the last part is a comment it makes sense to change it to the names of the users so you remember which line referrs to who:

ssh-rsa AAA...acFHU= mike
ssh-rsa AAA...acBDH= james

This is all well and good but at the moment Mike and James can also SSH directly into the svn account so we want to restrict their access. Each line in the authorized_keys file can also have a command="COMMAND" section at the start. The subversion client executes svnserve -t when it SSHs in but you can also specify the executable to use explicitly:

command="/usr/bin/svnserve -t" ssh-rsa AAA...acFHU= mike
command="/usr/bin/svnserve -t" ssh-rsa AAA...acBDH= james

Now you might want to limit the users to a particular repository, you can do this with the -r option:

command="/usr/bin/svnserve -t -r /var/svn/" ssh-rsa AAA...acFHU= mike
command="/usr/bin/svnserve -t -r /var/svn/" ssh-rsa AAA...acBDH= james

James and Mike can access the repo repository with svn+ssh://svn@localhost/repo instead of svn+ssh://svn@localhost/var/svn/repo

You will probably want to limit the other things the users can do with the svn account so you’d probably add no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding to the command like this:

command="/usr/bin/svnserve -t -r /var/svn/",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acFHU= mike
command="/usr/bin/svnserve -t -r /var/svn/",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acBDH= james

So far, the svnserve command still doesn’t know which username it is supposed to be using so we add the –tunnel-user option to tell it explicitly:

command="/usr/bin/svnserve -t -r /var/svn/ --tunnel-user=mike",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acFHU= mike
command="/usr/bin/svnserve -t -r /var/svn/ --tunnel-user=james",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding ssh-rsa AAA...acBDH= james

At this point everything should now work as expected. Both James and Mike should be able to access and commit to the repository without needing a password and their commits should be logged as james and mike respectively because they are using different private keys and we are using the –tunnel-user approach. No other users should be able to access the repository except the svn user if you know the password.

Further reading:

http://svnbook.red-bean.com/en/1.1/ch06s03.html http://allyourtech.com/content/articles/23_12_2005_setting_up_subversion_and_tortoisesvn.php http://allyourtech.com/content/articles/24_12_2005_ditching_the_password_prompts_in_tortoisesvn.php http://svn.collab.net/repos/svn/trunk/notes/ssh-tricks http://svn.haxx.se/dev/archive-2004-03/0253.shtml http://bitworking.org/news/Getting_subversion_svn_ssh____to_work_with_PuTTY http://www.linuxfromscratch.org/blfs/view/svn/server/svnserver.html

For testing it is also possible to set the $SVN_SSH variable specifying the private key to use with -i so that you can try connecting as different users. For example:

export SVN_SSH="ssh -i /home/james/.ssh/mike.key

OK, so in my first article I showed how to install Xen on Hetzner, in this one I’ll show how to configure it.

I want a setup where I can have 6 virtual machines, each accessible on the internet and each with their own IP address. At the same time I want to be able to have any number of virtual machines on a private subnet and use NAT to forward specific ports from the physical server (Dom0) to the individual guests. At the same time all the virtual machines have to be able to communicate with themselves and each other.

First of all you need to specify the memory that Dom0 should take up, otherwise it quickly uses all your free memory and you don’t have any for your virtual machines. I chose 256Mb. You can set it with this command:

sudo xm mem-set 0 256M

You’d have to run this every time the machine boots so it is much easier to just edit your /boot/grub/menu.lst file and add the dom0_mem=256M to the kernel option so that it gets set when the kernel loads.

I also found I got this error when I loaded lots of virtual machines:

Error: Device 0 (vif) could not be connected. Backend device not found.

This might have been because I’d run out of loopback devices so we need to increase the number allowed by adding max_loop=32 to the module options when loading.

You can correct both these problems at once by editing the grub menu. Here’s how the relevant section from my menu.lst looks after the changes:

title           Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-4-xen-amd64
root            (hd0,1)
kernel          /boot/xen-3.0.3-1-amd64.gz dom0_mem=256M
module          /boot/vmlinuz-2.6.18-4-xen-amd64 root=/dev/sda2 ro console=tty0 max_loop=32
module          /boot/initrd.img-2.6.18-4-xen-amd64
savedefault

You should reboot at this stage.

Secondly I want to remove the firewall rules I created earlier. The last thing you want when you are struggling with a complex set up is a load of extra rules to confuse things. I removed them like this:

sudo -i
iptables -Z
iptables -X
iptables -F
exit

I don’t want them coming back when I reboot so I run this so that an empty rules file is run temporarily:

sudo mv /etc/iptables.up.rules /etc/iptables.default.rules
sudo touch /etc/iptables.empty.rules
sudo ln -s /etc/iptables.empty.rules /etc/iptables.up.rules

Right, now we can configure Xen. Your /etc/xen/xend-config.sxp looks like this:

# -*- sh -*-

(network-script network-route)
(vif-script vif-route)

So, now you create some new domains. The first one is going to be for a private network:

sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm1 --ip 10.0.0.1 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

This whirrs away for a while and eventually you have a disk.img and swap.img in /var/xen/domains/vm1 and a config file in /etc/xen/vm1.cfg. You can start it up like this:

sudo xm create -c /etc/xen/vm1.cfg

Login as root, set a new password and then:

ping google.com

You should get lots of replies and no lost packets. Try pinging the IP address of Dom0 too, it should work fine. So, that’s one virtual machine set up on a private IP address not accessible to the public.

You can now follow this tutorial to setup a Nginx and a Pylons application on the server and have port 80 forwarded from Dom0 to the virtual machine. You’ll need to put any iptables rules back into the iptables.up.rules if you want them to work when the server restarts.

You can create as many virtual machines as you like in this way. Here’s another called vm3 on 10.0.0.3. Again, vm3 should be able to ping google.com, Dom0 and vm1:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm3 --ip 10.0.0.3 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

General Infomation
--------------------
Hostname       :  vm3
Distribution   :  etch
Fileystem Type :  ext3

Size Information
----------------
Image size     :  5Gb
Swap size      :  1024Mb
Image type     :  sparse
Memory size    :  512Mb
Kernel path    :  /boot/vmlinuz-2.6.18-4-xen-amd64
Initrd path    :  /boot/initrd.img-2.6.18-4-xen-amd64

Networking Information
----------------------
IP Address 1   : 10.0.0.3
Netmask        : 255.255.255.0
Gateway        : 10.0.0.254


Creating swap image: /var/xen/domains/vm3/swap.img
Done

Creating disk image: /var/xen/domains/vm3/disk.img
Done

Creating ext3 filesystem on /var/xen/domains/vm3/disk.img
Done

Installing your system with debootstrap mirror http://ftp.freenet.de/debian/
Done

Running hooks
Done

No role script specified.  Skipping

Creating Xen configuration file
Done
All done


Logfile produced at:
         /var/log/xen-tools/vm3.log

Again, you can setup any forwarding rules so that ports on the virtual machine can be accessed from Dom0. You should be able to ping 10.0.0.1 (if it is running) from this virtual machine and it should be able ping you.

Next, lets setup the public virtual machines. The hosting company have provided a range of IPs from 78.47.146.249 to 78.47.146.254. These are on a different subnet from my server so I might have had to follow Steve’s Xen setup here. Luckily though these IPs are already routed straight to my server so I don’t need to worry. Also, as someone pointed out in comment #16 on that page, you don’t need to waste an IP on a "bridge" because you can add the Dom0 IP as a route on the virtual machine.

So without further ado, here’s what you need to do. First create another vitual machine (or you could edit the settings on the old one). You might expect to be able to use a command like this to simply generate the new virtual machine:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm4 --ip 78.47.146.251 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

By the way, if you try to boot it and it fails with:

Error: Device 0 (vif) could not be connected. Backend device not found.

it might mean you already have a virtual machine running with the same address.

Once you’ve booted the virtual machine, change the networking settings to use these details by editing /etc/network/interfaces. Replace this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 gateway
 netmask 255.255.255.248

With this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 network 78.47.146.248
 netmask 255.255.255.248
 up route add 78.46.35.5 dev eth0
 up route add default gw 78.46.35.5
 down route del default gw 78.46.35.5
 down route del 78.46.35.5 dev eth0

Notice that you don’t need to specify a gateway, but you do need a network. The gateway routes are added manually in the up commands and removed in the down commands:

/etc/init.d/networking restart

You should now be able to ping google and all the other servers and what is more, if you ping 78.47.146.251 from anywhere else on the internet, the server will respond because vm4 is now publicly accessible on the internet under that IP address.

Following on from my last article I’ve now received the email with the free IP addresses:

Dear Mr. James Gardner,

Below you will find your additional IP addresses added to
the server 78.46.35.5.

Please note that you can use the subnet only for this server.

IP: 78.47.146.248
Mask: 255.255.255.248
Broadcast: 78.47.146.255

Useable IP addresses:
78.47.146.249 to 78.47.146.254

Instructions for installing additional IP addresses
you will find at the following link:
http://wiki.hetzner.de/index.php/Zusätzliche_IP-Adressen (in German)

Our competent support team is ready to answer your questions.

Please visit our website at http://www.hetzner.de/support.html.

This entry will show how to set up virtual ethernet interfaces for each of the IPs so they all resolve to the server. Note this isn’t what you want for Xen, see my next entry for that.

Following the link results in this page (translated). Which explains that you might have 6 IP addresses rather than 5 depending on the network setup. Looks like I have 6 so that seems good.

The Debian instructions for setting up these IP addresses are translated here but you’ll want to compare them to the original because Google reformats the config files in an attempt to translate them.

The basic technique is to create a virtual ethernet interface for each IP address so that the server can connect on each IP address. You do this by adding them to the /etc/network/interfaces config file. Here’s what mine looks like before:

### Hetzner Online AG - installimage
# Loopback device:
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules

# device: eth0
auto eth0
iface eth0 inet static
  address 78.46.35.5
  broadcast 78.46.35.31
  netmask 255.255.255.224
  gateway 78.46.35.1

# default route to access subnet
up route add -net 78.46.35.0 netmask 255.255.255.224 gw 78.46.35.1 eth0

and after:

### Hetzner Online AG - installimage
# Loopback device:
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules

# device: eth0
auto eth0
iface eth0 inet static
  address 78.46.35.5
  broadcast 78.46.35.31
  netmask 255.255.255.224
  gateway 78.46.35.1

# Now set up the virtual interfaces for the other IP addresses:

auto eth0:0
iface eth0:0 inet static
address 78.47.146.249
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:1
iface eth0:1 inet static
address 78.47.146.250
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:2
iface eth0:2 inet static
address 78.47.146.251
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:3
iface eth0:3 inet static
address 78.47.146.252
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:4
iface eth0:4 inet static
address 78.47.146.253
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:5
iface eth0:5 inet static
address 78.47.146.254
broadcast 78.47.146.255
netmask 255.255.255.248

# End of the virtual interfaces setup

# default route to access subnet
up route add -net 78.46.35.0 netmask 255.255.255.224 gw 78.46.35.1 eth0

Do a quick ping test from another machine - the IP addresses won’t work yet:

james@dirac:~$ ping 78.47.146.254
PING 78.47.146.254 (78.47.146.254) 56(84) bytes of data.

--- 78.47.146.254 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2008ms

Now you need to restart networking and cross your fingers. If your networking config is broken you’ll have to use the same rescue boot technique I described earlier for SSH:

sudo /etc/init.d/networking restart

Now if you ping the same IP your server should respond:

james@dirac:~$ ping 78.47.146.254
PING 78.47.146.254 (78.47.146.254) 56(84) bytes of data.
64 bytes from 78.47.146.254: icmp_seq=1 ttl=56 time=38.6 ms
64 bytes from 78.47.146.254: icmp_seq=2 ttl=56 time=57.3 ms

--- 78.47.146.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 38.687/48.003/57.319/9.316 ms

Your IP adresses are now setup correctly to point to the server.

I’m not sure how useful this setup actually is as I’m using Xen now, but it is here in case it helps someone.

I bought a new server from Hetzner (German) on Sunday 4th November and on the evening of the 5th I received my sign in details by email in both German and English for the server, already installed with the 64bit version of Debian Etch. This post documents my progress setting it up.

Verwaltung                           Administration
        Sicherheit                           Security
        E-Mail-Adressen                      Emails
        Support-Anfragen                     Support requests
        Status-Benachrichtigung              Status Notification
        Newsletter                           Newsletter
        RIPE-Registration                    RIPE-Registration
Rechnungen                           Bills
        Offene Posten                        Open Item
Traffic-Statistik                    Traffic Statistics
        Traffic-Limit-Reporting              Traffic limit Reporting
        Tagesbericht                         Daily Report
        Monatsbericht                        Monthly Report
        Jahresbericht                        Annual Report
Leistungsübersicht                   Performance Overview
        Resetaufträge                        Reset orders
        Rescuesystem                         Rescue System
        VNC-Installation                     VNC-Installation
        neuen Server bestellen               New server order
        Separater Admin-Zugang               Separate Admin Access
Reverse-DNS-Einträge                 Reverse-DNS
        Eintrag anlegen                      Entry
        Eintrag löschen                      Delete Entry
Dokumentation                        Documentation
        Daten-Export-Schnittstelle           Data Export interface

``Automatischen Hardware-Reset auslösen`` - Perform an automatic hardware reset.

Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  520M  346G   1% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
udev                   10M   36K   10M   1% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
Debian-40-etch-64-minimal:~# cat /etc/issue
Debian GNU/Linux 4.0 \n \l

Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
             total       used       free     shared    buffers     cached
Mem:          3926         73       3853          0         12         19
-/+ buffers/cache:         41       3885
Swap:         2055          0       2055

james@dirac:~$ ssh root@doppler.3aims.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
50:10:6b:5f:dc:e7:45:17:73:00:d0:50:f0:0e:48:be.
Please contact your system administrator.
Add correct host key in /home/james/.ssh/known_hosts to get rid of this message.
Offending key in /home/james/.ssh/known_hosts:1
RSA host key for doppler.3aims.com has changed and you have requested strict checking.
Host key verification failed.

Linux rescue 2.6.22.2 #2 SMP Tue Aug 28 09:28:15 CEST 2007 x86_64

------------------------------------------------------------------

  Welcome to the Hetzner Rescue System.

  This Rescue System is based on Debian 4.0 (etch) with a newer
  kernel. You can install software like in a normal system.

  To install a new operating system from one of our prebuilt
  images, run 'installimage' and follow the instructions.

  For more information take a look at http://wiki.hetzner.de

------------------------------------------------------------------

root@rescue ~ #

        Hetzner Online AG - installimage


#~ server will be installed now. this will take a few minutes.
#~ you can abort at any time with CTRL+C ..
#~ ( init) ~ reading vars...                            [ OK ]
#~ ( 1/11) ~ deleting partitions...                     [ OK ]
#~ ( 2/11) ~ creating partitions and fstab...           [ OK ]
#~ ( 3/11) ~ formatting partitions...                   [ OK ]
#~ ( 4/11) ~ mounting partitions...                     [ OK ]
#~ ( 5/11) ~ extracting imagefile from local...         [ OK ]
#~ ( 6/11) ~ setting up network config for eth0...      [ OK ]
#~ ( 7/11) ~ chrooting some commands...                 [ OK ]
#~ ( 8/11) ~ clearing logfiles...                       [ OK ]
#~ ( 9/11) ~ setting up some files...                   [ OK ]
#~ (10/11) ~ setting up rootpassword...                 [ OK ]
#~ (11/11) ~ setting up bootloader grub...              [ OK ]


#~~~ INSTALLATION COMPLETE ~~~#
you may now reboot into your new system
you can login to your new system with the same
password as you logged in into the rescue system


root@rescue ~ #

apt-get install xen-linux-system-2.6.18-4-xen-amd64
reboot

# -*- sh -*-
(network-script network-route)
(vif-script vif-route)

/etc/init.d/xend restart

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  612M  346G   1% /
tmpfs                 1.7G     0  1.7G   0% /lib/init/rw
udev                   10M   36K   10M   1% /dev
tmpfs                 1.7G     0  1.7G   0% /dev/shm
Debian-40-etch-64-minimal:~#  cat /etc/issue
Debian GNU/Linux 4.0 \n \l

Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-4-xen-amd64 #1 SMP Fri May 4 02:40:51 UTC  2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
             total       used       free     shared    buffers     cached
Mem:          3366        200       3165          0          1         15
-/+ buffers/cache:        182       3183
Swap:         2055          0       2055
Debian-40-etch-64-minimal:~# fdisk -l

Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         262     2104514+  82  Linux swap / Solaris
/dev/sda2             263       48641   388604317+  83  Linux

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Debian-40-etch-64-minimal:~# fdisk -u /dev/sdb

The number of cylinders for this disk is set to 48641.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-781422767, default 63):
Using default value 63
Last sector or +size or +sizeM or +sizeK (63-781422767, default 781422767):
Using default value 781422767

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Debian-40-etch-64-minimal:~# /sbin/mkfs -t ext3 /dev/sdb1
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
48840704 inodes, 97677838 blocks
4883891 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2981 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

/dev/sdb1 /mount/sdb1 ext3 defaults 0 0

mkdir /mount
mkdir /mount/sdb1
mount /dev/sdb1

Debian-40-etch-64-minimal:~# fdisk -l

Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         262     2104514+  82  Linux swap / Solaris
/dev/sda2             263       48641   388604317+  83  Linux

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       48642   390711352+  83  Linux
Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  612M  346G   1% /
tmpfs                 1.7G     0  1.7G   0% /lib/init/rw
udev                   10M   40K   10M   1% /dev
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/sdb1             367G  195M  348G   1% /mount/sdb1

Much better, 674Gb free space.

doppler.3aims.com.  1440  A   78.46.35.5

78.46.35.5 doppler doppler.3aims.com

Debian-40-etch-64-minimal:~# vim /etc/hostname
Debian-40-etch-64-minimal:~# vim /etc/hosts
Debian-40-etch-64-minimal:~# hostname doppler

Debian-40-etch-64-minimal:~# apt-get install sudo
Debian-40-etch-64-minimal:~# adduser james
Adding user `james' ...
Adding new group `james' (1000) ...
Adding new user `james' (1000) with group `james' ...
Creating home directory `/home/james' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for james
Enter the new value, or press ENTER for the default
        Full Name []: James Gardner
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [y/N] y

Debian-40-etch-64-minimal:~# su james
james@doppler:/root$

$ sudo dpkg-reconfigure locales

james@doppler:/root$ sudo tzconfig
Your current time zone is set to Europe/Berlin
Do you want to change that? [n]: y

Please enter the number of the geographic area in which you live:


        1) Africa                       7) Australia

        2) America                      <img src='http://jimmyg.org/wp-includes/images/smilies/icon_cool.gif' alt='8)' class='wp-smiley' /> Europe

        3) US time zones                9) Indian Ocean

        4) Canada time zones            10) Pacific Ocean

        5) Asia                         11) Use System V style time zones

        6) Atlantic Ocean               12) None of the above


Then you will be shown a list of cities which represent the time zone
in which they are located. You should choose a city in your time zone.

Number: 8

Amsterdam Andorra Athens Belfast Belgrade Berlin Bratislava Brussels
Bucharest Budapest Chisinau Copenhagen Dublin Gibraltar Guernsey Helsinki
Isle_of_Man Istanbul Jersey Kaliningrad Kiev Lisbon Ljubljana London
Luxembourg Madrid Malta Mariehamn Minsk Monaco Moscow Nicosia Oslo Paris
Podgorica Prague Riga Rome Samara San_Marino Sarajevo Simferopol Skopje
Sofia Stockholm Tallinn Tirane Tiraspol Uzhgorod Vaduz Vatican Vienna
Vilnius Volgograd Warsaw Zagreb Zaporozhye Zurich

Please enter the name of one of these cities or zones
You just need to type enough letters to resolve ambiguities
Press Enter to view all of them again
Name: [] London
Your default time zone is set to 'Europe/London'.
Local time is now:      Fri Nov  9 16:53:52 GMT 2007.
Universal Time is now:  Fri Nov  9 16:53:52 UTC 2007.

Port 30000         <-- change to a port other than 22
PermitRootLogin no
AllowUsers james

PasswordAuthentication no

doppler:~# /etc/init.d/ssh restart
Restarting OpenBSD Secure Shell server: sshd.

ssh james@doppler.3aims.com -p 30000

mount /dev/sda2 /mnt -t ext3

james@doppler:~$ sudo -i
Password:
doppler:~# whoami
root

doppler:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables-save > /etc/iptables.current.rules

iptables-restore < /etc/iptables.test.rules

doppler:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     0    --  anywhere             anywhere
REJECT     0    --  anywhere             loopback/8          reject-with icmp-port-unreachable
ACCEPT     0    --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:30000
ACCEPT     icmp --  anywhere             anywhere            icmp echo-request
LOG        0    --  anywhere             anywhere            limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: '
REJECT     0    --  anywhere             anywhere            reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     0    --  anywhere             anywhere            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     0    --  anywhere             anywhere

iptables-save > /etc/iptables.up.rules

...
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules
...

sudo apt-get update
sudo apt-get upgrade