Xen Routing with Public Static IPs and a Private virtual network

Posted:	2007-11-12 16:53
Tags:	Hosting

OK, so in my first article I showed how to install Xen on Hetzner, in this one I'll show how to configure it.

I want a setup where I can have 6 virtual machines, each accessible on the internet and each with their own IP address. At the same time I want to be able to have any number of virtual machines on a private subnet and use NAT to forward specific ports from the physical server (Dom0) to the individual guests. At the same time all the virtual machines have to be able to communicate with themselves and each other.

First of all you need to specify the memory that Dom0 should take up, otherwise it quickly uses all your free memory and you don't have any for your virtual machines. I chose 256Mb. You can set it with this command:

sudo xm mem-set 0 256M

You'd have to run this every time the machine boots so it is much easier to just edit your /boot/grub/menu.lst file and add the dom0_mem=256M to the kernel option so that it gets set when the kernel loads.

I also found I got this error when I loaded lots of virtual machines:

Error: Device 0 (vif) could not be connected. Backend device not found.

This might have been because I'd run out of loopback devices so we need to increase the number allowed by adding max_loop=32 to the module options when loading.

You can correct both these problems at once by editing the grub menu. Here's how the relevant section from my menu.lst looks after the changes:

title           Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-4-xen-amd64
root            (hd0,1)
kernel          /boot/xen-3.0.3-1-amd64.gz dom0_mem=256M
module          /boot/vmlinuz-2.6.18-4-xen-amd64 root=/dev/sda2 ro console=tty0 max_loop=32
module          /boot/initrd.img-2.6.18-4-xen-amd64
savedefault

You should reboot at this stage.

Secondly I want to remove the firewall rules I created earlier. The last thing you want when you are struggling with a complex set up is a load of extra rules to confuse things. I removed them like this:

sudo -i
iptables -Z
iptables -X
iptables -F
exit

I don't want them coming back when I reboot so I run this so that an empty rules file is run temporarily:

sudo mv /etc/iptables.up.rules /etc/iptables.default.rules
sudo touch /etc/iptables.empty.rules
sudo ln -s /etc/iptables.empty.rules /etc/iptables.up.rules

Right, now we can configure Xen. Your /etc/xen/xend-config.sxp looks like this:

# -*- sh -*-

(network-script network-route)
(vif-script vif-route)

So, now you create some new domains. The first one is going to be for a private network:

sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm1 --ip 10.0.0.1 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

This whirrs away for a while and eventually you have a disk.img and swap.img in /var/xen/domains/vm1 and a config file in /etc/xen/vm1.cfg. You can start it up like this:

sudo xm create -c /etc/xen/vm1.cfg

Login as root, set a new password and then:

ping google.com

You should get lots of replies and no lost packets. Try pinging the IP address of Dom0 too, it should work fine. So, that's one virtual machine set up on a private IP address not accessible to the public.

You can now follow this tutorial to setup a Nginx and a Pylons application on the server and have port 80 forwarded from Dom0 to the virtual machine. You'll need to put any iptables rules back into the iptables.up.rules if you want them to work when the server restarts.

You can create as many virtual machines as you like in this way. Here's another called vm3 on 10.0.0.3. Again, vm3 should be able to ping google.com, Dom0 and vm1:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm3 --ip 10.0.0.3 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

General Infomation
--------------------
Hostname       :  vm3
Distribution   :  etch
Fileystem Type :  ext3

Size Information
----------------
Image size     :  5Gb
Swap size      :  1024Mb
Image type     :  sparse
Memory size    :  512Mb
Kernel path    :  /boot/vmlinuz-2.6.18-4-xen-amd64
Initrd path    :  /boot/initrd.img-2.6.18-4-xen-amd64

Networking Information
----------------------
IP Address 1   : 10.0.0.3
Netmask        : 255.255.255.0
Gateway        : 10.0.0.254


Creating swap image: /var/xen/domains/vm3/swap.img
Done

Creating disk image: /var/xen/domains/vm3/disk.img
Done

Creating ext3 filesystem on /var/xen/domains/vm3/disk.img
Done

Installing your system with debootstrap mirror http://ftp.freenet.de/debian/
Done

Running hooks
Done

No role script specified.  Skipping

Creating Xen configuration file
Done
All done


Logfile produced at:
         /var/log/xen-tools/vm3.log

Again, you can setup any forwarding rules so that ports on the virtual machine can be accessed from Dom0. You should be able to ping 10.0.0.1 (if it is running) from this virtual machine and it should be able ping you.

Next, lets setup the public virtual machines. The hosting company have provided a range of IPs from 78.47.146.249 to 78.47.146.254. These are on a different subnet from my server so I might have had to follow Steve's Xen setup here. Luckily though these IPs are already routed straight to my server so I don't need to worry. Also, as someone pointed out in comment #16 on that page, you don't need to waste an IP on a "bridge" because you can add the Dom0 IP as a route on the virtual machine.

So without further ado, here's what you need to do. First create another vitual machine (or you could edit the settings on the old one). You might expect to be able to use a command like this to simply generate the new virtual machine:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm4 --ip 78.47.146.251 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

By the way, if you try to boot it and it fails with:

Error: Device 0 (vif) could not be connected. Backend device not found.

it might mean you already have a virtual machine running with the same address.

Once you've booted the virtual machine, change the networking settings to use these details by editing /etc/network/interfaces. Replace this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 gateway
 netmask 255.255.255.248

With this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 network 78.47.146.248
 netmask 255.255.255.248
 up route add 78.46.35.5 dev eth0
 up route add default gw 78.46.35.5
 down route del default gw 78.46.35.5
 down route del 78.46.35.5 dev eth0

Notice that you don't need to specify a gateway, but you do need a network. The gateway routes are added manually in the up commands and removed in the down commands:

/etc/init.d/networking restart

You should now be able to ping google and all the other servers and what is more, if you ping 78.47.146.251 from anywhere else on the internet, the server will respond because vm4 is now publicly accessible on the internet under that IP address.

Update: After starting and stopping lots of virtual machines I started getting errors saying Error: Device 2049 (vbd) could not be connected. Backend device not found. I ran echo loop max_loop=64 >> /etc/modules then /dev/MAKEDEV and rebooted. Then it worked but I'm not sure if this was due to rebooting or the loop changes.

(view source)