Xen Routing with Public Static IPs *and* a Private virtual network
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
:Posted: 2007-11-12 16:53
:Tags: Hosting
OK, so in my first article I showed how to install `Xen on Hetzner
`_,
in this one I'll show how to configure it.
I want a setup where I can have 6 virtual machines, each accessible on the
internet and each with their own IP address. At the same time I want to be able
to have any number of virtual machines on a private subnet and use NAT to
forward specific ports from the physical server (Dom0) to the individual
guests. At the same time all the virtual machines have to be able to
communicate with themselves and each other.
First of all you need to specify the memory that Dom0 should take up, otherwise
it quickly uses all your free memory and you don't have any for your virtual
machines. I chose 256Mb. You can set it with this command::
sudo xm mem-set 0 256M
You'd have to run this every time the machine boots so it is much easier to
just edit your ``/boot/grub/menu.lst`` file and add the ``dom0_mem=256M`` to
the ``kernel`` option so that it gets set when the kernel loads.
I also found I got this error when I loaded lots of virtual machines::
Error: Device 0 (vif) could not be connected. Backend device not found.
This might have been because I'd run out of loopback devices so we need to
increase the number allowed by adding ``max_loop=32`` to the module options
when loading.
You can correct both these problems at once by editing the grub menu. Here's
how the relevant section from my ``menu.lst`` looks after the changes::
title Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-4-xen-amd64
root (hd0,1)
kernel /boot/xen-3.0.3-1-amd64.gz dom0_mem=256M
module /boot/vmlinuz-2.6.18-4-xen-amd64 root=/dev/sda2 ro console=tty0 max_loop=32
module /boot/initrd.img-2.6.18-4-xen-amd64
savedefault
You should reboot at this stage.
Secondly I want to remove the firewall rules I created earlier. The last thing
you want when you are struggling with a complex set up is a load of extra rules
to confuse things. I removed them like this::
sudo -i
iptables -Z
iptables -X
iptables -F
exit
I don't want them coming back when I reboot so I run this so that an empty
rules file is run temporarily::
sudo mv /etc/iptables.up.rules /etc/iptables.default.rules
sudo touch /etc/iptables.empty.rules
sudo ln -s /etc/iptables.empty.rules /etc/iptables.up.rules
Right, now we can configure Xen. Your ``/etc/xen/xend-config.sxp`` looks like
this::
# -*- sh -*-
(network-script network-route)
(vif-script vif-route)
So, now you create some new domains. The first one is going to be for a private network::
sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm1 --ip 10.0.0.1 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb
This whirrs away for a while and eventually you have a ``disk.img`` and
``swap.img`` in ``/var/xen/domains/vm1`` and a config file in
``/etc/xen/vm1.cfg``. You can start it up like this::
sudo xm create -c /etc/xen/vm1.cfg
Login as ``root``, set a new password and then::
ping google.com
You should get lots of replies and no lost packets. Try pinging the IP address
of Dom0 too, it should work fine. So, that's one virtual machine set up on a
private IP address not accessible to the public.
You can now follow this tutorial to setup a `Nginx and a Pylons application
`_
on the server and have port 80 forwarded from Dom0 to the virtual machine.
You'll need to put any iptables rules back into the iptables.up.rules if you
want them to work when the server restarts.
You can create as many virtual machines as you like in this way. Here's another
called ``vm3`` on ``10.0.0.3``. Again, ``vm3`` should be able to ping
google.com, Dom0 and ``vm1``::
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm3 --ip 10.0.0.3 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb
General Infomation
--------------------
Hostname : vm3
Distribution : etch
Fileystem Type : ext3
Size Information
----------------
Image size : 5Gb
Swap size : 1024Mb
Image type : sparse
Memory size : 512Mb
Kernel path : /boot/vmlinuz-2.6.18-4-xen-amd64
Initrd path : /boot/initrd.img-2.6.18-4-xen-amd64
Networking Information
----------------------
IP Address 1 : 10.0.0.3
Netmask : 255.255.255.0
Gateway : 10.0.0.254
Creating swap image: /var/xen/domains/vm3/swap.img
Done
Creating disk image: /var/xen/domains/vm3/disk.img
Done
Creating ext3 filesystem on /var/xen/domains/vm3/disk.img
Done
Installing your system with debootstrap mirror http://ftp.freenet.de/debian/
Done
Running hooks
Done
No role script specified. Skipping
Creating Xen configuration file
Done
All done
Logfile produced at:
/var/log/xen-tools/vm3.log
Again, you can setup any forwarding rules so that ports on the virtual machine
can be accessed from Dom0. You should be able to ping 10.0.0.1 (if it is
running) from this virtual machine and it should be able ping you.
Next, lets setup the public virtual machines. The hosting company have provided
a range of IPs from 78.47.146.249 to 78.47.146.254. These are on a different
subnet from my server so I might have had to follow `Steve's Xen setup here
`_. Luckily though these IPs
are already routed straight to my server so I don't need to worry. Also, as
someone pointed out in `comment #16
`_ on that page,
you don't need to waste an IP on a "bridge" because you can add the Dom0 IP as
a route on the virtual machine.
So without further ado, here's what you need to do. First create another vitual
machine (or you could edit the settings on the old one). You might expect to be
able to use a command like this to simply generate the new virtual machine::
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm4 --ip 78.47.146.251 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb
By the way, if you try to boot it and it fails with::
Error: Device 0 (vif) could not be connected. Backend device not found.
it might mean you already have a virtual machine running with the same address.
Once you've booted the virtual machine, change the networking settings to use these
details by editing ``/etc/network/interfaces``. Replace this::
# The primary network interface
auto eth0
iface eth0 inet static
address 78.47.146.251
gateway
netmask 255.255.255.248
With this::
# The primary network interface
auto eth0
iface eth0 inet static
address 78.47.146.251
network 78.47.146.248
netmask 255.255.255.248
up route add 78.46.35.5 dev eth0
up route add default gw 78.46.35.5
down route del default gw 78.46.35.5
down route del 78.46.35.5 dev eth0
Notice that you don't need to specify a gateway, but you do need a network. The
gateway routes are added manually in the up commands and removed in the down
commands::
/etc/init.d/networking restart
You should now be able to ping google and all the other servers and what is
more, if you ping 78.47.146.251 from anywhere else on the internet, the server
will respond because vm4 is now publicly accessible on the internet under
that IP address.
Update: After starting and stopping lots of virtual machines I started getting errors saying ``Error: Device 2049 (vbd) could not be connected. Backend device not found``. I ran ``echo loop max_loop=64 >> /etc/modules`` then ``/dev/MAKEDEV`` and rebooted. Then it worked but I'm not sure if this was due to rebooting or the loop changes.
Comments
========
links for 2007-11-14 « Bloggitation
--------------------------------------------
:Posted: 2007-11-14 01:24
[...] Xen Routing with Public Static IPs *and* a Private virtual network (tags: linux xen sysadmin) [...]
:URL: http://zhesto.wordpress.com/2007/11/14/links-for-2007-11-14/
Erich
--------
:Posted: 2007-12-22 05:08
Hi there,
Thanks for this great article.
My requirement is to run multiple DomU's on public IP Addresses for things like mail servers and stuff, but I also have a further requirement to run multiple DomU's on private IP addresses for other services.
With your help, I've been able to get a couple of DomU's working on public IP's and there was no problem in that regard, however, the DomU's on the private IP addresses simply won't connect beyond the Dom0. I can ping the Dom0 and ssh to it from the DomU, but can't ping google.com (or anything else) or ssh to an external server. I can do these things from the Dom0 and from the DomU's with public IP addresses.
I've googled, but have not found and answer and was wondering if there was something you could suggest.
Cheers
Erich
thejimmyg
------------
:Posted: 2008-01-11 12:19
I found I had to manually add a route to the Dom0 (which I did in the /etc/network/interfaces file). Make sure the IP address of the Dom0 in that file is correct (78.46.35.5 in my case) and that you've added the route commands correctly then run:
ifdown eth0
ifup eth0
and see if that helps perhaps?
:URL: http://jimmyg.org