James Gardner

OK, so in my first article I showed how to install Xen on Hetzner, in this one I’ll show how to configure it.

I want a setup where I can have 6 virtual machines, each accessible on the internet and each with their own IP address. At the same time I want to be able to have any number of virtual machines on a private subnet and use NAT to forward specific ports from the physical server (Dom0) to the individual guests. At the same time all the virtual machines have to be able to communicate with themselves and each other.

First of all you need to specify the memory that Dom0 should take up, otherwise it quickly uses all your free memory and you don’t have any for your virtual machines. I chose 256Mb. You can set it with this command:

sudo xm mem-set 0 256M

You’d have to run this every time the machine boots so it is much easier to just edit your /boot/grub/menu.lst file and add the dom0_mem=256M to the kernel option so that it gets set when the kernel loads.

I also found I got this error when I loaded lots of virtual machines:

Error: Device 0 (vif) could not be connected. Backend device not found.

This might have been because I’d run out of loopback devices so we need to increase the number allowed by adding max_loop=32 to the module options when loading.

You can correct both these problems at once by editing the grub menu. Here’s how the relevant section from my menu.lst looks after the changes:

title           Xen 3.0.3-1-amd64 / Debian GNU/Linux, kernel 2.6.18-4-xen-amd64
root            (hd0,1)
kernel          /boot/xen-3.0.3-1-amd64.gz dom0_mem=256M
module          /boot/vmlinuz-2.6.18-4-xen-amd64 root=/dev/sda2 ro console=tty0 max_loop=32
module          /boot/initrd.img-2.6.18-4-xen-amd64
savedefault

You should reboot at this stage.

Secondly I want to remove the firewall rules I created earlier. The last thing you want when you are struggling with a complex set up is a load of extra rules to confuse things. I removed them like this:

sudo -i
iptables -Z
iptables -X
iptables -F
exit

I don’t want them coming back when I reboot so I run this so that an empty rules file is run temporarily:

sudo mv /etc/iptables.up.rules /etc/iptables.default.rules
sudo touch /etc/iptables.empty.rules
sudo ln -s /etc/iptables.empty.rules /etc/iptables.up.rules

Right, now we can configure Xen. Your /etc/xen/xend-config.sxp looks like this:

# -*- sh -*-

(network-script network-route)
(vif-script vif-route)

So, now you create some new domains. The first one is going to be for a private network:

sudo mkdir /var/xen
sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm1 --ip 10.0.0.1 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

This whirrs away for a while and eventually you have a disk.img and swap.img in /var/xen/domains/vm1 and a config file in /etc/xen/vm1.cfg. You can start it up like this:

sudo xm create -c /etc/xen/vm1.cfg

Login as root, set a new password and then:

ping google.com

You should get lots of replies and no lost packets. Try pinging the IP address of Dom0 too, it should work fine. So, that’s one virtual machine set up on a private IP address not accessible to the public.

You can now follow this tutorial to setup a Nginx and a Pylons application on the server and have port 80 forwarded from Dom0 to the virtual machine. You’ll need to put any iptables rules back into the iptables.up.rules if you want them to work when the server restarts.

You can create as many virtual machines as you like in this way. Here’s another called vm3 on 10.0.0.3. Again, vm3 should be able to ping google.com, Dom0 and vm1:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm3 --ip 10.0.0.3 --netmask 255.255.255.0 --gateway 10.0.0.254 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

General Infomation
--------------------
Hostname       :  vm3
Distribution   :  etch
Fileystem Type :  ext3

Size Information
----------------
Image size     :  5Gb
Swap size      :  1024Mb
Image type     :  sparse
Memory size    :  512Mb
Kernel path    :  /boot/vmlinuz-2.6.18-4-xen-amd64
Initrd path    :  /boot/initrd.img-2.6.18-4-xen-amd64

Networking Information
----------------------
IP Address 1   : 10.0.0.3
Netmask        : 255.255.255.0
Gateway        : 10.0.0.254


Creating swap image: /var/xen/domains/vm3/swap.img
Done

Creating disk image: /var/xen/domains/vm3/disk.img
Done

Creating ext3 filesystem on /var/xen/domains/vm3/disk.img
Done

Installing your system with debootstrap mirror http://ftp.freenet.de/debian/
Done

Running hooks
Done

No role script specified.  Skipping

Creating Xen configuration file
Done
All done


Logfile produced at:
         /var/log/xen-tools/vm3.log

Again, you can setup any forwarding rules so that ports on the virtual machine can be accessed from Dom0. You should be able to ping 10.0.0.1 (if it is running) from this virtual machine and it should be able ping you.

Next, lets setup the public virtual machines. The hosting company have provided a range of IPs from 78.47.146.249 to 78.47.146.254. These are on a different subnet from my server so I might have had to follow Steve’s Xen setup here. Luckily though these IPs are already routed straight to my server so I don’t need to worry. Also, as someone pointed out in comment #16 on that page, you don’t need to waste an IP on a "bridge" because you can add the Dom0 IP as a route on the virtual machine.

So without further ado, here’s what you need to do. First create another vitual machine (or you could edit the settings on the old one). You might expect to be able to use a command like this to simply generate the new virtual machine:

sudo xen-create-image --debootstrap --dir=/var/xen --size=5Gb --memory=512Mb --fs=ext3 --dist=etch --hostname=vm4 --ip 78.47.146.251 --netmask 255.255.255.248 --initrd=/boot/initrd.img-2.6.18-4-xen-amd64 --kernel=/boot/vmlinuz-2.6.18-4-xen-amd64 --mirror=http://ftp.freenet.de/debian/ --swap=1024Mb

By the way, if you try to boot it and it fails with:

Error: Device 0 (vif) could not be connected. Backend device not found.

it might mean you already have a virtual machine running with the same address.

Once you’ve booted the virtual machine, change the networking settings to use these details by editing /etc/network/interfaces. Replace this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 gateway
 netmask 255.255.255.248

With this:

# The primary network interface
auto eth0
iface eth0 inet static
 address 78.47.146.251
 network 78.47.146.248
 netmask 255.255.255.248
 up route add 78.46.35.5 dev eth0
 up route add default gw 78.46.35.5
 down route del default gw 78.46.35.5
 down route del 78.46.35.5 dev eth0

Notice that you don’t need to specify a gateway, but you do need a network. The gateway routes are added manually in the up commands and removed in the down commands:

/etc/init.d/networking restart

You should now be able to ping google and all the other servers and what is more, if you ping 78.47.146.251 from anywhere else on the internet, the server will respond because vm4 is now publicly accessible on the internet under that IP address.

Following on from my last article I’ve now received the email with the free IP addresses:

Dear Mr. James Gardner,

Below you will find your additional IP addresses added to
the server 78.46.35.5.

Please note that you can use the subnet only for this server.

IP: 78.47.146.248
Mask: 255.255.255.248
Broadcast: 78.47.146.255

Useable IP addresses:
78.47.146.249 to 78.47.146.254

Instructions for installing additional IP addresses
you will find at the following link:
http://wiki.hetzner.de/index.php/Zusätzliche_IP-Adressen (in German)

Our competent support team is ready to answer your questions.

Please visit our website at http://www.hetzner.de/support.html.

This entry will show how to set up virtual ethernet interfaces for each of the IPs so they all resolve to the server. Note this isn’t what you want for Xen, see my next entry for that.

Following the link results in this page (translated). Which explains that you might have 6 IP addresses rather than 5 depending on the network setup. Looks like I have 6 so that seems good.

The Debian instructions for setting up these IP addresses are translated here but you’ll want to compare them to the original because Google reformats the config files in an attempt to translate them.

The basic technique is to create a virtual ethernet interface for each IP address so that the server can connect on each IP address. You do this by adding them to the /etc/network/interfaces config file. Here’s what mine looks like before:

### Hetzner Online AG - installimage
# Loopback device:
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules

# device: eth0
auto eth0
iface eth0 inet static
  address 78.46.35.5
  broadcast 78.46.35.31
  netmask 255.255.255.224
  gateway 78.46.35.1

# default route to access subnet
up route add -net 78.46.35.0 netmask 255.255.255.224 gw 78.46.35.1 eth0

and after:

### Hetzner Online AG - installimage
# Loopback device:
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules

# device: eth0
auto eth0
iface eth0 inet static
  address 78.46.35.5
  broadcast 78.46.35.31
  netmask 255.255.255.224
  gateway 78.46.35.1

# Now set up the virtual interfaces for the other IP addresses:

auto eth0:0
iface eth0:0 inet static
address 78.47.146.249
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:1
iface eth0:1 inet static
address 78.47.146.250
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:2
iface eth0:2 inet static
address 78.47.146.251
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:3
iface eth0:3 inet static
address 78.47.146.252
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:4
iface eth0:4 inet static
address 78.47.146.253
broadcast 78.47.146.255
netmask 255.255.255.248

auto eth0:5
iface eth0:5 inet static
address 78.47.146.254
broadcast 78.47.146.255
netmask 255.255.255.248

# End of the virtual interfaces setup

# default route to access subnet
up route add -net 78.46.35.0 netmask 255.255.255.224 gw 78.46.35.1 eth0

Do a quick ping test from another machine - the IP addresses won’t work yet:

james@dirac:~$ ping 78.47.146.254
PING 78.47.146.254 (78.47.146.254) 56(84) bytes of data.

--- 78.47.146.254 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2008ms

Now you need to restart networking and cross your fingers. If your networking config is broken you’ll have to use the same rescue boot technique I described earlier for SSH:

sudo /etc/init.d/networking restart

Now if you ping the same IP your server should respond:

james@dirac:~$ ping 78.47.146.254
PING 78.47.146.254 (78.47.146.254) 56(84) bytes of data.
64 bytes from 78.47.146.254: icmp_seq=1 ttl=56 time=38.6 ms
64 bytes from 78.47.146.254: icmp_seq=2 ttl=56 time=57.3 ms

--- 78.47.146.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 38.687/48.003/57.319/9.316 ms

Your IP adresses are now setup correctly to point to the server.

I’m not sure how useful this setup actually is as I’m using Xen now, but it is here in case it helps someone.

I’ve had an instinctive feel for a long time that some URL structures are better than others and whilst URLs which map to the structure of the code on the filesystem are clearly bad I thought it would be interesting to think about exactly what makes a good URL structure and what doesn’t, both from a usability and technical point of view. My motivation for this today rather than at any other time is that I’m writing a chapter on Routes for the Pylons Book so I thought it would make useful background reading.

First a definition of the parts of a URL:

http://jimmyg.org:80/some/url#fragment?foo=bar
|--|   |---------|--|--------|--------|------|
 |          |     |     |        |       |
protocol    |    port   |     fragment   |
       domain name  path info      query string

1. Describe the content

An obvious URL is a great URL. If a user can glance at a link in an email and know what it contains you have done your job. This means choosing URL parts which accurately describe what is contained in each folder and always using a descriptive word rather than an ID in the URL. For example, if you were designing a blog you should try to use apr instead of 04 to represent April and you should use the name of a category rather than its ID. This makes your URLs more intuitive to your users and give search engines a better chance of understanding what the page is about.

You might think that direct use of URLs is likely to decrease as people use search engines and social bookmarking sites more frequently but research this year by Edward Cutrell and Zhiwei Guan from Microsoft Research where they conducted an eyetracking study of search engine use that found that people spend 24% of their gaze time looking at the URLs in the search results. If your URLs describe their content, uses can make a better guess about whether or not your content is what they are after.

2. Keep it short

Try to keep your URLs as short as possible without breaking any of the other tips here. Short URLs are easier to type in or to copy and paste into documents and emails. If possible, keeping URLs to less than 80 characters is ideal so that users can pase URLs into email without having to use URL shortening tools like qurl.com or tinyurl.com.

3. Hyphens separate best

It is best to use single words in each part of a URL but if you have to use multiple words, for example for the title of a blog post, then hyphens are the best characters to use to separate the words. e.g. /2007/nov/my-blog-post-title/. Unfortunately the - character cannot be used in Python keywords so if you intend to use the URL fragments as Python controller names or actions you might want to convert them to _ characters first. Incidentally using hyphens to separate words is also the most readable way of separating terms in CSS styles.

4. Static-looking URLs are best

Regardless of how your content is actually generated it is worth structuring URLs so that they don’t contain lots of &, = and ? characters which most visitors won’t properly understand. If you can write a URL like ?type=food&category=apple as /food/apple then users can see much more quickly what is about.

5. Keeping URLs lowercase makes your life easier

The protocol and domain name parts of a URL can technically be entered in any case but the part after the # is case sensitive. How a particular server treats anything between the two depends on the server, operating system and what the URL resolves to. UNIX is case-sensitive, while Windows isn’t so if the URL resolves to a file, Windows servers will generally allow any case whilst UNIX ones won’t. Query string parameters are also case sensitive. You can generally save yourself a headache by keeping everything lowercase and issuing a 404 for anything which isn’t. Of course if you are writing a wiki where the page names depend on the capitalisation then you’ll need to make the URLs case sensitive.

6. Keep the underlying technology out of the URL

Your users don’t care which specific technology you are using to generate your pages or whether it is a .html or .xhtml so the basic rule is don’t use a file extension for dynamically generated pages unless you are doing something clever in your application internally like determining the format to represent the content based on the extension. It is also generally best to choose names which represent what the URL is rather than its technology so you might consider style and script to be better choices than rather than css and js for your CSS and JavaScript files.

7. Use singular terms rather than plural

This is a matter of personal preference but rather than having a URL like /people/james use /person/james. It is likely that the last part of a URL will describe one thing, so the previous parts of the URL should describe that thing too. In this case james is a person, not a people so /person/james is more appropriate. You can use this convention throughout your application in naming controllers, database tables etc.

8. Only use Disambiguated URLs

Any piece of content should have one and only one definitive URL, with any alternatives acting as a permanent redirect. In the past features like Apache’s DirectoryIndex have meant that if you entered a URL which resolved to a folder, the default document for that folder would be served. This means that two URLs would exist for one resource (discussed by J Tauber here). To make matters worse servers are configured so that http://www.example.com/someresource and http://example.com/someresource both point to the same resource. This means there can easily be 4 URLs for the same resource.

There are three good reasons why this is bad:

• Browser or server caches will have to cache 4 versions of the page. Put another way this means they can’t improve performance if you visit a different version of the same URL the second time.
• All versions of the page will be treated by web browsers as different resources so the user’s browsing history won’t be accurate.
• Search engines and social bookmarking sites give pages that are linked to more frequently a higher rank. If you have 4 different URLs for the same page you are effectively dividing your rank by 4.

9. Never change a URL

Otherwise your users won’t be able to find the page you bookmarked and any page rank you built up in social bookmarking sites or search engines will be lost. If you absolutely have to change a URL, ensure you set up a permanent 301 redirect to the new one so that your user’s don’t get 404 errors. The w3c put it best: Cool URLs don’t change.

10. Treat the URL as UI.

Navigation links, sidebars and tabs are all well and good but if you have a good URL structure your users should be able to navigate your site by changing parts of the URL. There are a few rules about how best to do this:

• Ensure that for every part of the path info a user might remove, a useful page is returned.

  For example if the URL /2007/nov/entry gives a blog entry, 2007/nov might give a list of all the November entries and /2007 might give a list of all entries from 2007.

• Never have a URL on your domain which gives a 500 Error

  It doesn’t take a genius to realise that you don’t want any URL in your website to crash cause a server error but developers don’t always think about what will happen if a user starts hacking the URL to contain different values. For example if you have a URL /food/apple and a user changes it to /food/pizza when the application is only set up to deal with fruit it should give a 404, not a 500. If you’ve followed rule 3. above then all the variables your application code uses to generate the content will be part of the path info portion of the URL so you can issue a 404 Page Not Found response if someone enters an invalid value.

  Users are much more likely to stop trying to guess URLs if they get a 500 page because they are worried they might be breaking something and the moment they stop hacking the URL it has lost its usefulness as a UI component. Obviously you can’t issue 404s if a variable in a query string is incorrect because that would imply that whether or not the resource existed depended on the query string which it shouldn’t.

I hope that’s useful. If you have any extra tips, feel free to leave them in the comments and if you have any extra evidence to support any of these tips I’d be interested to hear it. I think the most important tip though it that you should use common sense when designing a URL structure and don’t apply any of the tips too rigidly, after all you know your application’s and user’s requirements better than me so you are better placed to make a judgment about what will work best for you.

On Dom0 set up port forwarding from port 80 to the DomU running NginX, in this case 10.0.0.1:

sudo -i
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 80 -j DNAT --to 10.0.0.1:80
exit

These get lost on each reboot unless you put them into a script in /etc/network/if-up.d/iptables but that’s OK for the moment.

First install Nginx:

sudo apt-get install nginx

If you visit the URL of the Dom0 server you should see the Welcome to nginx! message served from Nginx on the DomU.

Next you need to setup Python and Pylons. Install python:

sudo apt-get install python mercurial

Get vitualenv:

wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-0.9.1.tar.gz
tar zxfv virtualenv-0.9.1.tar.gz
cp virtualenv-0.9.1/virtualenv.py ./
rm virtualenv-0.9.1.tar.gz
rm -r virtualenv-0.9.1

Decide where you want your Pylons applications. I like mine in /var/web so lets create a test project:

sudo mkdir /var/web
sudo mkdir /var/web/test
sudo chown james:james /var/web/test

Setup a virtual environment:

james@vm2:~$ python virtualenv.py /var/web/test
New python executable in /var/web/test/bin/python
Installing setuptools.............................done.
james@vm2:~$ cd /var/web/test

Now you can install Pylons:

mkdir dep
cd dep
hg clone http://pylonshq.com/hg/pylons-dev
cd pylons-dev
../../bin/python setup.py develop

Now create a project:

mkdir src
cd src
../bin/paster create -t pylons Test

Now edit Test/test/config/middleware.py and set use_webop=True. Serve the app with:

cd Test/
../../bin/python setup.py develop
../../bin/easy_install sqlalchemy                        <-- So that beaker doesn't break
../../bin/paster serve --reload development.ini

That’s it. Now you just need to setup Nginx to reverse proxy to the paste server running on port 5000.

Edit the /etc/nginx/nginx.conf file so that the location / part looks like this:

location / {
    proxy_redirect          off;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffer_size       4k;
    proxy_buffers           4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass  http://127.0.0.1:5000;
    proxy_redirect  default;
}

You’ll then need to restart nginx and the paster server:

sudo /etc/init.d/nginx restart
../../bin/paster serve --reload development

If you press refresh in your browser you should see the Pylons welcome page.

I bought a new server from Hetzner (German) on Sunday 4th November and on the evening of the 5th I received my sign in details by email in both German and English for the server, already installed with the 64bit version of Debian Etch. This post documents my progress setting it up.

Verwaltung                           Administration
        Sicherheit                           Security
        E-Mail-Adressen                      Emails
        Support-Anfragen                     Support requests
        Status-Benachrichtigung              Status Notification
        Newsletter                           Newsletter
        RIPE-Registration                    RIPE-Registration
Rechnungen                           Bills
        Offene Posten                        Open Item
Traffic-Statistik                    Traffic Statistics
        Traffic-Limit-Reporting              Traffic limit Reporting
        Tagesbericht                         Daily Report
        Monatsbericht                        Monthly Report
        Jahresbericht                        Annual Report
Leistungsübersicht                   Performance Overview
        Resetaufträge                        Reset orders
        Rescuesystem                         Rescue System
        VNC-Installation                     VNC-Installation
        neuen Server bestellen               New server order
        Separater Admin-Zugang               Separate Admin Access
Reverse-DNS-Einträge                 Reverse-DNS
        Eintrag anlegen                      Entry
        Eintrag löschen                      Delete Entry
Dokumentation                        Documentation
        Daten-Export-Schnittstelle           Data Export interface

``Automatischen Hardware-Reset auslösen`` - Perform an automatic hardware reset.

Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  520M  346G   1% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
udev                   10M   36K   10M   1% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
Debian-40-etch-64-minimal:~# cat /etc/issue
Debian GNU/Linux 4.0 \n \l

Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-5-amd64 #1 SMP Tue Oct 2 20:37:02 UTC 2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
             total       used       free     shared    buffers     cached
Mem:          3926         73       3853          0         12         19
-/+ buffers/cache:         41       3885
Swap:         2055          0       2055

james@dirac:~$ ssh root@doppler.3aims.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
50:10:6b:5f:dc:e7:45:17:73:00:d0:50:f0:0e:48:be.
Please contact your system administrator.
Add correct host key in /home/james/.ssh/known_hosts to get rid of this message.
Offending key in /home/james/.ssh/known_hosts:1
RSA host key for doppler.3aims.com has changed and you have requested strict checking.
Host key verification failed.

Linux rescue 2.6.22.2 #2 SMP Tue Aug 28 09:28:15 CEST 2007 x86_64

------------------------------------------------------------------

  Welcome to the Hetzner Rescue System.

  This Rescue System is based on Debian 4.0 (etch) with a newer
  kernel. You can install software like in a normal system.

  To install a new operating system from one of our prebuilt
  images, run 'installimage' and follow the instructions.

  For more information take a look at http://wiki.hetzner.de

------------------------------------------------------------------

root@rescue ~ #

        Hetzner Online AG - installimage


#~ server will be installed now. this will take a few minutes.
#~ you can abort at any time with CTRL+C ..
#~ ( init) ~ reading vars...                            [ OK ]
#~ ( 1/11) ~ deleting partitions...                     [ OK ]
#~ ( 2/11) ~ creating partitions and fstab...           [ OK ]
#~ ( 3/11) ~ formatting partitions...                   [ OK ]
#~ ( 4/11) ~ mounting partitions...                     [ OK ]
#~ ( 5/11) ~ extracting imagefile from local...         [ OK ]
#~ ( 6/11) ~ setting up network config for eth0...      [ OK ]
#~ ( 7/11) ~ chrooting some commands...                 [ OK ]
#~ ( 8/11) ~ clearing logfiles...                       [ OK ]
#~ ( 9/11) ~ setting up some files...                   [ OK ]
#~ (10/11) ~ setting up rootpassword...                 [ OK ]
#~ (11/11) ~ setting up bootloader grub...              [ OK ]


#~~~ INSTALLATION COMPLETE ~~~#
you may now reboot into your new system
you can login to your new system with the same
password as you logged in into the rescue system


root@rescue ~ #

apt-get install xen-linux-system-2.6.18-4-xen-amd64
reboot

# -*- sh -*-
(network-script network-route)
(vif-script vif-route)

/etc/init.d/xend restart

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  612M  346G   1% /
tmpfs                 1.7G     0  1.7G   0% /lib/init/rw
udev                   10M   36K   10M   1% /dev
tmpfs                 1.7G     0  1.7G   0% /dev/shm
Debian-40-etch-64-minimal:~#  cat /etc/issue
Debian GNU/Linux 4.0 \n \l

Debian-40-etch-64-minimal:~# uname -a
Linux Debian-40-etch-64-minimal 2.6.18-4-xen-amd64 #1 SMP Fri May 4 02:40:51 UTC  2007 x86_64 GNU/Linux
Debian-40-etch-64-minimal:~# free -m
             total       used       free     shared    buffers     cached
Mem:          3366        200       3165          0          1         15
-/+ buffers/cache:        182       3183
Swap:         2055          0       2055
Debian-40-etch-64-minimal:~# fdisk -l

Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         262     2104514+  82  Linux swap / Solaris
/dev/sda2             263       48641   388604317+  83  Linux

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Debian-40-etch-64-minimal:~# fdisk -u /dev/sdb

The number of cylinders for this disk is set to 48641.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-781422767, default 63):
Using default value 63
Last sector or +size or +sizeM or +sizeK (63-781422767, default 781422767):
Using default value 781422767

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Debian-40-etch-64-minimal:~# /sbin/mkfs -t ext3 /dev/sdb1
mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
48840704 inodes, 97677838 blocks
4883891 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2981 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

/dev/sdb1 /mount/sdb1 ext3 defaults 0 0

mkdir /mount
mkdir /mount/sdb1
mount /dev/sdb1

Debian-40-etch-64-minimal:~# fdisk -l

Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1         262     2104514+  82  Linux swap / Solaris
/dev/sda2             263       48641   388604317+  83  Linux

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       48642   390711352+  83  Linux
Debian-40-etch-64-minimal:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             365G  612M  346G   1% /
tmpfs                 1.7G     0  1.7G   0% /lib/init/rw
udev                   10M   40K   10M   1% /dev
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/sdb1             367G  195M  348G   1% /mount/sdb1

Much better, 674Gb free space.

doppler.3aims.com.  1440  A   78.46.35.5

78.46.35.5 doppler doppler.3aims.com

Debian-40-etch-64-minimal:~# vim /etc/hostname
Debian-40-etch-64-minimal:~# vim /etc/hosts
Debian-40-etch-64-minimal:~# hostname doppler

Debian-40-etch-64-minimal:~# apt-get install sudo
Debian-40-etch-64-minimal:~# adduser james
Adding user `james' ...
Adding new group `james' (1000) ...
Adding new user `james' (1000) with group `james' ...
Creating home directory `/home/james' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for james
Enter the new value, or press ENTER for the default
        Full Name []: James Gardner
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [y/N] y

Debian-40-etch-64-minimal:~# su james
james@doppler:/root$

$ sudo dpkg-reconfigure locales

james@doppler:/root$ sudo tzconfig
Your current time zone is set to Europe/Berlin
Do you want to change that? [n]: y

Please enter the number of the geographic area in which you live:


        1) Africa                       7) Australia

        2) America                      <img src='http://jimmyg.org/wp-includes/images/smilies/icon_cool.gif' alt='8)' class='wp-smiley' /> Europe

        3) US time zones                9) Indian Ocean

        4) Canada time zones            10) Pacific Ocean

        5) Asia                         11) Use System V style time zones

        6) Atlantic Ocean               12) None of the above


Then you will be shown a list of cities which represent the time zone
in which they are located. You should choose a city in your time zone.

Number: 8

Amsterdam Andorra Athens Belfast Belgrade Berlin Bratislava Brussels
Bucharest Budapest Chisinau Copenhagen Dublin Gibraltar Guernsey Helsinki
Isle_of_Man Istanbul Jersey Kaliningrad Kiev Lisbon Ljubljana London
Luxembourg Madrid Malta Mariehamn Minsk Monaco Moscow Nicosia Oslo Paris
Podgorica Prague Riga Rome Samara San_Marino Sarajevo Simferopol Skopje
Sofia Stockholm Tallinn Tirane Tiraspol Uzhgorod Vaduz Vatican Vienna
Vilnius Volgograd Warsaw Zagreb Zaporozhye Zurich

Please enter the name of one of these cities or zones
You just need to type enough letters to resolve ambiguities
Press Enter to view all of them again
Name: [] London
Your default time zone is set to 'Europe/London'.
Local time is now:      Fri Nov  9 16:53:52 GMT 2007.
Universal Time is now:  Fri Nov  9 16:53:52 UTC 2007.

Port 30000         <-- change to a port other than 22
PermitRootLogin no
AllowUsers james

PasswordAuthentication no

doppler:~# /etc/init.d/ssh restart
Restarting OpenBSD Secure Shell server: sshd.

ssh james@doppler.3aims.com -p 30000

mount /dev/sda2 /mnt -t ext3

james@doppler:~$ sudo -i
Password:
doppler:~# whoami
root

doppler:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables-save > /etc/iptables.current.rules

iptables-restore < /etc/iptables.test.rules

doppler:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     0    --  anywhere             anywhere
REJECT     0    --  anywhere             loopback/8          reject-with icmp-port-unreachable
ACCEPT     0    --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:www
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:https
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:30000
ACCEPT     icmp --  anywhere             anywhere            icmp echo-request
LOG        0    --  anywhere             anywhere            limit: avg 5/min burst 5 LOG level debug prefix `iptables denied: '
REJECT     0    --  anywhere             anywhere            reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     0    --  anywhere             anywhere            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     0    --  anywhere             anywhere

iptables-save > /etc/iptables.up.rules

...
auto lo
iface lo inet loopback
pre-up iptables-restore < /etc/iptables.up.rules
...

sudo apt-get update
sudo apt-get upgrade

This blog post just documents one way of creating a package. This is for one named SQLAlchemyManager which is some experimental SQLAlchemy middleware I’m developing. Package names are normally lowercase but the project itself can use CamelCase so in this example the actual package is called sqlalchemymanager.

First create the directory structure:

mkdir SQLAlchemyManager
cd SQLAlchemyManager
mkdir docs
mkdir sqlalchemymanager
mkdir examples
mkdir tests

This tutorial assumes you are creating the new structure in an existing SVN tree. If you aren’t you should now import the directories into a new Subversion repository.

Then create the setup.cfg file:

[egg_info]
tag_build = dev
tag_svn_revision = true

These options tell setuptools that this is a development release, and that the SVN revision number should be included in the package version. This will create an egg with dev-r139.egg or similar at the end of the filename. When you want to make a production release you should comment out these lines.

Next you need to set up ez_setup.py so that if your project’s users don’t have setuptools it will be automatically installed when they try to use it. You can do this svn:externals definition so that subversion will automatically include the latest files you need:

ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup

You can set this by executing this command in your project directory:

svn propedit svn:externals .

And then adding the line shown above to the file that comes up for editing. Then, whenever you update your project, ez_setup will be updated as well.

Next you need a setup.py file which contains most of the metadata about the project. This one is setup to make use of the ez_setup.py you just set up if the user doesn’t have setup tools. Importing ez_setup in this way doesn’t cause problems with Buildout either:

try:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages

import sys, os

version = '0.0'

setup(
    name='SQLAlchemyManager',
    version=version,
    description="",
    long_description="""\
    """,
    # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[],
    keywords='',
    author='',
    author_email='',
    url='',
    license='',
    packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
    include_package_data=True,
    zip_safe=False,
    install_requires=[
    ],
    entry_points="""
    """,
)

You can enter information for all the fields listed above but version, install_requires and long_description are particularly interesting.

version

This shoud be a string containing the version number of the package in three parts. The first part is the maor version, the second part the minor version and the third part the revision. In this case I’m developing towards a 1.0 release so the major relase will be 0 until the software is stable enough for 1.0. This is the first version so the minor version is 1 and I haven’t released any revisions so I’ll call that 0. The version string is therefore "0.1.0".

install_requires

This should be a comma separated list of packages and their version numbers for all the packages the package you require depend on. SQLAlchemyManager depends on SQLAlchemy 0.4 so I’d add "SQLAlchemy>=0.4,<=0.4.99" to the install list. Notice that I specify <=0.4.99. The idea behind choosing this is that any incompatible changes to SQLAlchemy should result in a new minor version of SQLAlchemy being released so if my package works with 0.4 it should also work with 0.4.5, 0.4.9 and even 0.4.99 if there are that many revisions. It might not work with 0.5 though. You might think I could specify <0.5 here but 0.5a 0.5dev“ and 0.5rc1 are all treated as less that 0.5 by easy install so the recommendation is to use the .99 format instead. You could choose <=0.4.99999 if you wanted but pacjages rarely have more than about 30 revisions so 99 is virtually always fine.

long_description

This is handy because it allow you to write detailed information in reStructuredText format. This information then forms the main text for the page which will appear on the Cheeseshop which effectively means you don’t need to create a website for your project because you can keep all information in the long description. We’ll see an example of a long description at the end.

Now you need to write some documentation. I create all the docs with a .txt extension and using Windows line endings so that the information can be easily read by both Windows and Linux users.

First create the README.txt file:

See the ``docs/index.txt`` for information.

Then the CHANGELOG.txt file:

Changes
=======

0.1.0 (**svn**)

* First version

Every time you make a change you should add it to the change log, I put the most recent change at the top. Every time you create a new version you should add a new section. The (**svn**) label marks which is the version in subversion. You should remember to move this if you create a new version or remove it if you create a production release.

Now the documentation in docs/index.txt:

SQLAlchemyManager
+++++++++++++++++++++++

.. contents ::

Summary
=======

* Provides a sensible way of using SQLAlchemy in WSGI applications

Get Started
===========

* Download from the `SQLAlchemyManager Cheeseshop page <http://python.org/pypi/SQLAlchemyManager>`_
* Install with ``easy_install SQLAlchemyManager``.

Author
======

`James Gardner <http://jimmyg.org/>`_ james at pythonweb dot org

You can now create the long description. Here’s one way to do it. At the top of the setup.py file you can do this:

def read(*rnames):
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

long_description = (
    "\n"+read('docs/index.txt')
    + '\n'
    + read('CHANGELOG.txt')
    + '\n'
    'License\n'
    '=======\n'
    + read('LICENSE.txt')
    + '\n'
    'Download\n'
    '========\n'
)

Then update ensure long_description=long_description in the setup() function. This will then automatically create the long description from the files in your project, helping to make life easier.

You can now create the files your project will use in the sqlalchemy directory and check them all in.

Finally you can add the LICENSE.txt file, preferably using one of the Open Source Licenses, the MIT license being the one I generally use. You should update the license line in setup.py too.

You’ll also want to specify a summary which will appear in the index page on the cheeseshop. Choose a few words to describe the project and put them in the description argument string.

svn co http://somedomain/svn/SQLAlchemyManager/tags tags
svn export http://somedomain/svn/SQLAlchemyManager/trunk 0.1.0
mv 0.1.0 tags/
svn add tags/0.1.0
svn ci tags/0.1.0 -m "Creating the 0.1.0 tag"

svn ci tags/0.1.0 -m "Final changes for the 0.1.0 release"

svn export http://somedomain/svn/SQLAlchemyManager/tags/0.1.0 0.1.0
cd 0.1.0
python setup.py sdist bdist_egg register upload

I bought the HP LaserJet 1022 for £139 from PC world today. Not exactly cheap I know but I wanted a decent reliable printer to replace my melted Samsung ML-1610 which always gave slightly grey prints. The HP 1022 does exactly what it says on the tin. It produces nice quality black and white prints very quickly.

The OpenPrinting page here suggests it works perfectly with Linux. It works perfectly now with Debian Etch but just following the Gnome CUPS wizard didn’t do it. In playing around I performed the following and but I’m not sure which step was the one that made it work.

• Used the standard Gnome add printer wizard (gnome-cups-add) which auto-detected the printer and suggested the foo2zjs driver.
• Installed sudo apt-get install foo2zjs. Not sure why it why it wasn’t already installed since I’d selected it in the wizzard.
• Plugged it into a USB 2.0 plug rather than 1.1
• Visited http://localhost:631/printers/ and clicked "Print Test Page"

It then printed the CUPS test page and still works fine even when connected to a USB 1.1 port. It still won’t print a test page from the Gnome CUPS manager though but prints from anywhere else so it is possible it was working straight away anyway before I started trying other things.

Anyway, bottom line is that it seems to work perfectly on Debian Etch so you can buy with confidence.