James Gardner


Mercurial Hosting with mod_wsgi

Posted in Uncategorized by thejimmyg on the August 30th, 2008

If you choose to host a set of mercurial repositories using mod_wsgi you will probably his this exception:

mod_wsgi (pid=2189): Exception occurred processing WSGI script 'script/hgweb.wsgi'.
Traceback (most recent call last):
  File "python2.5/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 73, in __call__
    return self.run_wsgi(req)
  File "python2.5/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 98, in run_wsgi
    req.write(self.makeindex(req, tmpl))
  File "python2.5/site-packages/mercurial/hgweb/request.py", line 84, in write
    for part in thing:
  File "python2.5/site-packages/mercurial/templater.py", line 137, in __call__
    item = iters[0].next()
  File "python2.5/site-packages/mercurial/templater.py", line 120, in _process
    for i in v:
  File "python2.5/site-packages/mercurial/hgweb/hgwebdir_mod.py", line 175, in entries
    u.warn(_('error reading %s/.hg/hgrc: %s\\n' % (path, e)))
  File "python2.5/site-packages/mercurial/ui.py", line 443, in warn
    self.write_err(*msg)
  File "python2.5/site-packages/mercurial/ui.py", line 386, in write_err
    if not sys.stdout.closed: sys.stdout.flush()
IOError: sys.stdout access restricted by mod_wsgi

If you just change these lines in mercurial/ui.py around 386 from this:

def write_err(self, *args):
    try:
        if not sys.stdout.closed: sys.stdout.flush()
        for a in args:
            sys.stderr.write(str(a))
        # stderr may be buffered under win32 when redirected to files,
        # including stdout.
        if not sys.stderr.closed: sys.stderr.flush()
    except IOError, inst:
        if inst.errno != errno.EPIPE:
            raise

So that the Except clause at the end looks like this:

except IOError, inst:
    if not (inst.errno == errno.EPIPE or "access restricted by mod_wsgi" in str(inst)):
        raise

Then everything seems to work nicely.

Maintaining Scope in JavaScript Event Handlers via a Closure

Posted in JavaScript by thejimmyg on the August 29th, 2008

The example below and its comments should be self-explanatory:

<html>
<head></head>
<body>
<!--
This example demonstrates how to use a JavaScript closure to ensure that
an event handler can access variables passed to a create_handler() function
from the scope of the code in which the handler is assigned.

Ordinarily if you didn't use the closure and the variable which was
passed to the function changed its value the handler would
have the new value, not the value of the variable at the time
it was passed to the handler.

Here we only pass one argument but you could modify create_handler()
to pass more variables if needed.
-->
<div id="keep_element">Click me! I keep my value.</div>
<div id="lose_element">Click me! I lose my value.</div>
<script language="javascript">
function create_handler(arg){
    // Return a reference to an anonymous inner function created
    // with a function expression:-
    return (function(e){
        // This inner function is to be executed on the event
        // and when it is executed it can read, and act upon, the
        // parameters passed to the outer function, as well as have
        // access to the event.
        alert(arg+' Plus, we\'ve got access to the event: '+e);
    });
}
// Get the elements we want to add the handler to
var keep_elem = document.getElementById('keep_element');
var lose_elem = document.getElementById('lose_element');
// Create a variable
var msg = 'The original value.';
// Add the onclick handler so that when it is called it uses the value
// a at the time the handler was added.
lose_elem.onclick = function(e) {
    alert(msg+' Plus, we\'ve got access to the event: '+e);
};
keep_elem.onclick = create_handler(msg);
// Set a to a different value, this won't affect the value which
// is is appears in the alert box when the element is clicked.
msg = 'The value has changed.';
</script>
</body>
</html>

Frameworks such as YUI must do something similar with their automatic scope correction functionality.

Mercurial Bundle Compression Error on Debian

Posted in Debian, Mercurial by thejimmyg on the August 26th, 2008

If you get this error when pushing to a repository:

hg push
pushing to https://mercuriate.3aims.com/project/...
http authorization required
realm: Dev Repo
user: xxx
password:
searching for changes
unknown bundle compression type

It is because you need to upgrade to mercurial 1.0 in backports:

$ wget http://www.backports.org/debian/pool/main/m/mercurial/mercurial-common_1.0.1-2~bpo40+1_all.deb
$ wget http://www.backports.org/debian/pool/main/m/mercurial/mercurial_1.0.1-2~bpo40+1_amd64.deb
$ dpkg -i mercurial-common_1.0.1-2~bpo40+1_all.deb mercurial_1.0.1-2~bpo40+1_amd64.deb

Building Firefox 3 on Mac OS X

Posted in Web, Software Releases, Mercurial by thejimmyg on the August 24th, 2008

First of all read the main build page at http://developer.mozilla.org/en/Mac_OS_X_Build_Prerequisites

I already have the following installed:

Getting Started

Download and install MacPorts 1.6.0 for your platform from http://www.macports.org/install.php

Install packages needed for grabbing and building Firefox:

$ sudo /opt/local/bin/port sync
$ sudo /opt/local/bin/port install mercurial libidl autoconf213

The second stage takes a long time because it builds lots of dependencies. It will stop three times while handling Python. Just keep running the command until everything works. Here’s the output I received:

james-gardners-macbook-air:~ james$ sudo /opt/local/bin/port sync
james-gardners-macbook-air:~ james$ sudo /opt/local/bin/port install mercurial libidl autoconf213
--->  Fetching bzip2
--->  Attempting to fetch bzip2-1.0.5.tar.gz from http://www.bzip.org/1.0.5/
--->  Verifying checksum(s) for bzip2
--->  Extracting bzip2
--->  Applying patches to bzip2
--->  Configuring bzip2
--->  Building bzip2 with target all
--->  Staging bzip2 into destroot
--->  Installing bzip2 1.0.5_1
--->  Activating bzip2 1.0.5_1
--->  Cleaning bzip2
--->  Fetching expat
--->  Attempting to fetch expat-2.0.1.tar.gz from http://downloads.sourceforge.net/expat
--->  Verifying checksum(s) for expat
--->  Extracting expat
--->  Configuring expat
--->  Building expat with target all
--->  Staging expat into destroot
--->  Installing expat 2.0.1_0
--->  Activating expat 2.0.1_0
--->  Cleaning expat
--->  Fetching gperf
--->  Attempting to fetch gperf-3.0.3.tar.gz from http://ftp.gnu.org/gnu/gperf
--->  Verifying checksum(s) for gperf
--->  Extracting gperf
--->  Configuring gperf
--->  Building gperf with target all
--->  Staging gperf into destroot
--->  Installing gperf 3.0.3_0
--->  Activating gperf 3.0.3_0
--->  Cleaning gperf
--->  Fetching libiconv
--->  Attempting to fetch libiconv-1.12.tar.gz from http://ftp.gnu.org/gnu/libiconv
--->  Verifying checksum(s) for libiconv
--->  Extracting libiconv
--->  Applying patches to libiconv
--->  Configuring libiconv
--->  Building libiconv with target all
--->  Staging libiconv into destroot
--->  Installing libiconv 1.12_0
--->  Activating libiconv 1.12_0
--->  Cleaning libiconv
--->  Fetching ncursesw
--->  Attempting to fetch ncurses-5.6.tar.gz from http://ftp.gnu.org/gnu/ncurses
--->  Verifying checksum(s) for ncursesw
--->  Extracting ncursesw
--->  Applying patches to ncursesw
--->  Configuring ncursesw
--->  Building ncursesw with target all
--->  Staging ncursesw into destroot
--->  Installing ncursesw 5.6_1
--->  Activating ncursesw 5.6_1
--->  Cleaning ncursesw
--->  Fetching ncurses
--->  Verifying checksum(s) for ncurses
--->  Extracting ncurses
--->  Applying patches to ncurses
--->  Configuring ncurses
--->  Building ncurses with target all
--->  Staging ncurses into destroot
--->  Installing ncurses 5.6_0
--->  Activating ncurses 5.6_0
--->  Cleaning ncurses
--->  Fetching gettext
--->  Attempting to fetch gettext-0.17.tar.gz from http://ftp.gnu.org/gnu/gettext
--->  Verifying checksum(s) for gettext
--->  Extracting gettext
--->  Applying patches to gettext
--->  Configuring gettext
--->  Building gettext with target all
--->  Staging gettext into destroot
--->  Installing gettext 0.17_3
--->  Activating gettext 0.17_3
--->  Cleaning gettext
--->  Fetching python25
--->  Attempting to fetch Python-2.5.2.tgz from http://www.python.org//ftp/python/2.5.2/
--->  Verifying checksum(s) for python25
--->  Extracting python25
--->  Applying patches to python25
--->  Configuring python25
--->  Building python25 with target all libpython2.5.dylib
--->  Staging python25 into destroot
--->  Installing python25 2.5.2_5+darwin_9
--->  Activating python25 2.5.2_5+darwin_9

To fully complete your installation and make python 2.5 the default, please run

    sudo port install python_select
    sudo python_select python25

--->  Cleaning python25
--->  Fetching py25-bz2
--->  Attempting to fetch Python-2.5.2.tar.bz2 from http://www.python.org/ftp/python/2.5.2/
--->  Verifying checksum(s) for py25-bz2
--->  Extracting py25-bz2
--->  Configuring py25-bz2
--->  Building py25-bz2 with target build
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-bz2/work/Python-2.5.2/Modules" &#038;& /opt/local/bin/python2.5 setup.py build " returned error 1
Command output: running build
running build_ext
building 'bz2' extension
creating build
creating build/temp.macosx-10.3-i386-2.5
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/include/ -I/opt/local/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c bz2module.c -o build/temp.macosx-10.3-i386-2.5/bz2module.o
unable to execute -DNDEBUG: No such file or directory
error: command '-DNDEBUG' failed with exit status 1

Error: The following dependencies failed to build: py25-bz2 py25-hashlib openssl zlib py25-zlib
Error: Status 1 encountered during processing.
james-gardners-macbook-air:~ james$ sudo /opt/local/bin/port install mercurial libidl autoconf213
Password:
--->  Building py25-bz2 with target build
--->  Staging py25-bz2 into destroot
--->  Installing py25-bz2 2.5.2_0
--->  Activating py25-bz2 2.5.2_0
--->  Cleaning py25-bz2
--->  Fetching zlib
--->  Attempting to fetch zlib-1.2.3.tar.bz2 from http://www.zlib.net/
--->  Attempting to fetch zlib-1.2.3.tar.bz2 from http://www.gzip.org/zlib/
--->  Verifying checksum(s) for zlib
--->  Extracting zlib
--->  Applying patches to zlib
--->  Configuring zlib
--->  Building zlib with target all
--->  Staging zlib into destroot
--->  Installing zlib 1.2.3_1
--->  Activating zlib 1.2.3_1
--->  Cleaning zlib
--->  Fetching openssl
--->  Attempting to fetch openssl-0.9.8h.tar.gz from http://www.openssl.org/source/
--->  Verifying checksum(s) for openssl
--->  Extracting openssl
--->  Applying patches to openssl
--->  Configuring openssl
--->  Building openssl with target all
--->  Staging openssl into destroot
--->  Installing openssl 0.9.8h_0
--->  Activating openssl 0.9.8h_0
--->  Cleaning openssl
--->  Fetching py25-hashlib
--->  Verifying checksum(s) for py25-hashlib
--->  Extracting py25-hashlib
--->  Configuring py25-hashlib
--->  Building py25-hashlib with target build
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_python_py25-hashlib/work/Python-2.5.2/Modules" &#038;& /opt/local/bin/python2.5 setup.py build " returned error 1
Command output: running build
running build_ext
building '_hashlib' extension
creating build
creating build/temp.macosx-10.3-i386-2.5
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/local/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _hashopenssl.c -o build/temp.macosx-10.3-i386-2.5/_hashopenssl.o
unable to execute -DNDEBUG: No such file or directory
error: command '-DNDEBUG' failed with exit status 1

Error: The following dependencies failed to build: py25-hashlib py25-zlib
Error: Status 1 encountered during processing.
james-gardners-macbook-air:~ james$ sudo /opt/local/bin/port install mercurial libidl autoconf213
Password:
--->  Building py25-hashlib with target build
--->  Staging py25-hashlib into destroot
--->  Installing py25-hashlib 2.5.2_0
--->  Activating py25-hashlib 2.5.2_0
--->  Cleaning py25-hashlib
--->  Fetching py25-zlib
--->  Verifying checksum(s) for py25-zlib
--->  Extracting py25-zlib
--->  Configuring py25-zlib
--->  Building py25-zlib with target build
--->  Staging py25-zlib into destroot
--->  Installing py25-zlib 2.5.2_0
--->  Activating py25-zlib 2.5.2_0
--->  Cleaning py25-zlib
--->  Fetching mercurial
--->  Attempting to fetch mercurial-1.0.1.tar.gz from http://www.selenic.com/mercurial/release/
--->  Verifying checksum(s) for mercurial
--->  Extracting mercurial
--->  Configuring mercurial
--->  Building mercurial with target build
--->  Staging mercurial into destroot
--->  Installing mercurial 1.0.1_1
--->  Activating mercurial 1.0.1_1
--->  Cleaning mercurial
--->  Fetching pkgconfig
--->  Attempting to fetch pkg-config-0.23.tar.gz from http://pkg-config.freedesktop.org/releases/
--->  Verifying checksum(s) for pkgconfig
--->  Extracting pkgconfig
--->  Configuring pkgconfig
--->  Building pkgconfig with target all
--->  Staging pkgconfig into destroot
--->  Installing pkgconfig 0.23_0
--->  Activating pkgconfig 0.23_0
--->  Cleaning pkgconfig
--->  Fetching glib2
--->  Attempting to fetch glib-2.16.5.tar.bz2 from http://mandril.creatis.insa-lyon.fr/linux/gnome.org/sources/glib/2.16/
--->  Verifying checksum(s) for glib2
--->  Extracting glib2
--->  Applying patches to glib2
--->  Configuring glib2
--->  Building glib2 with target all
--->  Staging glib2 into destroot
--->  Installing glib2 2.16.5_0+darwin_9
--->  Activating glib2 2.16.5_0+darwin_9
--->  Cleaning glib2
--->  Fetching libidl
--->  Attempting to fetch libIDL-0.8.10.tar.bz2 from http://mandril.creatis.insa-lyon.fr/linux/gnome.org/sources/libIDL/0.8/
--->  Verifying checksum(s) for libidl
--->  Extracting libidl
--->  Applying patches to libidl
--->  Configuring libidl
--->  Building libidl with target all
--->  Staging libidl into destroot
--->  Installing libidl 0.8.10_0
--->  Activating libidl 0.8.10_0
--->  Cleaning libidl
--->  Fetching autoconf213
--->  Attempting to fetch autoconf-2.13.tar.gz from http://ftp.gnu.org/gnu/autoconf
--->  Verifying checksum(s) for autoconf213
--->  Extracting autoconf213
--->  Configuring autoconf213
--->  Building autoconf213 with target all
--->  Staging autoconf213 into destroot
--->  Installing autoconf213 2.13_0
--->  Activating autoconf213 2.13_0
--->  Cleaning autoconf213

Now get the Firefox source code using Mercurial:

hg clone http://hg.mozilla.org/mozilla-central

To build it you will need to modify your path to include the MacPorts software:

export PATH=$PATH:/opt/local/bin

Now you can start the build:

cd mozilla-central
make -f client.mk build

This command will run for hours but when it is finished you will have a Firefox build.

Run the build in a terminal like this (you have to close any existing Friefox browsers though):

./obj-ff/dist/bin/firefox-bin

Run the debug build by clicking on the MinefieldDebug icon in the same directory.

And here it is running:

Minefield

SSH to a Debian Etch Virtual Machine in VirtualBox

Posted in Debian, Virtulization by thejimmyg on the August 20th, 2008

VirtualBox uses NAT networking so that the running machine appears to the outside world to be the same as the host. This isn’t a lot of use if you want to SSH from the host to the virtual machine. Luckily there is a work around using port forwarding described at http://mydebian.blogdns.org/?p=148

The essence of the technique is to type these commands to forward port 2222 to port 22 replacing <guestname> with the VirtualBox name for the guest, in my case "Etch 2":

$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP

You’ll need to power off and power on the virtual machine (not just restart it) then you will be able to SSH in with this:

$ ssh -p 2222 root@localhost

You can do the same with other ports such as 80:

$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/apache/HostPort" 8080
$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/apache/GuestPort" 80
$ VBoxManage setextradata <guestname> "VBoxInternal/Devices/pcnet/0/LUN#0/Config/apache/Protocol" TCP

You Apache server will be available at http://localhost:8080

It appears you can choose whatever name you like eg ssh, apache etc but if you use the same key twice the second settings will over-write the first.

This isn’t completely ideal so if anyone knows a good way of setting up bridged networking for VirtualBox on Mac OS X please let me know.

Twitter

Posted in Web by thejimmyg on the August 20th, 2008

OK, so I’ve finally joined the twitter generation and so far am rather liking it. Benefits are:

  • It makes you think about what you are doing
  • You make opinions about things to twitter about
  • You feel connected to those around you
  • You have a record of what you were doing during a day

The first day I joined I made a tweet about Richard Dawkins and shortly after recieved a message saying almightygod is now following you on Twitter!. Made me chuckle:

Hi, thejimmyg.

almightygod (almightygod) is now following your updates on Twitter.

Check out almightygod's profile here:

  http://twitter.com/almightygod

You may follow almightygod as well by clicking on the "follow" button.

Best,
Twitter

--
Turn off these emails at: http://twitter.com/account/notifications

Features I still feel I’m missing though:

  • The ability to tag tweets so people could just follow say my personal or techie tweets
  • The ability for a tweet to become a blog post if it becomes too long
  • Correct spelling errors in early tweets (perhaps with an icon saying the tweet had been changed)

Btw, twhirl is the best twitter client I’ve found so far (but it needs Air).

GNU Screen with no Wuff! on Mac OS X

Posted in Debian, Mac OS X by thejimmyg on the August 11th, 2008

I’ve switched back from Ubuntu to Mac OS X again. The lack of hardware support was just too frustrating (eg: touchpad without right click, having to reboot to use the CD-ROM, dual screen not working properly, keyboard maps incorrect, no microphone etc) so I instead decided to try to fix some of the problems which made me unhappy with Mac OS X in the first place. The main one is the inability to use GNU Screen with the Terminal because it gives Wuff! messages.

This is actually easy to fix as described here:

http://www.dansanderson.com/blog/2004/07/newbie-linuxmac.html

Full background information is described here:

http://www.ibb.net/~anne/keyboard.html

The tip is duplicated below:

Terminal uses "xterm-color" as its TERM, which, in my Debian Etch installation, had undesired (but common) values for kbs (backspace) and kdch1 (delete). I was able to fix it with:

  1. Log in to the affected Linux box.
  2. echo $TERM to confirm the terminfo setting. (Mac OS X Terminal should be using xterm-color.)
  3. infocmp >xterm-color
  4. Edit xterm-color, find kbs and make sure it reads kbs=\177, and find kdch1 and make sure it reads kdch1=\E[3~. Save.
  5. mkdir -p ~/.terminfo/x
  6. Temporarily set the TERMINFO environment variable to ~/.terminfo. In bash/zsh: export TERMINFO=~/.terminfo
  7. Test-compile the new terminfo file: tic xterm-color If successful, ~/.terminfo/x/xterm-color will be the compiled terminfo file.
  8. unset TERMINFO
  9. If desired, back up the original compiled terminfo file. Look for it as /etc/terminfo/x/xterm-color or /usr/lib/terminfo/x/xterm-color.
  10. Run tic xterm-color again as root (using su or sudo) to compile and install the new version in the official location.

I’ve tested this and it seems to work well.

Beginner’s Guide to SSH Keys with SSH2

Posted in Hosting, Sysadmin by thejimmyg on the July 26th, 2008

Ordinarily when I’m working with remote servers I just issue a command like this:

ssh james@example.com uptime
 14:49:47 up 21 days,  2:52,  0 users,  load average: 0.00, 0.14, 0.41

This runs the remote command uptime on the remote server as user james. You can also get a command prompt with:

ssh james@example.com

For both of these examples you have to remember and type the password for james each time. If you administer a lot of servers this can quickly become tedious. One solution is to use SSH keys.

Let’s say I have a laptop with a user james and at laptop.com and a server with a ian account at server.com. I want to be able to login to server.com as ian whilst logged into my laptop as james. To do this I need a SSH key pair. The private key stays on my laptop, the public key has to be added to a special authorized_keys2 file in the .ssh directory in ian’s home directory on the server.

Create the key pair on the laptop without entering a password. Just press enter twice at that point:

james@laptop:~$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/james/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/james/.ssh/id_dsa.
Your public key has been saved in /home/james/.ssh/id_dsa.pub.
The key fingerprint is:
cd:9a:39:7a:ec:7a:77:8e:a3:09:85:aa:b3:e7:b7:93 james@laptop

This creates two files: /home/james/.ssh/id_dsa and /home/james/.ssh/id_dsa.pub. The id_dsa.pub file is the public key which you should place on any server you want to be able to sign in to without a password. The id_dsa file is your private key which stays on the laptop. You should never distribute or publish the private key because anyone with access to it will be able to login to any servers which have the public key set up. Having said that I’m going to show you what both keys look like because I’m not actually using these specific keys myself:

james@laptop:~$ cat /home/james/.ssh/id_dsa
-----BEGIN DSA PRIVATE KEY-----
MIIBugIBAAKBgQC9psPVFyJeJey3AjtjTa6Ep/E1NA/BhPwmQakI0vafkjO9w/HK
Z0VuHd8svp0fn/KcRCcgQhd0mFFRxU8JLf57GoVcPvRmHanFs0t6AqZmb6wpy3Zv
fmYFtnZThCgrUSbNFf6DvvRh3Cp+7OsUKP+MHJE0NI0XvG3m/VQbeeuSbwIVAJeC
6m1aXZ4EL/xaM25yk3C8qpPZAoGACCPPnaevWG7gWI7fG/aEGW6uwAAjKIWrO4Fc
vTzt1fXZcXXDsiHSSQ7sbbpyeStNMjW2tTLS7SXcGld8JbGiTt2aLh6X8n9aMa2f
lHRZjYGLylIFSHuj+L7y4qAYXt740l/ADYwjegeTzcD5QtR2GvgEmlCIkywAdZ9o
DcCc0i0CgYAlDA66+66CEKi/mn6R1cc3UeZ0fADGZeqWgZjamzRlXCmPleA1UFZB
lR40YUDUFfZ4a+p1JJpW9epNXmcVnbZsbfw+aVhR4EAgw4DN89hvilgD9Qjr8UN2
Ocxz5D5dIQrV+oK79CEobLs/uL2A/lofTZndzhSSYUdxy8wHBaNtawIUcnmU5Uzm
JCiJBTwWgUqBTRLzzs0=
-----END DSA PRIVATE KEY-----
james@laptop:~$ cat /home/james/.ssh/id_dsa.pub
ssh-dss AAAAB3NzaC1kc3MAAACBAL2mw9UXIl4l7LcCO2NNroSn8TU0D8GE/CZBqQjS9p+SM73D8cpnRW4d3yy+nR+f8pxEJyBCF3SYUVHFTwkt/nsahVw+9GYdqcWzS3oCpmZvrCnLdm9+ZgW2dlOEKCtRJs0V/oO+9GHcKn7s6xQo/4wckTQ0jRe8beb9VBt565JvAAAAFQCXguptWl2eBC/8WjNucpNwvKqT2QAAAIAII8+dp69YbuBYjt8b9oQZbq7AACMohas7gVy9PO3V9dlxdcOyIdJJDuxtunJ5K00yNba1MtLtJdwaV3wlsaJO3ZouHpfyf1oxrZ+UdFmNgYvKUgVIe6P4vvLioBhe3vjSX8ANjCN6B5PNwPlC1HYa+ASaUIiTLAB1n2gNwJzSLQAAAIAlDA66+66CEKi/mn6R1cc3UeZ0fADGZeqWgZjamzRlXCmPleA1UFZBlR40YUDUFfZ4a+p1JJpW9epNXmcVnbZsbfw+aVhR4EAgw4DN89hvilgD9Qjr8UN2Ocxz5D5dIQrV+oK79CEobLs/uL2A/lofTZndzhSSYUdxy8wHBaNtaw== james@laptop
james@laptop:~$

Notice that the public key is actually just one long line in three parts, a type, the public key itself and a comment. In this case the comment is james@laptop but you can change this to anything you like which helps you remember which private key the public key is associated with.

Let’s set up the public key on the ian account on server.com:

james@laptop:~$ scp /home/james/.ssh/id_dsa.pub ian@server.com:james_laptop_public_key.pub
Enter password:

You’ll have to enter the password because the key isn’t yet set up. Next we need to add this public key to ian’s .ssh/authorized_keys2 file. This file can have multiple entries because it is possible that ian wants to allow access to many different user accounts on different machines. This is why we didn’t copy the id_dsa.pub directly to /home/ian/.ssh/authorized_keys2 which we could have done if we were sure the file didn’t exist or was empty.

SSH into the ian@server.com account and enter your password:

james@laptop:~$ ssh ian@server.com

If the ian account has never used SSH before you should run this command and make a connection with your regular password to create the .ssh directory on the remote server:

ian@server:~$ ssh laptop.com

This will create a .ssh dir in ian’s home directory with the proper permissions. Now we need to append the james_laptop_public_key.pub file to the authorized_keys2:

ian@server:~$ cat james_laptop_public_key.pub >> ~/.ssh/authorized_keys2
ian@server:~$ exit
logout
Connection to server.com closed.
james@laptop:~$

At this point the public key is set up on the remote server and the private key is set up on the laptop. You can now ssh or scp from the james account on the laptop to the ian account on the server without typing a password:

james@laptop:~$ ssh ian@server.com
ian@server:~$

Caution!

It is very bad practice to actually have passwordless keys set up in this way because anyone who hacks your laptop will also have access to your server without needing the password. Read on to learn about ssh-agent.

A Word About Permissions

It is very important that no-one other than you can read your private key id_dsa file because anyone with that file will also be able to access your server. For that reason you should always make sure the file has permissions of 600:

james@laptop:~$ chmod 600 /home/james/.ssh/id_dsa

Multiple Servers

You can repeat the process described above to add your public key to the authorized_keys2 files on lots of different user accounts on lots of different servers. You only need the one private key though. This will allow you to log in to all those different accounts using the same private key and without needing a password.

Be aware though, if using a passwordless key for access to one server is a bad idea for security, using the same key on multiple servers is even worse because a hacker then has access to even more servers.

Multiple Private Keys

Although the SSH private key we’ve been using so far is designed to allow you to sign in to lots of different servers you can also create a different private key for each one if you prefer. To create a new private key you’ll need to specify a filename, otherwise you will overwrite the id_dsa and id_dsa.pub files you’ve just created:

james@laptop:~$ ssh-keygen -t dsa -f identity

You’ll need to add the contents of identity.pub to the end of the authorized_keys2 file on the remote server as you did before. Now you need to specify the key to use when you use an ssh command:

james@laptop:~$ ssh -i identity ian@server.com

Passworded Keys

It is much safer to use a key with a password. To generate a key with a password run this command but this time enter a password when prompted:

james@laptop:~$ ssh-keygen -t dsa

This will over-write the existing passwordless private key and public key. You’ll need to add the new public key line to the authorized_keys2 files of all the accounts on all the servers you wish to have access to. (You should probably remove the old lines too).

Now when you try to ssh in to the server you will be prompted for a password. This is the password of the key, not of the account on the server:

james@laptop:~$ ssh ian@sever.com
Enter passphrase for key '/home/james/.ssh/id_dsa':

If you are using a desktop environment such as GNOME (or are using Mac OS X for example) you might get a popup which asks you to enter the password instead of entering the key password on the command line as shown in the example above. Once you have entered it, GNOME will remember the password for the current session so you don’t have to enter it again.

You can achieve a similar result if you are just working on the command line by using a program called ssh-agent:

james@laptop:~$ ssh-agent bash

This will put you in a bash shell which is spawned by ssh-agent. Next you’ll need to add your key:

james@laptop:~$ ssh-add

This will try and add the default key id_dsa to the key manager. To add a key with a different name, enter:

james@laptop:~$ ssh-add /location/of/key

After this, the ssh-add program will ask you for your password. After you entered your password the key is loaded in the key manager ssh-agent.

You can now login to the remote servers without entering your password again. As you can see this set up is much safer than just using a passwordless key to start with and it isn’t significantly more hassle.

You can have a look at your currently loaded keys with:

james@laptop:~$ ssh-add -l
1024 45:1b:f8:87:15:31:2f:4a:38:c5:f3:b0:e5:de:62:8f  (DSA)

Single Purpose Keys

Now having keys with passwords and working with ssh-agent is fine for every day use but useless for automated tasks such as cron jobs because the cron script would need to have the password listed in plain text.

Luckily there is a way around this. One feature of sshd is that you can specify options in the authorized_keys2 file.

Create a new private key without a password this time:

james@laptop:~$ ssh-keygen -t dsa -f .ssh/single_use_identity

Edit the public key .ssh/single_use_identity.pub so that the comment says something like james@laptop_single_use_identity otherwise once it is copied to the server it might be difficult to identify.

Copy the public key to the remote server and add it to the end of the auhorized_keys2 file as before.

Once the sign in works correctly find the public key line representing the private key you are using in the remote server’s authorized_keys2 file again and add the following right at the front of the line with a single space between it and the ssh-dss line that is currently at the front:

command="uptime",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

The above text needs to be all on one line with no carriage returns in it. When the user signs in they will only be able to run the command listed in the command option. Let’s try it:

james@laptop:~$ ssh ian@server.com -i single_use_identity
 16:30:39 up 21 days,  4:33,  0 users,  load average: 0.04, 0.01, 0.00
Connection to server.com closed.

As you can see, this time the uptime command was run and then the connection was closed. The key can now only be used to execute the uptime command.

Disabling SSH Agent

If you ignored the advice to create a new private key for the single use identity and instead just modified the old one you might now find you can’t sign in to the server because it keeps printing the uptime and closing the connection. To sign into the server now you will need to disable ssh-agent. You can do this with this command:

james@laptop:~$ ssh-add -d
Identity removed: /home/james/.ssh/id_dsa (/home/james/.ssh/id_dsa.pub)

You can now sign in as usual:

james@laptop:~$ ssh ian@server.com
Agent admitted failure to sign using the key.
ian@server.com's password:

If you are using GNOME or Leopard you might need to move your private and public keys temporarly and logout of your desktop session and log back in again to force the agent to forget your identity:

james@laptop:~$ mv .ssh/id_dsa .ssh/id_dsa.bak
james@laptop:~$ mv .ssh/id_dsa.pub .ssh/id_dsa.pub.bak
james@laptop:~$ ssh ian@server.com
ian@serve.com's password:

Then you can change the remote authorized_keys2 file back, logout, and move the public and private key back:

ian@server:~# exit
logout
Connection to server.com closed.
james@laptop:~$ mv .ssh/id_dsa.bak .ssh/id_dsa
james@laptop:~$ mv .ssh/id_dsa.pub.bak .ssh/id_dsa.pub

Handling Backups

If you want to use a single purpose key to handle backups with rsync you should add a cron job on your laptop to run the rsync command:

james@laptop:~$ crontab -e

Add this line:

0 10 * * * rsync -aHxvz --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x -i /home/james/.ssh/single_use_identity" ian@server.com:/home/ian /home/james/ian_backup/

This will then invoke rsync on the laptop to back up information from the server at 10am every day using the SSH private key for james on the laptop.

At the moment though the private key is associated with a public key on the server which is only capable of running the uptime command. In order for rsync to work the server needs to be allowed to run the rsync –server command to start the rsync. We need to change the line in the authorized_keys2 file on the server to look like this:

command="/home/ian/secure-rsync",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

(Note you might need to log out of your session and back in to get your agent to forget your identity so you can sign in with a password again as mentioned earlier)

Next create the /home/ian/secure-rsync script on the remote server and make it look like this:

#!/bin/sh

case "$SSH_ORIGINAL_COMMAND" in
    *\&#038;* | *\;* | *\|*)
        echo "Access denied"
        ;;
    rsync\ --server*)
        $SSH_ORIGINAL_COMMAND
        ;;
    *)
        echo "Access denied"
        ;;
esac

Note: This script doesn’t display correctly in Wordpress. Copy it from here instead: http://www.barryodonovan.com/misc/publications/lg/104/

This script ensures that only the rsync server is allowed to be run. Make it executable:

ian@server:~$ chmod a+x secure-rsync

Now test the rsync command from the laptop:

james@laptop:~$ rsync -aHxvz --delete --progress --numeric-ids -e "ssh -c arcfour -o Compression=no -x -i /home/james/.ssh/single_use_identity" ian@server.com:/home/ian /home/james/ian_backup/

If rsync isn’t installed you can install it with:

sudo apt-get install rsync

It will need to be installed on both the laptop and the server.

If it works without a password you can leave your cron job to run and it will provide daily backups (providing your laptop is on and connected to the internet at the time of the backup!)

One thing to note is that if you have multiple keys installed on the server and laptop it is possible that rsync is happening over the default connection rather than the passwordless single use one. You can check it isn’t by temporarily moving the id_dsa and id_dsa.pub files as mentioned earlier whilst testing the rsync command.

That’s it, hopefully everything you need to know to be productive with SSH keys. I’m by no means on expert on all this so feel free to point out any mistakes I’ve made in the comments.

Troubleshooting

Any time things don’t work just add the -v switch which means "verbose". SSH will display exaclty what is going on which makes debugging much easier. A common problem is that one of the computers is using SSH 1 rather than SSH 2. This guide was for SSH 2 so if you are trying to use it with an SSH 1 computer I’m afraid it won’t work.

Revolution is coming… maybe!

Posted in Wild Speculation by thejimmyg on the June 21st, 2008

Yesterday I had a good discussion with people from the hub about the way I feel that something is going to change radically with the internet in the next 2-5 years. I think the source for this feeling is my belief that the internet is there, that in technical terms almost everything I ever wanted to create for the internet now exists.

This can only mean two things - technology will continue to become modularized enabling more and better-fitting solutions to problems to be achieved more cheaply or the way humans interact with technology will change so that the symbiosis of human and computer will become more efficient, not by technology better fitting the needs of humans but with humans better learning how to adapt to technology. In reality both are likely to happen but I think the major change will be in the realisation for the need for humans to adapt to the new world of technology which is a field which barely exists at the moment. IT will shift to be a small part about creating solutions with technology and a large part in changing human behaviour to better use the technological tools available. I want to be part of this revolution.

Python Odness

Posted in Python by thejimmyg on the June 16th, 2008

You probably already know that variables in Python are effectively passed by reference:

a = ['this', 'is', 'a', 'list']

def change(arg):
    arg.append('with')
    arg.append('new')
    arg.append('variables')
    print arg is a

change(a)
print ' '.join(a)

This program would result in the variable a being changed even though the appending of the extra words happens in the scope of the change() function. As you can see from the first print statement this is because the arg variable really is the same thing as the a variable, not a copy of it as would happen in C++.

The output from the program is:

True
this is a list with new variables

This behaviour is fairly understandable but I want to show you something you might not expect but which you really should be aware of. Consider this code, what would you expect to be printed the first time this function was called?

def test(a=[], b={}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

If you guessed [], {} then you would be correct. Try calling it again. What do you expect the second time?

The answer is might surprise you. It turns out to be [a], {1: ‘b’} because even though you are calling test() with no arguments, the previous call changed the values of the empty [] and {} to different values. The second time the function was called the definition was effectively:

def test(a=['a'], b={1: 'b'}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

As you keep calling test() the list and dictionary keep getting bigger. Now you might be suprised by this and wonder what would happen if you pass ordinary variables into the function. Well it appends a to the first one and adds a new key to the second one but this time because the variables aren’t the actual ones which make up the function definition the function definition isn’t changed. If you call test() again the variables from the function definition will go on incrementing. Here’s the full source:

def test(a=[], b={}):
    print a, b
    a.append('a')
    b[len(a)] = 'b'

test()
test()
test()
print
c = []
d = {}
test(c, d)
print 'Output: ',  c, d
test()

And the full output:

[] {}
['a'] {1: 'b'}
['a', 'a'] {1: 'b', 2: 'b'}

[] {}
Output:  ['a'] {1: 'b'}
['a', 'a', 'a'] {1: 'b', 2: 'b', 3: 'b'}

So, how do you avoid this problem?

  1. Never use mutable variables as defaults to functions.

Instead you can do something like this:

def test(a=None, b=None):
    if a is None:
        a = []
    if b is None:
        b = {}

This is very safe and always works.

  1. Make copies of the variables:

    def test(a=[], b={}):
        a = a[:]
        b = b.copy()
    

By creating local variables with the same names as the defaults you can no longer (easily) accidentally change the variables defined as defaults. The drawback is that you make copies of any arguments which are passed to the function which then uses up more memory.

  1. Be very careful not to actually change the values of the defaults.

This isn’t a particularly good idea because someone else looking at your code might not be aware of what you are doing and might change the value without realising.

I’d imagine quite a few libraries which allow mutable variables as default values might have this particular problem. In my own code I try to only use None as a default option. Do other people have any handy workarounds?

Next Page »