IronPython, Jython, Scala and Python: A Fairly Meaningless Server Performance Comparison

IronPython

As I mentioned a second ago, IronPython 2.0 is a full implementation of Python 2.5 targetting the .NET platform so it runs on Mono but never seems to be installed or set up correctly. The IronPython talk at PyCon has some pointers to help you though.

By far the easiest way of setting it up is to use the Novell 32bit installer (even if you are on a 64bit platform). Download it and get it started like this:

$ wget http://ftp.novell.com/pub/mono/archive/1.9.1/linux-installer/2/mono-1.9.1_2-installer.bin
$ chmod 755 mono-1.9.1_2-installer.bin
$ ./mono-1.9.1_2-installer.bin

You don't need to have root priveledges to run it as long as you install it somewhere you have write permissions to such as your home directory.

A GTK+ based wizard begins:

After accepting the license agreement you are asked to specify a directory:

You are asked whether to add mono to your path. I chose no so that if I ever install a different version of Mono, they won't interfere with each other. I changed the value to No.

This means every time I want to use Mono I have to run this command in a shell:

$ source /home/james/mono-1.9.1/bin/setup.sh

You are then asked to confirm your choices:

and then Mono is installed:

After a couple of minutes the wizard is complete but unfortunately there is an error:

It seems you also need to install some extra libraries but the installer doesn't tell you about this to start with. Glitz is easy to install:

$ sudo apt-get install libgail-common libglitz1

On Ubunut Hardy 8.04 (the version of Linux I used) I have libgaiutil18 but Mono requires version 17. There is a post at http://ubuntuforums.org/showthread.php?s=53a5632ac203fbea1b019175db73aa6f&t=673835&page=2 describing how to install the version for Etch but I didn't fancy that. I'm happy to continue without being able to use GTK for the timebeing, instead I just hacked a symblic link togther to hope for the best:

$ sudo ln -s /usr/lib/libgailutil.so.18 /usr/lib/libgailutil.so.17

Now let's try IronPython 2.0. Set up the current console:

$ source /home/james/mono-1.9.1/bin/setup.sh

Then run the ipy2 command. The first thing you'll notice is that it takes a few seconds to load.

$ ipy2
IronPython console: IronPython 2.0A5 (2.0.11011.00) on .NET 2.0.50727.42
Copyright (c) Microsoft Corporation. All rights reserved.
>>>

You'll also quickly notice that not all of the standard library is present, even though most of it is:

>>> import doctest
Traceback (most recent call last):
ImportError: No module named doctest

>>> import cgitb
Traceback (most recent call last):
ImportError: No module named cgitb

You'll need to add the Python 2.5 libraries to your path:

>>> import sys
>>> sys.path.append('/usr/lib/python2.5')
>>> sys.path.append('/usr/lib/python2.5/site-packages')
>>> import doctest
>>> import cgitb

Now that everything is present let's do a simple server benchmark. Here is a simple WSGI application and server named test3.py:

from wsgiref.util import setup_testing_defaults
from wsgiref.simple_server import make_server

def simple_app(environ, start_response):
    status = '200 OK'
    headers = [('Content-type', 'text/html')]
    start_response(status, headers)
    return ['Hello World!']

httpd = make_server('', 8001, simple_app)
print "Serving on port 8001..."
httpd.serve_forever()

Running the server with Python 2.5:

$ python test3.py

and benchmarking 1000 requests with Apache Bench (albeit from the same machine):

$ ab -n 1000 -c 10 http://localhost:8001/

gives a best score from 3 runs of 4.4ms/request

Now with IronPython:

$ ipy2 test3.py

The same test with a best of 3 gives 16.2ms/request.

I was rather suprised by this, IronPython 2 (alpha) (32bit) was 4 times slower than CPython (64bit) on the same machine even though the test didn't rely very heavily on Python and was primarily a network test.

It should be noted that even though the server performance was 4 times slower, generally using ipy2 felt a lot, lot longer than Python. Importing modules and starting the interpreter took a comparative age.

Jython

This got me wondering what Jython 2.5 Beta 3 would be like with the same test so here goes.

First installing Java was much simpler:

$ sudo apt-get install openjdk-6-jdk

Installing Jython was easy too. Get it and start the installer like this:

$ wget http://kent.dl.sourceforge.net/sourceforge/jython/jython_installer-2.5b3.jar
$ java -jar jython_installer-2.5b3.jar

The graphical installer seemed to have far more steps than it really needed as you can see below:

You need to accept the license:

Choose an installation type (I chose Standard):

Specify the install directory, again I chose a directory in my home directory:

I'm using the OpenJDK:

I wasn't sure why these options appeared to be editable but since they were auto-generated I continued:

Then the installation starts:

The README is shown:

and you are done:

After running the installer you have a jython command on your path. The first time you load it a few jar files are processed:

$ /home/james/jython2.5b3/bin/jython
*sys-package-mgr*: processing new jar, '/home/james/jython2.5b3/jython.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/resources.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/rt.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/jsse.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/jce.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/charsets.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/rhino.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/ext/sunjce_provider.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/ext/dnsns.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/ext/sunpkcs11.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/ext/localedata.jar'
*sys-package-mgr*: processing new jar, '/usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar'
Jython 2.5b3 (Release_2_5beta3:6092, Mar 10 2009, 15:34:57)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_0
Type "help", "copyright", "credits" or "license" for more information.
>>>

This time the whole standard library appears to be present including doctest and cgitb (Woohoo!). As with IronPython, importing the modules for the first time takes a while as they are processed.

The second time you run jython it loads much faster without processing the jars it found the first time:

$ /home/james/jython2.5b3/bin/jython
Jython 2.5b3 (Release_2_5beta3:6092, Mar 10 2009, 15:34:57)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_0
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now for the benchmark:

$ /home/james/jython2.5b3/bin/jython test3.py

The result from a best of 3 was 19ms/request which was in the same ballbark as the mono version.

Scala

OK, so since I seem to be experimenting with lots of different frameworks, how about one more.

You can download and install scala like this:

$ wget http://www.scala-lang.org/downloads/distrib/files/scala-2.7.3.final-installer.jar
$ java -jar scala-2.7.3.final-installer.jar

If you don't run the installer with root priveledges you need to change the path to a directory you have access to. I chose /home/james/scala-2.7.3.

Here are the installer screenshots:

Choose a language:

See the welcome message:

Not sure we need a welcome message and a helpful screen telling us how to quit the installer but there you go:

Agree to the license:

Again, I put Scala in my home directory instead of the default shown below because I didn't run the installer as root:

Choosing requirements is easy!

The scala install step takes literally ages (20 mins on my machine) and I'm not sure why.

Not sure what this screen was for:

Finished!

Interestingly there is some great documentation installed along with the language. Have a look at the PDFs if you install it yourself:

$ ls /home/james/scala-2.7.3/scala-2.7.3.final/doc/scala-documentation/
ScalaByExample.pdf  ScalaReference.pdf  ScalaTutorial.pdf

Since I can't run the same test I had to write a server in Scala. First I found tiscaf, a small Scala web server written without Java dependencies. Download it (copy here) >`_):

$ wget http://gaydenko.com/scala/tiscaf/files/tiscaf-0.2.4.zip

Then build it:

$ unzip tiscaf-0.2.4.zip
$ cd tiscaf-0.2.4
$ mkdir -p classes
$ /home/james/scala-2.7.3/scala-2.7.3.final/bin/scalac -d classes src/zgs/httpd/*.scala \
                  src/zgs/httpd/let/*.scala \
                  src/zgs/utl/*.scala \
                  src/zgs/sync/*.scala \

Then run it:

$ /home/james/scala-2.7.3/scala-2.7.3.final/bin/scala -classpath classes zgs.httpd.HomeServerStart

The best of 3 time this time is 10ms. Faster than both the Hello World examples even though this version is displaying an entire directory listing page and doing GZip encoding, very impressive!

Java Servlets in Scala

Now on a roll I thought I'd try a Scala example which uses the Java servlet API. I followed the tutorial at http://burak.emir.googlepages.com/scalaservlet.html and downloaded the sample code from http://burak.emir.googlepages.com/scalaServletHowTo-0.9.tgz (copy here). This was the most difficult software to get working out of all the three because I've never used Tomcat before and whilst I could find plenty of detailed information, I couldn't find any good introductions relevant to Ubuntu Hardy. Actually it turns out to be very easy:

$ sudo apt-get install ant tomcat5.5 tomcat5.5-webapps tomcat5.5-admin

You then just place the .war files in /usr/share/tomcat-5.5/webapps and restart tomcat:

$ sudo /etc/init.d/tomcat-5.5 restart

I benchmarked the Hello World example (which also calculates and prints the current time) and was astonished to have to do 10000 requests to keep the benchmark running long enough to be consistent. The best of 3 attempts at 10,000 requests was 0.9ms, some 20x faster than my simple Python server running via Jython. To put this in perspective, serving Hello World directly from memcached compiled directly into Nginx takes 0.8ms/request so this really is blazing.

Java Servlets in Jython

I would have liked to have benchmarked the same Python app running under mod_jy (http://opensource.xhaus.com/wiki/modjy/) but unfortunately I couldn't get it working with Tomcat 5.5. The example application kept getting the following error and never having been near Tomcat before, I didn't really know where to start. This didn't really matter though because all my experiments were done anyway:

HTTP Status 500 -

type Exception report

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

javax.servlet.ServletException: Servlet.init() for servlet modjy threw exception
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
    org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
    org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
    org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
    org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    java.lang.Thread.run(Thread.java:636)

root cause

java.security.AccessControlException: access denied (java.util.PropertyPermission * read,write)
    java.security.AccessControlContext.checkPermission(AccessControlContext.java:342)
    java.security.AccessController.checkPermission(AccessController.java:553)
    java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    java.lang.SecurityManager.checkPropertiesAccess(SecurityManager.java:1269)
    java.lang.System.getProperties(System.java:599)
    com.xhaus.modjy.ModjyJServlet.init(ModjyJServlet.java:84)
    javax.servlet.GenericServlet.init(GenericServlet.java:211)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:616)
    org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:244)
    java.security.AccessController.doPrivileged(Native Method)
    javax.security.auth.Subject.doAsPrivileged(Subject.java:537)
    org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:276)
    org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
    org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:115)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
    org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
    org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
    org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
    org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
    org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    java.lang.Thread.run(Thread.java:636)

note The full stack trace of the root cause is available in the Apache Tomcat/5.5 logs.
Apache Tomcat/5.5

If I have time I'll try it again with Tomcat 6 which is what the mod_jy instructions suggest you use.

Summary

I found the experimenting really interesting. IronPython and Jython, whilst very impressive projects didn't have the performance I expected, particularly after having heard rumours that IronPython sometimes out-performed Python. Those times must be fairly few and far between. I was also astonished by the speed of Scala and Java and was very interested in the Java eco-system which, once you get through all the marketting, has some very sensible products.

Out of IronPython, Jython and Scala the one I am now most interested in is Jython which would allow me access to all the interesting Java software that exists for "enterprise" applications whilst giving me the flexibility of Python and access to all my existing code, albiet for a performance penalty which as I mentioned at the start of this post, doesn't hugely really affect most well-written real world applications anyway.

I know Philip Jenvey has been doing some great work on Jython and there are instructions on how to run Pylons under Jython as part of the Pylons 0.9.7 documentaion. Well done to the Jython developers.

Hope you found that as interesting as I did.