Oracle DB XML (was Sleepycat)
At Geek Night 5 I bumped into an old friend of mine John Snelson who I haven’t seen for ages. He’s now working for Oracle on DB XML, a project he’s been involved with for a long time since the Sleepycat days. I promised to have a play with the Python bindings so for anyone following in my footsteps, here’s how you do it, it’s a lot easier than it appears from the docs and forum posts…
Download the latest version from the link below by clicking the "Free Download" button in the top right: http://www.oracle.com/technology/products/berkeley-db/xml/index.html
The dbxml-2.3.10.tar.gz file is about 30.9Mb. You’ll also need a standard GNU toolchain, I happen to have the following installed although you may not need them all:
sudo apt-get install build-essential make libc6-dev gcc g++ patch zlib1g-dev libncurses5-dev libssl-dev flex bison sharutils subversion python gettext autoconf2.13 pkg-config
Next extract the source and build it:
tar zxfv dbxml-2.3.10.tar.gz cd dbxml-2.3.10 sh buildall.sh
This takes quite a while (about an hour on my Thinkpad R50e laptop) but compiles without any problems producing an install directory with bin, docs, lib and include directories containing everything you need.
Now for the Python bindings which are equally as easy because the main distribution also comes with a compatible source tree for the Python bsddb library. Compile and install it all like this:
cd dbxml/src/python python setup.py build python setup.py install cd bsddb3-4.5.0 python setup.dbxml.py build python setup.dbxml.py install
If you were to have any problems the README files contain plenty of tips. To test the Python bindings you simply need to add the library directory to your LD_LIBRARY_PATH since the original buildall.sh script is polite enough not to actually install the files to the system locations (although there is an option to have it do this). In my case this meant:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/james/Desktop/dbxml-2.3.10/install/lib
Now you can test the example:
cd ../../../../examples/python python examples.py test
Everything should work perfectly.
Now that I’ve got the software compiled I need to start using it ;-). Here were some links I found were a useful introduction to native XML databases but the links on the Oracle page linked to earlier are no doubt worth a read too.
http://www-128.ibm.com/developerworks/xml/library/x-mxd4.html http://www.rpbourret.com/xml/XMLAndDatabases.htm http://www.w3.org/TR/xquery-use-cases/ http://www.regdeveloper.co.uk/2007/07/18/berkeley_db_xml/
Flex Builder Linux Alpha 2 on Debian Etch
I went to the Flex User Group http://ria.meetup.com/7/calendar/6862191/?a=wm1_rsvp yesterday because AIR is one of the technologies I’m most excited about at the moment. I was slightly surprised by the demographic, I think I was expecting a crowd of open source developers, much like you’d find at a hack day or geek event but somehow they didn’t seem as excited about the technologies Adobe are beginning to open up as I did. I wonder whether this is because most of the people who use Adobe products are less technical than the average person you might see at other events and therefore less interested in the new coding opportunities Flex and Air bring.
I got the slight impression that some people were just there for the free beer provided by Adobe (although maybe that’s just because I was sitting near the back by the bar) and others (including myself probably) had a feeling of "why should I do Adobe the favour of switching to flex". I think Adobe’s answer might be along the lines of "we’re doing enoughm just look at the numbers and if you don’t want to take advantage of it that’s up to you" so some of the questions seemed slightly hostile, but maybe I’m reading too much into it and applying my views to my impression of everyone else’s!
I was also slightly surprised by some of the attitudes of the speakers who made frequent references to a designer/developer divide saying things like "you’ll only be impressed with that if you are a developer". Maybe the audience for Adobe products does feature this divide but it surprised me nonetheless and I was surprised it was something the speakers wanted to encourage.
Anyway, despite the above observations I did find the event very interesting and am pleased Adobe are moving towards being more open. One question I asked in the Q&A section was "What are the timescales for AIR on Linux". The answer I got back was "soon" and it appears James Ward is as keen to get AIR on Linux as I am as it is his primary desktop too. Still, I pointed out that if Adobe wants to attract open source developers a Linux port would be pretty essential. Let’s hope it happens "soon" as James promised because developing AIR on FlexBuilder on VMWare is very painful! James also suggested I try the Flex Builder Alpha for Linux so here goes…
First install Java. To do this you’ll need to edit your apt sources list to use the non-free repository:
sudo vim /etc/apt/sources.list
Mine looks like this:
deb http://ftp.uk.debian.org/debian/ etch main non-free deb-src http://ftp.uk.debian.org/debian/ etch main non-free deb http://security.debian.org/ etch/updates main contrib deb-src http://security.debian.org/ etch/updates main contrib
Next install Java 5:
sudo apt-get update sudo apt-get install sun-java5-jdk
(You can probably get away with the JRE if you prefer, in which case install sun-java5-jre instead).
You’ll have to agree to the license:
Package configuration
┌───────────────────────┤ Configuring sun-java5-bin ├───────────────────────┐
│ │
│ Operating System Distributor License for Java v1.1 (DLJ) ↑
│ ▮
│ Operating System Distributor License for Java version 1.1 (DLJ) ▒
│ ▒
│ SUN MICROSYSTEMS, INC. ("SUN") IS WILLING TO LICENSE THE JAVA PLATFORM ▒
│ STANDARD EDITION DEVELOPER KIT ("JDK" - THE "SOFTWARE") TO YOU ONLY UPON ▒
│ THE CONDITION THAT YOU ACCEPT ALL OF THE TERMS CONTAINED IN THIS LICENSE ▒
│ AGREEMENT (THE "AGREEMENT"). PLEASE READ THE AGREEMENT CAREFULLY. BY ▒
│ INSTALLING, USING, OR DISTRIBUTING THIS SOFTWARE, YOU ACCEPT ALL OF THE ▒
│ TERMS OF THE AGREEMENT. ▒
│ ▒
│ 1. DEFINITIONS. "Software" means the code identified above in binary ▒
│ form, any other machine readable materials including, but not ▒
│ limited to, libraries, source files, header files, and data files), ↓
│
│ <Ok>
│ │
└───────────────────────────────────────────────────────────────────────────┘
Test Java is working by typing java -version at the command line. You should see something like this:
$ java -version java version "1.5.0_10" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03) Java HotSpot(TM) Client VM (build 1.5.0_10-b03, mixed mode, sharing)
Next you’ll need Eclipse 3.3 or higher. Go to http://www.eclipse.org and download the Linux version (I chose Eclipse IDE for Java EE Developers). Extract the files and launch the program:
tar zxfv eclipse-jee-europa-fall2-linux-gtk.tar.gz cd eclipse ./eclipse
You’ll be asked where you want to create a workspace. I stuck with the default of /home/james/workspace. Eventually Eclipse loads. Once you are happy it is working you should exit it and you can finally you can get around to actually installing Flex Builder.
Download the alpha 2 from the Adobe website: http://labs.adobe.com/downloads/flexbuilder_linux.html then run these commands:
chmod 755 flexbuilder_linux_install_a2_121807.bin ./flexbuilder_linux_install_a2_121807.bin
You’ll see the following output:
Preparing to install... Extracting the installation resources from the installer archive... Configuring the installer for this system's environment... Launching installer...
Then the installer launches and after a few screens you are asked to choose your eclipse folder. This will be the location of the eclipse folder you just extracted. You’ll also be asked to install Flash Player 9 and you should choose to do so, even if you already have it because this version has debugging built-in.
The installation finishes with some errors which aren’t displayed. Instead you are asked to view the installation log but not told where this is and I couldn’t find it! The installer doesn’t tell you how to launch Flex Builder either, but it turns out you just load Eclipse again and Choose Help->Flex Builder whereupon you are asked to enter a license key and told you only have 65 days remaining. After all that effort it would have been nice to know this was only a 65 day trial if it really is. After clicking OK nothing happened (didn’t seem to be able to connect to flexstart.adobe.com) so I exited and tried again.
To get started you need to create a new project. Chose Project… from the File menu and choose a Flex Builder project. There is no support for AIR or BlazeDS so you can only create Flex applications targeting Flash Player 9 but this seems like a good start.
Here’s a screenshot:
http://jimmyg.org/wp-content/uploads/2008/01/flex_builder_running.png
So I think we can add Debian Etch to the list of platforms where Flex Builder works. Good work Adobe and keep it up!
P.S. Was that Evan Davis I spotted at the event? P.P.S Submit bugs!
EuroPython Days 3 and 4
I spent most of days 3 and 4 catching up with people I knew and exploring Vilnius itself. I spent some more time discussing the request response integration with Pylons and Django and went to the Death to Zope Instances sprint where I was delighted to see other developers (including the SchoolTool people) also implementing variations on the Zope on Paste theme and keen to agree a standard solution.
The lightning talks were as interesting as ever with my particular interest being taken by PyDoctor (a source code documentation tool) and a really impressive 3D modelling engine. The keynote on MySQL was interesting (My should really be pronounced “me” because it is the name of the daughter of the other co-founder of the project).
EuroPython Day 1
Well, here I am in Vilnius in Lithuania at EuroPython 2007. I’ve already bumped into a load of people I already know and have been pleased to see Pylons used in three of the talks already!
Here are some notes on the talks I’ve been to:
Extending Python with EasyExtend
I was actually planning on going to Object databases for Python but I ended up in the wrong room but was pleased I did. EasyExtend is a Python package which allows you to create new language syntax. This can be used to implement Python 2.5 language features in Python 2.4 or create a new programming language completely. I’m particularly interested in it because it could be used to restrict access to certain Python features so could potentially be used to provide scripting for users using a cut-down and safe version of Python. This might be handy in say a web-based spreadsheet application where a custom language could be implemented for the formulae.
Seamless object persistence with ZODB
I hear a lot about ZODB because Zope is used extensively on other projects at the Institute of Child Health where I work but I’ve never actually used it myself. This talk was a good introduction. Basically you have to ensure you use special persistant classes for lists and the like in order for ZODB to keep track of changes you make to them. As long as all your objects are derived from the special persistent classes all works well. One interesting method you can use on transactions is doomed() which will prevent any further commits but will still allow you to access all the data as it currently stands for use in debugging code.
The Storm Object-Relational Mapper
This talk outlined Storm, a new object relational mapper created by Canonical which hadn’t been widely announced to the community before. Key features are that it works with normal Python classes, is very simple and has more lines of unit tests than it does actual code so is likely to be very stable. It has also been used in production by Canonical for over a year I think this would be my object relational mapper of choice if I were to use an ORM. It has similar functionality to SQLObject or Elixir, is much simpler and more stable than SQLAlchemy and is well supported by Canonical. The speaker also claimed it was very very simple to implement new drivers and had a very nice SQL query layer. Definitely worth a look if you are interested in ORMs.
Case study of a Pylons project
In this session Max Ischenko discussed his experiences of converting a Ukranian developer’s site written in PHP to use Pylons. Overall Max seemed very pleased with the how Pylons performed although he found he had to do quite a lot of the groundwork himself. View the slides if you are interested in his thoughts.
KSS, Ajax development with style
KSS effectively lets designers and HTML writers handle JavaScript functionality in the same way they would use CSS and provides a clean interface to allow developers to provide the functionality those designers can use. This was one of the most interesting projects I saw at EuroPython and I spent a couple of hours discussing it afterwards with Jeroen Vloothius, a Plone developer who works at Pareto.
KSS lets you apply JavaScript to elements via CSS-style stylesheets called KSS files. This means you abstract away all the JavaScript used by your appliaction into simple rules that even a non-technical user can apply to their HTML in the same way they would apply CSS. It also means that you can use any JavaScript library you like to implement the KSS which means if you swap JavaScript libraries later you have a clean interface to implement.
The second important feature if that KSS lets you determine the actions to perform after an AJAX request on the server side via an HTML document returned to the browser rather than needing to do it client side. This is tremendously useful because after all it is the developers who need control over what happens after an AJAX request, not the designers. If you’ve ever come across a problem where someone has used some nice bit of JavaScript to update a status but failed to notice that the status is also displayed on other parts of the page which need updating too then you’ll understand why this is useful.
KSS is already used in Plone and Jeroen applied it to the Pylons QuickWiki tutorial code to demonstrate that it was cross-framework too. While demoing it one of the Python developers in the audience asked how he managed to get it running so fast (it was just running the standard Pylons server though) so it just goes to show that if you can use AJAX effectively it can really impress people and KSS represents a great tool for effective AJAX use.
One feature which would be nice are the ability to update the URL on each click (a la Dojo) to provide an AJAX history in such a way that all the links can use an AJAX replace but that if the user ever hit refresh on the current URL they would see exactly the same page. I’m having a think about how this paradigm could be implemented in Pylons.
Using FormEncode for web data validation
This talk was given by a friend of mine, Graham Stratton and gave a good insight into how to use FormEncode. The key points were that FormEncode itself, despite its name, is simply a tool for converting data from a particular format to Python and back again. In the context of web forms, FormEncode converts form data to Python objects and can then covert Python objects back from Python to web form values again. How the form data is then displayed is up to the developer but FormEncode also comes with a tool called htmlfill which can parse an HTML fragment, extract field names and generate a new HTML fragment with all the values and errors set, ready to redisplay. There is no requirement to use htmlfill, but it is a useful approach for populating fields in a very generic way.
This talk by Philipp von Weitershausen really interested me. What he has been trying to do is take Zope 3’s WSGI support and see how far he can integrate Zope 3 with Paste in an effort to end relience on Zope 3 instances and instead encourage developers to use Zope 3 more like a library where they import, setup and use only those bits they need. Philipp produced some paster create templates to create a sample project which can then be served by paster serve. This also meant that he could add all the Pylons middleware to Zope 3 and at one point he demonstrated a TurboGears app and a Zope app mounted using Paste to a WSGI app and then served together on the same server. I hadn’t realised just how far Zope had come in terms of its refactoring and as a result much of Zope 3 may very well become accessible to projects such as Pylons in the very near future, particularly with this sort of work going on. If any members of the Zope community want WSGI, Paste or Pylons advice related to this sort of opening up, please get in touch, I’m keen to help.
Other Talks
In the afternoon I saw py.test: towards interactive, distributed and rapid testing and unittest is Broken. There seem to be lots of testing tools available at the moment and nosetest and py.test in particular seem very similar. My instinct is to wait a bit longer to see how the various alternatives settle down.
Keynote
After the talks Simon Willison delivered his keynote on OpenID and as well as pointing out my AuthKit package as a Python OpenID implementation he announced that he would be working with the Pylons community to help unify the Django request and response with the Pylons one. If things go well we might have a chance to sprint on this on Wednesday or Thursday. I’ve heard Simon speak about OpenID quite a few times now but every time his arguments get more refined and his vision for decentralised social networks based on OpenID and web services becomes more focussed. Simon encouraged developers to work towards an open alternative to the Facebook platform. As always, I agree with virtually everything he said.
After the keynote and a quick chat to Nat and Simon I headed out into Vilnius with Graham to explore the city based on some of Natalie’s recommendations and Graham and I eventually ended up in the traditional-style bar/restaurant which Guido had chosen for the evening. It was made mainly of wood and had a tree growing inside. Beer was on Google and we met two developers who had travelled all the way from South Africa.
London Hack Day
So I’ve been here at Alexandra Palace in North London at the Hack Day organised by Yahoo! and BBC Backstage since 9:30 yesterday morning. I managed to get about 4 hours sleep but apart from that have been hacking away on various things, meeting new people (including Jacob Kaplan-Moss from Django) and meeting people I know already. By the way wii tennis on a conference projector is truly fantastic.
Every delegate has been asked to try to hack together a project which uses the BBC or Yahoo! APIs. Our team has produced is an image recognition engine called Fruitr which determines the type of fruit in a picture and recommends a recipe which uses the fruit. The comparison engine searches all images on Flickr which are tagged with fruitr and determines the picture on Flickr which best matches the picture the user uploaded. It then looks at the other tags associated with the image on Flickr to determine the type of fruit from the other tags. Once the fruit type has been determined the site fetches a recipe which uses the fruit with a JSON request to another server.
As well as the web interface the engine also has an email interface so you can email your picture to recipe@fruitr.com to receive a recipe via email. This means the site also works with photos taken on a mobile phoned and emailed on.
The great thing about the system is that the accuracy can be improved by other users uploading good quality fruit images to Flickr and tagging them with the fruitr tag.
Why not have a go at Fruitr.co.uk. If you need a test image I’ve uploaded a few here.
Identity 2.0 - mashup*
I went to the Identity 2.0 mashup at BT by St Pauls yesterday as part of my commitment to try to get more involved in the London start-up scene. I met some very friendly people including Ian Forrester, Stephan Tual and Paul Walsh and was pleased to catch up with Simon Willison and Natalie Downe.
Although I’m approaching the issue of digital identity from a technical point of view most people who attended were looking at it from a business perspective and frankly seemed a bit behind the times. Overall I was surprised at just how much disagreement there was about about the various concepts surrounding identity. It still means very different things to different people and whilst there was a general sense that OpenID is likely to play a major role in a future identity system for the web, no-one seemed quite sure how. Many people I talked to still had concerns about the basic concepts of the technology and whether it was even secure enough to be able to adopt in real commercial systems.
I still have a lot of faith in OpenID but the one thing that seems clear is that OpenID in its current form isn’t workable for the needs of most businesses. Whilst companies may be willing to provide OpenIDs they aren’t so keen on accepting them to grant access to systems where users have any sort of sensitive data like phone numbers or addresses. Of course at the moment OpenID isn’t really aimed at those markets but I don’t think it can really take off until it is.
Anyway, here are some of the points that stuck in my mind:
- Your identity is worth £85,000 to a fraudster - not because that’s what you have in your account but because that is how much credit they could get by pretending to be you.
- The value of your identity to a business is not in your identity itself but their ability to tie up different pieces of information about you in order to provide more focussed advertising or marketing.
- Potential revenue sources from digital identity are closely related to being able to provide solutions to areas of risk.
- About 55% of the “UK digital mainstream” are concerned about identity but only 4% do anything about it - a consumer product which was capable of saying “protects you from identity theft” would therefore be likely to sell nicely the same way household shredders do at the moment.
- For consumers identity isn’t an intellectual issue, it is an emotive issue.
- Finding language to describe digital identity to ordinary people is a key challenge but at the same time if we were all used to using a digital identity we wouldn’t need a language to describe it. After all, most people don’t know what a URL is but they can tell you their myspace page.
As an aside. I’ve had £700 stolen from my account over the last few days by a fraudster withdrawing cash from cashpoints in Milan. I still have my card so the person must have been using a clone. The experience has made me even more aware about of the flaws in current digital identity systems and the risks involved if they fail.