BuildOut: A C++ Executable to run Python Scripts

UPDATE: 2007-09-30 Because this blog entry doesn't make a lot of sense on its own I've bundled up the functionality into a new recipe called pylons_sandbox which you can use yourself.

I've been using Buildout recently for deploying my Pylons applications. Using the zc.recipe.egg recipe results in a series of scripts being added to your Buildout bin directory. These scripts set up all the paths to the installed eggs and then run the script itself but there are a number of problems:

1. Scripts don't get created for dependent packages
2. Error messages are often hidden when running bin/buildout
3. The Python interpreters created by Buildout aren't real executables so can't be used in #! lines in other scripts

Lets look at each in turn.

To solve the first problem simply list all the eggs of packages you want the scripts for in the buildout.cfg file then Buildout will add them.

To solve the second problem always run bin/buildout -vvv for very verbose output.

Solving the third problem is a little trickier. On Windows the setuptools package creates .exe files for each of the scripts it installs and these .exe files then execute the script. Below is the C++ code to create an executable for Buildout which works in a similar way:

/*
 * Buildout Launcher
 * +++++++++++++++++
 *
 * This application excutes a python script in the same directory as the
 * application. This is useful because it effectively turns a Python script
 * into a real executable which you can use on the #! line of other scripts.
 *
 * The script to be executed should have the same name as the the filename of
 * this compiled program but with a .py extension added to the end. The real
 * Python interpreter used to execute the script is dermined from the script's
 * #! line or /usr/bin/python is used as a fallback if no Python interpreter
 * can be found.
 *
 * The Python interpreters generated by Buildout are actually just Python
 * scripts so this application allows them to be run from a real executable.
 *
 * Compile this file with the following command:
 *
 *     g++ launcher.cc -o launcher
 *
 * Copyright James Gardner. MIT license. No warranty to the maximum extent
 * possible under the law.
 *
 */

#include <vector>
#include <string>
#include <unistd.h>
#include <fstream>

using namespace std;
int main(int argc,char *argv[])
{
    vector<string> args;
    int i;
    args.push_back("python");
    for (i=0;i<argc;i++)
        args.push_back(argv[i]);
    args[1] = strcat(argv[0], ".py");
    char *new_argv[argc+1];
    for (int i=0 ; i<argc+1 ; i++)  {
        new_argv[i] = (char *)args[i].c_str();
        //printf("i: %d, val: %s\n", i, new_argv[i]);
    }
    new_argv[argc+1] = NULL;
    vector<string> text_file;
    ifstream ifs(new_argv[1]);
    string temp;
    string temp_short;
    getline(ifs, temp);
    if (strncmp((char *)temp.c_str(), "#!", 2)) {
        /* default to /usr/bin/python if no #! header */
        temp_short = "/usr/bin/python";
    } else {
        temp_short = temp.substr(2,(temp.length()-2));
    }
    char python[temp_short.length()];
    strcpy(python, (char *)temp_short.c_str());
    //printf("%s\n", python);
    return execv(python, new_argv);
}

To use this code first install build-essentials:

apt-get install build-essentials

Then compile it:

g++ launcher.cc -o launcher

Then add .py to each of the script names in the bin/buildout directory and create a copy of the launcher application for each of the scripts with the original name of the script. When you execute program it will look for the corresponding .py file and analyse the #! line it contains to try to use the most appropriate version of Python, falling back to /usr/bin/python if there is no #! line. It then uses the Python executable chosen to execute the script which in turn sets up the paths and calls the appropriate function so that the script behaves properly.

This all works nicely but the current working directory where the initial script can be found is not on sys.path so you need to modify the buildout-generated scripts themselves. Add this after the existing path setups:

import os
sys.path.append(os.getcwd())

Now everything works nicely and the Buildout-generated Python interpreter script can be used from a real executable.

At some point I'll try to bundle this up into a proper egg recipe so that you don't have to handle the steps manually. I'll also write some background about Buildout to put this blog entry in context.

(view source)