James Gardner

The easy way to install Nginx is like this:

sudo aptitude install nginx

The problem is that this installs an old version (0.4) and doesn’t have SSL support built in. Nginx is designed to be super fast so and modules need to be compiled in manually. This means you need remove the Nginx package if you want SSL support.

If you try sudo apt-get remove –purge nginx you get this error:

Reading package lists... Done
Building dependency tree... Done
The following packages will be REMOVED
  nginx*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 0B of archives.
After unpacking 524kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 26362 files and directories currently installed.)
Removing nginx ...
Stopping nginx: invoke-rc.d: initscript nginx, action "stop" failed.
dpkg: error processing nginx (--purge):
 subprocess pre-removal script returned error exit status 1
Starting nginx: 2007/12/07 14:45:03 [emerg] 29441#0: unknown directive "ssl" in /etc/nginx/nginx.conf:30
invoke-rc.d: initscript nginx, action "start" failed.
dpkg: error while cleaning up:
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 nginx
E: Sub-process /usr/bin/dpkg returned an error code (1)

The solution is to edit /var/lib/dpkg/info/nginx.prerm and comment out the line with invoke-rc.d which gives the stop statement for nginx. If you run the command again it will remove successfully, purging your nginx.conf config file too. If this isn’t what you want make a backup first.

You can then remove your logs with:

sudo rm -rf /var/log/nginx/

Now to install Nginx 0.6 from source. Fisrt you will need some build tools:

sudo aptitude install build-essential

Then we need some libraries:

sudo aptitude install libpcre3 libpcre3-dev libpcrecpp0 libssl-dev zlib1g-dev

Check what the latest version is at http://nginx.net. At the time of writing the latest stable version is 0.5.33 so download that:

wget http://sysoev.ru/nginx/nginx-0.5.33.tar.gz
tar zxfv nginx-0.5.33.tar.gz
cd nginx-0.5.33

Now you are ready to compile Nginx. As mentioned earlier many of the options are set at compile time so have a look at the Compile Time Options page and decide which you need. We are going to set two options. The first is:

--with-http_ssl_module

This enables the SSL module. The second option is to customise where Nginx is installed to. The default is /usr/local/nginx but this means the Nginx binary will be /usr/local/nginx/sbin/nginx which isn’t on the PATH. A better place is /usr/local/sbin where it is easily accessible by root or by users with sudo access:

--sbin-path=/usr/local/sbin

Now we are ready to configure:

./configure --sbin-path=/usr/local/sbin --with-http_ssl_module

There is a useful configuration summary:

nginx path prefix: "/usr/local/nginx"
nginx binary file: "/usr/local/sbin"
nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
nginx pid file: "/usr/local/nginx/logs/nginx.pid"
nginx error log file: "/usr/local/nginx/logs/error.log"
nginx http access log file: "/usr/local/nginx/logs/access.log"
nginx http client request body temporary files: "/usr/local/nginx/client_body_temp"
nginx http proxy temporary files: "/usr/local/nginx/proxy_temp"
nginx http fastcgi temporary files: "/usr/local/nginx/fastcgi_temp"

Then compile and install:

make
sudo make install

The first thing I do next is create a backup of the original file:

sudo cp /usr/local/nginx/conf/nginx.conf  /usr/local/nginx/conf/nginx.conf.bak

Then start the server:

sudo /usr/local/sbin/nginx

If you visit http://yourdomain.com you should see the Nginx welcome message Welcome to nginx!. If you don’t it might be because you have another server such as Apache running.

To kill the server you can use the pid file:

sudo kill `cat /usr/local/nginx/logs/nginx.pid`

Notice this uses the ` character (normally underneath Esc) not the ‘ character. Obviously it is nicer to have an init.d setup so you can start and stop Nginx the same way as Apache and other servers. We’ll use the script from Slicehost which is itself based on the Debian package:

wget http://articles.slicehost.com/assets/2007/10/19/nginx
sudo chmod +x nginx
sudo mv nginx /etc/init.d

Now we can add this script to the runlevels:

sudo /usr/sbin/update-rc.d -f nginx defaults

Now you can start, stop and restart Nginx with these commands as normal:

sudo /etc/init.d/nginx start
sudo /etc/init.d/nginx stop
sudo /etc/init.d/nginx restart

Now let’s setup Nginx to proxy to a Pylons application. Replace these lines in /usr/local/nginx/conf/nginx.conf:

location / {
    root   html;
    index  index.html index.htm;
}

with these:

location / {
    proxy_redirect          off;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Port $server_port;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffer_size       4k;
    proxy_buffers           4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass  http://127.0.0.1:5000;
    proxy_redirect  default;
}

Update: Added X-Forwarded-Port and X-Forwarded-Host to the above since I use them both so much.

This will proxy all requests to another server running on port 5000, for example a running Pylons application. To test this you should start your Pylons application (making sure debug is set to false if this is a production setup):

paster serve development.ini

Now restart Nginx:

sudo /etc/init.d/nginx restart

and f you visit the site you should see your Pylons application.

Now for the SSL:

sudo aptitude install openssh-server

First create a the key, pem and certificate files:

openssl genrsa 1024 > host.key
chmod 400 host.key
openssl req -new -x509 -nodes -sha1 -days 365 -key host.key > host.cert
cat host.cert host.key > host.pem
chmod 400 host.pem

Make sure the Common Name is the same as the domain name the certificate is for.

Now we need to setup SSL on Nginx. This is described in detail here but it really just requires making the server section look like this:

worker_processes 1;
http {
    ...
    server {
        listen               443;
        ssl                  on;
        ssl_certificate      /path/to/host.pem;
        ssl_certificate_key  /path/to/host.key;
        keepalive_timeout    70;
    }
}

Restart Nginx again:

sudo /etc/init.d/nginx restart

Now if you visit http://yourdomain.com you’ll get no response but if you visit https://yourdomain.com you’ll get a certificate warning because the SSL certificate is not signed by a certificate authority. Click OK and you will have access to your application.

The final part of the setup is to setup a static file which redirects from the http version of the site to the https version. The easy way to do this is by setting up another server listening on port 80 and creating an error 404 page which redirects to the https version. Add this to the nginx config just before the existing server:

server {
    listen 80;
    server_name  yourdomain.com;
    error_page  404              /404.html;
    location = /404.html {
        root   /path/to/directory/containing/404/doc/;
    }
}

Then create a 404.html file which looks something like this but adjusted for your URL instead of yourdomain.com:

<html>
<head>
<meta http-equiv="refresh" content="2;URL=https://yourdomain.com/">

</head>
<body>
<h2>Redirecting</h2>
<p>You are being redirected to the <a href="https://yourdomain.com/">secure version of this site.</a></p>
</body>
</html>

Then restart Nginx again.

You should now have a secure Pylons app on the https port and a redirect page on the http port.

Further Reading:

http://articles.slicehost.com/2007/10/19/debian-etch-installing-nginx http://ubuntuforums.org/showthread.php?t=453053 http://sudhanshuraheja.com/2007/09/remove-nginx-from-ubuntu-fiesty-fawn.html http://www.rkblog.rk.edu.pl/w/p/pylons-and-nginx/

On Dom0 set up port forwarding from port 80 to the DomU running NginX, in this case 10.0.0.1:

sudo -i
iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 80 -j DNAT --to 10.0.0.1:80
exit

These get lost on each reboot unless you put them into a script in /etc/network/if-up.d/iptables but that’s OK for the moment.

First install Nginx:

sudo apt-get install nginx

If you visit the URL of the Dom0 server you should see the Welcome to nginx! message served from Nginx on the DomU.

Next you need to setup Python and Pylons. Install python:

sudo apt-get install python mercurial

Get vitualenv:

wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-0.9.1.tar.gz
tar zxfv virtualenv-0.9.1.tar.gz
cp virtualenv-0.9.1/virtualenv.py ./
rm virtualenv-0.9.1.tar.gz
rm -r virtualenv-0.9.1

Decide where you want your Pylons applications. I like mine in /var/web so lets create a test project:

sudo mkdir /var/web
sudo mkdir /var/web/test
sudo chown james:james /var/web/test

Setup a virtual environment:

james@vm2:~$ python virtualenv.py /var/web/test
New python executable in /var/web/test/bin/python
Installing setuptools.............................done.
james@vm2:~$ cd /var/web/test

Now you can install Pylons:

mkdir dep
cd dep
hg clone http://pylonshq.com/hg/pylons-dev
cd pylons-dev
../../bin/python setup.py develop

Now create a project:

mkdir src
cd src
../bin/paster create -t pylons Test

Now edit Test/test/config/middleware.py and set use_webop=True. Serve the app with:

cd Test/
../../bin/python setup.py develop
../../bin/easy_install sqlalchemy                        <-- So that beaker doesn't break
../../bin/paster serve --reload development.ini

That’s it. Now you just need to setup Nginx to reverse proxy to the paste server running on port 5000.

Edit the /etc/nginx/nginx.conf file so that the location / part looks like this:

location / {
    proxy_redirect          off;
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffer_size       4k;
    proxy_buffers           4 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    proxy_pass  http://127.0.0.1:5000;
    proxy_redirect  default;
}

You’ll then need to restart nginx and the paster server:

sudo /etc/init.d/nginx restart
../../bin/paster serve --reload development

If you press refresh in your browser you should see the Pylons welcome page.

This blog post just documents one way of creating a package. This is for one named SQLAlchemyManager which is some experimental SQLAlchemy middleware I’m developing. Package names are normally lowercase but the project itself can use CamelCase so in this example the actual package is called sqlalchemymanager.

First create the directory structure:

mkdir SQLAlchemyManager
cd SQLAlchemyManager
mkdir docs
mkdir sqlalchemymanager
mkdir examples
mkdir tests

This tutorial assumes you are creating the new structure in an existing SVN tree. If you aren’t you should now import the directories into a new Subversion repository.

Then create the setup.cfg file:

[egg_info]
tag_build = dev
tag_svn_revision = true

These options tell setuptools that this is a development release, and that the SVN revision number should be included in the package version. This will create an egg with dev-r139.egg or similar at the end of the filename. When you want to make a production release you should comment out these lines.

Next you need to set up ez_setup.py so that if your project’s users don’t have setuptools it will be automatically installed when they try to use it. You can do this svn:externals definition so that subversion will automatically include the latest files you need:

ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup

You can set this by executing this command in your project directory:

svn propedit svn:externals .

And then adding the line shown above to the file that comes up for editing. Then, whenever you update your project, ez_setup will be updated as well.

Next you need a setup.py file which contains most of the metadata about the project. This one is setup to make use of the ez_setup.py you just set up if the user doesn’t have setup tools. Importing ez_setup in this way doesn’t cause problems with Buildout either:

try:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages

import sys, os

version = '0.0'

setup(
    name='SQLAlchemyManager',
    version=version,
    description="",
    long_description="""\
    """,
    # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[],
    keywords='',
    author='',
    author_email='',
    url='',
    license='',
    packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
    include_package_data=True,
    zip_safe=False,
    install_requires=[
    ],
    entry_points="""
    """,
)

You can enter information for all the fields listed above but version, install_requires and long_description are particularly interesting.

version

This shoud be a string containing the version number of the package in three parts. The first part is the maor version, the second part the minor version and the third part the revision. In this case I’m developing towards a 1.0 release so the major relase will be 0 until the software is stable enough for 1.0. This is the first version so the minor version is 1 and I haven’t released any revisions so I’ll call that 0. The version string is therefore "0.1.0".

install_requires

This should be a comma separated list of packages and their version numbers for all the packages the package you require depend on. SQLAlchemyManager depends on SQLAlchemy 0.4 so I’d add "SQLAlchemy>=0.4,<=0.4.99" to the install list. Notice that I specify <=0.4.99. The idea behind choosing this is that any incompatible changes to SQLAlchemy should result in a new minor version of SQLAlchemy being released so if my package works with 0.4 it should also work with 0.4.5, 0.4.9 and even 0.4.99 if there are that many revisions. It might not work with 0.5 though. You might think I could specify <0.5 here but 0.5a 0.5dev“ and 0.5rc1 are all treated as less that 0.5 by easy install so the recommendation is to use the .99 format instead. You could choose <=0.4.99999 if you wanted but pacjages rarely have more than about 30 revisions so 99 is virtually always fine.

long_description

This is handy because it allow you to write detailed information in reStructuredText format. This information then forms the main text for the page which will appear on the Cheeseshop which effectively means you don’t need to create a website for your project because you can keep all information in the long description. We’ll see an example of a long description at the end.

Now you need to write some documentation. I create all the docs with a .txt extension and using Windows line endings so that the information can be easily read by both Windows and Linux users.

First create the README.txt file:

See the ``docs/index.txt`` for information.

Then the CHANGELOG.txt file:

Changes
=======

0.1.0 (**svn**)

* First version

Every time you make a change you should add it to the change log, I put the most recent change at the top. Every time you create a new version you should add a new section. The (**svn**) label marks which is the version in subversion. You should remember to move this if you create a new version or remove it if you create a production release.

Now the documentation in docs/index.txt:

SQLAlchemyManager
+++++++++++++++++++++++

.. contents ::

Summary
=======

* Provides a sensible way of using SQLAlchemy in WSGI applications

Get Started
===========

* Download from the `SQLAlchemyManager Cheeseshop page <http://python.org/pypi/SQLAlchemyManager>`_
* Install with ``easy_install SQLAlchemyManager``.

Author
======

`James Gardner <http://jimmyg.org/>`_ james at pythonweb dot org

You can now create the long description. Here’s one way to do it. At the top of the setup.py file you can do this:

def read(*rnames):
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

long_description = (
    "\n"+read('docs/index.txt')
    + '\n'
    + read('CHANGELOG.txt')
    + '\n'
    'License\n'
    '=======\n'
    + read('LICENSE.txt')
    + '\n'
    'Download\n'
    '========\n'
)

Then update ensure long_description=long_description in the setup() function. This will then automatically create the long description from the files in your project, helping to make life easier.

You can now create the files your project will use in the sqlalchemy directory and check them all in.

Finally you can add the LICENSE.txt file, preferably using one of the Open Source Licenses, the MIT license being the one I generally use. You should update the license line in setup.py too.

You’ll also want to specify a summary which will appear in the index page on the cheeseshop. Choose a few words to describe the project and put them in the description argument string.

svn co http://somedomain/svn/SQLAlchemyManager/tags tags
svn export http://somedomain/svn/SQLAlchemyManager/trunk 0.1.0
mv 0.1.0 tags/
svn add tags/0.1.0
svn ci tags/0.1.0 -m "Creating the 0.1.0 tag"

svn ci tags/0.1.0 -m "Final changes for the 0.1.0 release"

svn export http://somedomain/svn/SQLAlchemyManager/tags/0.1.0 0.1.0
cd 0.1.0
python setup.py sdist bdist_egg register upload

The YUI toolkit comes with a very flexible autocomplete control but a common requirement is for an autocomplete control that submits the ID associated with a text value rather than the text value itself, much like a select field submits the option value, not the contents the user selects from a drop down list.

Luckily this is fairly easy to achieve using a forceSelection option, a hidden field, and a custom itemSelectEvent handler.

First setup the imports as described on the YUI AutoComplete page:

<!--CSS file (default YUI Sam Skin) -->
<link type="text/css" rel="stylesheet"
    href="http://yui.yahooapis.com/2.3.0/build/autocomplete/assets/skins/sam/autocomplete.css">

<!-- Dependencies -->
<script type="text/javascript"
    src="http://yui.yahooapis.com/2.3.0/build/yahoo-dom-event/yahoo-dom-event.js"></script>

<!-- OPTIONAL: Connection (required only if using XHR DataSource) -->
<script type="text/javascript"
    src="http://yui.yahooapis.com/2.3.0/build/connection/connection-min.js"></script>

<!-- OPTIONAL: Animation (required only if enabling animation) -->
<script type="text/javascript"
    src="http://yui.yahooapis.com/2.3.0/build/animation/animation-min.js"></script>

<!-- OPTIONAL: External JSON parser from http://www.json.org/ (enables JSON validation) -->
<script type="text/javascript" src="http://www.json.org/json.js"></script>

<!-- Source file -->
<script type="text/javascript"
    src="http://yui.yahooapis.com/2.3.0/build/autocomplete/autocomplete-min.js"></script>

Now let’s set up the data structure in a Pylons controller action which the YUI component will access to populate the find as you type select dropdown. You could write similar code for Rails or PHP, it doesn’t have to be Pylons. Notice that the @jsonify decorator converts the Python data structure we return to valid JSON:

@jsonify
def get_data(self):
    return {
        "ResultSet": {
            "totalResultsAvailable":"100",
            "totalResultsReturned":2,
            "firstResultPosition":1,
            "Result": [
                {
                    "ID": "897",
                    "Title":"foo",
                    "Summary":"... When foo' is used in connection with bar' it has generally traced...",
                    "Url":"http:\/\/www.catb.org\/~esr\/jargon\/html\/F\/foo.html",
                    "ModificationDate":1072684800,
                    "MimeType":"text\/html"
                },
                {
                    "ID": "492",
                    "Title":"Foo Fighters",
                    "Summary":"Official site with news, tour dates, discography, store, community, and more.",
                    "Url":"http:\/\/www.foofighters.com\/",
                    "ModificationDate":1138521600,
                    "MimeType":"text\/html"
                }
            ]
        }
    }

Now we need to write the code to generate the autocomplete control. YUI uses an existing text field as the autocomplete field and this will contain whatever text is looked up. Our application requires the corresponding ID so we need a hidden field to hold that value. The hidden field should have the name you want the ID to be submitted as, the text field can have any name because it doesn’t contain the data the application needs. YUI also needs a container <div> which it populates with the results. Ours is called myContainer.

Here’s some HTML to achive this:

<form action="/fayt/submit">

<p><label for="myInput">Sponsor Name<br /></label></p>

<div id="dashboard_autocomplete" style="clear:both; padding-bottom: 20px; width: 400px;">
    <input id="myInput_id" type="hidden" name="myInput_id" />
    <input id="myInput" type="text" name="item">
    <div id="myContainer"></div>
</div>

<input type="submit" name="submit" value="submit" />
</form>

Now let’s add some JavaScript. This can go after the HTML above:

<script type="text/javascript">
    var mySchema = ["ResultSet.Result","Title","ID","Url","ModificationDate"];
    var myDataSource = new YAHOO.widget.DS_XHR("/fayt/get_data", mySchema);
    myDataSource.responseType = YAHOO.widget.DS_XHR.TYPE_JSON;
    var myAutoComp = new YAHOO.widget.AutoComplete("myInput","myContainer", myDataSource);
</script>

The first line sets up a schema, the first item of which specifies where the actual results are in the data structrue returned (in this case they are in the Result list of the ResultSet dictionary), the subsequent entries specify values within each result which might be displayed in the control.

Test this example by entering foo into the field. It works as expected returning two options and allowing you to select one but the hidden myInput_id field doesn’t get populated.

To populate the hidden field we need to create a callback function and subscribe it to the autocomplete control’s itemSelectEvent. Add the following code at the end of the JavaScript above, just before the </script> tag:

function fnCallback(e, args) {
    YAHOO.util.Dom.get("myInput_id").value = args[2][1];
 }
myAutoComp.itemSelectEvent.subscribe(fnCallback);

Now, when an item is selected from the autocomplete control, fnCallback gets called with two arguments. The first is an event object e and the second is a list of arguments described here. These are:

oSelf

<YAHOO.widget.AutoComplete> The AutoComplete instance.

elItem

<HTMLElement> The selected <li> element item.

oData

<Object> The data returned for the item, either as an object, or mapped from the schema into an array.

In this case we want to access the data returned for the item, which we can access as args[2]. Because of the way we set up mySchema earlier the ID field is the second item in the list (you don’t count the first, "ResultSet.Result" because it isn’t one of the data items). We can therefore access the ID of the selected item as args[2][1] (JavaScript arrays are counted from 0 so the second item is numbered 1). All that remains is to assign this ID to the field which is what the YAHOO.util.Dom.get("myInput_id").value = args[2][1]; line does.

Now that the control is working properly let’s write the code to get the ID in a Pylons controller action:

def submit(self):
    id = request.params.get('myInput_id')
    item = request.params.get('item')
    return "The submitted ID was: %s, the selected item was %s" % (id, item)

There are various options you can use to spice up the control. Try adding some of these at the end of the JavaScript, before the </script> tag:

myAutoComp.useShadow = true;
myAutoComp.forceSelection = true;
myAutoComp.useIFrame = true;

The second of these options ensures that a user selects one of the options from the list rather than entering any old value. This is essential if you want the autocomplete control to be able to replace an actual select control. The third puts the content in an iFrame so that on IE, any form elements beneath the div generated during the autocomplete do not show through.

You can also specify a function to change how the dropdown container formats the infomation. You could use this to display images, make certain parts of the text bold etc. Here’s a simple example:

// This function puts the title in bold if the modification date is
// after 1100000000
myAutoComp.formatResult = function(aResultItem, sQuery) {
    var sKey = aResultItem[0]; // the entire result key

    // We aren't using these two in this example but it is useful
    // to know how to get them.
    var sKeyQuery = sKey.substr(0, sQuery.length); // the query itself
    var sKeyRemainder = sKey.substr(sQuery.length); // the rest of the result

    if (aResultItem[3] > 1100000000) {
        var aMarkup = [
           "<div id='ysearchresult'>",
           "<span style='font-weight:bold'>",
           sKey[0],
           "</span>",
           "</div>"
        ];
     } else {
       var aMarkup = [
           "<div id='ysearchresult'>",
           "<span style='font-weight:normal'>",
           sKey[0],
           "</span>",
           "</div>"
       ]
     }
     return (aMarkup.join(""));
};

That’s it. You now have a fully customisable autocomplete control which can be used to replace ordinary HTML <select> fields in cases where there are too many items to easily list in a select alone.

If you spot any mistakes or have suggestions for improvements please feel free to leave a comment.

It is sometimes rather useful to be able to populate a form with static text which you don’t want the user to be able to edit. FormEncode’s htmlfill.render() does have an optoin for text_as_default which you can set to true which will render any unknown input fields as text. It does try to render the form field but needs this patch to render the value:

Index: formencode/htmlfill.py
===================================================================
--- formencode/htmlfill.py      (revision 3008)
+++ formencode/htmlfill.py      (working copy)
@@ -436,9 +436,15 @@
             if value is None:
                 value = self.get_attr(attrs, 'value', '')
             self.set_attr(attrs, 'value', value)
-            self.write_tag('input', attrs, startend)
+           self.write_text(self.get_attr(attrs, 'value'))

I like to go further and implement support for a static field (it feels less hacky) just add this to the end of the patch:

             self.skip_next = True
             self.add_key(name)
+        elif t == 'static':
+            self.set_attr(attrs, 'value', value or
+                          self.get_attr(attrs, 'value', ''))
+            self.write_text(self.get_attr(attrs, 'value'))
+            self.skip_next = True
+            self.add_key(name)
         else:
             assert 0, "I don't know about this kind of <input>: %s (pos: %s)" \
                    % (t, self.getpos())

Then you can create fields like this:

<input type="static" name="name" />

and htmlfilll will replace them with the text value associated with the name key.

I’m pleased to announce the release of AuthKit 0.4.0. It has taken me a lot longer than it should have but it is out. You can download AuthKit here. Please note that the config file format has changed a bit since 0.3 so have a look at the Pylons Book chapters to read about the new version.

Please let me know about any bugs and I’ll release a 0.4.1 if necessary.

There are two problems with uploading files over the web using a simple HTML Form

• You can only upload one file at once
• You don’t get any idea of the upload progress

There are ways of solving these problems using AJAX and a hidden iframe (I might write one up if I get a chance) but Flash 8 provides a better solution because it allows you to select multiple files from a single dialog box so that if you want to upload a hole directory you don’t have to select each of the files individually. Flash 8 can also keep track of the upload progress on the client side so you don’t need to implement any server-side feedback system as the file is being uploaded. You can also specify a maximum file size.

There are a number of open source implementations which expose this Flash 8 functionality via JavaScript. The one I use is SWFUpload.

If you follow on from my last upload example you can replace the index() action with this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head>

    <script type="text/javascript" src="jscripts/SWFUpload/SWFUpload.js"></script>
    <script type="text/javascript" src="jscripts/example_callbacks.js"></script>
    <script type="text/javascript">
            var swfu;
            var swfu2;
            window.onload = function() {
                    // Max settings
                    swfu = new SWFUpload({
                            upload_script : "/up/upload",
                            target : "SWFUploadTarget",
                            flash_path : "jscripts/SWFUpload/SWFUpload.swf",
                            allowed_filesize : 3072000,     // 3000 MB
                            allowed_filetypes : "*.*",
                            allowed_filetypes_description : "All files...",
                            browse_link_innerhtml : "Browse",
                            upload_link_innerhtml : "Upload queue",
                            browse_link_class : "swfuploadbtn browsebtn",
                            upload_link_class : "swfuploadbtn uploadbtn",
                            flash_loaded_callback : 'swfu.flashLoaded',
                            upload_file_queued_callback : "fileQueued",
                            upload_file_start_callback : 'uploadFileStart',
                            upload_progress_callback : 'uploadProgress',
                            upload_file_complete_callback : 'uploadFileComplete',
                            upload_file_cancel_callback : 'uploadFileCancelled',
                            upload_queue_complete_callback : 'uploadQueueComplete',
                            upload_error_callback : 'uploadError',
                            upload_cancel_callback : 'uploadCancel',
                            auto_upload : false
                    });
            }
    </script>
    <style type="text/css">
            .swfuploadbtn {
                    display: block;
                    width: 100px;
                    padding: 0 0 0 20px;
            }
            .browsebtn { background: url(/images/add.png) no-repeat 0 4px; }
            .uploadbtn {
                    display: none;
                    background: url(/images/accept.png) no-repeat 0 4px;
            }
            .cancelbtn {
                    display: block;
                    width: 16px;
                    height: 16px;
                    float: right;
                    background: url(/images/cancel.png) no-repeat;
            }
            #cancelqueuebtn {
                    display: block;
                    display: none;
                    background: url(/images/cancel.png) no-repeat 0 4px;
                    margin: 10px 0;
            }
            #SWFUploadFileListingFiles ul {
                    margin: 0;
                    padding: 0;
                    list-style: none;
            }
            .SWFUploadFileItem {
                    display: block;
                    width: 230px;
                    height: 70px;
                    float: left;
                    background: #eaefea;
                    margin: 0 10px 10px 0;
                    padding: 5px;

            }
            .fileUploading { background: #fee727; }
            .uploadCompleted { background: #d2fa7c; }
            .uploadCancelled { background: #f77c7c; }
            .uploadCompleted .cancelbtn, .uploadCancelled .cancelbtn {
                    display: none;
            }
            span.progressBar {
                    width: 200px;
                    display: block;
                    font-size: 10px;
                    height: 4px;
                    margin-top: 2px;
                    margin-bottom: 10px;
                    background-color: #CCC;
            }
    </style>
</head>
<body>
    <h1><span>SWFUpload Testpage</span></h1>
            <h2>Upload Example</h2>
            <p>Replace contents of a div with links for uploading and browsing,
            degrades gracefully if flash/javascript isn't accessible</p>
            <div id="SWFUploadTarget">
                    <form action="upload.php" method="post" enctype="multipart/form-data">

                            <input type="file" name="Filedata" id="Filedata" />
                            <input type="submit" value="upload test" />
                    </form>
            </div>
            <h4 id="queueinfo">Queue is empty</h4>
            <div id="SWFUploadFileListingFiles"></div>
            <br class="clr" />
            <a class="swfuploadbtn" id="cancelqueuebtn"
            href="javascript:cancelQueue();">Cancel queue</a>
</body>
</html>

You’ll also need to download the SWFUpload source distribution and copy the jscripts directory into your Pylons project public directory so that the /jscripts/ links in the HTML work correctly. The example also uses some images which are attached to this entry and should be put in the public/images directory so that the /images/ URLs resolve.

Accept icon Add icon Cancel icon Progress bar

If you have Flash 8 or above and run this example you’ll now be able to upload multiple files to the unmodified controller. You can change the CSS and the images to change how the upload behaves and there are plenty of hooks for you to tie what’s happening in Flash to your page’s JavaScript.

Sample Pylons File Upload Project

File uploads are one of those things in Python which are still rather tricky to handle. First of all you need a form like this one with enctype="multipart/form-data" and a file input field:

<form action="up" method="post" enctype="multipart/form-data">
Upload file: <input type="file" name="myfile" /> <br />
             <input type="submit" name="submit" value="Submit" />
</form>

In your Python code you can then get the uploaded file data like this:

import cgi
form_data = cgi.FieldStorage()
file_data = form_data['myfile'].value

and you can write it somewhere like this:

fp =open('some/file','wb')
fp.write(file_data)
fp.close()

This is all well and good but what if you want to stream that data to a service such as Amazon S3 or what if you want to provide feedback to the user about how much of the file has been uploaded? The example here can’t help with this because you can only access the data once the whole file is uploaded.

Here’s how you can solve this problem. It is a bit of a nasty solution because you need to create your own file class which calls your callback. You also need your own FieldStorage class which has its make_file() method overridden to use the open file object you supply instead of the tempfile it would use by default. It also only works with forms with one file field but it demonstrates the principles on which you can build your own solution.

Here is a Pylons controller using this system:

import logging
from upload.lib.base import *
log = logging.getLogger(__name__)

import os
import shutil
import cgi

class ProgressFile(file):
    def write(self, *k, **p):
        if hasattr(self, 'callback'):
            self.callback(self, *k, **p)
        return file.write(self, *k,**p)

    def set_callback(self, callback):
        self.callback = callback

def stream(file_object):

    class CustomFieldStorage(cgi.FieldStorage):
        def make_file(self, binary=None):
            self.open_file = file_object
            return self.open_file

    return CustomFieldStorage

class UpController(BaseController):

    def index(self):
        return """
            <html>
            <body>
            <h1>Upload</h1>
            <form action="up" method="post" enctype="multipart/form-data">
            Upload file: <input type="file" name="myfile" /> <br />
                         <input type="submit" name="submit" value="Submit" />
            </form>
            </body>
            </html>
        """

    def upload(self):

        def callback(file, *k, **p):
            log.debug("Logged %s", [file.tell()])

        fp = ProgressFile('somefile', 'wb')
        fp.set_callback(callback)
        custom_field_storage = stream(fp)(
            environ=request.environ,
            strict_parsing=True,
            fp=request.environ['wsgi.input']
        )
        fp.close()
        return 'done'

As the file is uploaded it will now get streamed to the open some/file object as required and the calback() function gets called on every write so that you can find out how much data has been written with file.tell(). If you try this you will see the file uploads fine and you receive the done message. The output logs then look something like this as each write() call is logged:

13:36:22,447 DEBUG [upload.controllers.up] Logged [30011568L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011619L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011661L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011662L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011702L]
13:36:22,473 DEBUG [upload.controllers.up] Logged [30011762L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011812L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011876L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011942L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30012006L]
13:36:22,475 DEBUG [upload.controllers.up] Logged [30012048L]
13:36:22,475 DEBUG [upload.controllers.up] Logged [30012049L]

In most of my Pylons apps nowadays I make use of a few tricks form handling tricks that allow me to re-use the same fields template for creating new objects and updating existing ones. Here’s how it works.

First of all I write a Mako template with the fields fragments called fields_fragment.mako:

## -*- coding: utf-8 -*-
<p>
    <label for="approvalgranted">Date Approval Granted<br /></label>
    <input type="text" name="approvalgranted" />
</p>
<p>
    <label for="note">Note<br /></label>
    <input type="text" name="note" />
</p>
<p>
    <label for="enteredby">Last edited by<br /></label>
    <input type=unknown name="enteredby" />
</p>

Then I create two templates to use the fields, one for adding a record, the other for updating it. Here is add.mako:

## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>

<h2>Add Requirement</h2>

${h.form(h.url(controller='studyapprovals', action='add', study_id=c.study_id), method='post')}
    ${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
    <p><input type="submit" value="Add &raquo;" name="go" class="button" /></p>
${h.end_form()}

and update.mako:

## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>

<h2>Update Requirement</h2>

${h.form(h.url(controller='studyapprovals', action='edit', id=c.id,  study_id=c.study_id), method='post')}
     ${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
    <p><input type="submit" value="Save" name="go" class="button" /></p>
${h.end_form()}

Notice the lines with ## -*- coding: utf-8 -*- specify the files are encoded with the UTF-8 chracterset which means I can use any Unicode characters I like in the templates. Also notice that both templates inherit from a file /base/index.mako which provides the bulk of the HTML for the page.

The important parts to draw your attention to are the lines:

<%namespace file="fields_fragment.mako" name="fields" import="*"/>

and:

${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}

The first is effectively an import of the shared fields_fragment.mako file as the namespace fields. I can then output the contents of that file by writing ${fields.body()} in the template to render the body of the fields_fragment.mako template but most of the time that’s not what I want to do.

Working with forms is mainly about handling the validation and then repopulating the form with the entered values and error messages. My controllers handle the validation and set c.values and c.errors so I can use formencode.htmlfill to populate the forms. The only complication is that in order to pass the value of calling fields.body() capture the render() function I have to capture it, calling it simply renders the output directly which isn’t what I’m after. This is why the call to capture is made.

Checkboxes are tricky in web applications because often you want a value of either True or False to be associated with a variable and you want to handle that with a checkbox which is either ticked or not but an unticked checkbox doesn’t submit a value at all which can be a pain for validation.

FormEncode can handle this situation like this:

>>> import formencode
>>>
>>> class MySchema(formencode.Schema):
...     ticked = formencode.validators.StringBoolean(if_missing=False)

You can then do:

>>> MySchema().to_python({})
{'ticked': False}
>>> MySchema().to_python({'ticked': 'true'})
{'ticked': True}

Both these states are handled fine by HTMLFill too:

>>> import formencode.htmlfill
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':False})
'<input type="checkbox" name="ticked" />'
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':True})
'<input type="checkbox" name="ticked" checked="checked" />'

Occasionally you run into a situation where if a checkbox is ticked you want to run some extra validation. For this you can use FormEncode’s RequireIfMissing or “ RequireIfPresent“ validators documented here.