James Gardner

So, today’s been an interesting day. I finally decided on Sunday against building my future projects with Amazon web services on the basis that I’m more likely to be able to build a faster responding and more tailored solution to each of my needs with other tools. I decided to investigate MogileFS as an alternative to Amazon S3 and at the same time I decided to start work on a new set of Python tools to be named "good ingredients" which would basically be very low level components using a more genericised version of the WSGI spec which also allows services (something I’ve been thinking about recently). I mentioned some of my ideas to some of the Rails community including one of the core comitters but didn’t get a lot of interest. I’m not too suprised becuase I haven’t had a lot of interest from the Pylons community either and the Rails community are far more in favour of abstractions than the Pylons community and abstractions are precicely what good ingredients were not going to be about.

Anyway, the plan was to put together existing tools that provide the sorts of very scalable infrastructure which would otherwise require something like AWS and expose useful a useful low-level web framework stack with the good ingredients.

Then after all this planning Google go an release their App Engine which offers both a scalable infrastructure and a new set of Python modules which seem farily low-level and support WSGI. Still the infrastructure isn’t customisable and the modules don’t use anything akin to the modular WSGI services API I am planning so I think it makes sense for me to carry on with my ideas anyway. Interesting times though.

This blog post just documents one way of creating a package. This is for one named SQLAlchemyManager which is some experimental SQLAlchemy middleware I’m developing. Package names are normally lowercase but the project itself can use CamelCase so in this example the actual package is called sqlalchemymanager.

First create the directory structure:

mkdir SQLAlchemyManager
cd SQLAlchemyManager
mkdir docs
mkdir sqlalchemymanager
mkdir examples
mkdir tests

This tutorial assumes you are creating the new structure in an existing SVN tree. If you aren’t you should now import the directories into a new Subversion repository.

Then create the setup.cfg file:

[egg_info]
tag_build = dev
tag_svn_revision = true

These options tell setuptools that this is a development release, and that the SVN revision number should be included in the package version. This will create an egg with dev-r139.egg or similar at the end of the filename. When you want to make a production release you should comment out these lines.

Next you need to set up ez_setup.py so that if your project’s users don’t have setuptools it will be automatically installed when they try to use it. You can do this svn:externals definition so that subversion will automatically include the latest files you need:

ez_setup svn://svn.eby-sarna.com/svnroot/ez_setup

You can set this by executing this command in your project directory:

svn propedit svn:externals .

And then adding the line shown above to the file that comes up for editing. Then, whenever you update your project, ez_setup will be updated as well.

Next you need a setup.py file which contains most of the metadata about the project. This one is setup to make use of the ez_setup.py you just set up if the user doesn’t have setup tools. Importing ez_setup in this way doesn’t cause problems with Buildout either:

try:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages

import sys, os

version = '0.0'

setup(
    name='SQLAlchemyManager',
    version=version,
    description="",
    long_description="""\
    """,
    # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
    classifiers=[],
    keywords='',
    author='',
    author_email='',
    url='',
    license='',
    packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
    include_package_data=True,
    zip_safe=False,
    install_requires=[
    ],
    entry_points="""
    """,
)

You can enter information for all the fields listed above but version, install_requires and long_description are particularly interesting.

version

This shoud be a string containing the version number of the package in three parts. The first part is the maor version, the second part the minor version and the third part the revision. In this case I’m developing towards a 1.0 release so the major relase will be 0 until the software is stable enough for 1.0. This is the first version so the minor version is 1 and I haven’t released any revisions so I’ll call that 0. The version string is therefore "0.1.0".

install_requires

This should be a comma separated list of packages and their version numbers for all the packages the package you require depend on. SQLAlchemyManager depends on SQLAlchemy 0.4 so I’d add "SQLAlchemy>=0.4,<=0.4.99" to the install list. Notice that I specify <=0.4.99. The idea behind choosing this is that any incompatible changes to SQLAlchemy should result in a new minor version of SQLAlchemy being released so if my package works with 0.4 it should also work with 0.4.5, 0.4.9 and even 0.4.99 if there are that many revisions. It might not work with 0.5 though. You might think I could specify <0.5 here but 0.5a 0.5dev“ and 0.5rc1 are all treated as less that 0.5 by easy install so the recommendation is to use the .99 format instead. You could choose <=0.4.99999 if you wanted but pacjages rarely have more than about 30 revisions so 99 is virtually always fine.

long_description

This is handy because it allow you to write detailed information in reStructuredText format. This information then forms the main text for the page which will appear on the Cheeseshop which effectively means you don’t need to create a website for your project because you can keep all information in the long description. We’ll see an example of a long description at the end.

Now you need to write some documentation. I create all the docs with a .txt extension and using Windows line endings so that the information can be easily read by both Windows and Linux users.

First create the README.txt file:

See the ``docs/index.txt`` for information.

Then the CHANGELOG.txt file:

Changes
=======

0.1.0 (**svn**)

* First version

Every time you make a change you should add it to the change log, I put the most recent change at the top. Every time you create a new version you should add a new section. The (**svn**) label marks which is the version in subversion. You should remember to move this if you create a new version or remove it if you create a production release.

Now the documentation in docs/index.txt:

SQLAlchemyManager
+++++++++++++++++++++++

.. contents ::

Summary
=======

* Provides a sensible way of using SQLAlchemy in WSGI applications

Get Started
===========

* Download from the `SQLAlchemyManager Cheeseshop page <http://python.org/pypi/SQLAlchemyManager>`_
* Install with ``easy_install SQLAlchemyManager``.

Author
======

`James Gardner <http://jimmyg.org/>`_ james at pythonweb dot org

You can now create the long description. Here’s one way to do it. At the top of the setup.py file you can do this:

def read(*rnames):
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

long_description = (
    "\n"+read('docs/index.txt')
    + '\n'
    + read('CHANGELOG.txt')
    + '\n'
    'License\n'
    '=======\n'
    + read('LICENSE.txt')
    + '\n'
    'Download\n'
    '========\n'
)

Then update ensure long_description=long_description in the setup() function. This will then automatically create the long description from the files in your project, helping to make life easier.

You can now create the files your project will use in the sqlalchemy directory and check them all in.

Finally you can add the LICENSE.txt file, preferably using one of the Open Source Licenses, the MIT license being the one I generally use. You should update the license line in setup.py too.

You’ll also want to specify a summary which will appear in the index page on the cheeseshop. Choose a few words to describe the project and put them in the description argument string.

svn co http://somedomain/svn/SQLAlchemyManager/tags tags
svn export http://somedomain/svn/SQLAlchemyManager/trunk 0.1.0
mv 0.1.0 tags/
svn add tags/0.1.0
svn ci tags/0.1.0 -m "Creating the 0.1.0 tag"

svn ci tags/0.1.0 -m "Final changes for the 0.1.0 release"

svn export http://somedomain/svn/SQLAlchemyManager/tags/0.1.0 0.1.0
cd 0.1.0
python setup.py sdist bdist_egg register upload

It is sometimes rather useful to be able to populate a form with static text which you don’t want the user to be able to edit. FormEncode’s htmlfill.render() does have an optoin for text_as_default which you can set to true which will render any unknown input fields as text. It does try to render the form field but needs this patch to render the value:

Index: formencode/htmlfill.py
===================================================================
--- formencode/htmlfill.py      (revision 3008)
+++ formencode/htmlfill.py      (working copy)
@@ -436,9 +436,15 @@
             if value is None:
                 value = self.get_attr(attrs, 'value', '')
             self.set_attr(attrs, 'value', value)
-            self.write_tag('input', attrs, startend)
+           self.write_text(self.get_attr(attrs, 'value'))

I like to go further and implement support for a static field (it feels less hacky) just add this to the end of the patch:

             self.skip_next = True
             self.add_key(name)
+        elif t == 'static':
+            self.set_attr(attrs, 'value', value or
+                          self.get_attr(attrs, 'value', ''))
+            self.write_text(self.get_attr(attrs, 'value'))
+            self.skip_next = True
+            self.add_key(name)
         else:
             assert 0, "I don't know about this kind of <input>: %s (pos: %s)" \
                    % (t, self.getpos())

Then you can create fields like this:

<input type="static" name="name" />

and htmlfilll will replace them with the text value associated with the name key.

I’m pleased to announce the release of AuthKit 0.4.0. It has taken me a lot longer than it should have but it is out. You can download AuthKit here. Please note that the config file format has changed a bit since 0.3 so have a look at the Pylons Book chapters to read about the new version.

Please let me know about any bugs and I’ll release a 0.4.1 if necessary.

There are two problems with uploading files over the web using a simple HTML Form

• You can only upload one file at once
• You don’t get any idea of the upload progress

There are ways of solving these problems using AJAX and a hidden iframe (I might write one up if I get a chance) but Flash 8 provides a better solution because it allows you to select multiple files from a single dialog box so that if you want to upload a hole directory you don’t have to select each of the files individually. Flash 8 can also keep track of the upload progress on the client side so you don’t need to implement any server-side feedback system as the file is being uploaded. You can also specify a maximum file size.

There are a number of open source implementations which expose this Flash 8 functionality via JavaScript. The one I use is SWFUpload.

If you follow on from my last upload example you can replace the index() action with this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head>

    <script type="text/javascript" src="jscripts/SWFUpload/SWFUpload.js"></script>
    <script type="text/javascript" src="jscripts/example_callbacks.js"></script>
    <script type="text/javascript">
            var swfu;
            var swfu2;
            window.onload = function() {
                    // Max settings
                    swfu = new SWFUpload({
                            upload_script : "/up/upload",
                            target : "SWFUploadTarget",
                            flash_path : "jscripts/SWFUpload/SWFUpload.swf",
                            allowed_filesize : 3072000,     // 3000 MB
                            allowed_filetypes : "*.*",
                            allowed_filetypes_description : "All files...",
                            browse_link_innerhtml : "Browse",
                            upload_link_innerhtml : "Upload queue",
                            browse_link_class : "swfuploadbtn browsebtn",
                            upload_link_class : "swfuploadbtn uploadbtn",
                            flash_loaded_callback : 'swfu.flashLoaded',
                            upload_file_queued_callback : "fileQueued",
                            upload_file_start_callback : 'uploadFileStart',
                            upload_progress_callback : 'uploadProgress',
                            upload_file_complete_callback : 'uploadFileComplete',
                            upload_file_cancel_callback : 'uploadFileCancelled',
                            upload_queue_complete_callback : 'uploadQueueComplete',
                            upload_error_callback : 'uploadError',
                            upload_cancel_callback : 'uploadCancel',
                            auto_upload : false
                    });
            }
    </script>
    <style type="text/css">
            .swfuploadbtn {
                    display: block;
                    width: 100px;
                    padding: 0 0 0 20px;
            }
            .browsebtn { background: url(/images/add.png) no-repeat 0 4px; }
            .uploadbtn {
                    display: none;
                    background: url(/images/accept.png) no-repeat 0 4px;
            }
            .cancelbtn {
                    display: block;
                    width: 16px;
                    height: 16px;
                    float: right;
                    background: url(/images/cancel.png) no-repeat;
            }
            #cancelqueuebtn {
                    display: block;
                    display: none;
                    background: url(/images/cancel.png) no-repeat 0 4px;
                    margin: 10px 0;
            }
            #SWFUploadFileListingFiles ul {
                    margin: 0;
                    padding: 0;
                    list-style: none;
            }
            .SWFUploadFileItem {
                    display: block;
                    width: 230px;
                    height: 70px;
                    float: left;
                    background: #eaefea;
                    margin: 0 10px 10px 0;
                    padding: 5px;

            }
            .fileUploading { background: #fee727; }
            .uploadCompleted { background: #d2fa7c; }
            .uploadCancelled { background: #f77c7c; }
            .uploadCompleted .cancelbtn, .uploadCancelled .cancelbtn {
                    display: none;
            }
            span.progressBar {
                    width: 200px;
                    display: block;
                    font-size: 10px;
                    height: 4px;
                    margin-top: 2px;
                    margin-bottom: 10px;
                    background-color: #CCC;
            }
    </style>
</head>
<body>
    <h1><span>SWFUpload Testpage</span></h1>
            <h2>Upload Example</h2>
            <p>Replace contents of a div with links for uploading and browsing,
            degrades gracefully if flash/javascript isn't accessible</p>
            <div id="SWFUploadTarget">
                    <form action="upload.php" method="post" enctype="multipart/form-data">

                            <input type="file" name="Filedata" id="Filedata" />
                            <input type="submit" value="upload test" />
                    </form>
            </div>
            <h4 id="queueinfo">Queue is empty</h4>
            <div id="SWFUploadFileListingFiles"></div>
            <br class="clr" />
            <a class="swfuploadbtn" id="cancelqueuebtn"
            href="javascript:cancelQueue();">Cancel queue</a>
</body>
</html>

You’ll also need to download the SWFUpload source distribution and copy the jscripts directory into your Pylons project public directory so that the /jscripts/ links in the HTML work correctly. The example also uses some images which are attached to this entry and should be put in the public/images directory so that the /images/ URLs resolve.

Accept icon Add icon Cancel icon Progress bar

If you have Flash 8 or above and run this example you’ll now be able to upload multiple files to the unmodified controller. You can change the CSS and the images to change how the upload behaves and there are plenty of hooks for you to tie what’s happening in Flash to your page’s JavaScript.

Sample Pylons File Upload Project

File uploads are one of those things in Python which are still rather tricky to handle. First of all you need a form like this one with enctype="multipart/form-data" and a file input field:

<form action="up" method="post" enctype="multipart/form-data">
Upload file: <input type="file" name="myfile" /> <br />
             <input type="submit" name="submit" value="Submit" />
</form>

In your Python code you can then get the uploaded file data like this:

import cgi
form_data = cgi.FieldStorage()
file_data = form_data['myfile'].value

and you can write it somewhere like this:

fp =open('some/file','wb')
fp.write(file_data)
fp.close()

This is all well and good but what if you want to stream that data to a service such as Amazon S3 or what if you want to provide feedback to the user about how much of the file has been uploaded? The example here can’t help with this because you can only access the data once the whole file is uploaded.

Here’s how you can solve this problem. It is a bit of a nasty solution because you need to create your own file class which calls your callback. You also need your own FieldStorage class which has its make_file() method overridden to use the open file object you supply instead of the tempfile it would use by default. It also only works with forms with one file field but it demonstrates the principles on which you can build your own solution.

Here is a Pylons controller using this system:

import logging
from upload.lib.base import *
log = logging.getLogger(__name__)

import os
import shutil
import cgi

class ProgressFile(file):
    def write(self, *k, **p):
        if hasattr(self, 'callback'):
            self.callback(self, *k, **p)
        return file.write(self, *k,**p)

    def set_callback(self, callback):
        self.callback = callback

def stream(file_object):

    class CustomFieldStorage(cgi.FieldStorage):
        def make_file(self, binary=None):
            self.open_file = file_object
            return self.open_file

    return CustomFieldStorage

class UpController(BaseController):

    def index(self):
        return """
            <html>
            <body>
            <h1>Upload</h1>
            <form action="up" method="post" enctype="multipart/form-data">
            Upload file: <input type="file" name="myfile" /> <br />
                         <input type="submit" name="submit" value="Submit" />
            </form>
            </body>
            </html>
        """

    def upload(self):

        def callback(file, *k, **p):
            log.debug("Logged %s", [file.tell()])

        fp = ProgressFile('somefile', 'wb')
        fp.set_callback(callback)
        custom_field_storage = stream(fp)(
            environ=request.environ,
            strict_parsing=True,
            fp=request.environ['wsgi.input']
        )
        fp.close()
        return 'done'

As the file is uploaded it will now get streamed to the open some/file object as required and the calback() function gets called on every write so that you can find out how much data has been written with file.tell(). If you try this you will see the file uploads fine and you receive the done message. The output logs then look something like this as each write() call is logged:

13:36:22,447 DEBUG [upload.controllers.up] Logged [30011568L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011619L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011661L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011662L]
13:36:22,448 DEBUG [upload.controllers.up] Logged [30011702L]
13:36:22,473 DEBUG [upload.controllers.up] Logged [30011762L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011812L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011876L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30011942L]
13:36:22,474 DEBUG [upload.controllers.up] Logged [30012006L]
13:36:22,475 DEBUG [upload.controllers.up] Logged [30012048L]
13:36:22,475 DEBUG [upload.controllers.up] Logged [30012049L]

This is actually very easy. First install Ted, an old UNIX text editor with this command:

$ sudo apt-get install ted

You’ll also need Ghostscript but this is probably already installed, if not run:

$ sudo apt-get install gs-afpl

Then download the rtf2pdf.sh script from ftp://ftp.nluug.nl/pub/editors/ted/rtf2pdf.sh and run this command:

$ sh rtf2pdf.sh file.rtf output.pdf

and that’s it, very easy and it actually works. I’ve attached a text version of the rtf2pdf.sh script here as well.

Here’s a very short program which uses this from Python:

import os

def convert(rtf, pdf):
    pipe = os.popen("sh rtf2pdf.sh %s %s"%(rtf, pdf), "r")
    error = pipe.read()
    if error:
        print "ERROR. An error occurred here is the description..."
        print
        print error

if __name__ == '__main__':
    convert('test.rtf', 'test.pdf')

In most of my Pylons apps nowadays I make use of a few tricks form handling tricks that allow me to re-use the same fields template for creating new objects and updating existing ones. Here’s how it works.

First of all I write a Mako template with the fields fragments called fields_fragment.mako:

## -*- coding: utf-8 -*-
<p>
    <label for="approvalgranted">Date Approval Granted<br /></label>
    <input type="text" name="approvalgranted" />
</p>
<p>
    <label for="note">Note<br /></label>
    <input type="text" name="note" />
</p>
<p>
    <label for="enteredby">Last edited by<br /></label>
    <input type=unknown name="enteredby" />
</p>

Then I create two templates to use the fields, one for adding a record, the other for updating it. Here is add.mako:

## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>

<h2>Add Requirement</h2>

${h.form(h.url(controller='studyapprovals', action='add', study_id=c.study_id), method='post')}
    ${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
    <p><input type="submit" value="Add &raquo;" name="go" class="button" /></p>
${h.end_form()}

and update.mako:

## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>

<h2>Update Requirement</h2>

${h.form(h.url(controller='studyapprovals', action='edit', id=c.id,  study_id=c.study_id), method='post')}
     ${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
    <p><input type="submit" value="Save" name="go" class="button" /></p>
${h.end_form()}

Notice the lines with ## -*- coding: utf-8 -*- specify the files are encoded with the UTF-8 chracterset which means I can use any Unicode characters I like in the templates. Also notice that both templates inherit from a file /base/index.mako which provides the bulk of the HTML for the page.

The important parts to draw your attention to are the lines:

<%namespace file="fields_fragment.mako" name="fields" import="*"/>

and:

${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}

The first is effectively an import of the shared fields_fragment.mako file as the namespace fields. I can then output the contents of that file by writing ${fields.body()} in the template to render the body of the fields_fragment.mako template but most of the time that’s not what I want to do.

Working with forms is mainly about handling the validation and then repopulating the form with the entered values and error messages. My controllers handle the validation and set c.values and c.errors so I can use formencode.htmlfill to populate the forms. The only complication is that in order to pass the value of calling fields.body() capture the render() function I have to capture it, calling it simply renders the output directly which isn’t what I’m after. This is why the call to capture is made.

Checkboxes are tricky in web applications because often you want a value of either True or False to be associated with a variable and you want to handle that with a checkbox which is either ticked or not but an unticked checkbox doesn’t submit a value at all which can be a pain for validation.

FormEncode can handle this situation like this:

>>> import formencode
>>>
>>> class MySchema(formencode.Schema):
...     ticked = formencode.validators.StringBoolean(if_missing=False)

You can then do:

>>> MySchema().to_python({})
{'ticked': False}
>>> MySchema().to_python({'ticked': 'true'})
{'ticked': True}

Both these states are handled fine by HTMLFill too:

>>> import formencode.htmlfill
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':False})
'<input type="checkbox" name="ticked" />'
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':True})
'<input type="checkbox" name="ticked" checked="checked" />'

Occasionally you run into a situation where if a checkbox is ticked you want to run some extra validation. For this you can use FormEncode’s RequireIfMissing or “ RequireIfPresent“ validators documented here.

Following on from my previous post about representing multiple subtopics as a comma separated list in a table field using a custom aggregate function in PostgreSQL, this post is about how to use FormEncode to validate a UI for the structure.

Our form consists of two fields: the first is the topic name and the second are the subtopics which are implemented as a load of checkboxes which the user can tick depending on which subtopics are relevant.

For the sake of this discussion the topic field is just a hidden field with the name of the topic although in real life it might be a dropdown select box which when changed, triggers an AJAX callback to load the appropriate sub-topic checkboxes.

The FormEncode schema looks like this:

class StudyTopic(formencode.Schema):
    allow_extra_fields = True
    filter_extra_fields = True

    topic = formencode.validators.String()
    subtopics = formencode.ForEach(formencode.validators.Int())

The important thing here is to notice the use of the ForEach validator which will apply the Int validator to each of the values submitted for the subtopics field.

If this was our HTML with Dementia and Parkinson’s checked:

<input type="hidden" name="topic" value="disease" />

<input type="checkbox" name="subtopics" value="1" checked="checked" />Dementia
<input type="checkbox" name="subtopics" value="2" />Huntington's Disease
<input type="checkbox" name="subtopics" value="3" />Motor neurone disease
<input type="checkbox" name="subtopics" value="4" checked="checked" />Parkinson's Disease

Then the values 1 and 4 would be submitted for the subtopics and the value disease would be submitted for the topic.

If you ran the values through the schema you would get this:

{'topic': 'disease', subtopics': [1,4]}

Say you later want to repopulate the same HTML with these values:

defaults = {'topic': 'disease', 'subtopics': [1,3]}
errors = {}

You can do so like this:

htmlfill.render(html, defaults=defaults, errors=errors)

This works fine because HTMLFill knows how to handle the multiple subtopics with lots of checkboxes of different values.

Now imagine a variation on this theme where rather than using ForEach to handle validation of any number of integers, you instead want to use it to have a repeating set of fields. Consider the Study schema below which uses ForEach to check any number of Person sets of fields:

class Person(Schema):
    firstname = String(not_empty=True)
    surname = String(not_empty=True)

class Study(Schema):
    allow_extra_fields = True
    filter_extra_fields = True
    pre_validators = [NestedVariables()]
    start_date = Date()
    people = ForEach(Person())

This time we are not just iterating over a single field, each set of Person fields contains both a firstname and a surname field. To handle this situation we need to name the fields which make up each Person according to the FormEncode nested variables specification.

Now when the form is submitted the NestedVariables pre-validator will decode the Person field names to produce a data structure that ForEach can validate. After validation you might get a data structure which looks like this:

{
    'start_date': date(2007,9,17),
    'people': [
        {'firstname': 'james', 'surname': 'gardner'},
        {'firstname': 'ian', 'surname': 'gardner'}
    ]
}

This is all very well but remember your form fields had to be named according to a certain specification in order for this to work so you have to do some extra work to get the values back into a format which HTMLFill can use. You do so like this:

defaults = variable_encode(defaults, add_repetitions=False)

You’ll also need to so the same to any errors:

errors = variable_encode(errors, add_repetitions=False)

Now you can use HTMLFill as normal:

return htmlfill.render(html, defaults=defaults, errors=errors)

This is a very handy trick if you want to use repeating sets of fields with FormEncode.

One question this begs though is whether you use the NestedVariables and variable_encode technique we’ve just used to validate the checkboxes used in the first example. The answer is "not easily".

The thing about the first example using checkboxes is that values are only submitted when the checkbox is ticked. This means only values which have been ticked will eventually get passed to HTMLFill. If you are using variable_encode HTMLFill will expect a value for each checkbox to be passed to it in the order the values should be set. Since there are missing values HTMLFill will not be able to match the ticked values to the names of the checkboxes except in the special case where you tick all the boxes up to a certain point and don’t tick any after that point in which case the submitted values happen to map correctly to the checkbox names.

You should therefore decide what sort of behaviour you want from ForEach, whether it is to validate a repeating set of fields, or to handle a checkbox group and then plan whether or not to use NestedVariables and variable_encode accordingly. Is this a limitation in FormEncode“ which needs fixing?