Manual

What is FormConvert?

• A set of ConversionKit converters for converting Unicode strings received from a form submission into Python objects

FormConvert can deal with the following types of form components:

• Form components which allow the user to input a single value per field name such as text fields, hidden fields and textareas
• Form components which allow the user to select a single value per field name from a set of possilbe options. For example a select field or radio button group
• Form components which allow the user to input zero or more values for a particular form field. For example multiple-value selects fields or checkbox groups.

FormConvert also contains tools to allow you to represent any data structure supported by the NestedRecords package. This allows you to represent extremely complex nested data structures (such as the ones you would naturally have to work with when using an SQL database) as HTML forms and have them seamlessly converted to Python data structures on submission.

FormConvert doesn’t care how the forms are generated, or even whether you are using an HTML form at all. It is really just a set of converters with a specific application to decoding data submitted over HTTP, but can be used in many other situations too. Many applications will install FormConvert just so they can re-use the ConversionKit converters it already defines.

In this documentation I’ll assume you are using FormConvert to handle HTML form submission. The examples will use FormBuild 2.0 to generate the HTML required but you could use any tool you preferred in your application.

A Trivial Example

Just so you can see the sorts of steps involved let’s go through a really trivial example.

To perform a conversion you use ConversionKit. ConversionKit is very generic and not much use on its own. Instead packages such as FormConvert, URLConvert, ConfigConvert, RecordConvert and NestedRecord build on ConversionKit to provide useful functionality in a specific problem domain.

To perform a conversion with ConversionKit or any of the tools built upon it involves four steps:

1. Create a Conversion object containing the value you wish to convert
2. Perform a conversion with a converter
3. Check if the conversion was successful
4. Read the result from the conversion object’s .result attribute if it was successful or the error from its .error attribute if it was not.

FormConvert is also closely tied to the RecordConvert and NestedRecord packages and is set up assuming you want to work with NestedRecord data structures in your application. You’ll see this later.

To generate our HTML fields we need some software. Install FormBuild (a tool which works well with FormConvert):

$ easy_install FormBuild>=2.0.1

Then let’s import some objects we need:

>>> from formbuild import Form
>>> from conversionkit import Conversion
>>> from stringconvert import stringToInteger

Now let’s create an HTML field with the name age:

>>> form = Form(values=dict(age='28'))
>>> form.text(name='age')
literal(u'<input id="age" name="age" type="text" value="28" />')

Imagine this form was submitted and you have to convert its value to to a Python integer. The web framework you are using should provide some interface to decode the HTTP post data or query string and provide a dictionary of Unicode values, keyed by the field names. If it doesn’t you can use a Request object provided by the WebOb package:

$ easy_install WebOb

Now let’s set up the object as if it is really handling a GET form submission by setting up a fake environment with the QUERY_STRING set up as it would be in a real example:

>>> from webob import Request
>>> fake_environ = {
...     'QUERY_STRING': 'age=3',
... }
>>> request = Request(fake_environ)
>>> request.params['age']
'3'

Now let’s actually perform the conversion, this is where ConversionKit comes in:

>>> conversion = Conversion(request.params['age'])
>>> conversion.perform(stringToInteger())
<conversionkit.Conversion object at 0x...>
>>> conversion.successful
True
>>> conversion.result
3
>>> result = Record(age = conversion.result)
>>> print result
{'age': 3}

As you can see the conversion was successful and the result was a Python integer. We then assembled a record to represent the form data submitted.

Tip

Before you continue it is strongly recommended you read the all the ConversionKit documentation, particularly the manual.

The Simple Cases

Now that you’ve seen how a conversion works when you convert each field manually and assemble the result into a record, let’s look at some more realistic cases.

Handling Multiple Fields

FormConvert is designed to work on sets of fields. If you are just converting one field you don’t need FormConvert you can just follow the trivial example in the previous section.

If you had to convert each field manually each time, using FormConvert would barely be quicker than writing your conversion code from scratch. This is where records come in.

In the simplest case FormConvert assumes that the form you are creating represents a row in a database table. One field name therefore corresponds to one field in the row. On top of this, FormConvert also assumes that you want to use the NestedRecord data model in your application and are therefore happy to restrict the field names you are using to those which are also valid Python variable names and which do not start with an _ character.

Tip

The beauty of FormConvert is that it is based on ConversionKit which means it can be used with any converters and not just the ones from RecordConvert. If you don’t want to work with RecordConvert records, and would rather use simple Python dictionaries and lists you can use the ConversionKit toDictionary() and toListOf() conversion factories instead. You might want to do this if you are using FormConvert in a legacy application or if you are trying to replace existing FormEncode schema which don’t have the same restrictions.

Having said that, if you are writing an application from scratch it is strongly recommended you use RecordConvert to deal with forms because it deliberately limits the combinations of data structures you can work with to just the ones which can be represented direclty in an SQL database and this in turn prevents you designing forms which are more complex than they need to be which in turn simplifies your application code and makes it more re-usable and maintainable because there are fewer combinations of cases to deal with.

To convert a set of fields you need to use a toRecord() converter. This is just like the toDictionary() converter which is documented in detail in the ConversionKit manual but it imposes restrictions on the names which can make up the keys of the dictionary. Now would be a good time to read the RecordConvert documentation if you haven’t done so already.

Let’s convert a form with fields for both name and age. You’ll need to write a converter capable of converting both of these fields. Luckily toRecord() can generate such a converter you if you tell it how you want each field converted. Here’s how you would create a suitable converter:

>>> from recordconvert import toRecord
>>> from stringconvert import stringToInteger, stringToString
>>>
>>> form_converter = toRecord(
...     converters = dict(
...         name=stringToString(min=3, max=30),
...         age=stringToInteger(),
...     )
... )

This creates a converter which will produce a Record object where the age field is an integer and the name field is a string with between 3 and 30 characters.

Notice that because all we are using records all the field names have to be valid Python names which means it is safe to use dict() rather than {} to generate a dictionary because the keys won’t contain values which can’t be passed as Python arguments.

Here’s a typical form submission from this form:

>>> fake_environ = {
...     'QUERY_STRING': 'age=28&name=James',
... }

Let’s imagine we are using WebOb in our application. First install WebOb:

$ easy_install WebOb

Then let’s create a dummy request using the WebOb pacakge:

>>> request = Request(fake_environ)

The form_converter object you’ve just created will expect a dictionary as the conversion’s value, not a WebOb request object so you need to use FormConvert’s multiDictToDict() converter to extract all the parameters:

>>> from formconvert import multiDictToDict
>>> from conversionkit import Conversion
>>>
>>> conversion = Conversion(request.params).perform(multiDictToDict)
>>> conversion.successful
True
>>> params = conversion.result
>>> params
{'age': '28', 'name': 'James'}

Now we can perform the conversion:

>>> conversion = Conversion(params).perform(form_converter)
<conversionkit.Conversion object at 0x...>
>>> conversion.successful
True
>>> conversion.result
{'age': 28, 'name': 'James'}

Notice that the age field has been converted to an integer.

Because the result is actually a Record object you can access the results as attributes:

>>> person = conversion.result
>>> person.name
'James'

Handling Empty Fields

You can set a default value for an empty field using the empty_errors argument to the toRecord() converter. For example, if the name is not allowed to be empty you could set up the converter like this:

>>> form_converter = toRecord(
...     converters = dict(
...         name=stringToString(min=3, max=30),
...         age=stringToInteger(),
...     ),
...     empty_errors = dict(
...         name = 'Please specify your name',
...     ),
... )

Let’s test it:

>>> fake_environ = {
...     'QUERY_STRING': 'age=28&name=',
... }
>>> request = Request(fake_environ)
>>> conversion = Conversion(request.params).perform(multiDictToDict)
>>> conversion.successful
True
>>> params = conversion.result
>>> params
{'age': '28', 'name': ''}
>>> conversion = Conversion(params).perform(form_converter)
<conversionkit.Conversion object at 0x...>
>>> conversion.successful
False
>>> conversion.error
'Please specify your name'

The toRecord() converter has a empty_defaults argument which allows you to specify default values for empty fields in a similar way.

Handling Missing Fields

Dealing with Checkboxes

Select Dropdowns and Radio Button Groups (Enums)

Form fields such as select dropdowns or radio groups allow a user to pick one value from a set of possible values. Your converter needs to ensure that the value submitted is one of the allowed values. You can do this with a oneOf() converter factory which is part of ConversionKit:

>>> from conversionkit import oneOf

Here’s an example:

>>> form = Form(values=dict(eye_colour='blue'))
>>> form.radio_group(name='eye_colour', options=[('blue', 'Blue'), ('brown', 'Brown'), ('green', 'Green')])
literal(u'<input type="radio" name="eye_colour" value="blue" checked /> Blue\n<input type="radio" name="eye_colour" value="brown" /> Brown\n<input type="radio" name="eye_colour" value="green" /> Green')
>>> from webob import Request
>>> fake_environ = {
...     'QUERY_STRING': 'eye_colour=blue',
... }
>>> request = Request(fake_environ)
>>> request.params['eye_colour']
'blue'
>>> conversion = Conversion(request.params['eye_colour'])
>>> conversion.perform(oneOf(['blue', 'brown', 'green']))
<conversionkit.Conversion object at 0x...>
>>> conversion.successful
True
>>> conversion.result
'blue'

If the use had submitted the value ‘grey’ the conversion would fail and you would get an error:

>>> fake_environ = {
...     'QUERY_STRING': 'eye_colour=grey',
... }
>>> request = Request(fake_environ)
>>> conversion = Conversion(request.params['eye_colour'])
>>> conversion.perform(oneOf(['blue', 'brown', 'green']))
<conversionkit.Conversion object at 0x...>
>>> conversion.successful
False
>>> conversion.error
'The value submitted is not one of the allowed values'

Handling Nested Structures

RecordConvert defines two classes: Record(), which behaves like a Python dictionary but supports attribute access to the values and places restricitons on the names you can use as keys, and ListOfRecords() which behaves like a Python lists but proxies attribute access to the first item in the list. The idea is that you model all internal data strucutres using only records and lists of records, no records with records as keys, no lists which don’t contain records and no lists which contain different types of records. Although this appears to make certain types of problem much harder, once you get used to writing the converters it actually makes everything tremendously simple.

In the NestedRecord data model there are no such thing as single values or lists of single values. The simplest object is a record and the simplest record is one with one key and one value:

{key: value}

The simplest list is an empty list:

[]

but after that all keys must be records. This means the next simplest list looks like this:

[{key: value}]

Records can have lists of records as their values so you can have data structures like this:

[{key: [{key: value}]}]

You can’t have any other data structure. No dictionaries with dictionaries for keys for example.

NestedRecord comes in to provide a way to represent the nested data structures of dictionaries and lists of dictionaries as a flattened data structure by using a convention on the key names. It turns out this convention is very useful when representing data as HTML forms because the HTTP protocol only supports key-value pairs of data.

Unfortunately there are a few edge cases where using a “pure” NestedRecord structure doesn’t work. In these cases FormConvert steps in to perform the necessary conversions. In this manual you’ll learn what the edge cases are and how to use FormConvert to solve them.

If you aren’t committed to use a NestedRecord data structure as the model for your application, FormConvert is unlikely to be a great deal of use to you though.

Handling Multiple Fields With The Same Name

Sometimes the form you are converting has multiple fields with the same name but the web framework you are using might not always pass those on to ConversionKit. As an example look what happens with WebOb when you submit two fields with the same name:

>>> fake_environ = {
...     'QUERY_STRING': 'age=28&eye_colour=blue&eye_colour=green',
... }
>>> request = Request(fake_environ)
>>> print dict(request.params)
{'age': '28', 'eye_colour': 'green'}

Notice that only the second eye colour appears in the result. If you want to access all the values for eye colour you do so like this:

>>> request.params.getall('eye_colour')
['blue', 'green']

Here’s a function for extracing all the data:

>>> def params_to_dict(params):
...     result = {}
...     for k in params.keys():
...         v = params.getall(k)
...         if len(v) == 1:
...             result[k] = v[0]
...         else:
...             result[k] = v
...     return result
>>> params_to_dict(request.params)
{'age': '28', 'eye_colour': ['blue', 'green']}

Although the eye_colour is not correctly a list object and could be converted using a converter generated by the toListOf() converter factory, a better approach for handling repeating items if by their ID using a the Repeatable Nested Records encoding described in the RecordConvert documenation. Let’s look at this in the next sections.

Introducing The Repeatable Nested Relationships Encoding

The RecordConvert documentation defines a Repeatable Nested Relationships encoding which allows nested data structures to be represented as a single set of key value pairs. This turns out to be very useful for handling HTML forms.

The encoding is described in detail in the RecordConvert documentation.

Handling FieldSets

Sometimes you want a form with fields from more than one database table. If the field names from each table were different you could get away with having them all in the same form but if some of the columns have the same names you need a more sophisticated approach.

In the application you will want to deal with two sets of records so the best thing to do is give each record a name and encode them. The encoded field names can then be used in the form and then decded into the two records once the data is submitted. Here’s an example of encoding the data for use in the form. We only show the field for the address firstname, but notice how the field name for the input field uses the encoded key name:

>>> from recordconvert import Record, ListOfRecords
>>> from nestedrecord import encode
>>> account = Record(
...     username='jim',
...     password=123456,
... )
>>> address1=Record(
...     number = 12,
...     city = 'London',
... )
>>> data = Record(
...     address=ListOfRecords(address1),
...     account=ListOfRecords(account),
... )
>>> encoded_data = encode(data)
>>> print encoded_data
{'address[0].number': 12, 'account[0].password': 123456, 'address[0].city': 'London', 'account[0].username': 'jim'}

Imagine the data was submitted from an HTML form. We could use a pre-converter from FromConvert to extract the data directly from the request to convert it into a dictionary. First install WebOb:

$ easy_install WebOb

Then let’s create a dummy request using the WebOb pacakge:

>>> import StringIO
>>> # Should really encode v below but the strings we've chosen are safe anyway
>>> query_string = '&'.join(['%s=%s'%(k,v) for k, v in encoded_data.items()])
>>> print query_string
address[0].number=12&account[0].password=123456&address[0].city=London&account[0].username=jim
>>> fake_environ = {
...     'QUERY_STRING': query_string,
...     'wsgi.url_scheme': 'http',
...     'SERVER_NAME': 'FakeServer',
...     'SERVER_PORT': '80',
...     'wsgi.input': StringIO.StringIO(),
... }

The dictionary produced after accessing the params from the WebOb Request API looks like this:

>>> request = Request(fake_environ)
>>> print request.params
NestedMultiDict([('address[0].number', '12'), ('account[0].password', '123456'), ('address[0].city', 'London'), ('account[0].username', 'jim')])

Now let’s use the request in a pre-validator:

>>> from formconvert import multiDictToDict
>>> from conversionkit import chainConverters
>>> contact_to_dictionary = chainConverters(
...     multiDictToDict(),
...     splitName('name'),
...     toDictionary(
...         converters={
...             'firstname': noConversion(),
...             'lastname': noConversion(),
...             'email': StringToEmail(),
...         }
...     )
... )




>>> form = Form(values=encoded_data)
>>> print form.text(name='address.number')
<input id="addressnumber" name="address.number" type="text" />

Let’s pass the entire request as the argument to convert:

>>> print Conversion(request.GET).perform(contact_to_dictionary).result
{'lastname': 'Garnder', 'firstname': 'James', 'email': u'james@example.com'}

As you can see, the MultiDictToDict() pre-converter is called to produce a dictionary from the request the SplitName('name') converter is called.

After you’ve learned about post-converters we’ll come back to this example to discuss when it is best to use pre- and post-handlers and when it is best to use the chainConverters tool you learned about earlier.

You can learn more about how the FromConvert and RecordConvert packages help with web development in their own documentation.

One-to-One Relationships

• Builds a record inside a record

One-to-Many Relationships

• Builds a record with a list of records as one of the keys

Many-to-Many Relationships

A many-to-many relationship is when one or more instances of one entity are related to zero or more instances of another. For example, consider the relationship in a wiki between pages and tags. A tag can appear or many pages and a page can have many different tags.

When dealing with forms though, you only ever deal with one entity at once. For example you are either editing a page and changing its tags or editing the tag to change which pages it is on. You are never dealing with tags and pages at the same time. This means that handling a many to many mapping in a form is exaclty the same as handling a one-to-many mapping from the point of view of the entity being changed. The only difference is that with a many-to-many mapping you will need two forms, one to handle each entity.

If you think about how you would implement a many-to-many mapping in a relational database, the reasoning becomes clear. You actually introduce a third table and each entitiy has a one to many relationship with the third table which is why a many-to-many mapping can be decomposed into two one-to-many mappings for the purposes of form handling.

Handling Errors

Errors are a little different from data structures. There can be an error associated with any field as well as for every record and list of records. The encoding for errors reflects this.

The encode_error() function

encode_error()

Takes a nested conversion and encodes any errors into a KERNS-like format where keys representing an overall error at a particular depth are also allowed.

Here are some examples of the encode_error() function in action on the conversions performed earlier in the chapter.

>>> from nestedrecord import encode_error
>>> error1 = Conversion(case1).perform(case1_converter)
>>> encode_error(error1)
{'key-0.key': "invalid literal for int() with base 10: 'value'", 'key-1.key': "invalid literal for int() with base 10: 'value'", 'key-1': 'The key field is invalid', 'key-0': 'The key field is invalid', 'key': 'Some of the items were not valid'}
>>> encode_error(error2)
{'key.key': "invalid literal for int() with base 10: 'value'", 'key': 'The key field is invalid'}
>>> encode_error(error3)
Traceback (most recent call last):
  ...
ConversionKitError: Standlone lists are not supported by kerns format
>>> encode_error(error4)
Traceback (most recent call last):
  ...
ConversionKitError: Standlone lists are not supported by kerns format
>>> encode_error(error_simple)
{'key': "invalid literal for int() with base 10: 'value'"}

Handling the Difficult Cases

Handling Image Buttons

So far this example hasn’t been very interesting because most of the steps had to be performed manually, we manually extracted the age variable from the request params, manually passed it to a stringToInteger() converter and then manually assembled the Record object we want to work with in the application. In more complex cases we might also have to:

• Strip the .x and .y values from names submitted by most browsers when you use an image button to submit a form
• Decode a NestedRecord data structure
• Treat certain fields as being IDs of other Records

We can simplify the process with FormConvert. First let’s see an example where the

>>> from formconvert import multiDictToDict, formDecode
>>> from conversionkit import Conversion, chainConverters
>>> conversion = Conversion(request.params).perform(chainConverters(multiDictToDict, formDecode))
>>> if conversion.successful:
...     print conversion.result
{'age': '3'}

Handling Multiple Fields With The Same Name

Handling Checkbox Groups

Handling Combo Boxes