AuthKit 0.4.0
I’m pleased to announce the release of AuthKit 0.4.0. It has taken me a lot longer than it should have but it is out. You can download AuthKit here. Please note that the config file format has changed a bit since 0.3 so have a look at the Pylons Book chapters to read about the new version.
Please let me know about any bugs and I’ll release a 0.4.1 if necessary.
Skype 1.4 (Beta) on Debian Linux
First install some dependencies:
$ sudo apt-get install libqt4-core libqt4-gui
Then download Skype from http://www.skype.com/download/skype/linux/ and install it like this:
$ sudo dpkg --install skype-debian_1.4.0.99-1_i386.deb
If all goes well you can run Skype like this:
$ skype
You’ll see a license agreement which you have to accept and then the Skype application.
If you don’t have an account you’ll need to sign up, otherwise enter your username and password and you are good to go.
If you have any problems check the terminal window you loaded Skype in for error messages. It is also worth loading the Gnome Sound Recorder and trying to record from the microphone to check your system is correctly recording and playing back sounds:
$ gnome-sound-recorder
Again, check the terminal window for any error messages. If sound recording isn’t working you should load the Gnome Volume Control application to ensure that your microphone is on and that the volume on it is turned up. You can load it with Applications->Sound and Video->Volume Control.
I found I had to change devices to use the OSS mixer before Skype worked. You do this in the Volume Control application’s menu. Chose File->Change Device.
Once sound is working correctly in Gnome, it is likely to work in Skype.
Multiple File Uploads with Progress Meters using Flash 8 and JavaScript
There are two problems with uploading files over the web using a simple HTML Form
- You can only upload one file at once
- You don’t get any idea of the upload progress
There are ways of solving these problems using AJAX and a hidden iframe (I might write one up if I get a chance) but Flash 8 provides a better solution because it allows you to select multiple files from a single dialog box so that if you want to upload a hole directory you don’t have to select each of the files individually. Flash 8 can also keep track of the upload progress on the client side so you don’t need to implement any server-side feedback system as the file is being uploaded. You can also specify a maximum file size.
There are a number of open source implementations which expose this Flash 8 functionality via JavaScript. The one I use is SWFUpload.
If you follow on from my last upload example you can replace the index() action with this:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head>
<script type="text/javascript" src="jscripts/SWFUpload/SWFUpload.js"></script>
<script type="text/javascript" src="jscripts/example_callbacks.js"></script>
<script type="text/javascript">
var swfu;
var swfu2;
window.onload = function() {
// Max settings
swfu = new SWFUpload({
upload_script : "/up/upload",
target : "SWFUploadTarget",
flash_path : "jscripts/SWFUpload/SWFUpload.swf",
allowed_filesize : 3072000, // 3000 MB
allowed_filetypes : "*.*",
allowed_filetypes_description : "All files...",
browse_link_innerhtml : "Browse",
upload_link_innerhtml : "Upload queue",
browse_link_class : "swfuploadbtn browsebtn",
upload_link_class : "swfuploadbtn uploadbtn",
flash_loaded_callback : 'swfu.flashLoaded',
upload_file_queued_callback : "fileQueued",
upload_file_start_callback : 'uploadFileStart',
upload_progress_callback : 'uploadProgress',
upload_file_complete_callback : 'uploadFileComplete',
upload_file_cancel_callback : 'uploadFileCancelled',
upload_queue_complete_callback : 'uploadQueueComplete',
upload_error_callback : 'uploadError',
upload_cancel_callback : 'uploadCancel',
auto_upload : false
});
}
</script>
<style type="text/css">
.swfuploadbtn {
display: block;
width: 100px;
padding: 0 0 0 20px;
}
.browsebtn { background: url(/images/add.png) no-repeat 0 4px; }
.uploadbtn {
display: none;
background: url(/images/accept.png) no-repeat 0 4px;
}
.cancelbtn {
display: block;
width: 16px;
height: 16px;
float: right;
background: url(/images/cancel.png) no-repeat;
}
#cancelqueuebtn {
display: block;
display: none;
background: url(/images/cancel.png) no-repeat 0 4px;
margin: 10px 0;
}
#SWFUploadFileListingFiles ul {
margin: 0;
padding: 0;
list-style: none;
}
.SWFUploadFileItem {
display: block;
width: 230px;
height: 70px;
float: left;
background: #eaefea;
margin: 0 10px 10px 0;
padding: 5px;
}
.fileUploading { background: #fee727; }
.uploadCompleted { background: #d2fa7c; }
.uploadCancelled { background: #f77c7c; }
.uploadCompleted .cancelbtn, .uploadCancelled .cancelbtn {
display: none;
}
span.progressBar {
width: 200px;
display: block;
font-size: 10px;
height: 4px;
margin-top: 2px;
margin-bottom: 10px;
background-color: #CCC;
}
</style>
</head>
<body>
<h1><span>SWFUpload Testpage</span></h1>
<h2>Upload Example</h2>
<p>Replace contents of a div with links for uploading and browsing,
degrades gracefully if flash/javascript isn't accessible</p>
<div id="SWFUploadTarget">
<form action="upload.php" method="post" enctype="multipart/form-data">
<input type="file" name="Filedata" id="Filedata" />
<input type="submit" value="upload test" />
</form>
</div>
<h4 id="queueinfo">Queue is empty</h4>
<div id="SWFUploadFileListingFiles"></div>
<br class="clr" />
<a class="swfuploadbtn" id="cancelqueuebtn"
href="javascript:cancelQueue();">Cancel queue</a>
</body>
</html>
You’ll also need to download the SWFUpload source distribution and copy the jscripts directory into your Pylons project public directory so that the /jscripts/ links in the HTML work correctly. The example also uses some images which are attached to this entry and should be put in the public/images directory so that the /images/ URLs resolve.
Accept icon Add icon Cancel icon Progress bar
If you have Flash 8 or above and run this example you’ll now be able to upload multiple files to the unmodified controller. You can change the CSS and the images to change how the upload behaves and there are plenty of hooks for you to tie what’s happening in Flash to your page’s JavaScript.
File Uploads in Python
File uploads are one of those things in Python which are still rather tricky to handle. First of all you need a form like this one with enctype="multipart/form-data" and a file input field:
<form action="up" method="post" enctype="multipart/form-data">
Upload file: <input type="file" name="myfile" /> <br />
<input type="submit" name="submit" value="Submit" />
</form>
In your Python code you can then get the uploaded file data like this:
import cgi form_data = cgi.FieldStorage() file_data = form_data['myfile'].value
and you can write it somewhere like this:
fp =open('some/file','wb')
fp.write(file_data)
fp.close()
This is all well and good but what if you want to stream that data to a service such as Amazon S3 or what if you want to provide feedback to the user about how much of the file has been uploaded? The example here can’t help with this because you can only access the data once the whole file is uploaded.
Here’s how you can solve this problem. It is a bit of a nasty solution because you need to create your own file class which calls your callback. You also need your own FieldStorage class which has its make_file() method overridden to use the open file object you supply instead of the tempfile it would use by default. It also only works with forms with one file field but it demonstrates the principles on which you can build your own solution.
Here is a Pylons controller using this system:
import logging
from upload.lib.base import *
log = logging.getLogger(__name__)
import os
import shutil
import cgi
class ProgressFile(file):
def write(self, *k, **p):
if hasattr(self, 'callback'):
self.callback(self, *k, **p)
return file.write(self, *k,**p)
def set_callback(self, callback):
self.callback = callback
def stream(file_object):
class CustomFieldStorage(cgi.FieldStorage):
def make_file(self, binary=None):
self.open_file = file_object
return self.open_file
return CustomFieldStorage
class UpController(BaseController):
def index(self):
return """
<html>
<body>
<h1>Upload</h1>
<form action="up" method="post" enctype="multipart/form-data">
Upload file: <input type="file" name="myfile" /> <br />
<input type="submit" name="submit" value="Submit" />
</form>
</body>
</html>
"""
def upload(self):
def callback(file, *k, **p):
log.debug("Logged %s", [file.tell()])
fp = ProgressFile('somefile', 'wb')
fp.set_callback(callback)
custom_field_storage = stream(fp)(
environ=request.environ,
strict_parsing=True,
fp=request.environ['wsgi.input']
)
fp.close()
return 'done'
As the file is uploaded it will now get streamed to the open some/file object as required and the calback() function gets called on every write so that you can find out how much data has been written with file.tell(). If you try this you will see the file uploads fine and you receive the done message. The output logs then look something like this as each write() call is logged:
13:36:22,447 DEBUG [upload.controllers.up] Logged [30011568L] 13:36:22,448 DEBUG [upload.controllers.up] Logged [30011619L] 13:36:22,448 DEBUG [upload.controllers.up] Logged [30011661L] 13:36:22,448 DEBUG [upload.controllers.up] Logged [30011662L] 13:36:22,448 DEBUG [upload.controllers.up] Logged [30011702L] 13:36:22,473 DEBUG [upload.controllers.up] Logged [30011762L] 13:36:22,474 DEBUG [upload.controllers.up] Logged [30011812L] 13:36:22,474 DEBUG [upload.controllers.up] Logged [30011876L] 13:36:22,474 DEBUG [upload.controllers.up] Logged [30011942L] 13:36:22,474 DEBUG [upload.controllers.up] Logged [30012006L] 13:36:22,475 DEBUG [upload.controllers.up] Logged [30012048L] 13:36:22,475 DEBUG [upload.controllers.up] Logged [30012049L]
Converting Word RTF to PDF on Debian
This is actually very easy. First install Ted, an old UNIX text editor with this command:
$ sudo apt-get install ted
You’ll also need Ghostscript but this is probably already installed, if not run:
$ sudo apt-get install gs-afpl
Then download the rtf2pdf.sh script from ftp://ftp.nluug.nl/pub/editors/ted/rtf2pdf.sh and run this command:
$ sh rtf2pdf.sh file.rtf output.pdf
and that’s it, very easy and it actually works. I’ve attached a text version of the rtf2pdf.sh script here as well.
Here’s a very short program which uses this from Python:
import os
def convert(rtf, pdf):
pipe = os.popen("sh rtf2pdf.sh %s %s"%(rtf, pdf), "r")
error = pipe.read()
if error:
print "ERROR. An error occurred here is the description..."
print
print error
if __name__ == '__main__':
convert('test.rtf', 'test.pdf')
IE7 produces different HTTP Digest headers
Rohan Barnett has been pointing out a bug in AuthKit he has experienced for quite a while. For some reason AuthKit’s digest implementation worked in every browser apart from IE7. All that happens is that IE keeps showing you the pop to enter your username and password.
Today I finally tracked down the problem. IE7’s headers look like this:
Digest username="asd",realm="Test Realm",nonce="b99bd0d4c8c5f9d0448910ea6ab5be28",uri="/private",cnonce="9e1a7dc289b9ce26090ae88cb5433cff",nc=00000005,response="1709e5d0616f43a63c3c77b0866ef3d3",qop="auth",opaque="b41c8bf75355adb523d1b05f8f2707d8"
Every other browser’s look like something like this with spaces after the commas:
Digest username="asd", realm="Test Realm", nonce="7f05810618404280262353bb51bd998c", uri="/private", response="e8e97e5d335af696ab3dd169da02b7b8", opaque="d0837206408ac92195012fb0ce233e91", qop=auth, nc=00000002, cnonce="9d6a68265679ac27"
The AuthKit code was splitting on ", " not on ",". I’ve corrected this, fixing ticket #31 and it now works but I thought I’d blog about it here in case anyone else experiences the same issue.
Ideal Email Client
I’m fed up with the way I currently handle my email and am about to look for some better solutions. Before I do I want to make a note of the problems I’m facing and what I think the ideal solution would be so that when I get some free time sometime next year I can start building a platform that meets my needs (and therefore probably everyone else’s too).
- I’m subscribed to about 20 mailing lists and so get hundreds of emails a day. These are filtered into various folders and every now and again I check each to see if there is anything relevant to me. This ends up in me missing lots of important emails or not getting a chance to answer a pertinent question until lots of other people have chipped in and confused the issue. As a result some people just cc me into emails sent to lists. This doesn’t help either because then when faced with a huge inbox I can’t instantly tell who is writing to me directly and who has just cc’d me int an email which probably isn’t such a high priority. The result is that I miss import ant emails directly from people who also contribute to the mailing list communities.
- Emails don’t normally fit nicely into one category so putting them in folders is really very little use. Instead tags are needed where emails can be tagged according to various criteria. Tags themselves can be grouped so that related tags appear together in the interface. When you receive an email you can then interface wise you just click on the various tags related to that email and they can fly off the tag list onto the email. You can also drag and drop them to change how you want to categorize the tags. The main interface to the emails then isn’t the date or the sender but the tag.
- I work with the same people on different projects so there is no easy way to categorize messages according to the project. What is needed is a way for people to categorize the emails they are sending. One way would be to send emails to different addresses depending on the project. Better would be to provide the same tag interface for sending as used for taging mails in the inbox then the recipient can also use those tags to categorize the email.
- When looking through emails three processes are happening: prioritizing, categorizing and dealing with. Interfaces need to be able to deal with those three different modes of operation.
- Emails need to be accessible via a web interface as well as via POP and IMAP. The web interface needs to minimise bandwidth so the email can be browsed over a crap connection at a client. It should also support offline access via something like Google Gears.
- Emails from people you haven’t heard from for ages need highlighting so that you don’t miss them because when scanning through your inbox unfamiliar names can easily missed. This means the email client needs built-in contacts lists supporting tagging.
- People sometimes just cc you into emails so that they don’t have to explain what’s actually going on directly to you. This isn’t helpful because you can rarely pick up the deeper politics behind a situation from email cc’s and you end up treating all emails from the related parties with the same lack of interest when occasionally the emails are really important and relevant to you.
- You need to be able to select which email address the email appears to have come from even if it is from a different account. The default should be the email address the email was sent to.
OK, that’s the rant over.. better get back to dealing with some emails…
Re-Using Fields in a Pylons, Mako and FormEncode Workflow
In most of my Pylons apps nowadays I make use of a few tricks form handling tricks that allow me to re-use the same fields template for creating new objects and updating existing ones. Here’s how it works.
First of all I write a Mako template with the fields fragments called fields_fragment.mako:
## -*- coding: utf-8 -*-
<p>
<label for="approvalgranted">Date Approval Granted<br /></label>
<input type="text" name="approvalgranted" />
</p>
<p>
<label for="note">Note<br /></label>
<input type="text" name="note" />
</p>
<p>
<label for="enteredby">Last edited by<br /></label>
<input type=unknown name="enteredby" />
</p>
Then I create two templates to use the fields, one for adding a record, the other for updating it. Here is add.mako:
## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>
<h2>Add Requirement</h2>
${h.form(h.url(controller='studyapprovals', action='add', study_id=c.study_id), method='post')}
${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
<p><input type="submit" value="Add »" name="go" class="button" /></p>
${h.end_form()}
and update.mako:
## -*- coding: utf-8 -*-
<%inherit file="/base/index.mako" />
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
<!% import formencode.htmlfill %>
<h2>Update Requirement</h2>
${h.form(h.url(controller='studyapprovals', action='edit', id=c.id, study_id=c.study_id), method='post')}
${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
<p><input type="submit" value="Save" name="go" class="button" /></p>
${h.end_form()}
Notice the lines with ## -*- coding: utf-8 -*- specify the files are encoded with the UTF-8 chracterset which means I can use any Unicode characters I like in the templates. Also notice that both templates inherit from a file /base/index.mako which provides the bulk of the HTML for the page.
The important parts to draw your attention to are the lines:
<%namespace file="fields_fragment.mako" name="fields" import="*"/>
and:
${formencode.htmlfill.render(capture(fields.body), c.values, c.errors)}
The first is effectively an import of the shared fields_fragment.mako file as the namespace fields. I can then output the contents of that file by writing ${fields.body()} in the template to render the body of the fields_fragment.mako template but most of the time that’s not what I want to do.
Working with forms is mainly about handling the validation and then repopulating the form with the entered values and error messages. My controllers handle the validation and set c.values and c.errors so I can use formencode.htmlfill to populate the forms. The only complication is that in order to pass the value of calling fields.body() capture the render() function I have to capture it, calling it simply renders the output directly which isn’t what I’m after. This is why the call to capture is made.
Handling a Checkbox in FormEncode
Checkboxes are tricky in web applications because often you want a value of either True or False to be associated with a variable and you want to handle that with a checkbox which is either ticked or not but an unticked checkbox doesn’t submit a value at all which can be a pain for validation.
FormEncode can handle this situation like this:
>>> import formencode >>> >>> class MySchema(formencode.Schema): ... ticked = formencode.validators.StringBoolean(if_missing=False)
You can then do:
>>> MySchema().to_python({})
{'ticked': False}
>>> MySchema().to_python({'ticked': 'true'})
{'ticked': True}
Both these states are handled fine by HTMLFill too:
>>> import formencode.htmlfill
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':False})
'<input type="checkbox" name="ticked" />'
>>> formencode.htmlfill.render('''<input type="checkbox" name="ticked" />''', {'ticked':True})
'<input type="checkbox" name="ticked" checked="checked" />'
Occasionally you run into a situation where if a checkbox is ticked you want to run some extra validation. For this you can use FormEncode’s RequireIfMissing or “ RequireIfPresent“ validators documented here.
Multiple Checkboxes With FormEncode
Following on from my previous post about representing multiple subtopics as a comma separated list in a table field using a custom aggregate function in PostgreSQL, this post is about how to use FormEncode to validate a UI for the structure.
Our form consists of two fields: the first is the topic name and the second are the subtopics which are implemented as a load of checkboxes which the user can tick depending on which subtopics are relevant.
For the sake of this discussion the topic field is just a hidden field with the name of the topic although in real life it might be a dropdown select box which when changed, triggers an AJAX callback to load the appropriate sub-topic checkboxes.
The FormEncode schema looks like this:
class StudyTopic(formencode.Schema):
allow_extra_fields = True
filter_extra_fields = True
topic = formencode.validators.String()
subtopics = formencode.ForEach(formencode.validators.Int())
The important thing here is to notice the use of the ForEach validator which will apply the Int validator to each of the values submitted for the subtopics field.
If this was our HTML with Dementia and Parkinson’s checked:
<input type="hidden" name="topic" value="disease" /> <input type="checkbox" name="subtopics" value="1" checked="checked" />Dementia <input type="checkbox" name="subtopics" value="2" />Huntington's Disease <input type="checkbox" name="subtopics" value="3" />Motor neurone disease <input type="checkbox" name="subtopics" value="4" checked="checked" />Parkinson's Disease
Then the values 1 and 4 would be submitted for the subtopics and the value disease would be submitted for the topic.
If you ran the values through the schema you would get this:
{'topic': 'disease', subtopics': [1,4]}
Say you later want to repopulate the same HTML with these values:
defaults = {'topic': 'disease', 'subtopics': [1,3]}
errors = {}
You can do so like this:
htmlfill.render(html, defaults=defaults, errors=errors)
This works fine because HTMLFill knows how to handle the multiple subtopics with lots of checkboxes of different values.
Now imagine a variation on this theme where rather than using ForEach to handle validation of any number of integers, you instead want to use it to have a repeating set of fields. Consider the Study schema below which uses ForEach to check any number of Person sets of fields:
class Person(Schema):
firstname = String(not_empty=True)
surname = String(not_empty=True)
class Study(Schema):
allow_extra_fields = True
filter_extra_fields = True
pre_validators = [NestedVariables()]
start_date = Date()
people = ForEach(Person())
This time we are not just iterating over a single field, each set of Person fields contains both a firstname and a surname field. To handle this situation we need to name the fields which make up each Person according to the FormEncode nested variables specification.
Now when the form is submitted the NestedVariables pre-validator will decode the Person field names to produce a data structure that ForEach can validate. After validation you might get a data structure which looks like this:
{
'start_date': date(2007,9,17),
'people': [
{'firstname': 'james', 'surname': 'gardner'},
{'firstname': 'ian', 'surname': 'gardner'}
]
}
This is all very well but remember your form fields had to be named according to a certain specification in order for this to work so you have to do some extra work to get the values back into a format which HTMLFill can use. You do so like this:
defaults = variable_encode(defaults, add_repetitions=False)
You’ll also need to so the same to any errors:
errors = variable_encode(errors, add_repetitions=False)
Now you can use HTMLFill as normal:
return htmlfill.render(html, defaults=defaults, errors=errors)
This is a very handy trick if you want to use repeating sets of fields with FormEncode.
One question this begs though is whether you use the NestedVariables and variable_encode technique we’ve just used to validate the checkboxes used in the first example. The answer is "not easily".
The thing about the first example using checkboxes is that values are only submitted when the checkbox is ticked. This means only values which have been ticked will eventually get passed to HTMLFill. If you are using variable_encode HTMLFill will expect a value for each checkbox to be passed to it in the order the values should be set. Since there are missing values HTMLFill will not be able to match the ticked values to the names of the checkboxes except in the special case where you tick all the boxes up to a certain point and don’t tick any after that point in which case the submitted values happen to map correctly to the checkbox names.
You should therefore decide what sort of behaviour you want from ForEach, whether it is to validate a repeating set of fields, or to handle a checkbox group and then plan whether or not to use NestedVariables and variable_encode accordingly. Is this a limitation in FormEncode“ which needs fixing?