Manual

Introduction

I have about 30 packages I maintain and it can become a pain to manually build test and release each one, creating the documentation and uploading it to a webiste. BuildKit helps automate this process for packages set up in a certain way.

Creating a package

Edit the new_package.py script in the example directory of the source distribution to set appropriate variables at the bottom. Then run it like this to create a package in the current directory:

$ ~/env/bin/python new_package.py

The package generated includes support for eggs and has Sphinx documentation built in.

Tip

The script above was originally created from a directory structure using TemplateGenerator.

Now edit the variables at the bottom of the package_to_repo.py script in the example directroy and run it to convert the package to the format expected by BuildKit and to turn it into a mercurial repository:

$ ~/env/bin/python package_to_repo.py

Setting a website theme

By default the theme for Sphinx to use is called default but you change this to another theme by changing the THEME variable in new_package.py before you create your pakage or by changing the html_theme variable in docs/source/conf.py afterwards.

Building

Here are some example of buildkit’s use once a package in the structure generated by package_to_repo.py has been created.

My setup involves all the packages being kept in the same base directory along with two other folders:

eggs

The .egg or .tar.gz files representing distributable files from each of the packages

eggs_3rd_party

A directory containing third party eggs which the packages depend on. Importantly, this directory must contain the virtualenv.py script from http://pylonsbook.com/virtualenv.py and the setuptools-0.6c11-py2.6.egg or equivalent for the version of Python you want to use from http://pypi.python.org/pypi/setuptools/0.6c11. You’ll need pip-0.6.1.tar.gz too. You should download these now if you don’t already have them.

By keeping these two directories it is possible to build and test each package entirely from the distributable files on your local machine without any need for an internet connection. This means that you can be confident that your applications can be installed using only the files from the two directories.

Let’s set up a directory strucutre for testing:

>>> import os
>>> import buildkit
>>> test_dir = os.getcwd()

Let’s import the buildkit module and set up the base variable to point to the base directory in test:

>>> base = os.path.join(test_dir, 'base')

Let’s create a new package called ExamplePackage in the base directory, removing it first if it already exists:

>>> import sys
>>> python = sys.executable
>>> if os.path.exists(os.path.join(base, 'ExamplePackage')):
...     buildkit.run(
...         ['rm', '-rf', 'ExamplePackage'],
...         cwd=base,
...         surpress_stdout=True,
...         surpress_stderr=True,
...     )
>>> example_dir = os.path.abspath(os.path.join(base, '..', '..', 'example'))
>>> buildkit.run(
...     [python, 'new_package.py'],
...     cwd=example_dir,
...     surpress_stdout=True,
...     surpress_stderr=True,
... )
>>> buildkit.run(
...     [python, 'package_to_repo.py'],
...     cwd=example_dir,
...     surpress_stdout=True,
...     surpress_stderr=True,
... )
>>> buildkit.run(
...     ['mv', 'ExamplePackage', '../test/base'],
...     cwd=example_dir,
...     surpress_stdout=True,
...     surpress_stderr=True,
... )

We also need the virtualenv.py and setuptools-0.6c11-py2.6.egg files.

>>> egg_path = os.path.join(base, 'eggs_3rd_party', 'setuptools-0.6c11-py2.6.egg')
>>> if not os.path.exists(egg_path):
...     buildkit.run(
...         ['wget', 'http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg'],
...         cwd=os.path.join(base, 'eggs_3rd_party'),
...         surpress_stdout=True,
...         surpress_stderr=True,
...     )
>>> pip_path = os.path.join(base, 'eggs_3rd_party', 'pip-0.6.1.tar.gz')
>>> if not os.path.exists(pip_path):
...     buildkit.run(
...         ['wget', 'http://pypi.python.org/packages/source/p/pip/pip-0.6.1.tar.gz'],
...         cwd=os.path.join(base, 'eggs_3rd_party'),
...         surpress_stdout=True,
...         surpress_stderr=True,
...     )
>>> virtualenv_path = os.path.join(base, 'eggs_3rd_party', 'virtualenv.py')
>>> if not os.path.exists(virtualenv_path):
...     buildkit.run(
...         ['wget', 'http://pylonsbook.com/virtualenv.py'],
...         cwd=os.path.join(base, 'eggs_3rd_party'),
...         surpress_stdout=True,
...         surpress_stderr=True,
...     )

Now we can test some of the ways BuildKit can be used.

Get the status of a module:

>>> buildkit.status(os.path.join(base, 'ExamplePackage'))

Pull in the latest updates and update:

>>> # Code below doesn't work yet, needs pexpect:
>>> # buildkit.pull(os.path.join(base, 'ExamplePackage'))
>>> # buildkit.update(os.path.join(base, 'ExamplePackage'))

Note: The buildkit.update() function currently fails if a username and password are required.

Remove the build and dist directories from the ExamplePackage package:

>>> buildkit.remove_dist(os.path.join(base, 'ExamplePackage'))

Build the .egg and .tar.gz files to be distributed:

>>> buildkit.build_dist(
...     path=os.path.join(base, 'ExamplePackage'),
...     third_party_eggs_path=os.path.join(base, 'eggs_3rd_party'),
...     python='python2.6',
...     setuptools_version='setuptools-0.6c11-py2.6.egg',
... )
Copying .../setuptools-0.6c11-py2.6.egg to '.../base/ExamplePackage/trunk'
Building in .../base/ExamplePackage/trunk

Create a virtual Python environment, installing the egg from the .tar.gz distribution and run the tests:

>>> buildkit.test_dist(
...     requirement='ExamplePackage==0.1.0',
...     path=os.path.join(base, 'ExamplePackage'),
...     eggs_path=os.path.join(base, 'eggs'),
...     third_party_eggs_path=os.path.join(base, 'eggs_3rd_party'),
...     python='python2.6',
...     test_requirement=None,
... )
Creating virtual environment in .../test/base/ExamplePackage/trunk/dist
Installing eggs
Extracting source
Running tests in '...'

Search the directory base for any .tar.gz files and copy them to the eggs` directory:

>>> buildkit.prepare_sources(base, os.path.join(base, 'eggs'))
Copying .../ExamplePackage-0.1.0.tar.gz to .../test/base/eggs/ExamplePackage-0.1.0.tar.gz
Copying .../test/base/eggs_3rd_party/pip-0.6.1.tar.gz to .../test/base/eggs/pip-0.6.1.tar.gz

Perform the remove_dist(), build_dist() and test_dist() steps in one go:

>>> buildkit.handle_one(
...     os.path.join(base, 'ExamplePackage'),
...     'ExamplePackage', # Or you can use full requirements like 'ExamplePackage==0.1.0' here
...     eggs_path=os.path.join(base, 'eggs'),
...     third_party_eggs_path=os.path.join(base, 'eggs_3rd_party'),
... )
Removing .../test/base/ExamplePackage/trunk/dist
Building in .../test/base/ExamplePackage/trunk
Creating virtual environment in .../test/base/ExamplePackage/trunk/dist
Installing eggs
Extracting source
Running tests in '.../test/base/ExamplePackage/trunk/dist/ExamplePackage-0.1.0/test'

Parse the PKG_INFO file:

>>> pkg_info = buildkit.get_pkg_info('ExamplePackage', os.path.join(base, 'ExamplePackage'))

>>> buildkit.build_html_docs('ExamplePackage', os.path.join(base, 'ExamplePackage'))

Update a website with the documentation:

>>> docs_path = os.path.join(test_dir, 'website', 'code')
>>> template_path = os.path.join(test_dir, 'website', 'Templates', 'main.dwt')
>>> buildkit.copy_html_docs(
...     docs_path,
...     name='ExamplePackage',
...     path=os.path.join(base, 'ExamplePackage'),
...     version=pkg_info['Version'][0],
... )
>>> buildkit.package_index_page(
...     eggs_path=os.path.join(base, 'eggs'),
...     package_docs_path = os.path.join(docs_path, 'ExamplePackage'.lower()),
...     template_path = template_path,
...     pkg_info=pkg_info,
... )
>>> buildkit.egg_index_page(
...     docs_path = docs_path,
...     template_path = template_path,
... )
>>> buildkit.main_index_page(
...     docs_path = docs_path,
...     template_path = template_path,
...     exclude = ['eggs', 'flows'],
... )

Writing your own build scripts

There’s an example called build.py in the examples directory of the source distribution which you can put in your base directory to control builds. You can use it like this to build and test a single package, creating the web pages and adding the source distribution to the eggs directory:

python build.py BuildKit

You can use it like this to build all the packages the script can find:

python build.py