Introduction

CommandTool is a Python package which provides the commandtool module for parsing command line options. It is different from the getopt and optparse modules in the Python standard library in that it:

• is specifically designed for cases where you want to handle sub-commands
• contains tools for automatically creating command line help and man pages from docstrings written in reStructuredText
• can deal with sub-command aliases
• can work out whether an option is related to the program itself or a sub-command based on its position
• can deal with configuration options which provide defaults if particular command line options are missing

CommandTool isn’t useful if:

• you aren’t using sub-commands
• you want the same option to have a different meaning in different sub-commands (although CommandTool is great if you want an option name to have a consistent meaning across all your sub-commands).

CommandTool requires access to the following command line tools for the generation of command line help text from reStructuredText:

• elinks

It also depends on these packages for the reStructuredText parsing for man page creation and help text generation:

• docutils

To get you familiar with CommandTool let’s start by looking at the command line API. Once you’ve seen some examples you’ll move on to look at the Python API used to produce them.

Using the command line

Command line programs written with CommandTool are executed like this:

PROGRAM_NAME [PROGRAM_OPTIONS] SUB-COMMAND [COMMAND_OPTIONS] ARGUMENTS

Users can put the program options, sub-command options and arguments in in any order and CommandTool will work out which are which. If both the program itself and the sub-command which will be run take an option with the same name, for example --help, if the option comes before the sub-command name it will be treated as a program option, otherwise it will be treated as a sub-command option.

Here’s an example:

$ find --help content -t .py two

Here find is the program name and content is the sub-command. The --help option is a program option because it comes before the sub-command name, the -t option is a sub-command option because it comes afterwards. The value .py is associated with the -t option and the only argument is the word two which is the word to search for.

If you hadn’t set find up as an executable Python script you could run the file directly from the command line with Python instead:

$ python find.py --help content -t .py two

Core concepts

Aliases

Sub-commands can have aliases which are typically two letter “shortcuts”. For example, if ct was an alias for content, the same sub-command could also be executed like this:

$ find --help ct -t .py two

Internal Variables

When using CommandTool, the sub-command options set on the command line affect a series of internal variables you set up. For example, the verboseness (how much output it prints) of an application might be important so you could set up an internal variable named verbosity. The options -q, --quiet, -v and --verbose could all then be set up to affect the one internal variable, verbosity.

You can write code to determine the affect each of the options has, but the point is that you’ll never end up dealing with -q, --quiet, -v and --verbose in your application code. Instead, after all the CommandTool processing you’ll end up with a variable called verbosity and its value will have been determined by the options set.

This approach means that you never have to deal with command line options in your application code because by the time it is called, you are only dealing with internal variables which exaclty map to the functionality your APIs need, and not the command line API necessary to set them effectively. This helps keep code simple.

APIs

A CommandTool program has three API levels:

• CommandTool itself which does the parsing and provides data structures so you can set up the internal variables
• The sub-command handlers you write (one for each sub-command) which process the data values from CommandTool and set up the internal variables
• The Python API which directly uses the internal variables set up

The idea is that because of this set up, the Python API called by the sub-command handlers is identical to the API your program itself needs because the arguments to the functions or classes are simply the internal variables you have set up in your sub-command handlers.

Metavars

Some command line options take a value, such values are called metavars in CommandTool terminology.

In the find example we’ve been using so far if we defined that the -t option set the metavar FILETYPE, then the FILETYPE metavar would contain the value .py after the sub-commands used so far had been run.

By giving metvars names it is also possible to set them in a configuration file rather than on the command line. You’ll see how this is done later when we look at the CommandTool API.

CommandTool API

In this section you’ll build the find.py program used in the examples so far. Find will have two sub-commands, name and content. The first is used to search for a particular string within filenames, the second for words inside files. The name sub-command will have two aliases: n and nm and the content sub-command will just have the alias ct.

Let’s start by building the functions which will actually do the finding and then look at how to use CommandTool to enable that functionality to be controlled from the command line.

Building the API functions

The Python API looks like this (I know it isn’t great functionality, this is about the command line API, not the implementation of a search!):

import os.path
import logging
log = logging.getLogger('find')

def name(letters, start_directory='/home/james'):
    found = []
    log.info(
        'Finding files in %r whose filenames contain the letters %r',
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)):
            log.debug('Trying %r', filename)
            if letters in filename:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

def content(letters, start_directory='/home/james', file_type='.txt'):
    found = []
    log.info(
        'Finding %r files in %r which contain the letters %r',
        file_type,
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)) and filename.endswith(file_type):
            log.debug('Trying %r', filename)
            fp = open(os.path.join(start_directory, filename), 'r')
            data = fp.read()
            fp.close()
            if letters in data:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

The Thinking Process

In order to work out how to structure the command line options you need to answer the following questions:

1. What are the sub-commands I need?
2. What aliases do I want to set up?
3. What are the internal variables used in each sub-command?
4. What options are needed to control each of the internal variables?
5. Are any of the options not optional?

Let’s go through each of the numbered questions above to decide how to structure the command line options:

1. In this case we only have two functions and it is clear each represents a sub-command. Let’s call the sub-commands name and content.

   We’ll need to write a sub-command handler for each of these sub-commands. Sub-command handlers always take this form where subcommand is replaced with the name of the subcommand.

   def handle_command_subcommand(
       option_sets,
       command_options,
       args
   ):
       ...
   

   In our case the sub-command handlers will be called handle_command_name() and handle_command_content().

   In order to let CommandTool know the sub-command handlers available you create a dictionary called command_handler_factories with keys which match the sub-command name and values which are functions which when called, return the appropriate handler.

   The reason a function which returns a handler is specified rather than just the handler itself is that for large applications you may wish to import just the one sub-command handler required from another module rather than having to have all sub-command handlers present. This can be more efficient.

   Let’s set up the handlers directly, we can do that with the makeHandler() function:

   from commandtool import makeHandler
   
   command_handler_factories = {
       'name': makeHandler(handle_command_name),
       'content': makeHandler(handle_command_content),
   }
   

   Note

   A better API might just be to define a get_handler() factory which returns the correct sub-command handler?

2. Now let’s think about sub-command aliases. Let’s have aliases of n and nm for name and ct for content.We can now define the sub-command alias map. It looks like this:

   aliases = {
       'name': ['n', 'nm'],
       'content': ['ct'],
   }
   

   Tip

   If you didn’t have any aliases for the sub-commands you would still need to have an entry in the alias map, but the list would be empty, for example:

   aliases = {
       'name': [],
       'content': [],
   }
   

3. Next let’s see which internal variables are used. Looking at the arguments for the two functions we need to be able to control the variables letters, start_directory`` and file_type. The letters and start_directory internal variables are used in both sub-commands but the file_type internal variable is only used in the content sub-command.

4. Out of all the internal variables, only the letters argument is required. It can be entered on the command line as an argument rather than an option. This means we only need to set up options for start_directory and file_type.

You now have your alias map and sub-command handler factories, you know what your internal vairables will be called, and which API function params will be specified as arguments on the command line.

Now let’s set up the option set to specify which options affect which internal variables.

Option sets

Let’s look at the options definition we’ll need for the two optional internal variables we identified, start_directory and file_type:

option_sets = {
    'start_directory': [
        dict(
            type = 'command',
            long = ['--start-directory'],
            short = ['-s'],
            metavar = 'START_DIRECTORY',
        ),
    ],
    'file_type': [
        dict(
            type = 'command',
            long = ['--file-type'],
            short = ['-t'],
            metavar = 'FILE_TYPE',
        ),
    ],
}

Notice that:

• the options are grouped by the internal variable name they affect
• you can specify both long and short option names for the same internal variable
• you can specify more than one set of options for the same internal variable if you prefer
• options sets where the options set a value have a metavar assoicated with them

The letters internal variable will be specified on the command line directly without any option assoicated with it so it doesn’t appear in the option set.

Users of your program are likely to be used to expecting --help and --verbose or --quiet options. Although you don’t strictly need to support these options, this documentation for CommandTool assumes you will.

You’ll need to add these options to the option sets:

'help': [
    dict(
        type = 'shared',
        long = ['--help'],
        short = ['-h'],
    ),
],
'verbose': [
    dict(
        type = 'program',
        long = ['--verbose'],
        short = ['-v'],
    ),
    dict(
        type = 'program',
        long = ['--quiet'],
        short = ['-q'],
    ),
],

As you can see, the help internal variable can be treated as a program option or a command option depending on where on the command line it appears. It is therefore has a type of shared. The verbose internal variable is a program option but it has two command line options which affect it, --quiet and --verbose. How these options affect the internal variable is implemented in code later on. Usually this internal variable affects which log level gets used to print output to the console.

Tip

each option sets a type specifying whether the option can only be used in programs ('programs'), only be used in a sub-command ('command') or can be used in either ('shared')

There are some other things you should be aware of:

• you can’t use the same option in two option sets
• although you can specify more than one long and short option, only the first of each appears in any help output by default

The final option we’ll add affects an internal variable named config and can be used to specify a file which specifies metavars directly. This is really handy as it allows you to use a config file as a default for certain options you might not want to have to specify explicitly. Add this to the option set too:

'config': [
    dict(
        type = 'command',
        long = ['--config'],
        short = ['-c'],
        metavar = 'CONFIG',
    ),
],

Basic Parsing and Program Handler

Now, let’s do some basic parsing to see how the commands are used.

CommandTool also has a help system which we’ll look at later where help messages are obtained from a help sub-module of your application or a class instance. If this is missing the doc-strings are used directly.

Create the find.py file and add the following:

import getopt
import os
import sys

from commandtool import parse_html_config
from commandtool import parse_command_line
from commandtool import handle_program
from commandtool import makeHandler
from commandtool import strip_docsting
from commandtool import option_names_from_option_list
from commandtool import set_error_on

# Import the help strings
import help_strings

import os.path
import logging
log = logging.getLogger('find')

def name(letters, start_directory='/home/james'):
    found = []
    log.info(
        'Finding files in %r whose filenames contain the letters %r',
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)):
            log.debug('Trying %r', filename)
            if letters in filename:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

def content(letters, start_directory='/home/james', file_type='.txt'):
    found = []
    log.info(
        'Finding %r files in %r which contain the letters %r',
        file_type,
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)) and filename.endswith(file_type):
            log.debug('Trying %r', filename)
            fp = open(os.path.join(start_directory, filename), 'r')
            data = fp.read()
            fp.close()
            if letters in data:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

def handle_command_name(
    option_sets,
    command_options,
    args
):
    pass

def handle_command_content(
    option_sets,
    command_options,
    args
):
    pass

command_handler_factories = {
    'name': makeHandler(handle_command_name),
    'content': makeHandler(handle_command_content),
}

aliases = {
    'name': ['n', 'nm'],
    'content': ['ct'],
}

option_sets = {
    'start_directory': [
        dict(
            type = 'command',
            long = ['--start-directory'],
            short = ['-s'],
            metavar = 'START_DIRECTORY',
        ),
    ],
    'file_type': [
        dict(
            type = 'command',
            long = ['--file-type'],
            short = ['-t'],
            metavar = 'FILE_TYPE',
        ),
    ],
    'help': [
        dict(
            type = 'shared',
            long = ['--help'],
            short = ['-h'],
        ),
    ],
    'verbose': [
        dict(
            type = 'program',
            long = ['--verbose'],
            short = ['-v'],
        ),
        dict(
            type = 'program',
            long = ['--quiet'],
            short = ['-q'],
        ),
    ],
    'config': [
        dict(
            type = 'command',
            long = ['--config'],
            short = ['-c'],
            metavar = 'CONFIG',
        ),
    ],
}

program_help = """\
usage: ``%(program)s COMMAND [OPTIONS] ARGS``

Commands (aliases):

 :name:     search for letters in filenames
 :content:  search for letters in content

Try \`%(program)s COMMAND --help' for help on a specific command."""

if __name__ == '__main__':
    try:
        program_options, command_options, command, args = parse_command_line(
            option_sets,
            aliases,
        )
        program_name = os.path.split(sys.argv[0])[1]
        handle_program(
            command_handler_factories=command_handler_factories,
            option_sets=option_sets,
            aliases=aliases,
            program_options=program_options,
            command_options=command_options,
            command=command,
            args=args,
            program_name=program_name,
            help=help_strings,
        )
    except getopt.GetoptError, err:
        # print help information and exit:
        print str(err)
        if command:
            print "Try `%(program)s %(command)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
                'command': command,
            }
        else:
            print "Try `%(program)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
            }
        sys.exit(2)

You’ll update the sub-command handlers with real code in a minute.

Notice that because the call to handle_program() is within a try... except block, any of the option handlers, command handlers or any other code can raise a getopt.GetoptError exception which will result in the error being printed to the screen and a short message printed about how to get help to use the correct options. Because of this, command handlers should raise a getopt.GetoptError exception if there is any problem handling the command line options so that the user can make a correction.

Before we update the command handlers, let’s think about config files.

Dealing with Config Files

In addition to all the command options users can also specify defaults in a config file. If a command doesn’t recieve a neccessary metavar on the command line it can load it from the config file.

The metavars used to label the values associated with options on the command line in the options sets are used as keys in the config file.

You’ll see one of the option sets is for the internal variable config which sets the metavar CONFIG. When combined with a handler which parses the filename specified by the metavar CONFIG, the options can be automatically loaded if the -c or --config options are used.

In the command handler you can then do something like this to set a default value for the a metavar if it isn’t specified on the command line.

from commandtool import parse_html_config

def handle_name(options, command_options, args):
    ...
    config = {}
    if command_options.has_key('config'):
        config = parse_html_config(command_options['config'][0]['value'])
    internal_vars = {}
    ...
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = command_options['start_directory'][0]['value']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = config['START_DIRECTORY']
    ...

Notice that whether the value of START_DIRECTORY is extracted from the command line or on the config file, it is still run through the handler so the rest of the application beyond the handle_name() function remains the same.

Now let’s implement the command handlers.

Command Handlers

The job of the command handlers is to look at the options used for each internal variable and call the appropriate Python API function as a result. The docstring is used as a basis for the help text for that sub-command.

Here’s what they look like:

def handle_command_name(
    option_sets,
    command_options,
    args
):
    """\
    usage: ``%(program)s [PROGRAM_OPTIONS] name [OPTIONS] LETTERS``

    search START_DIRECTORY for a filename matching LETTERS

    Arguments:
      :``LETTERS``: The letters to search for

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type \`%(program)s --help'
    """
    set_error_on(
        command_options,
        allowed=['config', 'start_directory']
    )
    config = {}
    if command_options.has_key('config'):
        config = parse_html_config(command_options['config'][0]['value'])
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['value']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = config['START_DIRECTORY']
    for found in name(**internal_vars):
        print found

def handle_command_content(
    option_sets,
    command_options,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] content [OPTIONS] LETTERS

    search START_DIRECTORY for a files containing LETTERS in their content

    Arguments:
      :``LETTERS``: The letters to search for

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -t, --file-type=FILE_TYPE                the file types to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type \`%(program)s --help'"""
    set_error_on(
        command_options,
        allowed=['config', 'start_directory', 'file_type']
    )
    config = {}
    if command_options.has_key('config'):
        config = parse_html_config(command_options['config'][0]['value'])
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['value']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = config['START_DIRECTORY']
    if command_options.has_key('file_type'):
        internal_vars['file_type'] = \
           command_options['file_type'][0]['value']
    for found in content(**internal_vars):
        print found

Notice that the docstring for each forms the help text for that command. In a real world applicaiton this approach is much simpler than trying to automatically generate help options (as the optparse modules does).

Within each command handler you can choose to set an error if unexpected options are set. There is a helper function called set_error_on() to help you with this. It will set an error if any options have been set for internal variables you weren’t expecting. You can also use it like this:

internal_vars = set_error_on(
    command_options,
    allowed=['config', 'site_directory', 'file_type'],
)

If the allowed internal variable has a metavar associated with it, the value specified on the command line is set to the vaule, otherwise the value True is assigned. This is handy because in many cases this is all the processing you need. If you have more than one option affecting a variable you’ll need to process it in code instead.

Command Options

It’s worth spending a minute looking at the command_options variable which gets passed to the sub-command handlers.

If we run this command:

$ python find.py -t .py ct two --config test.html

The command_options variable looks like this:

 {
     'file_type': [
         {
             'value': '.py',
             'short': ['-t'],
             'pos': 0,
             'name': '-t',
             'type': 'command',
             'internal': 'file_type',
             'long': ['--file-type'],
             'metavar': 'FILE_TYPE'
        }
    ],
    'config': [
        {
            'value': 'test.html',
            'short': ['-c'],
            'pos': 1,
            'name': '--config',
            'type': 'command',
            'internal': 'config',
            'long': ['--config'],
            'metavar': 'CONFIG'
        }
    ]
}

You can see that command_options is a dictionary keyed by the internal variable name. Each corresponding key is a list where each item represents one of the options specified on the command line which affected that variable. Within each item the short, type, metavar, long and internal keys are just the keys you specified in the option set for that option. The value, pos and name keys are more interesting and have the following meanings:

value

The text entered on the command line for that option (only present if the option takes a metavar).

pos

The position on the command line the option was found, excluding sub-command names and arguments.

name

The version of the long or short command which was used.

Summary of the Processing Pipeline

With these features the following actions happen:

• Prepare option_sets, aliases and option_handlers
• Pass them to the parse_command_line() function to have them parsed
• If the options specified aren’t valid an error messages is displayed
• If successful and there are some command options, the options are passed through any required handlers to format the values specified
• Otherwise the options are organised into the four variables: program_options, command_options, command, args. The program_options and command_options are dictionaries with the internal variables as their key and a list of all the options which affect that variable as the value. The items in each option list are actually dictionaries with other useful information about the option, such as its position on the original command line, the metavar, help text etc.
• This parsed data can then be sent to a command handler which checks the options are valid for the particular command, sets the internal values it requires based on the options, calls the real Python API with the internal variables and formats the result for display on the command line.

Testing the Example

Let’s test the basic help options:

$ python find.py
No command specified.
Try `find.py --help' for more information.

$ python find.py --help
usage: find.py COMMAND [OPTIONS] ARGS

Commands (aliases):

   name:     search for letters in filenames
   content:  search for letters in content

Try `find.py COMMAND --help' for help on a specific command.

$ python find.py name
Expected exactly one argument, LETTERS
Try `find.py name --help' for more information.

$ python find.py name --help
usage: find.py [PROGRAM_OPTIONS] name [OPTIONS] LETTERS

search START_DIRECTORY for a filename matching LETTERS

Arguments:

         LETTERS:  The letters to search for

Options:

         -s, --start-directory=START_DIRECTORY
                  the directory to search
         -c, --config=CONFIG
                  the config file to use

For PROGRAM_OPTIONS type `find.py --help'

Let’s check that putting --help in front of the command name triggers the program help, not the name command help:

$ python find.py --help name
usage: find.py COMMAND [OPTIONS] ARGS

Commands (aliases):

   name:     search for letters in filenames
   content:  search for letters in content

Try `find.py COMMAND --help' for help on a specific command.

It does.

So far so good, now let’s try the command out for real.

To test the example let’s set up a sample directory to search:

$ mkdir start_here
$ cd start_here
$ cat << EOF >> one.txt
This is a sample file which contains the text 'one'.
EOF
$ cat << EOF >> two.txt
This is a sample file which contains the text 'two'.
EOF
$ cat << EOF >> two.py
# This is a Python file which contains the text 'two'.
EOF
$ cd ../

Now let’s find all files with two in their names within the start_here directory:

$ python find.py name -s start_here
Expected exactly one argument, LETTERS
Try `find.py name --help' for more information.

Oops, forgot the letters:

$ python find.py name -s start_here two
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

Great! It has found the files. How about using --start-directory instead of -s:

$ python find.py name --start-directory start_here two
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

No trouble. What about trying to confuse it by changing the order?

$ python find.py name two --start-directory start_here
INFO:find:Finding files in '/home/james/Desktop/start_here' whose filenames contain the letters 'two'
/home/james/Desktop/start_here/two.py
/home/james/Desktop/start_here/two.txt

Still works.

Now let’s try to search the contents of the files with the ct alias for the content command:

$ python find.py ct -s start_here two
INFO:find:Finding '.txt' files in '/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.txt

Yep, that’s correct because it only searches .txt files by default, let’s specify .py files instead and put the option in completely the wrong place:

$ python find.py -t .py ct two -s start_here
INFO:find:Finding '.py' files in u'/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.py

No trouble.

Now let’s introduce an html config file:

$ cat << EOF >> test.html
<html>
<head><title>Example Config File</title></head>
<body>
<h1>Example Config File</h1>
<table class="commandtool-config">
<tr><td class="metavar"><tt>START_DIRECTORY</tt></td>
    <td>The directory to begin the search</td>
    <td>/home/james/Desktop/start_here</td></tr>
</table>
</body>
</html>
EOF

You’ll need to update the path to point to your start_here directory. For this example, make sure test.html is in the same directory as find.py, not in the start_here directory.

Now let’s use this and not specify the -s option to see if it correctly gets the value from the config file:

$ python find.py -t .py ct two --config test.html
INFO:find:Finding '.py' files in u'/home/james/Desktop/start_here' which contain the letters 'two'
/home/james/Desktop/start_here/two.py

No problem, config file parsing works too.

Now let’s test the program options --verbose:

$ python find.py –verbose -t .py ct two –config test.html INFO:find:Finding ‘.py’ files in u’/home/james/Desktop/start_here’ which contain the letters ‘two’ DEBUG:find:Trying u’two.py’ DEBUG:find:Matched u’/home/james/Desktop/start_here/two.py’ /home/james/Desktop/start_here/two.py

As you can see, DEBUG messages get printed in addition to INFO messages. Now with --quiet:

$ python find.py –quiet -t .py ct two –config test.html /home/james/Desktop/start_here/two.py

No messages get printed at all. Let’s check we can put these program options after the command:

$ python find.py -t .py ct two --config test.html -q
/home/james/Desktop/start_here/two.py

Yes we can. What happens if we specify them both together:

$ python find.py -t .py ct two --config test.html -qv
Only specify one of -q, -v
Try `find.py content --help' for more information.

As you can see the comman line API is very flexible.

Dealing with Help

Users of command line programs usually expect to be able to get a summary of how to use the program by specifying a --help option. They also expect a man page about the program. CommandTool can help with this.

When you call the handle_program() function you can specify a help argument. This should be any Python object whose attributes have the names of the sub-commands you use and contain the help text to display for that sub-command. If no appropriate help text can be found, the particular command handler’s docstring is used. Either way, the help text is stripped to remove leading whitespace and is expected to be escaped so it is suitable for Python string formatting (ie % characters must be written %%). Any string written %(program)s is then replaced with the program name so that the help text will be correct regardless of the method the program was invoked.

Since the handle_program() function takes care of handling any cases where a sub-command isn’t specified you can’t change its docstring (well you can but you shouldn’t). Instead you should define a program_help variable in your file and set the help text as the value as that variable. For example, add this just before the if __name__ == '__main__' line in find.py:

program_help = """\
usage: ``%(program)s COMMAND [OPTIONS] ARGS``

Commands (aliases):

 :name:     search for letters in filenames
 :content:  search for letters in content

Try \`%(program)s COMMAND --help' for help on a specific command."""

Make sure you list the sub-commands with a : character before and after the name so that it gets formatted correctly in both HTML and as a man page.

Although you could hard-code the help object so that it contains exactly what you want to display, CommandTool also comes with a create_man program, itself written with CommandTool, to automatically generate plain-text help strings and man pages from docstrings written in reStructuredText.

Using the commandtool.help tool

Creating the command line help

You create a help module containing formatted plain text help from your reStructuredText docstrings. To do so you need to have docutils installed and you need the elinks command line program:

$ easy_install docutils
$ sudo apt-get install elinks

You also need the module containing your command handlers to be installed or on your PYTHONPATH. You can modify your PYTHONPATH like this:

$ export PYTHONPATH=$PYTHONPATH:/path/to/directory

You can now extract the docstrings like this:

$ python -m commandtool.help commands2help find help_strings.py

The first argument after commands2help is the module path of the module containing the command handlers. Since we are in the same directory as find.py we just enter find. Otherwise you would enter the same string as you would use if you were importing a module with a Python import statement, for example path.to.module.with.command_handlers.

The second argument is the name of the file to output the formatted help strings to. Any existing file at this path will be overwritten.

The create_man program will look for the command_handlers variable within the module you specified and use that to find the command handler functions and their docstrings. It will also look for the program_help string for the program help.

If there are any problems with your reStructuredText syntax you’ll see an error:

<string>:13: (WARNING/2) Inline interpreted text or phrase reference start-string without end-string.

Hint: You may need to escape literal backticks ` with a \ character in front of them.

Once everything is working the help_strings.py file will be generated.

To change the program to use this new help file you need to change the handle_program() call in find.py from this if you haven’t already got the help line present:

handle_program(
    command_handler_factories=command_handler_factories,
    option_sets=option_sets,
    aliases=aliases,
    program_options=program_options,
    command_options=command_options,
    command=command,
    args=args,
    program_name=program_name,
)

to this:

import help
handle_program(
    command_handler_factories=command_handler_factories,
    option_sets=option_sets,
    aliases=aliases,
    program_options=program_options,
    command_options=command_options,
    command=command,
    args=args,
    program_name=program_name,
    help=help_strings,
)

(Notice the import statement and that we’ve added the help argument).

You can now test the help:

$ python find.py --help
usage: find.py COMMAND [OPTIONS] ARGS

Commands (aliases):

   name:     search for letters in filenames
   content:  search for letters in content

Try `find.py COMMAND --help' for help on a specific command.

Notice that the help output isn’t quite the same as the reStructuredText markup, it has been nicely formatted for optimum display on an 80 character terminal. Internally this happens by turing the docstring into HTML and then using the elinks browser to render that HTML.

Creating a man page

Now you’ve seen how to generate help text from the reStructuredText docstrings let’s look at how to generate man pages.

You need to have the programs gzip, man installed for this to work. The code also uses the manpage.py writer from the rst2man project (in the docutils SVN repository) and the docutils module itself.

$ easy_install docutils
$ sudo apt-get install man gzip

Once again you need to setup the PYTHONPATH:

$ export PYTHONPATH=$PYTHONPATH:/path/to/directory

Type this for help:

$ python -m commandtool.help --help

Here’s an example of creating a find.rst file:

$ python -m commandtool.help commands2rst find find -e james@example.com -c "Copyright 2009 James Gardner, All Rights Reserved" -o "3aims Ltd" -V 0.1  -D "" -d "A simple tool for finding words in filenames and file contents."

The first find argument after commands2rst refers to the module path as before, the second is the name of the command as it should be documented in the man page. They are often the same. There are quite a few different options. Here’s the help text explaining them all (or see the man page for full info):

Options:

   -e, --email     author email
   -V, --version   software version
   -s, --program-description
                   one line program description
   -d, --description
                   full description
   -o, --organization
                   organization name
   -O, --options   options
   -a, --address   organization address
   -D, --date      date (defaults to current date if not specified)
   -c, --copyright
                   copyright
   -t, --rest      extra information to add
   -S, --synopsis  program synopsis
   -g, --group     man group (defaults to text processing)
   -i, --section   man page section number (defaults to 1)

You then convert that .rst file to a man page.

$ mkdir man1
$ python -m commandtool.help rst2man find.rst

View the man page like this:

$ man -M ./ find

Further Reading

See also:

• http://jimmyg.org/blog/2009/python-command-line-interface-(cli)-with-sub-commands.html
• http://jimmyg.org/blog/2009/generating-man-pages-from-restructuredtext.html