Writing a Python Command Line Interface (cli) with Sub-Commands

Aliases

I want the commands to be able to be referenced by aliases so that a user can run due rc rather than due recreate and have the same action performed.

Return Value

I want the parser to return four data structures:

program_options

A dictionary containing the PROGRAM_OPTIONS and any SHARED_OPTIONS associated with the program rather than the command being used (if any).

command_options

A dictionary of all the COMMAND_OPTIONS and any SHARED_OPTIONS associated with the command being run rather than program (if any).

command

The name of the command itself (not its alias)

args

A list of any extra arguments specified (excluding the COMMAND itself)

Specifying Available Options

In order for the command line parser to spot errors it needs to know all the program options and all the command options for the program and every possible command. I specifically don't want different commands to use the same option to mean different things as this could make the commands seem inconsistent to the user so having them all defined to start with is fine.

Internal Variables

Conceptually it is helpful to think about the set of command line options affecting an internal variable in the application. For example, both the --quiet and --verbose options can be thought of as affecting an internal variable called verbosity.

After the command line is parsed I'd like to be able to access the options by the internal variable they affect so that I can easily work out the correct value the verbosity variable has internally based on the options specified.

Metavars

Some options are just flags which don't have a value associated with them, for example --help, --quiet etc. Others do have a value and this can be given a label which I'm calling a metavar. For example --database=users. If we defined that the --database option set the metavar DATABASE, the DATABASE metavar would then contain the value users if this option was used.

Each internal variable can have more than one metavar associated with it. For example consider the case where you have an internal variable postition representing the lattitude and longitude coordinates where a photo was taken. The correspondind opitons to set these variables might be --latitude=LATITUDE, --longitude=LONGITUDE``, thus the position internal variable has two metavars associated with it, LATITUDE and LONGITUDE.

The same metavar can also affect more than one internal variable. For example if you had a metavar called BASE_PATH in a move command it might affect both the source and dest internal variables. It also means that options can be re-used amongst internal variables.

The only restriction is that the same metavar can't take two different values in the same command so you have to take care to ensure that metavars are given different names if they are allowed to appear in the same command with different values. Such a condition raises an Exception, instead of printing an error.

Config Files

Now that you have the concept of metvars associated with options affecting internal variables, it is easy to intoriduce the concept of a config file.

A config file is simply a format which, when parsed, results in a Python dictionary where the keys are the metavars and the values are the values associated with a metavar. For example, a config file might look like this:

LATITUDE  66° 33' 39'' N

On the command line the --latitude option would no longer need to be set because the value from the LATITUDE metavar in the config file could be used instead.

There is currently no way to specify flags like --help or --quiet in a config file and no way to specify arguments. I might implement such features if I come accross a suitable use case that can't be handled with the existing functionality.

Error and Try Message

If you enter a set of invalid options or arguments or a command which is not regonised, an error is displayed but is also helpful to print some information explaining how the user can get help to find out what the correct flag are. For example, look at how the mv command works on Linux:

$ mv /home/james/example
mv: missing destination file operand after `/home/james/example'
Try `mv --help' for more information.

The error message will be displayed by the parsing code but the try message can be configured.

Some command line programs print the usage and help when an error occurs but I value screen space and don't like that approach. Instead my applications will display the help if the --help flag is displayed anywhere on the command line so that you can just press the up arrow type --help and press enter to get the help without having to write a new command from scratch.

Help Text

The help text will take a different format depending on whether it is help with the program overall or with a specific command.

Help messages usually have the following components:

• usage - explains how to structure the options and what argument are expected
• description - explains what the command does and what the arguments are for
• options - explains what each of the options do

Program help will usually also have a section listing the available commands. For example, here's the svn program's help output:

$ svn --help
usage: svn <subcommand> [options] [args]
Subversion command-line client, version 1.5.1.
Type 'svn help <subcommand>' for help on a specific subcommand.
Type 'svn --version' to see the program version and RA modules
  or 'svn --version --quiet' to see just the version number.

Most subcommands take file and/or directory arguments, recursing
on the directories.  If no arguments are supplied to such a
command, it recurses on the current directory (inclusive) by default.

Available subcommands:
   add
   blame (praise, annotate, ann)
   cat
   changelist (cl)
   checkout (co)
   cleanup
   commit (ci)
   copy (cp)
   delete (del, remove, rm)
   diff (di)
   export
   help (?, h)
   import
   info
   list (ls)
   lock
   log
   merge
   mergeinfo
   mkdir
   move (mv, rename, ren)
   propdel (pdel, pd)
   propedit (pedit, pe)
   propget (pget, pg)
   proplist (plist, pl)
   propset (pset, ps)
   resolve
   resolved
   revert
   status (stat, st)
   switch (sw)
   unlock
   update (up)

Subversion is a tool for version control.
For additional information, see http://subversion.tigris.org/

The individual command's help will take a similar format but without the list of available sub-commands. For example, here's the help output from svn ci --help:

$ svn ci --help
commit (ci): Send changes from your working copy to the repository.
usage: commit [PATH...]

  A log message must be provided, but it can be empty.  If it is not
  given by a --message or --file option, an editor will be started.
  If any targets are (or contain) locked items, those will be
  unlocked after a successful commit.

Valid options:
  -q [--quiet]             : print nothing, or only summary information
  -N [--non-recursive]     : obsolete; try --depth=files or --depth=immediates
  --depth ARG              : limit operation by depth ARG ('empty', 'files',
                            'immediates', or 'infinity')
  --targets ARG            : pass contents of file ARG as additional args
  --no-unlock              : don't unlock the targets
  -m [--message] ARG       : specify log message ARG
  -F [--file] ARG          : read log message from file ARG
  --force-log              : force validity of log message source
  --editor-cmd ARG         : use ARG as external editor
  --encoding ARG           : treat value as being in charset encoding ARG
  --with-revprop ARG       : set revision property ARG in new revision
                             using the name[=value] format
  --changelist ARG         : operate only on members of changelist ARG
                             [aliases: --cl]
  --keep-changelists       : don't delete changelists after commit

Global options:
  --username ARG           : specify a username ARG
  --password ARG           : specify a password ARG
  --no-auth-cache          : do not cache authentication tokens
  --non-interactive        : do no interactive prompting
  --config-dir ARG         : read user configuration files from directory ARG

The optparse module generates these help messages for you with a small degree of flexibility to customise how they appear. There are two problems with this for more complex applications:

• You might want extra information displayed or help formatted in a different way
• You might not want to expose every possible option in the help text, instead encouraging the user to use the most up-to-date API

It is easy to generate the help messages yourself as static text anyway so I'd recommend doing that rather than trying to auto-generate help.

In order to make this process easier though, the docstring of the function which handles the specific command can be used as the help text, that way you only need to maintain the help text in one place.

Python API

The Python API looks like this (I know it isn't great functionality, this is about the command line API, not the implementation of a search!):

import os.path
import logging
log = logging.getLogger('find')

def name(letters, start_directory='/home/james'):
    found = []
    log.info(
        'Finding files in %r whose filenames contain the letters %r',
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)):
            log.debug('Trying %r', filename)
            if letters in filename:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

def content(letters, start_directory='/home/james', file_type='.txt'):
    found = []
    log.info(
        'Finding %r files in %r which contain the letters %r',
        file_type,
        start_directory,
        letters
    )
    for filename in os.listdir(start_directory):
        if not os.path.isdir(os.path.join(start_directory, filename)) and filename.endswith(file_type):
            log.debug('Trying %r', filename)
            fp = open(os.path.join(start_directory, filename), 'r')
            data = fp.read()
            fp.close()
            if letters in data:
                log.debug(
                    'Matched %r',
                    os.path.join(start_directory, filename)
                )
                found.append(os.path.join(start_directory, filename))
    return found

You can see that there are three internal variables being used: start_directory, letters and file_type. In both cases the variable letters is required. It should therefore be implemented as an argument rather than an option. start_directory and file_type can be options.

Option Sets

We will also want an internal variable called verbose which will be configured by program options and affect the log level which gets printed to the console. We'll also want a help variable which should affect whether the help text for the program or the command should be shown.

Let's look at the options definition we'll need:

option_sets = {
    'start_directory': [
        dict(
            type = 'command',
            long = ['--start-directory'],
            short = ['-s'],
            metavar = 'START_DIRECTORY',
        ),
    ],
    'file_type': [
        dict(
            type = 'command',
            long = ['--file-type'],
            short = ['-t'],
            metavar = 'FILE_TYPE',
        ),
    ],
    'help': [
        dict(
            type = 'shared',
            long = ['--help'],
            short = ['-h'],
        ),
    ],
    'config': [
        dict(
            type = 'command',
            long = ['--config'],
            short = ['-c'],
            metavar = 'CONFIG',
        ),
    ],
    'verbose': [
        dict(
            type = 'program',
            long = ['--verbose'],
            short = ['-v'],
        ),
        dict(
            type = 'program',
            long = ['--quiet'],
            short = ['-q'],
        ),
    ]
}

Notice that:

• the options are grouped by the internal variable name they affect
• you can specify both long and short option names for the same internal variable
• you can specify more than one set of options for the same internal variable if you prefer
• each option set a type specifying whether the option can only be used in programs ('programs'), only be used in command ('command') or can be used in either ('shared')
• options sets where the options set a value have a metavar assoicated with them

There are some other things you should be aware of:

• you can't use the same option in two option sets
• although you can specify more than one long and short option, only the first of each appears in any help output by default

Aliases

The aliases are easy to set up, they look like this:

aliases = {
    'name': ['n', 'nm'],
    'content': ['c', 'ct'],
}

Metavar Handlers

Sometimes it is useful to have some pre-processing performed on option values (metavars). For example you might like to convert all path-related metavars from the value specified on the command line into absolute, normalized paths before handling them in the application. Handlers can perform this task for you.

Handlers are simply functions which take metavar value as an argument and return a result. They are passed to the parse_command_line() function as a dictionary keyed by the metavar they operate on.

For example, here is a set of metavar handlers for converting the START_DIRECTORY metavar into an absolute, normalised path. They can also set errors by raising a getopt.GetoptError exception.

import getopt
from commandtool import parse_html_config

def uniform_path(path):
    return os.path.normpath(os.path.abspath(path))

metavar_handlers = {
    'START_DIRECTORY': uniform_path,
    'CONFIG': parse_html_config
}

Basic Parsing and Program Handler

Now, let's do some basic parsing to see how the commands are used. We'll have one function called the program handler which will be responsible for dealing with any program options and, if necessary, calling the appropriate command handler to handle the options for the command chosen.

import sys

from commandtool import parse_command_line

def handle_program(
    metavar_handlers,
    command_handlers,
    option_sets,
    aliases,
    program_options,
    command_options,
    command,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] COMMAND [OPTIONS] ARGS

    Commands (aliases):
     name (n, nm)      find filenames containing a particular word
     content (c, ct)   find files whose content contains a particular word

    Try `%(program)s COMMAND --help' for help on a specific command."""
    # First, are they asking for program help?
    if program_options.has_key('help'):
        # if so provide it no matter what other options are given
        print strip_docsting(
            handle_program.__doc__ % {
                'program': os.path.split(sys.argv[0])[1],
            }
        )
        sys.exit(0)
    else:
        if not command:
            raise getopt.GetoptError("No command specified.")
        # Are they asking for command help:
        if command_options.has_key('help'):
            # if so provide it no matter what other options are given
            print strip_docsting(
                command_handlers[command].__doc__ % {
                    'program': os.path.split(sys.argv[0])[1],
                }
            )
            sys.exit(0)
        if program_options.has_key('verbose'):
            verbose_options = []
            for option in program_options['verbose']:
                verbose_options.append(option['name'])
            verbose_option_names = option_names_from_option_list(
                [option_sets['verbose'][0]]
            )
            quiet_option_names = option_names_from_option_list(
                [option_sets['verbose'][1]]
            )
            if len(verbose_options) > 1:
                raise getopt.GetoptError(
                    "Only specify one of %s"%(
                        ', '.join(verbose_options)
                    )
                )
            elif verbose_options[0] in quiet_option_names:
                logging.basicConfig(level=logging.ERROR)
            elif verbose_options[0] in verbose_option_names:
                logging.basicConfig(level=logging.DEBUG)
        else:
            logging.basicConfig(level=logging.INFO)
        # Now handle the command options and arguments
        command_handlers[command](metavar_handlers, option_sets, command_options, args)

if __name__ == '__main__':
    try:
        command = None
        prog_opts, command_opts, command, args = parse_command_line(
            option_sets,
            aliases,
            metavar_handlers = metavar_handlers,
        )
        handle_program(
            metavar_handlers,
            command_handlers,
            option_sets,
            aliases,
            prog_opts,
            command_opts,
            command,
            args
        )
    except getopt.GetoptError, err:
        # print help information and exit:
        print str(err)
        if command:
            print "Try `%(program)s %(command)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
                'command': command,
            }
        else:
            print "Try `%(program)s --help' for more information." % {
                'program': os.path.split(sys.argv[0])[1],
            }
        sys.exit(2)

Notice that because the call to handle_program() is within a try... except block, any of the option handlers, command handlers or any other code can raise a getopt.GetoptError exception which will result in the error being printed to the screen and a short message printed about how to get help to use the correct options.

Dealing with Config Files

In addition to all the command options, I want users to be able to specify defaults in a config file. If a command doesn't recieve a neccessary command on the command line it can load it from the config file.

The metavars used to label the values associated with options on the command line earlier can also be used as keys in the config file as if the option that specifies the metavar has been added on the command line.

This merging doesn't happen automatically though, if a command handler wants to use the config file it handles the merging itself.

You'll see one of the option sets is for the internal variable config which sets the metavar CONFIG. When combined with a handler which parses the filename specified by the metavar CONFIG, the options can be automatically loaded if the -c or --config options are used.

In the command handler you can then do something like this to set a default value for the a metavar if it isn't specified on the command line.

def handle_name(options, command_options, args):
    ...
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    ...
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    ...

Notice that whether the value of START_DIRECTORY is extracted from the command line or on the config file, it is still run through the handler so the rest of the application beyond the handle_name() function remains the same.

Command Handlers

Now let's look at the command handlers. Their job is to look at the options used for each internal variable and call the appropriate Python API function to generate a result. They must then format the result for output on the command line.

Here's what they look like:

from commandtool import strip_docsting
from commandtool import option_names_from_option_list
from commandtool import set_error_on

def handle_command_name(
    metavar_handlers,
    option_sets,
    command_options,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] name [OPTIONS] LETTERS

    search DIRECTORY for a filename matching LETTERS

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type `%(program)s --help'
    """
    set_error_on(
        command_options,
        allowed=['config', 'start_directory']
    )
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = \
           metavar_handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    for found in name(**internal_vars):
        print found

def handle_command_content(
    metavar_handlers,
    option_sets,
    command_options,
    args
):
    """\
    usage: %(program)s [PROGRAM_OPTIONS] content [OPTIONS] LETTERS

    search DIRECTORY for a files containing LETTERS in their content

    Options:
     -s, --start-directory=START_DIRECTORY    the directory to search
     -t, --file-type=FILE_TYPE                the file types to search
     -c, --config=CONFIG                      the config file to use

    For PROGRAM_OPTIONS type `%(program)s --help'"""
    set_error_on(
        command_options,
        allowed=['config', 'start_directory', 'file_type']
    )
    config = {}
    if command_options.has_key('config'):
        config = command_options['config'][0]['handled']
    internal_vars = {}
    if not len(args)==1:
        raise getopt.GetoptError('Expected exactly one argument, LETTERS')
    internal_vars['letters'] = args[0]
    if command_options.has_key('start_directory'):
        internal_vars['start_directory'] = \
           command_options['start_directory'][0]['handled']
    elif config.has_key('START_DIRECTORY'):
        internal_vars['start_directory'] = metavar_handlers['START_DIRECTORY'](
            config['START_DIRECTORY']
        )
    if command_options.has_key('file_type'):
        internal_vars['file_type'] = \
           command_options['file_type'][0]['value']
    for found in content(**internal_vars):
        print found

command_handlers = {
    'name': handle_command_name,
    'content': handle_command_content,
}

Notice that the docstring for each forms the help text for that command. In a real world applicaiton this approach is much simpler than trying to automatically generate help options. In this example, display of help text for the command is actually handled in handle_program() but if you wanted to be more consistent you could handle it in one of the command handlers.

Within each command handler you can choose to set an error if unexpected options are set. There is a helper function called set_error_on() to help you with this. It will set an error if any options have been set for internal variables you weren't expecting. It is used like this:

set_error_on(
    command_options,
    allowed=['config', 'site_directory', 'file_type'],
)

Summary of the Processing Pipeline

With these features the following actions happen:

• Prepare option_sets, aliases and option_handlers
• Pass them to the parse_command_line() function to have them parsed
• If the options specified aren't valid an error messages is displayed
• If successful and there are some command options, the options are passed through any required handlers to format the values specified
• Otherwise the options are organised into the four variables: program_options, command_options, command, args. The program_options and command_options are dictionaries with the internal variables as their key and a list of all the options which affect that variable as the value. The items in each option list are actually dictionaries with other useful information about the option, such as its position on the original command line, the metavar, help text etc.
• This parsed data can then be sent to a command handler which checks the options are valid for the particular command, sets the internal values it requires based on the options, calls the real Python API with the internal variables and formats the result for display on the command line.