Merge lp:~mwhudson/launchpad/ec2-entrypoints into lp:launchpad

[Michael Hudson-Doyle]

[Michael Hudson-Doyle (mwhudson)]

Hi Jono,

You've seen this branch before. Since last time:

1) A good degree of tidying, mostly as discussed.
2) ec2 help works in most ways that I think it should.
3) No tests :( The fact that the ec2 stuff only works with 2.4+ bites, basically.

[Jonathan Lange (jml)]

Hey Michael,

We've already talked & had an email conversation about this, so I've been brief in my comments. Apologies if it looks like brusqueness.

A fair few questions, so marking as 'needs fixing', even though the general direction is great and the vast majority of the code is fine.

jml

> === added file 'lib/devscripts/ec2test/builtins.py'
> --- lib/devscripts/ec2test/builtins.py 1970-01-01 00:00:00 +0000
> +++ lib/devscripts/ec2test/builtins.py 2009-09-18 03:16:29 +0000
> @@ -0,0 +1,404 @@

Copyright notice, __all__, metaclass thingy.

> +import pdb
> +
> +from bzrlib.commands import Command
> +from bzrlib.errors import BzrCommandError
> +from bzrlib.help import help_commands
> +from bzrlib.option import ListOption, Option
> +
> +import socket
> +
> +from devscripts.ec2test.credentials import EC2Credentials
> +from devscripts.ec2test.instance import (
> + AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
> +from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
> +
> +

A comment before each of these options explaining what they are for would
probably help future maintainers.

> +branch_option = ListOption(
> + 'branch', type=str, short_name='b', argname='BRANCH',
> + help=('Branches to include in this run in sourcecode. '
> + 'If the argument is only the project name, the trunk will be '
> + 'used (e.g., ``-b launchpadlib``). If you want to use a '
> + 'specific branch, if it is on launchpad, you can usually '
> + 'simply specify it instead (e.g., '
> + '``-b lp:~username/launchpadlib/branchname``). If this does '
> + 'not appear to work, or if the desired branch is not on '
> + 'launchpad, specify the project name and then the branch '
> + 'after an equals sign (e.g., '
> + '``-b launchpadlib=lp:~username/launchpadlib/branchname``). '
> + 'Branches for multiple projects may be specified with '
> + 'multiple instances of this option. '
> + 'You may also use this option to specify the branch of launchpad '
> + 'into which your branch may be merged. This defaults to %s. '
> + 'Because typically the important branches of launchpad are owned '
> + 'by the launchpad-pqm user, you can shorten this to only the '
> + 'branch name, if desired, and the launchpad-pqm user will be '
> + 'assumed. For instance, if you specify '
> + '``-b launchpad=db-devel`` then this is equivalent to '
> + '``-b lp:~launchpad-pqm/launchpad/db-devel``, or the even longer'
> + '``-b launchpad=lp:~launchpad-pqm/launchpad/db-devel``.'
> + % (TRUNK_BRANCH,)))
> +
> +
> +machine_id_option = Option(
> + 'machine', short_name='m', type=str,
> + help=('The AWS machine identifier (AMI) on which to base this run. '
> + 'You should typically only have to supply this if you are '
> + 'testing new AWS images. Defaults to trying to find the most '
> + 'recent one with an approved owner.'))
> +
> +
> +def _convert_instance_type(arg):
> + """Ensure that `arg` is acceptable as an instance type."""
> + if arg not in ...

review: Needs Fixing

[Michael Hudson-Doyle (mwhudson)]

Jonathan Lange wrote:
> Review: Needs Fixing
> Hey Michael,
>
> We've already talked & had an email conversation about this, so I've been brief in my comments. Apologies if it looks like brusqueness.
>
> A fair few questions, so marking as 'needs fixing', even though the general direction is great and the vast majority of the code is fine.
>
> jml
>
>> === added file 'lib/devscripts/ec2test/builtins.py'
>> --- lib/devscripts/ec2test/builtins.py 1970-01-01 00:00:00 +0000
>> +++ lib/devscripts/ec2test/builtins.py 2009-09-18 03:16:29 +0000
>> @@ -0,0 +1,404 @@
>
> Copyright notice, __all__, metaclass thingy.

Doh!

>> +import pdb
>> +
>> +from bzrlib.commands import Command
>> +from bzrlib.errors import BzrCommandError
>> +from bzrlib.help import help_commands
>> +from bzrlib.option import ListOption, Option
>> +
>> +import socket
>> +
>> +from devscripts.ec2test.credentials import EC2Credentials
>> +from devscripts.ec2test.instance import (
>> + AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
>> +from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
>> +
>> +
>
> A comment before each of these options explaining what they are for would
> probably help future maintainers.

I added one comment, and beefed up the help strings a bit. I think if
any are still confusing, the help should probably be improved...

>> +branch_option = ListOption(
>> + 'branch', type=str, short_name='b', argname='BRANCH',
>> + help=('Branches to include in this run in sourcecode. '
>> + 'If the argument is only the project name, the trunk will be '
>> + 'used (e.g., ``-b launchpadlib``). If you want to use a '
>> + 'specific branch, if it is on launchpad, you can usually '
>> + 'simply specify it instead (e.g., '
>> + '``-b lp:~username/launchpadlib/branchname``). If this does '
>> + 'not appear to work, or if the desired branch is not on '
>> + 'launchpad, specify the project name and then the branch '
>> + 'after an equals sign (e.g., '
>> + '``-b launchpadlib=lp:~username/launchpadlib/branchname``). '
>> + 'Branches for multiple projects may be specified with '
>> + 'multiple instances of this option. '
>> + 'You may also use this option to specify the branch of launchpad '
>> + 'into which your branch may be merged. This defaults to %s. '
>> + 'Because typically the important branches of launchpad are owned '
>> + 'by the launchpad-pqm user, you can shorten this to only the '
>> + 'branch name, if desired, and the launchpad-pqm user will be '
>> + 'assumed. For instance, if you specify '
>> + '``-b launchpad=db-devel`` then this is equivalent to '
>> + '``-b lp:~launchpad-pqm/launchpad/db-devel``, or the even longer'
>> + '``-b launchpad=lp:~launchpad-pqm/launchpad/db-devel``.'
>> + % (TRUNK_BRANCH,)))
>> +
>> +
>> +machine_id_option = Option(
>> + 'machine', short_name='m', type=str,
>> + help=('The AWS machine identifier (AMI) on which to base this run. '
>> + 'You should typically only have to supply this if you ar...

[Jonathan Lange (jml)]

You have three blank lines after _get_branches_and_test_branch, rather than the regulation two. Other than that, it looks great. Well done!

review: Approve

=== modified file 'lib/devscripts/ec2test/__init__.py'
--- lib/devscripts/ec2test/__init__.py	2009-09-14 05:43:04 +0000
+++ lib/devscripts/ec2test/__init__.py	2009-09-18 02:23:13 +0000
@@ -5,13 +5,11 @@
 
 __metaclass__ = type
 
-__all__ = [
-    'error_and_quit',
-    'main',
-    ]
+__all__ = []
 
+from bzrlib.plugin import load_plugins
+load_plugins()
 import paramiko
-import sys
 
 #############################################################################
 # Try to guide users past support problems we've encountered before
@@ -21,11 +19,3 @@
 # maybe add similar check for bzrlib?
 # End
 #############################################################################
-
-def error_and_quit(msg):
-    """Print error message and exit."""
-    sys.stderr.write(msg)
-    sys.exit(1)
-
-from devscripts.ec2test.commandline import main
-main # shut up pyflakes

=== added file 'lib/devscripts/ec2test/builtins.py'
--- lib/devscripts/ec2test/builtins.py	1970-01-01 00:00:00 +0000
+++ lib/devscripts/ec2test/builtins.py	2009-09-18 03:16:29 +0000
@@ -0,0 +1,404 @@
+import pdb
+
+from bzrlib.commands import Command
+from bzrlib.errors import BzrCommandError
+from bzrlib.help import help_commands
+from bzrlib.option import ListOption, Option
+
+import socket
+
+from devscripts.ec2test.credentials import EC2Credentials
+from devscripts.ec2test.instance import (
+    AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
+from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
+
+
+branch_option = ListOption(
+    'branch', type=str, short_name='b', argname='BRANCH',
+    help=('Branches to include in this run in sourcecode. '
+          'If the argument is only the project name, the trunk will be '
+          'used (e.g., ``-b launchpadlib``).  If you want to use a '
+          'specific branch, if it is on launchpad, you can usually '
+          'simply specify it instead (e.g., '
+          '``-b lp:~username/launchpadlib/branchname``).  If this does '
+          'not appear to work, or if the desired branch is not on '
+          'launchpad, specify the project name and then the branch '
+          'after an equals sign (e.g., '
+          '``-b launchpadlib=lp:~username/launchpadlib/branchname``). '
+          'Branches for multiple projects may be specified with '
+          'multiple instances of this option. '
+          'You may also use this option to specify the branch of launchpad '
+          'into which your branch may be merged.  This defaults to %s. '
+          'Because typically the important branches of launchpad are owned '
+          'by the launchpad-pqm user, you can shorten this to only the '
+          'branch name, if desired, and the launchpad-pqm user will be '
+          'assumed.  For instance, if you specify '
+          '``-b launchpad=db-devel`` then this is equivalent to '
+          '``-b lp:~launchpad-pqm/launchpad/db-devel``, or the even longer'
+          '``-b launchpad=lp:~launchpad-pqm/launchpad/db-devel``.'
+          % (TRUNK_BRANCH,)))
+
+
+machine_id_option = Option(
+    'machine', short_name='m', type=str,
+    help=('The AWS machine identifier (AMI) on which to base this run. '
+          'You should typically only have to supply this if you are '
+          'testing new AWS images. Defaults to trying to find the most '
+          'recent one with an approved owner.'))
+
+
+def _convert_instance_type(arg):
+    """Ensure that `arg` is acceptable as an instance type."""
+    if arg not in AVAILABLE_INSTANCE_TYPES:
+        raise BzrCommandError('Unknown instance type %r' % arg)
+    return arg
+
+
+instance_type_option = Option(
+    'instance', short_name='i', type=_convert_instance_type,
+    param_name='instance_type',
+    help=('The AWS instance type on which to base this run. '
+          'Available options are %r. Defaults to `%s`.' %
+          (AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE)))
+
+
+debug_option = Option(
+    'debug', short_name='d',
+    help=('Drop to pdb trace as soon as possible.'))
+
+
+trunk_option = Option(
+    'trunk', short_name='t',
+    help=('Run the trunk as the branch'))
+
+
+include_download_cache_changes_option = Option(
+    'include-download-cache-changes', short_name='c',
+    help=('Include any changes in the download cache (added or unknown) '
+          'in the download cache of the test run.  Note that, if you have '
+          'any changes in your download cache, trying to submit to pqm '
+          'will always raise an error.  Also note that, if you have any '
+          'changes in your download cache, you must explicitly choose to '
+          'include or ignore the changes.'))
+
+
+class EC2Command(Command):
+    """Subclass of `Command` that customizes usage to say 'ec2' not 'bzr'.
+
+    When https://bugs.edge.launchpad.net/bzr/+bug/431054 is fixed, we can
+    delete this class, or at least make it less of a copy/paste/hack of the
+    superclass.
+    """
+
+    def _usage(self):
+        """Return single-line grammar for this command.
+
+        Only describes arguments, not options.
+        """
+        s = 'ec2 ' + self.name() + ' '
+        for aname in self.takes_args:
+            aname = aname.upper()
+            if aname[-1] in ['$', '+']:
+                aname = aname[:-1] + '...'
+            elif aname[-1] == '?':
+                aname = '[' + aname[:-1] + ']'
+            elif aname[-1] == '*':
+                aname = '[' + aname[:-1] + '...]'
+            s += aname + ' '
+        s = s[:-1]      # remove last space
+        return s
+
+
+class cmd_test(EC2Command):
+    """Run the test suite in ec2."""
+
+    takes_options = [
+        branch_option,
+        trunk_option,
+        machine_id_option,
+        instance_type_option,
+        Option(
+            'file', short_name='f',
+            help=('Store abridged test results in FILE.')),
+        ListOption(
+            'email', short_name='e', argname='EMAIL', type=str,
+            help=('Email address to which results should be mailed.  Defaults to '
+                  'the email address from `bzr whoami`. May be supplied multiple '
+                  'times. The first supplied email address will be used as the '
+                  'From: address.')),
+        Option(
+            'noemail', short_name='n',
+            help=('Do not try to email results.')),
+        Option(
+            'test-options', short_name='o', type=str,
+            help=('Test options to pass to the remote test runner.  Defaults to '
+                  "``-o '-vv'``.  For instance, to run specific tests, you might "
+                  "use ``-o '-vvt my_test_pattern'``.")),
+        Option(
+            'submit-pqm-message', short_name='s', type=str, argname="MSG",
+            help=('A pqm message to submit if the test run is successful.  If '
+                  'provided, you will be asked for your GPG passphrase before '
+                  'the test run begins.')),
+        Option(
+            'pqm-public-location', type=str,
+            help=('The public location for the pqm submit, if a pqm message is '
+                  'provided (see --submit-pqm-message).  If this is not provided, '
+                  'for local branches, bzr configuration is consulted; for '
+                  'remote branches, it is assumed that the remote branch *is* '
+                  'a public branch.')),
+        Option(
+            'pqm-submit-location', type=str,
+            help=('The submit location for the pqm submit, if a pqm message is '
+                  'provided (see --submit-pqm-message).  If this option is not '
+                  'provided, the script will look for an explicitly specified '
+                  'launchpad branch using the -b/--branch option; if that branch '
+                  'was specified and is owned by the launchpad-pqm user on '
+                  'launchpad, it is used as the pqm submit location. Otherwise, '
+                  'for local branches, bzr configuration is consulted; for '
+                  'remote branches, it is assumed that the submit branch is %s.'
+                  % (TRUNK_BRANCH,))),
+        Option(
+            'pqm-email', type=str,
+            help=('Specify the email address of the PQM you are submitting to. '
+                  'If the branch is local, then the bzr configuration is '
+                  'consulted; for remote branches "Launchpad PQM '
+                  '<[e-mail]>" is used by default.')),
+        Option(
+            'postmortem', short_name='p',
+            help=('Drop to interactive prompt after the test and before shutting '
+                  'down the instance for postmortem analysis of the EC2 instance '
+                  'and/or of this script.')),
+        Option(
+            'headless',
+            help=('After building the instance and test, run the remote tests '
+                  'headless.  Cannot be used with postmortem '
+                  'or file.')),
+        debug_option,
+        Option(
+            'open-browser',
+            help=('Open the results page in your default browser')),
+        include_download_cache_changes_option,
+        ]
+
+    takes_args = ['test_branch?']
+
+    def run(self, test_branch=None, branch=[], trunk=False, machine=None,
+            instance_type=DEFAULT_INSTANCE_TYPE,
+            file=None, email=None, test_options='-vv', noemail=False,
+            submit_pqm_message=None, pqm_public_location=None,
+            pqm_submit_location=None, pqm_email=None, postmortem=False,
+            headless=False, debug=False, open_browser=False,
+            include_download_cache_changes=False):
+        if debug:
+            pdb.set_trace()
+        if trunk:
+            if test_branch is not None:
+                raise BzrCommandError(
+                    "Cannot specify both a branch to test and --trunk")
+            else:
+                test_branch = TRUNK_BRANCH
+        else:
+            if test_branch is None:
+                test_branch = '.'
+        if ((postmortem or file) and headless):
+            raise BzrCommandError(
+                'Headless mode currently does not support postmortem or file '
+                ' options.')
+        if noemail:
+            if email:
+                raise BzrCommandError(
+                    'May not supply both --no-email and an --email address')
+        else:
+            if email == []:
+                email = True
+        branches = [data.split('=', 1) for data in branch]
+
+        if headless and not (email or submit_pqm_message):
+            raise BzrCommandError(
+                'You have specified no way to get the results '
+                'of your headless test run.')
+
+
+        instance = EC2Instance(
+            EC2TestRunner.name, instance_type, machine)
+
+        runner = EC2TestRunner(
+            test_branch, email=email, file=file,
+            test_options=test_options, headless=headless,
+            branches=branches, pqm_message=submit_pqm_message,
+            pqm_public_location=pqm_public_location,
+            pqm_submit_location=pqm_submit_location,
+            open_browser=open_browser, pqm_email=pqm_email,
+            include_download_cache_changes=include_download_cache_changes,
+            instance=instance, vals=instance._vals)
+
+        instance.set_up_and_run(postmortem, not headless, runner.run_tests)
+
+
+class cmd_demo(EC2Command):
+    """Start a demo instance of Launchpad.
+
+    See https://wiki.canonical.com/Launchpad/EC2Test/ForDemos
+    """
+
+    takes_options = [
+        branch_option,
+        trunk_option,
+        machine_id_option,
+        instance_type_option,
+        Option(
+            'postmortem', short_name='p',
+            help=('Drop to interactive prompt after the test and before shutting '
+                  'down the instance for postmortem analysis of the EC2 instance '
+                  'and/or of this script.')),
+        debug_option,
+        include_download_cache_changes_option,
+        ListOption(
+            'demo', type=str,
+            help="Allow this netmask to connect to the instance."),
+        ]
+
+    takes_args = ['test_branch?']
+
+    def run(self, test_branch=None, branch=[], trunk=False, machine=None,
+            instance_type=DEFAULT_INSTANCE_TYPE, debug=False,
+            include_download_cache_changes=False, demo=None):
+        if debug:
+            pdb.set_trace()
+        if trunk:
+            if test_branch is not None:
+                raise BzrCommandError(
+                    "Cannot specify both a branch to test and --trunk")
+            else:
+                test_branch = TRUNK_BRANCH
+        else:
+            if test_branch is None:
+                test_branch = '.'
+        branches = [data.split('=', 1) for data in branch]
+
+        instance = EC2Instance.make(
+            EC2TestRunner.name, instance_type, machine, demo)
+
+        runner = EC2TestRunner(
+            test_branch, branches=branches,
+            include_download_cache_changes=include_download_cache_changes,
+            instance=instance, vals=instance._vals)
+
+        demo_network_string = '\n'.join(
+            '  ' + network for network in demo)
+
+        instance.set_up_and_run(
+            True, False, self.run_server, runner,
+            instance.hostname, demo_network_string)
+
+
+    def run_server(self, runner, hostname, demo_network_string):
+        runner.run_demo_server()
+        ec2_ip = socket.gethostbyname(hostname)
+        print (
+            "\n\n"
+            "********************** DEMO *************************\n"
+            "It may take 20 seconds for the demo server to start up."
+            "\nTo demo to other users, you still need to open up\n"
+            "network access to the ec2 instance from their IPs by\n"
+            "entering command like this in the interactive python\n"
+            "interpreter at the end of the setup. "
+            "\n  instance.security_group.authorize("
+            "'tcp', 443, 443, '10.0.0.5/32')\n\n"
+            "These demo networks have already been granted access on "
+            "port 80 and 443:\n" + demo_network_string +
+            "\n\nYou also need to edit your /etc/hosts to point\n"
+            "launchpad.dev at the ec2 instance's IP like this:\n"
+            "  " + ec2_ip + "    launchpad.dev\n\n"
+            "See "
+            "<https://wiki.canonical.com/Launchpad/EC2Test/ForDemos>."
+            "\n*****************************************************"
+            "\n\n")
+
+
+
+class cmd_update_image(EC2Command):
+    """Make a new AMI."""
+
+    takes_options = [
+        machine_id_option,
+        instance_type_option,
+        Option(
+            'postmortem', short_name='p',
+            help=('Drop to interactive prompt after the test and before shutting '
+                  'down the instance for postmortem analysis of the EC2 instance '
+                  'and/or of this script.')),
+        debug_option,
+        ListOption(
+            'extra-update-image-command', type=str,
+            help=('Run this command (with an ssh agent) on the image before '
+                  'running the default update steps.  Can be passed more than '
+                  'once, the commands will be run in the order specified.')),
+        ]
+
+    takes_args = ['ami_name']
+
+    def run(self, ami_name, machine=None, instance_type='m1.large',
+            debug=False, postmortem=False, extra_update_image_command=[]):
+        if debug:
+            pdb.set_trace()
+
+        credentials = EC2Credentials.load_from_file()
+
+        instance = EC2Instance.make(
+            EC2TestRunner.name, instance_type, machine,
+            credentials=credentials)
+        instance.check_bundling_prerequisites()
+
+        instance.set_up_and_run(
+            postmortem, True, self.update_image, instance,
+            extra_update_image_command, ami_name, credentials)
+
+    def update_image(self, instance, extra_update_image_command, ami_name,
+                     credentials):
+        user_connection = instance.connect_as_user()
+        user_connection.perform('bzr launchpad-login %(launchpad-login)s')
+        for cmd in extra_update_image_command:
+            user_connection.run_with_ssh_agent(cmd)
+        user_connection.run_with_ssh_agent(
+            "rsync -avp --partial --delete "
+            "--filter='P *.o' --filter='P *.pyc' --filter='P *.so' "
+            "devpad.canonical.com:/code/rocketfuel-built/launchpad/sourcecode/* "
+            "/var/launchpad/sourcecode/")
+        user_connection.run_with_ssh_agent(
+            'bzr pull -d /var/launchpad/test ' + TRUNK_BRANCH)
+        user_connection.run_with_ssh_agent(
+            'bzr pull -d /var/launchpad/download-cache lp:lp-source-dependencies')
+        user_connection.close()
+        root_connection = instance.connect_as_root()
+        root_connection.perform(
+            'deluser --remove-home %(USER)s', ignore_failure=True)
+        root_connection.close()
+        instance.bundle(ami_name, credentials)
+
+
+class cmd_help(EC2Command):
+    """Show general help or help for a command."""
+
+    aliases = ["?", "--help", "-?", "-h"]
+    takes_args = ["topic?"]
+
+    def run(self, topic=None):
+        """
+        Show help for the C{bzrlib.commands.Command} matching C{topic}.
+
+        @param topic: Optionally, the name of the topic to show.  Default is
+            to show some basic usage information.
+        """
+        if topic is None:
+            print >>self.outf, 'Usage:    ec2 <command> <options>'
+            print >>self.outf
+            print >>self.outf, 'Available commands:'
+            help_commands(self.outf)
+        else:
+            command = self.controller._get_command(None, topic)
+            if command is None:
+                print >>self.outf, "%s is an unknown command." % (topic,)
+            text = command.get_help_text()
+            if text:
+                print >>self.outf, text

=== removed file 'lib/devscripts/ec2test/commandline.py'
--- lib/devscripts/ec2test/commandline.py	2009-09-15 01:37:01 +0000
+++ lib/devscripts/ec2test/commandline.py	1970-01-01 00:00:00 +0000
@@ -1,369 +0,0 @@
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""The command line parsing and entrypoint for ec2test."""
-
-__metaclass__ = type
-__all__ = [
-    'main',
-    ]
-
-import code
-import optparse
-import socket
-import traceback
-# The rlcompleter and readline modules change the behavior of the python
-# interactive interpreter just by being imported.
-import readline
-import rlcompleter
-# Shut up pyflakes.
-rlcompleter
-
-import paramiko
-
-from devscripts.ec2test import error_and_quit
-from devscripts.ec2test.credentials import CredentialsError, EC2Credentials
-from devscripts.ec2test.instance import (
-    AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
-from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
-
-readline.parse_and_bind('tab: complete')
-
-
-# XXX: JonathanLange 2009-05-31: Strongly considering turning this into a
-# Bazaar plugin -- probably would make the option parsing and validation
-# easier.
-
-def run_with_instance(instance, run, demo_networks, postmortem):
-    """Call run(), then allow post mortem debugging and shut down `instance`.
-
-    :param instance: A running `EC2Instance`.  If `run` returns True, it will
-        be shut down before this function returns.
-    :param run: A callable that will be called with no arguments to do
-        whatever needs to be done with the instance.
-    :param demo_networks: ???
-    :param postmortem: If this flag is true, any exceptions will be caught and
-        an interactive session run to allow debugging the problem.
-    """
-    shutdown = True
-    try:
-        try:
-            shutdown = run()
-        except Exception:
-            # If we are running in demo or postmortem mode, it is really
-            # helpful to see if there are any exceptions before it waits
-            # in the console (in the finally block), and you can't figure
-            # out why it's broken.
-            traceback.print_exc()
-    finally:
-        try:
-            if demo_networks:
-                demo_network_string = '\n'.join(
-                    '  ' + network for network in demo_networks)
-                ec2_ip = socket.gethostbyname(instance.hostname)
-                print (
-                    "\n\n"
-                    "********************** DEMO *************************\n"
-                    "It may take 20 seconds for the demo server to start up."
-                    "\nTo demo to other users, you still need to open up\n"
-                    "network access to the ec2 instance from their IPs by\n"
-                    "entering command like this in the interactive python\n"
-                    "interpreter at the end of the setup. "
-                    "\n  runner.security_group.authorize("
-                    "'tcp', 443, 443, '10.0.0.5/32')\n\n"
-                    "These demo networks have already been granted access on "
-                    "port 80 and 443:\n" + demo_network_string +
-                    "\n\nYou also need to edit your /etc/hosts to point\n"
-                    "launchpad.dev at the ec2 instance's IP like this:\n"
-                    "  " + ec2_ip + "    launchpad.dev\n\n"
-                    "See "
-                    "<https://wiki.canonical.com/Launchpad/EC2Test/ForDemos>."
-                    "\n*****************************************************"
-                    "\n\n")
-            if postmortem:
-                console = code.InteractiveConsole(locals())
-                console.interact((
-                    'Postmortem Console.  EC2 instance is not yet dead.\n'
-                    'It will shut down when you exit this prompt (CTRL-D).\n'
-                    '\n'
-                    'Tab-completion is enabled.'
-                    '\n'
-                    'Test runner instance is available as `runner`.\n'
-                    'Also try these:\n'
-                    '  http://%(dns)s/current_test.log\n'
-                    '  ssh -A %(dns)s') %
-                                 {'dns': instance.hostname})
-                print 'Postmortem console closed.'
-        finally:
-            if shutdown:
-                instance.shutdown()
-
-
-def main():
-    parser = optparse.OptionParser(
-        usage="%prog [options] [branch]",
-        description=(
-            "Check out a Launchpad branch and run all tests on an Amazon "
-            "EC2 instance."))
-    parser.add_option(
-        '-f', '--file', dest='file', default=None,
-        help=('Store abridged test results in FILE.'))
-    parser.add_option(
-        '-n', '--no-email', dest='no_email', default=False,
-        action='store_true',
-        help=('Do not try to email results.'))
-    parser.add_option(
-        '-e', '--email', action='append', dest='email', default=None,
-        help=('Email address to which results should be mailed.  Defaults to '
-              'the email address from `bzr whoami`. May be supplied multiple '
-              'times. The first supplied email address will be used as the '
-              'From: address.'))
-    parser.add_option(
-        '-o', '--test-options', dest='test_options', default='-vv',
-        help=('Test options to pass to the remote test runner.  Defaults to '
-              "``-o '-vv'``.  For instance, to run specific tests, you might "
-              "use ``-o '-vvt my_test_pattern'``."))
-    parser.add_option(
-        '-b', '--branch', action='append', dest='branches',
-        help=('Branches to include in this run in sourcecode. '
-              'If the argument is only the project name, the trunk will be '
-              'used (e.g., ``-b launchpadlib``).  If you want to use a '
-              'specific branch, if it is on launchpad, you can usually '
-              'simply specify it instead (e.g., '
-              '``-b lp:~username/launchpadlib/branchname``).  If this does '
-              'not appear to work, or if the desired branch is not on '
-              'launchpad, specify the project name and then the branch '
-              'after an equals sign (e.g., '
-              '``-b launchpadlib=lp:~username/launchpadlib/branchname``). '
-              'Branches for multiple projects may be specified with '
-              'multiple instances of this option. '
-              'You may also use this option to specify the branch of launchpad '
-              'into which your branch may be merged.  This defaults to %s. '
-              'Because typically the important branches of launchpad are owned '
-              'by the launchpad-pqm user, you can shorten this to only the '
-              'branch name, if desired, and the launchpad-pqm user will be '
-              'assumed.  For instance, if you specify '
-              '``-b launchpad=db-devel`` then this is equivalent to '
-              '``-b lp:~launchpad-pqm/launchpad/db-devel``, or the even longer'
-              '``-b launchpad=lp:~launchpad-pqm/launchpad/db-devel``.'
-              % (TRUNK_BRANCH,)))
-    parser.add_option(
-        '-t', '--trunk', dest='trunk', default=False,
-        action='store_true',
-        help=('Run the trunk as the branch'))
-    parser.add_option(
-        '-s', '--submit-pqm-message', dest='pqm_message', default=None,
-        help=('A pqm message to submit if the test run is successful.  If '
-              'provided, you will be asked for your GPG passphrase before '
-              'the test run begins.'))
-    parser.add_option(
-        '--pqm-public-location', dest='pqm_public_location', default=None,
-        help=('The public location for the pqm submit, if a pqm message is '
-              'provided (see --submit-pqm-message).  If this is not provided, '
-              'for local branches, bzr configuration is consulted; for '
-              'remote branches, it is assumed that the remote branch *is* '
-              'a public branch.'))
-    parser.add_option(
-        '--pqm-submit-location', dest='pqm_submit_location', default=None,
-        help=('The submit location for the pqm submit, if a pqm message is '
-              'provided (see --submit-pqm-message).  If this option is not '
-              'provided, the script will look for an explicitly specified '
-              'launchpad branch using the -b/--branch option; if that branch '
-              'was specified and is owned by the launchpad-pqm user on '
-              'launchpad, it is used as the pqm submit location. Otherwise, '
-              'for local branches, bzr configuration is consulted; for '
-              'remote branches, it is assumed that the submit branch is %s.'
-              % (TRUNK_BRANCH,)))
-    parser.add_option(
-        '--pqm-email', dest='pqm_email', default=None,
-        help=('Specify the email address of the PQM you are submitting to. '
-              'If the branch is local, then the bzr configuration is '
-              'consulted; for remote branches "Launchpad PQM '
-              '<[e-mail]>" is used by default.'))
-    parser.add_option(
-        '-m', '--machine', dest='machine_id', default=None,
-        help=('The AWS machine identifier (AMID) on which to base this run. '
-              'You should typically only have to supply this if you are '
-              'testing new AWS images. Defaults to trying to find the most '
-              'recent one with an approved owner.'))
-    parser.add_option(
-        '-i', '--instance', dest='instance_type',
-        default=DEFAULT_INSTANCE_TYPE,
-        help=('The AWS instance type on which to base this run. '
-              'Available options are %r. Defaults to `%s`.' %
-              (AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE)))
-    parser.add_option(
-        '-p', '--postmortem', dest='postmortem', default=False,
-        action='store_true',
-        help=('Drop to interactive prompt after the test and before shutting '
-              'down the instance for postmortem analysis of the EC2 instance '
-              'and/or of this script.'))
-    parser.add_option(
-        '--headless', dest='headless', default=False,
-        action='store_true',
-        help=('After building the instance and test, run the remote tests '
-              'headless.  Cannot be used with postmortem '
-              'or file.'))
-    parser.add_option(
-        '-d', '--debug', dest='debug', default=False,
-        action='store_true',
-        help=('Drop to pdb trace as soon as possible.'))
-    # Use tabs to force a newline in the help text.
-    fake_newline = "\t\t\t\t\t\t\t"
-    parser.add_option(
-        '--demo', action='append', dest='demo_networks',
-        help=("Don't run tests. Instead start a demo instance of Launchpad. "
-              "You can allow multiple networks to access the demo by "
-              "repeating the argument." + fake_newline +
-              "Example: --demo 192.168.1.100 --demo 10.1.13.0/24" +
-              fake_newline +
-              "See" + fake_newline +
-              "https://wiki.canonical.com/Launchpad/EC2Test/ForDemos" ))
-    parser.add_option(
-        '--open-browser', dest='open_browser', default=False,
-        action='store_true',
-        help=('Open the results page in your default browser'))
-    parser.add_option(
-        '-c', '--include-download-cache-changes',
-        dest='include_download_cache_changes', action='store_true',
-        help=('Include any changes in the download cache (added or unknown) '
-              'in the download cache of the test run.  Note that, if you have '
-              'any changes in your download cache, trying to submit to pqm '
-              'will always raise an error.  Also note that, if you have any '
-              'changes in your download cache, you must explicitly choose to '
-              'include or ignore the changes.'))
-    parser.add_option(
-        '-g', '--ignore-download-cache-changes',
-        dest='include_download_cache_changes', action='store_false',
-        help=('Ignore any changes in the download cache (added or unknown) '
-              'in the download cache of the test run.  Note that, if you have '
-              'any changes in your download cache, trying to submit to pqm '
-              'will always raise an error.  Also note that, if you have any '
-              'changes in your download cache, you must explicitly choose to '
-              'include or ignore the changes.'))
-    parser.add_option(
-        '--update-image', dest='bundle', action='store',
-        help=('Start the image, update the system packages, sourcecode and '
-              'Launchpad branch then bundle, upload and register a new AMI '
-              'with the given name.'))
-    parser.add_option(
-        '--extra-update-image-command', dest='extra_update_image_commands',
-        action='append', metavar="CMD",
-        help=('Run this command (with an ssh agent) on the image before '
-              'running the default update steps.  Can be passed more than '
-              'once, the commands will be run in the order specified.'))
-    options, args = parser.parse_args()
-    if options.debug:
-        import pdb; pdb.set_trace()
-    if options.demo_networks:
-        # We need the postmortem console to open the ec2 instance's
-        # network access, and to keep the ec2 instance from being shutdown.
-        options.postmortem = True
-    if len(args) == 1:
-        if options.trunk:
-            parser.error(
-                'Cannot supply both a branch and the --trunk argument.')
-        branch = args[0]
-    elif len(args) > 1:
-        parser.error('Too many arguments.')
-    elif options.trunk:
-        branch = None
-    else:
-        branch = '.'
-    if ((options.postmortem or options.file or options.demo_networks)
-        and options.headless):
-        parser.error(
-            'Headless mode currently does not support postmortem, file '
-            'or demo options.')
-    if options.no_email:
-        if options.email:
-            parser.error(
-                'May not supply both --no-email and an --email address')
-        email = False
-    else:
-        email = options.email
-        if email is None:
-            email = True
-    if options.instance_type not in AVAILABLE_INSTANCE_TYPES:
-        parser.error('Unknown instance type.')
-    if options.branches is None:
-        branches = ()
-    else:
-        branches = [data.split('=', 1) for data in options.branches]
-
-    agent = paramiko.Agent()
-    keys = agent.get_keys()
-    if len(keys) == 0:
-        error_and_quit(
-            'You must have an ssh agent running with keys installed that '
-            'will allow the script to rsync to devpad and get your '
-            'branch.\n')
-    user_key = agent.get_keys()[0]
-
-    if options.demo_networks is None:
-        demo_networks = ()
-    else:
-        demo_networks = options.demo_networks
-
-    # Get the AWS identifier and secret identifier.
-    try:
-        credentials = EC2Credentials.load_from_file()
-    except CredentialsError, e:
-        error_and_quit(str(e))
-
-    instance = EC2Instance.make(
-        credentials, EC2TestRunner.name, instance_type=options.instance_type,
-        machine_id=options.machine_id, demo_networks=demo_networks)
-
-    if not options.bundle:
-        runner = EC2TestRunner(
-            branch, email=email, file=options.file,
-            test_options=options.test_options, headless=options.headless,
-            branches=branches,
-            pqm_message=options.pqm_message,
-            pqm_public_location=options.pqm_public_location,
-            pqm_submit_location=options.pqm_submit_location,
-            open_browser=options.open_browser, pqm_email=options.pqm_email,
-            include_download_cache_changes=options.include_download_cache_changes,
-            instance=instance, vals=instance._vals,
-            )
-        def run_tests():
-            runner.configure_system()
-            runner.prepare_tests()
-            if demo_networks:
-                runner.start_demo_webserver()
-            else:
-                runner.run_tests()
-            return not options.headless
-        run = run_tests
-    else:
-        instance.check_bundling_prerequisites()
-        def make_new_image():
-            user_connection = instance.connect_as_user()
-            user_connection.perform('bzr launchpad-login %(launchpad-login)s')
-            if options.extra_update_image_commands:
-                for cmd in options.extra_update_image_commands:
-                    user_connection.run_with_ssh_agent(cmd)
-            user_connection.run_with_ssh_agent(
-                "rsync -avp --partial --delete "
-                "--filter='P *.o' --filter='P *.pyc' --filter='P *.so' "
-                "devpad.canonical.com:/code/rocketfuel-built/launchpad/sourcecode/* "
-                "/var/launchpad/sourcecode/")
-            user_connection.run_with_ssh_agent(
-                'bzr pull -d /var/launchpad/test ' + TRUNK_BRANCH)
-            user_connection.run_with_ssh_agent(
-                'bzr pull -d /var/launchpad/download-cache lp:lp-source-dependencies')
-            user_connection.close()
-            root_connection = instance.connect_as_root()
-            root_connection.perform(
-                'deluser --remove-home %(USER)s', ignore_failure=True)
-            root_connection.close()
-            instance.bundle(options.bundle, credentials)
-            return True
-        run = make_new_image
-
-    instance.start()
-    instance.set_up_user(user_key)
-
-    run_with_instance(
-        instance, run, options.demo_networks, options.postmortem)

=== added file 'lib/devscripts/ec2test/controller.py'
--- lib/devscripts/ec2test/controller.py	1970-01-01 00:00:00 +0000
+++ lib/devscripts/ec2test/controller.py	2009-09-18 03:00:35 +0000
@@ -0,0 +1,178 @@
+# This file is incuded almost verbatim from commandant,
+# https://launchpad.net/commandant.  The only changes are removing some code
+# we don't use that depends on other parts of commandant.  When Launchpad is
+# on Python 2.5 we can include commandant as an egg.
+
+
+# Commandant is a framework for building command-oriented tools.
+# Copyright (C) 2009 Jamshed Kakar.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Infrastructure to run C{bzrlib.commands.Command}s and L{HelpTopic}s."""
+
+import os
+import sys
+
+from bzrlib.commands import run_bzr, Command
+
+
+class CommandRegistry(object):
+
+    def __init__(self):
+        self._commands = {}
+
+    def install_bzrlib_hooks(self):
+        """
+        Register this controller with C{Command.hooks} so that the controller
+        can take advantage of Bazaar's command infrastructure.
+
+        L{_list_commands} and L{_get_command} are registered as callbacks for
+        the C{list_commands} and C{get_commands} hooks, respectively.
+        """
+        Command.hooks.install_named_hook(
+            "list_commands", self._list_commands, "commandant commands")
+        Command.hooks.install_named_hook(
+            "get_command", self._get_command, "commandant commands")
+
+    def _list_commands(self, names):
+        """Hook to find C{bzrlib.commands.Command} names is called by C{bzrlib}.
+
+        @param names: A set of C{bzrlib.commands.Command} names to update with
+            names from this controller.
+        """
+        names.update(self._commands.iterkeys())
+        return names
+
+    def _get_command(self, command, name):
+        """
+        Hook to get the C{bzrlib.commands.Command} for C{name} is called by
+        C{bzrlib}.
+
+        @param command: A C{bzrlib.commands.Command}, or C{None}, to be
+            returned if a command matching C{name} can't be found.
+        @param name: The name of the C{bzrlib.commands.Command} to retrieve.
+        @return: The C{bzrlib.commands.Command} from the index or C{command}
+            if one isn't available for C{name}.
+        """
+        try:
+            local_command = self._commands[name]()
+        except KeyError:
+            for cmd in self._commands.itervalues():
+                if name in cmd.aliases:
+                    local_command = cmd()
+                    break
+            else:
+                return command
+        local_command.controller = self
+        return local_command
+
+    def register_command(self, name, command_class):
+        """Register a C{bzrlib.commands.Command} with this controller.
+
+        @param name: The name to register the command with.
+        @param command_class: A type object, typically a subclass of
+            C{bzrlib.commands.Command} to use when the command is invoked.
+        """
+        self._commands[name] = command_class
+
+    def load_module(self, module):
+        """Load C{bzrlib.commands.Command}s and L{HelpTopic}s from C{module}.
+
+        Objects found in the module with names that start with C{cmd_} are
+        treated as C{bzrlib.commands.Command}s and objects with names that
+        start with C{topic_} are treated as L{HelpTopic}s.
+        """
+        for name in module.__dict__:
+            if name.startswith("cmd_"):
+                sanitized_name = name[4:].replace("_", "-")
+                self.register_command(sanitized_name, module.__dict__[name])
+            elif name.startswith("topic_"):
+                sanitized_name = name[6:].replace("_", "-")
+                self.register_help_topic(sanitized_name, module.__dict__[name])
+
+
+class HelpTopicRegistry(object):
+
+    def __init__(self):
+        self._help_topics = {}
+
+    def register_help_topic(self, name, help_topic_class):
+        """Register a C{bzrlib.commands.Command} to this controller.
+
+        @param name: The name to register the command with.
+        @param command_class: A type object, typically a subclass of
+            C{bzrlib.commands.Command} to use when the command is invoked.
+        """
+        self._help_topics[name] = help_topic_class
+
+    def get_help_topic_names(self):
+        """Get a C{set} of help topic names."""
+        return set(self._help_topics.iterkeys())
+
+    def get_help_topic(self, name):
+        """
+        Get the help topic matching C{name} or C{None} if a match isn't found.
+        """
+        try:
+            help_topic = self._help_topics[name]()
+        except KeyError:
+            return None
+        help_topic.controller = self
+        return help_topic
+
+
+
+class CommandExecutionMixin(object):
+
+    def run(self, argv):
+        """Run the C{bzrlib.commands.Command} specified in C{argv}.
+
+        @raise BzrCommandError: Raised if a matching command can't be found.
+        """
+        run_bzr(argv)
+
+
+
+def import_module(filename, file_path, package_path):
+    """Import a module and make it a child of C{commandant_command}.
+
+    The module source in C{filename} at C{file_path} is copied to a temporary
+    directory, a Python package called C{commandant_command}.
+
+    @param filename: The name of the module file.
+    @param file_path: The path to the module file.
+    @param package_path: The path for the new C{commandant_command} package.
+    @return: The new module.
+    """
+    module_path = os.path.join(package_path, "commandant_command")
+    if not os.path.exists(module_path):
+        os.mkdir(module_path)
+
+    init_path = os.path.join(module_path, "__init__.py")
+    open(init_path, "w").close()
+
+    source_code = open(file_path, "r").read()
+    module_file_path = os.path.join(module_path, filename)
+    module_file = open(module_file_path, "w")
+    module_file.write(source_code)
+    module_file.close()
+
+    name = filename[:-3]
+    sys.path.append(package_path)
+    try:
+        return __import__("commandant_command.%s" % (name,), fromlist=[name])
+    finally:
+        sys.path.pop()

=== modified file 'lib/devscripts/ec2test/credentials.py'
--- lib/devscripts/ec2test/credentials.py	2009-09-14 05:14:44 +0000
+++ lib/devscripts/ec2test/credentials.py	2009-09-18 01:33:42 +0000
@@ -13,9 +13,11 @@
 
 import boto
 
+from bzrlib.errors import BzrCommandError
+
 from devscripts.ec2test.account import EC2Account
 
-class CredentialsError(Exception):
+class CredentialsError(BzrCommandError):
     """Raised when AWS credentials could not be loaded."""
 
     def __init__(self, filename, extra=None):

=== added file 'lib/devscripts/ec2test/entrypoint.py'
--- lib/devscripts/ec2test/entrypoint.py	1970-01-01 00:00:00 +0000
+++ lib/devscripts/ec2test/entrypoint.py	2009-09-18 03:10:25 +0000
@@ -0,0 +1,32 @@
+import readline
+import rlcompleter
+import sys
+
+from bzrlib.errors import BzrCommandError
+
+from devscripts.ec2test import builtins
+from devscripts.ec2test.controller import (
+    CommandRegistry, CommandExecutionMixin)
+
+# Shut up pyflakes.
+rlcompleter
+
+readline.parse_and_bind('tab: complete')
+
+class EC2CommandController(CommandRegistry, CommandExecutionMixin):
+    def __init__(self):
+        CommandRegistry.__init__(self)
+
+
+def main():
+    controller = EC2CommandController()
+    controller.install_bzrlib_hooks()
+    controller.load_module(builtins)
+
+    args = sys.argv[1:]
+    if not args:
+        args = ['help']
+    try:
+        controller.run(args)
+    except BzrCommandError, e:
+        sys.exit('ec2: ERROR: ' + str(e))

=== modified file 'lib/devscripts/ec2test/instance.py'
--- lib/devscripts/ec2test/instance.py	2009-09-14 05:43:04 +0000
+++ lib/devscripts/ec2test/instance.py	2009-09-18 02:23:13 +0000
@@ -8,6 +8,7 @@
     'EC2Instance',
     ]
 
+import code
 import glob
 import os
 import select
@@ -15,13 +16,15 @@
 import subprocess
 import sys
 import time
+import traceback
 
+from bzrlib.errors import BzrCommandError
 from bzrlib.plugins.launchpad.account import get_lp_login
 
 import paramiko
 
-from devscripts.ec2test import error_and_quit
 from devscripts.ec2test.sshconfig import SSHConfig
+from devscripts.ec2test.credentials import EC2Credentials
 
 DEFAULT_INSTANCE_TYPE = 'c1.xlarge'
 AVAILABLE_INSTANCE_TYPES = ('m1.large', 'm1.xlarge', 'c1.xlarge')
@@ -36,23 +39,45 @@
         pass
 
 
+def get_user_key():
+    """Get a SSH key from the agent.  Exit if not found.
+
+    This key will be used to let the user log in (as $USER) to the instance.
+    """
+    agent = paramiko.Agent()
+    keys = agent.get_keys()
+    if len(keys) == 0:
+        raise BzrCommandError(
+            'You must have an ssh agent running with keys installed that '
+            'will allow the script to rsync to devpad and get your '
+            'branch.\n')
+    user_key = agent.get_keys()[0]
+    return user_key
+
+
 class EC2Instance:
     """A single EC2 instance."""
 
     @classmethod
-    def make(cls, credentials, name, instance_type, machine_id, demo_networks):
+    def make(cls, name, instance_type, machine_id,
+             demo_networks=None, credentials=None):
         """Construct an `EC2Instance`.
 
-        :param credentials: An `EC2Credentials` object.
         :param name: The name to use for the key pair and security group for
             the instance.
         :param instance_type: One of the AVAILABLE_INSTANCE_TYPES.
-        :param machine_id: ???
-        :param demo_networks: ???
+        :param machine_id: The AMI to use, or None to do the usual regexp
+            matching.
+        :param demo_networks: The networks to add to the security group to
+            allow access to the instance.
+        :param credentials: An `EC2Credentials` object.
         """
         if instance_type not in AVAILABLE_INSTANCE_TYPES:
             raise ValueError('unknown instance_type %s' % (instance_type,))
 
+        if credentials is None:
+            credentials = EC2Credentials.load_from_file()
+
         # Make the EC2 connection.
         account = credentials.connect(name)
 
@@ -71,7 +96,7 @@
         vals = os.environ.copy()
         login = get_lp_login()
         if not login:
-            error_and_quit(
+            raise BzrCommandError(
                 'you must have set your launchpad login in bzr.')
         vals['launchpad-login'] = login
 
@@ -104,7 +129,7 @@
             return
         start = time.time()
         self.private_key = self._account.acquire_private_key()
-        self._account.acquire_security_group(
+        self.security_group = self._account.acquire_security_group(
             demo_networks=self._demo_networks)
         reservation = self._image.run(
             key_name=self._name, security_groups=[self._name],
@@ -123,7 +148,7 @@
             self._output = self._boto_instance.get_console_output()
             self.log(self._output.output)
         else:
-            error_and_quit(
+            raise BzrCommandError(
                 'failed to start: %s\n' % self._boto_instance.state)
 
     def shutdown(self):
@@ -241,6 +266,50 @@
         as_root('rm -fr /var/tmp/*')
         root_connection.close()
 
+    def set_up_and_run(self, postmortem, shutdown, func, *args, **kw):
+        """Start and set up, run `func` and then optionally shut down.
+
+        :param postmortem: If true, any exceptions will be caught and an
+            interactive session run to allow debugging the problem.
+        :param shutdown: If true, the instance will be shut down before this
+            function returns.
+        :param func: A callable that will be called when the instance is
+            running and a user account has been set up on it.
+        :param args: Passed to `func`.
+        :param kw: Passed to `func`.
+        """
+        user_key = get_user_key()
+        self.start()
+        try:
+            self.set_up_user(user_key)
+            try:
+                return func(*args, **kw)
+            except Exception:
+                # When running in postmortem mode, it is really helpful to see if
+                # there are any exceptions before it waits in the console (in the
+                # finally block), and you can't figure out why it's broken.
+                traceback.print_exc()
+        finally:
+            try:
+                if postmortem:
+                    console = code.InteractiveConsole(locals())
+                    console.interact((
+                        'Postmortem Console.  EC2 instance is not yet dead.\n'
+                        'It will shut down when you exit this prompt (CTRL-D).\n'
+                        '\n'
+                        'Tab-completion is enabled.'
+                        '\n'
+                        'EC2Instance is available as `instance`.\n'
+                        'Also try these:\n'
+                        '  http://%(dns)s/current_test.log\n'
+                        '  ssh -A %(dns)s') %
+                                     {'dns': self.hostname})
+                    print 'Postmortem console closed.'
+            finally:
+                if shutdown:
+                    self.shutdown()
+        
+
     def _copy_single_file(self, sftp, local_path, remote_dir):
         """Copy `local_path` to `remote_dir` on this instance.
 
@@ -281,7 +350,7 @@
         pattern = os.path.join(local_dir, pattern)
         matches = glob.glob(pattern)
         if len(matches) != 1:
-            error_and_quit(
+            raise BzrCommandError(
                 '%r must match a single %s file' % (pattern, file_kind))
         return matches[0]
 
@@ -290,12 +359,12 @@
         """
         local_ec2_dir = os.path.expanduser('~/.ec2')
         if not os.path.exists(local_ec2_dir):
-            error_and_quit(
+            raise BzrCommandError(
                 "~/.ec2 must exist and contain aws_user, aws_id, a private "
                 "key file and a certificate.")
         aws_user_file = os.path.expanduser('~/.ec2/aws_user')
         if not os.path.exists(aws_user_file):
-            error_and_quit(
+            raise BzrCommandError(
                 "~/.ec2/aws_user must exist and contain your numeric AWS id.")
         self.aws_user = open(aws_user_file).read().strip()
         self.local_cert = self._check_single_glob_match(

=== modified file 'lib/devscripts/ec2test/testrunner.py'
--- lib/devscripts/ec2test/testrunner.py	2009-09-15 00:14:56 +0000
+++ lib/devscripts/ec2test/testrunner.py	2009-09-18 02:23:13 +0000
@@ -14,9 +14,6 @@
 import re
 import sys
 
-
-from bzrlib.plugin import load_plugins
-load_plugins()
 from bzrlib.branch import Branch
 from bzrlib.bzrdir import BzrDir
 from bzrlib.config import GlobalConfig
@@ -169,12 +166,6 @@
         self.headless = headless
         self.include_download_cache_changes = include_download_cache_changes
         self.open_browser = open_browser
-        if headless and file:
-            raise ValueError(
-                'currently do not support files with headless mode.')
-        if headless and not (email or pqm_message):
-            raise ValueError('You have specified no way to get the results '
-                             'of your headless test run.')
 
         if test_options != '-vv' and pqm_message is not None:
             raise ValueError(
@@ -375,12 +366,6 @@
         sys.stdout.write(msg)
         sys.stdout.flush()
 
-    def shutdown(self):
-        if self.headless and self._running:
-            self.log('letting instance run, to shut down headlessly '
-                     'at completion of tests.\n')
-            return
-        return self._instance.shutdown()
 
     def configure_system(self):
         user_connection = self._instance.connect_as_user()
@@ -487,10 +472,13 @@
         # close ssh connection
         user_connection.close()
 
-    def start_demo_webserver(self):
+    def run_demo_server(self):
         """Turn ec2 instance into a demo server."""
+        self.configure_system()
+        self.prepare_tests()
         user_connection = self._instance.connect_as_user()
         p = user_connection.perform
+        p('make -C /var/launchpad/test schema')
         p('mkdir -p /var/tmp/bazaar.launchpad.dev/static')
         p('mkdir -p /var/tmp/bazaar.launchpad.dev/mirrors')
         p('sudo a2enmod proxy > /dev/null')
@@ -520,6 +508,8 @@
         user_connection.close()
 
     def run_tests(self):
+        self.configure_system()
+        self.prepare_tests()
         user_connection = self._instance.connect_as_user()
 
         # Make sure we activate the failsafe --shutdown feature.  This will

=== added file 'utilities/ec2'
--- utilities/ec2	1970-01-01 00:00:00 +0000
+++ utilities/ec2	2009-09-16 01:34:30 +0000
@@ -0,0 +1,17 @@
+#!/usr/bin/python
+
+# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Executable for the ec2 scripts."""
+
+__metaclass__ = type
+
+import os
+import sys
+
+sys.path.append(
+    os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+
+from devscripts.ec2test.entrypoint import main
+main()

=== modified file 'utilities/ec2test.py'
--- utilities/ec2test.py	2009-09-11 22:49:43 +0000
+++ utilities/ec2test.py	2009-09-18 03:10:03 +0000
@@ -1,17 +1,10 @@
-#!/usr/bin/python
+#!/bin/sh
 
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-"""Executable for the ec2test script."""
-
-__metaclass__ = type
-
-import os
-import sys
-
-sys.path.append(
-    os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
-
-from devscripts.ec2test import main
-main()
+echo "You should run $(dirname $0)/ec2 test" $@ "instead." >/dev/null 1>&2
+echo "Waiting for 5 seconds in case you're not reading this." >/dev/null 1>&2
+sleep 5
+
+exec $(dirname $0)/ec2 test "$@"