Merge lp:~jml/launchpad/dirty-parry into lp:launchpad

[Jonathan Lange]

[Jonathan Lange (jml)]

This branch works towards running the test suite in parallel. It takes a different approach to allenap's ec2-parry branch, hence the name "dirty-parry" -- it's a quick and dirty approach. It doesn't actually make the test suite runnable in parallel.

It adds two... abilities to the ec2 program. The first is that EC2TestRunner now populates a TestResult object, rather than simply dumping stuff to stdout. It does this by running the tests on the instance with a subunit test result object, and then parsing this output on the client side.

The other one is the ability to split the test suite up into N parts. This isn't really useful right now, but combining it with the previous ability basically gives us parallelization. Maybe I should do that before this branch lands.

[Michael Hudson-Doyle (mwhudson)]

Most of this branch looks fine... we have subunit in sourcecode already, right? However...

It seems this branch does 80% of what would actually be useful: why don't you actually boot num_instance instances? I guess this is the thing that you talked about with using Twisted or threads or whatever: you'd want to do this in parallel. That said, until num_instances >1 boots >1 instance, I'm not sure what the win from landing this would be.

review: Needs Information

[Jonathan Lange (jml)]

On Tue, Nov 10, 2009 at 2:18 PM, Michael Hudson
<email address hidden> wrote:
> Review: Needs Information
> Most of this branch looks fine... we have subunit in sourcecode already, right?  However...
>
> It seems this branch does 80% of what would actually be useful: why don't you actually boot num_instance instances?  I guess this is the thing that you talked about with using Twisted or threads or whatever: you'd want to do this in parallel.  That said, until num_instances >1 boots >1 instance, I'm not sure what the win from landing this would be.

OK, I'll make it work then.

jml

=== modified file 'lib/devscripts/ec2test/builtins.py'
--- lib/devscripts/ec2test/builtins.py	2009-10-16 11:07:37 +0000
+++ lib/devscripts/ec2test/builtins.py	2009-11-08 09:20:38 +0000
@@ -9,6 +9,8 @@
 import os
 import pdb
 import subprocess
+import sys
+import unittest
 
 from bzrlib.bzrdir import BzrDir
 from bzrlib.commands import Command
@@ -25,6 +27,7 @@
     AVAILABLE_INSTANCE_TYPES, DEFAULT_INSTANCE_TYPE, EC2Instance)
 from devscripts.ec2test.session import EC2SessionName
 from devscripts.ec2test.testrunner import EC2TestRunner, TRUNK_BRANCH
+from devscripts.utils import split_list
 
 
 # Options accepted by more than one command.
@@ -239,6 +242,12 @@
                 'If the branch is local, then the bzr configuration is '
                 'consulted; for remote branches "Launchpad PQM '
                 '<[e-mail]>" is used by default.')),
+        Option(
+            'num-instances', short_name='j', type=int,
+            help=(
+                'The number of EC2 instances to spread the tests across. '
+                'Incompatible with --headless. Currently only runs a '
+                'fraction of the tests.')),
         postmortem_option,
         Option(
             'headless',
@@ -254,23 +263,45 @@
 
     takes_args = ['test_branch?']
 
+    def _load_tests(self):
+        """Return the full list of tests."""
+        tests = subprocess.Popen(
+            ['./bin/test', '--list'], stdout=subprocess.PIPE).communicate()[0]
+        return tests.splitlines()[:-1]
+
     def run(self, test_branch=None, branch=None, trunk=False, machine=None,
             instance_type=DEFAULT_INSTANCE_TYPE,
             file=None, email=None, test_options='-vv', noemail=False,
             submit_pqm_message=None, pqm_public_location=None,
             pqm_submit_location=None, pqm_email=None, postmortem=False,
             headless=False, debug=False, open_browser=False,
-            include_download_cache_changes=False):
+            include_download_cache_changes=False, num_instances=1):
         if debug:
             pdb.set_trace()
         if branch is None:
             branch = []
         branches, test_branch = _get_branches_and_test_branch(
             trunk, branch, test_branch)
-        if ((postmortem or file) and headless):
+        if (postmortem or file) and headless:
             raise BzrCommandError(
-                'Headless mode currently does not support postmortem or file '
-                ' options.')
+                'Headless mode currently does not support postmortem or '
+                'file options.')
+        if num_instances > 1:
+            if headless:
+                raise BzrCommandError(
+                    'Headless mode currently does not support using more '
+                    'than one instance.')
+            if test_options != '-vv':
+                raise BzrCommandError(
+                    "Cannot specify non-default options when running with "
+                    "multiple instances.")
+            tests = split_list(self._load_tests(), num_instances)[0]
+            # XXX: JonathanLange 2009-11-08: Email logic doesn't work properly
+            # with this running only part of the test suite. Need to figure
+            # this out before full parallel support can work.
+            noemail = True
+        else:
+            tests = None
         if noemail:
             if email:
                 raise BzrCommandError(
@@ -301,9 +332,15 @@
             pqm_submit_location=pqm_submit_location,
             open_browser=open_browser, pqm_email=pqm_email,
             include_download_cache_changes=include_download_cache_changes,
-            instance=instance, launchpad_login=instance._launchpad_login)
+            instance=instance, launchpad_login=instance._launchpad_login,
+            tests=tests)
 
-        instance.set_up_and_run(postmortem, not headless, runner.run_tests)
+        result = unittest._TextTestResult(
+            unittest._WritelnDecorator(sys.stdout), 0, 2)
+        result.getDescription = lambda test: test.id()
+        instance.set_up_and_run(
+            postmortem, not headless, runner.run_tests, result)
+        result.printErrors()
 
 
 class cmd_land(EC2Command):

=== modified file 'lib/devscripts/ec2test/testrunner.py'
--- lib/devscripts/ec2test/testrunner.py	2009-10-04 15:35:16 +0000
+++ lib/devscripts/ec2test/testrunner.py	2009-11-08 09:20:38 +0000
@@ -13,6 +13,7 @@
 import pickle
 import re
 import sys
+import tempfile
 
 from bzrlib.branch import Branch
 from bzrlib.bzrdir import BzrDir
@@ -21,6 +22,8 @@
 from bzrlib.plugins.pqm.pqm_submit import (
     NoPQMSubmissionAddress, PQMSubmission)
 
+from devscripts.utils import TestResultToStream
+
 
 TRUNK_BRANCH = 'bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel'
 
@@ -115,7 +118,7 @@
                  pqm_submit_location=None,
                  open_browser=False, pqm_email=None,
                  include_download_cache_changes=None, instance=None,
-                 launchpad_login=None):
+                 launchpad_login=None, tests=None):
         """Create a new EC2TestRunner.
 
         This sets the following attributes:
@@ -137,6 +140,7 @@
         self.open_browser = open_browser
         self.file = file
         self._launchpad_login = launchpad_login
+        self._tests = tests
 
         trunk_specified = False
         trunk_branch = TRUNK_BRANCH
@@ -446,7 +450,19 @@
         # close ssh connection
         user_connection.close()
 
-    def run_tests(self):
+    def _copy_tests(self, connection, tests):
+        """Copy a list of tests to the server."""
+        tempdir = tempfile.mkdtemp()
+        testfilepath = os.path.join(tempdir, 'test-list')
+        testfile = open(testfilepath, 'wb')
+        for test in tests:
+            testfile.write(test)
+            testfile.write('\n')
+        testfile.close()
+        connection.sftp.put(testfilepath, '/var/launchpad/test-list')
+        return '/var/launchpad/test-list'
+
+    def run_tests(self, result):
         self.configure_system()
         self.prepare_tests()
         user_connection = self._instance.connect()
@@ -488,6 +504,11 @@
 
         # Add any additional options for ec2test-remote.py
         cmd.extend(self.get_remote_test_options())
+
+        if self._tests is not None:
+            test_path = self._copy_tests(user_connection, self._tests)
+            cmd.append('--load-list=%s' % test_path)
+
         self.log(
             'Running tests... (output is available on '
             'http://%s/)\n' % self._instance.hostname)
@@ -505,7 +526,8 @@
 
         # Run the remote script!  Our execution will block here until the
         # remote side disconnects from the terminal.
-        user_connection.perform(' '.join(cmd))
+        user_connection.perform(
+            ' '.join(cmd), out=TestResultToStream(result))
         self._running = True
 
         if not self.headless:
@@ -536,4 +558,7 @@
             # Run the normal testsuite with our Zope testrunner options.
             # ec2test-remote.py wants the extra options to be after a double-
             # dash.
-            return ('--', self.test_options)
+            if self._tests is None:
+                return ('--', '--subunit', self.test_options)
+            else:
+                return ('--', '--subunit')

=== added file 'lib/devscripts/utils.py'
--- lib/devscripts/utils.py	1970-01-01 00:00:00 +0000
+++ lib/devscripts/utils.py	2009-11-08 09:20:38 +0000
@@ -0,0 +1,58 @@
+# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+__all__ = [
+    'LineReceiver',
+    'split_list',
+    'TestResultToStream',
+    ]
+
+from itertools import cycle, izip
+
+import subunit
+
+
+def split_list(stuff, n):
+    """Split a 'stuff' into 'n' similarly-sized chunks."""
+    chunks = [[] for i in range(n)]
+    for thing, chunk in izip(stuff, cycle(chunks)):
+        chunk.append(thing)
+    return chunks
+
+
+class LineReceiver:
+    """Stream-like interface that calls the 'lineReceived' callback."""
+
+    delimiter = '\n'
+
+    def __init__(self, lineReceived):
+        self._buffer = ''
+        self._lineReceived = lineReceived
+
+    def write(self, data):
+        self._buffer += data
+        lines = self._buffer.split(self.delimiter)
+        self._buffer = lines.pop()
+        for line in lines:
+            self._lineReceived(line)
+
+    def flush(self):
+        pass
+
+    def close(self):
+        pass
+
+
+class TestResultToStream(LineReceiver):
+    """Wraps a result in a stream-like interface."""
+
+    def __init__(self, result):
+        self._subunit = subunit.TestProtocolServer(result)
+        super(TestResultToStream, self).__init__(self.lineReceived)
+
+    def lineReceived(self, line):
+        self._subunit.lineReceived(line + '\n')
+
+    def close(self):
+        self._subunit.lostConnection()