Merge lp:~jtv/launchpad/bug-499405-translationtemplates-buildmanager into lp:launchpad

[Jeroen T. Vermeulen]

BuildManager for generating translation templates.

[Jeroen T. Vermeulen (jtv)]

= Bug 499405 =

This branch implements a build manager for generating translation templates from a branch. It provides no real functionality on its own; rather it's part of an ongoing project. Pre-imp discussions were done IRL at the Wellington build-farm generalisation sprint.

A build manager is the piece of code that runs on a build slave to coordinate its work. It iterates through a finite-state machine describing the job to be done. An important thing to bear in mind is that this is not "Launchpad code": it runs on a slave machine that has neither the Launchpad modules nor access to the database.

For this particular job, the finite-state machine consists of 7 states:
1. INIT: Initialize.
2. UNPACK: Create a chroot directory by unpacking a chroot tarball.
3. MOUNT: Set up a realistic system environment inside the chroot.
4. UPDATE: Get the latest versions of the packages installed inside the chroot.
5. INSTALL: Add a few job-specific packages to the chroot.
6. GENERATE: Check out the branch and produce templates based on it.
7. CLEANUP: Return the slave to its original, clean state.

Of these, step 6 is the "payload." It invokes a separate script, also found in this branch, that checks out a branch and will eventually generate templates inside it. The script is run under a non-privileged user inside the slave's chroot, providing maximum insulation from untrusted code inside the branches.

The implementations for most of the other steps are inherited from the existing BuildSlave base class.

Testing the actual shell commands is rather hard, which is why other build managers seem to go largely untested. I did find a way to unit-test at least the iteration of the state machine: mock up the BuildManager method that normally forks off an action of the state machine and hands it to twistd. Instead of running the action and triggering a state transition in twistd, the test just logs and verifies the command that would have been run in a sub-process.

In order to make this testable, I had to make just one small change in the base class. It made changes to the filesystem in the current user's home directory. Instead, I made it record the current user's home directory in its constructor; the other methods now get the location of the home directory from that same field. This allows the test to substitute a temporary directory of its own.

I also added a manual test to kick off a templates generation build through XMLRPC. I'm told this can't be run as part of the normal test suite, but it can be run manually to see if anything breaks spectacularly. It's a blatant ripoff of a similar test that abentley implemented for his own build-manager class. You "make run" in one shell then while that's active, run the test script in another with one argument: the SHA1 of a chroot tarball that's in your local Librarian.

Finally, you'll notice a helper method makeTemporaryDirectory in the Launchpad version of the TestCase class. This breaks naming convention with the existing useTempDir, but matches the convention of both a similar method in bzr test cases and the naming style in the standard tempfile module. I did update useTempDir to make use o...

[Paul Hummer (rockstar)]

Hi Jeroen-

  This branch looks good. I do have some questions though, and would like to have answers before I Approve this branch.

=== added file 'lib/canonical/buildd/generate_translation_templates.py'
--- lib/canonical/buildd/generate_translation_templates.py 1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/generate_translation_templates.py 2010-01-21 12:50:24 +0000
@@ -0,0 +1,57 @@
+#! /usr/bin/python2.5
+# Copyright 2010 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os.path
+import sys
+from subprocess import call
+
+
+class GenerateTranslationTemplates:
+ """Script to generate translation templates from a branch."""
+
+ def __init__(self, branch_spec):
+ """Prepare to generate templates for a branch.
+
+ :param branch_spec: Either a branch URL or the path of a local
+ branch. URLs are recognized by the occurrence of ':'. In
+ the case of a URL, this will make up a path for the branch
+ and check out the branch to there.
+ """
+ self.home = os.environ['HOME']
+ self.branch_spec = branch_spec
+
+ def _getBranch(self):
+ """Set `self.branch_dir`, and check out branch if needed."""
+ if ':' in self.branch_spec:
+ # This is a branch URL. Check out the branch.
+ self.branch_dir = os.path.join(self.home, 'source-tree')
+ self._checkout(self.branch_spec)
+ else:
+ # This is a local filesystem path. Use the branch in-place.
+ self.branch_dir = self.branch_spec
+
+ def _checkout(self, branch_url):
+ """Check out a source branch to generate from.
+
+ The branch is checked out to the location specified by
+ `self.branch_dir`.
+ """
+ command = ['/usr/bin/bzr', 'checkout', branch_url, self.branch_dir]
+ return call(command, cwd=self.home)

Is there a reason you're not using bzrlib? I know that the build slaves have special security there, but if you have bzr there, you have bzrlib.

+
+ def generate(self):
+ """Do It. Generate templates."""
+ self._getBranch()
+ # XXX JeroenVermeulen 2010-01-19 bug=509557: Actual payload goes here.
+ return 0
+

Does this mean that this is basically a no-op right now? I'm okay with this code, but I'd rather you wait for it to actually do something than land it now.

+
+if __name__ == '__main__':
+ if len(sys.argv) != 2:
+ print "Usage: %s branch" % sys.argv[0]
+ print "Where 'branch' is a branch URL or directory."
+ sys.exit(1)
+ sys.exit(GenerateTranslationTemplates(sys.argv[1]).generate())

=== added file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'
--- lib/canonical/buildd/tests/test_generate_translation_templates.py 1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/tests/test_generate_translation_templates.py 2010-01-21 12:50:24 +0000
@@ -0,0 +1,57 @@
+# Copyright 2010 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os
+from unitte...

review: Needs Information

[Jeroen T. Vermeulen (jtv)]

> This branch looks good. I do have some questions though, and would like to
> have answers before I Approve this branch.

Thanks for the review—glad I could get someone who was involved in the sprint!

> === added file 'lib/canonical/buildd/generate_translation_templates.py'
> --- lib/canonical/buildd/generate_translation_templates.py 1970-01-01
> 00:00:00 +0000
> +++ lib/canonical/buildd/generate_translation_templates.py 2010-01-21
> 12:50:24 +0000
> @@ -0,0 +1,57 @@
> +#! /usr/bin/python2.5
> +# Copyright 2010 Canonical Ltd. This software is licensed under the
> +# GNU Affero General Public License version 3 (see the file LICENSE).
> +
> +__metaclass__ = type
> +
> +import os.path
> +import sys
> +from subprocess import call
> +
> +
> +class GenerateTranslationTemplates:
> + """Script to generate translation templates from a branch."""
> +
> + def __init__(self, branch_spec):
> + """Prepare to generate templates for a branch.
> +
> + :param branch_spec: Either a branch URL or the path of a local
> + branch. URLs are recognized by the occurrence of ':'. In
> + the case of a URL, this will make up a path for the branch
> + and check out the branch to there.
> + """
> + self.home = os.environ['HOME']
> + self.branch_spec = branch_spec
> +
> + def _getBranch(self):
> + """Set `self.branch_dir`, and check out branch if needed."""
> + if ':' in self.branch_spec:
> + # This is a branch URL. Check out the branch.
> + self.branch_dir = os.path.join(self.home, 'source-tree')
> + self._checkout(self.branch_spec)
> + else:
> + # This is a local filesystem path. Use the branch in-place.
> + self.branch_dir = self.branch_spec
> +
> + def _checkout(self, branch_url):
> + """Check out a source branch to generate from.
> +
> + The branch is checked out to the location specified by
> + `self.branch_dir`.
> + """
> + command = ['/usr/bin/bzr', 'checkout', branch_url, self.branch_dir]
> + return call(command, cwd=self.home)
>
> Is there a reason you're not using bzrlib? I know that the build slaves have
> special security there, but if you have bzr there, you have bzrlib.

Mainly laziness. But also, this being a corner of LP that's hard to test, I would like to minimize the risk of silly little mistakes that might not be reported back to me very effectively. Instead of writing my own python, I'm producing a command line here that I can just replicate in a shell when debugging.

By the way, I'm replacing the "checkout" with "export." All I need is the stored files; actual revision control is just useless baggage for this particular job.

> + def generate(self):
> + """Do It. Generate templates."""
> + self._getBranch()
> + # XXX JeroenVermeulen 2010-01-19 bug=509557: Actual payload goes
> here.
> + return 0
> +
>
> Does this mean that this is basically a no-op right now? I'm okay with this
> code, but I'd rather you wait for it to actually do something than land it
> now.

This is a pretty big project, so I'd...

[Paul Hummer (rockstar)]

>> + def _checkout(self, branch_url):
>> + """Check out a source branch to generate from.
>> +
>> + The branch is checked out to the location specified by
>> + `self.branch_dir`.
>> + """
>> + command = ['/usr/bin/bzr', 'checkout', branch_url, self.branch_dir]
>> + return call(command, cwd=self.home)
>>
>> Is there a reason you're not using bzrlib? I know that the build slaves have
>> special security there, but if you have bzr there, you have bzrlib.
>
>Mainly laziness. But also, this being a corner of LP that's hard to test, I would >like to minimize the risk of silly little mistakes that might not be reported back to >me very effectively. Instead of writing my own python, I'm producing a command line >here that I can just replicate in a shell when debugging.

So there are a few issues with calling the bzr runtime instead of using bzrlib. The first is that we do not want to use the system bzr. Launchpad packages a local copy of bzr for this very purpose. The second is that we aren't guaranteed we'll have the plugins we need on the system, but we make sure they're available to bzrlib. There are also performance advantages to using bzrlib specifically. I really think that bzrlib needs to be used in this situation.

Other than that, I'm happy with your other changes.

review: Needs Fixing (code)

[Jeroen T. Vermeulen (jtv)]

> So there are a few issues with calling the bzr runtime instead of using
> bzrlib. The first is that we do not want to use the system bzr. Launchpad
> packages a local copy of bzr for this very purpose. The second is that we
> aren't guaranteed we'll have the plugins we need on the system, but we make
> sure they're available to bzrlib. There are also performance advantages to
> using bzrlib specifically. I really think that bzrlib needs to be used in
> this situation.
>
> Other than that, I'm happy with your other changes.

This code runs on the slave, so Launchpad is not present. I think that also eliminates a lot of other complications there might otherwise be on the system.

[Paul Hummer (rockstar)]

I conferenced with abentley about this, so it's not just me saying "please use bzrlib" but also another code folk. If nothing else, it gives us a nice traceback, instead of a user friendly (and rather unhelpful) error message. It basically runs bzr as a subprocess, which could actually make things more complicated.

[Jeroen T. Vermeulen (jtv)]

Alright then... I've replaced the code with an XXX (referring to a bug, of course) to say that in future the code needs to do the equivalent of "bzr export" using bzrlib.

[Paul Hummer (rockstar)]

review: Approve (code)

=== modified file 'daemons/buildd-slave.tac'
--- daemons/buildd-slave.tac	2010-01-13 23:16:43 +0000
+++ daemons/buildd-slave.tac	2010-02-18 14:41:20 +0000
@@ -12,6 +12,7 @@
 from canonical.buildd.binarypackage import BinaryPackageBuildManager
 from canonical.buildd.sourcepackagerecipe import (
     SourcePackageRecipeBuildManager)
+from canonical.buildd.translationtemplates import TranslationTemplatesBuildManager
 from canonical.launchpad.daemons import tachandler
 
 from twisted.web import server, resource, static
@@ -29,6 +30,7 @@
 slave.registerBuilder(BinaryPackageBuildManager, "debian")
 slave.registerBuilder(BinaryPackageBuildManager, "binarypackage")
 slave.registerBuilder(SourcePackageRecipeBuildManager, "sourcepackagerecipe")
+slave.registerBuilder(TranslationTemplatesBuildManager, 'translation-templates')
 
 application = service.Application('BuildDSlave')
 builddslaveService = service.IServiceCollection(application)

=== modified file 'lib/canonical/buildd/debian/rules'
--- lib/canonical/buildd/debian/rules	2010-01-13 23:16:43 +0000
+++ lib/canonical/buildd/debian/rules	2010-02-18 14:41:20 +0000
@@ -21,10 +21,10 @@
 targetshare = $(target)/usr/share/launchpad-buildd
 pytarget = $(targetshare)/canonical/buildd
 
-pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py sourcepackagerecipe.py
+pyfiles = debian.py slave.py binarypackage.py utils.py __init__.py sourcepackagerecipe.py translationtemplates.py
 slavebins = unpack-chroot mount-chroot update-debian-chroot sbuild-package \
 scan-for-processes umount-chroot remove-build apply-ogre-model \
-override-sources-list buildrecipe
+override-sources-list buildrecipe generate_translation_templates.py
 
 BUILDDUID=65500
 BUILDDGID=65500

=== added file 'lib/canonical/buildd/generate_translation_templates.py'
--- lib/canonical/buildd/generate_translation_templates.py	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/generate_translation_templates.py	2010-02-18 14:41:20 +0000
@@ -0,0 +1,57 @@
+#! /usr/bin/python2.5
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os.path
+import sys
+from subprocess import call
+
+
+class GenerateTranslationTemplates:
+    """Script to generate translation templates from a branch."""
+
+    def __init__(self, branch_spec):
+        """Prepare to generate templates for a branch.
+
+        :param branch_spec: Either a branch URL or the path of a local
+            branch.  URLs are recognized by the occurrence of ':'.  In
+            the case of a URL, this will make up a path for the branch
+            and check out the branch to there.
+        """
+        self.home = os.environ['HOME']
+        self.branch_spec = branch_spec
+
+    def _getBranch(self):
+        """Set `self.branch_dir`, and check out branch if needed."""
+        if ':' in self.branch_spec:
+            # This is a branch URL.  Check out the branch.
+            self.branch_dir = os.path.join(self.home, 'source-tree')
+            self._checkout(self.branch_spec)
+        else:
+            # This is a local filesystem path.  Use the branch in-place.
+            self.branch_dir = self.branch_spec
+
+    def _checkout(self, branch_url):
+        """Check out a source branch to generate from.
+
+        The branch is checked out to the location specified by
+        `self.branch_dir`.
+        """
+        # XXX JeroenVermeulen 2010-02-11 bug=520651: Read the contents
+        # of the branch using bzrlib.
+
+    def generate(self):
+        """Do It.  Generate templates."""
+        self._getBranch()
+        # XXX JeroenVermeulen 2010-01-19 bug=509557: Actual payload goes here.
+        return 0
+
+
+if __name__ == '__main__':
+    if len(sys.argv) != 2:
+        print "Usage: %s branch" % sys.argv[0]
+        print "Where 'branch' is a branch URL or directory."
+        sys.exit(1)
+    sys.exit(GenerateTranslationTemplates(sys.argv[1]).generate())

=== modified file 'lib/canonical/buildd/slave.py'
--- lib/canonical/buildd/slave.py	2010-02-09 00:17:40 +0000
+++ lib/canonical/buildd/slave.py	2010-02-18 14:41:20 +0000
@@ -96,6 +96,11 @@
     """Build Daemon slave build manager abstract parent"""
 
     def __init__(self, slave, buildid):
+        """Create a BuildManager.
+
+        :param slave: A `BuildDSlave`.
+        :param buildid: Identifying string for this build.
+        """
         object.__init__(self)
         self._buildid = buildid
         self._slave = slave
@@ -104,6 +109,7 @@
         self._mountpath = slave._config.get("allmanagers", "mountpath")
         self._umountpath = slave._config.get("allmanagers", "umountpath")
         self.is_archive_private = False
+        self.home = os.environ['HOME']
 
     def runSubProcess(self, command, args):
         """Run a sub process capturing the results in the log."""
@@ -112,7 +118,7 @@
         childfds = {0: devnull.fileno(), 1: "r", 2: "r"}
         reactor.spawnProcess(
             self._subprocess, command, args, env=os.environ,
-            path=os.environ["HOME"], childFDs=childfds)
+            path=self.home, childFDs=childfds)
 
     def doUnpack(self):
         """Unpack the build chroot."""
@@ -146,10 +152,10 @@
         value keyed under the 'archive_private' string. If that value
         evaluates to True the build at hand is for a private archive.
         """
-        os.mkdir("%s/build-%s" % (os.environ["HOME"], self._buildid))
+        os.mkdir("%s/build-%s" % (self.home, self._buildid))
         for f in files:
             os.symlink( self._slave.cachePath(files[f]),
-                        "%s/build-%s/%s" % (os.environ["HOME"],
+                        "%s/build-%s/%s" % (self.home,
                                             self._buildid, f))
         self._chroottarfile = self._slave.cachePath(chroot)
 

=== added file 'lib/canonical/buildd/test_buildd_generatetranslationtemplates'
--- lib/canonical/buildd/test_buildd_generatetranslationtemplates	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/test_buildd_generatetranslationtemplates	2010-02-18 14:41:20 +0000
@@ -0,0 +1,33 @@
+#!/usr/bin/env python
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+#
+# Test script for manual use only.  Exercises the
+# TranslationTemplatesBuildManager through XMLRPC.
+
+import sys
+
+from xmlrpclib import ServerProxy
+
+if len(sys.argv) != 2:
+    print "Usage: %s <chroot_sha1>" % sys.argv[0]
+    print "Where <chroot_sha1> is the SHA1 of the chroot tarball to use."
+    print "The chroot tarball must be in the local Librarian."
+    print "See https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally"
+    sys.exit(1)
+
+chroot_sha1 = sys.argv[1]
+
+proxy = ServerProxy('http://localhost:8221/rpc')
+print proxy.info()
+print proxy.status()
+buildid = '1-2'
+build_type = 'translation-templates'
+filemap = {}
+args = {'branch_url': 'no-branch-here-sorry'}
+print proxy.build(buildid, build_type, chroot_sha1, filemap, args)
+#status = proxy.status()
+#for filename, sha1 in status[3].iteritems():
+#    print filename
+#proxy.clean()
+

=== modified file 'lib/canonical/buildd/tests/harness.py'
--- lib/canonical/buildd/tests/harness.py	2010-02-01 14:55:03 +0000
+++ lib/canonical/buildd/tests/harness.py	2010-02-18 14:41:20 +0000
@@ -79,8 +79,18 @@
     >>> s.echo('Hello World')
     ['Hello World']
 
-    >>> s.info()
-    ['1.0', 'i386', ['sourcepackagerecipe', 'binarypackage', 'debian']]
+    >>> info = s.info()
+    >>> len(info)
+    3
+    >>> print info[:2]
+    ['1.0', 'i386']
+
+    >>> for buildtype in sorted(info[2]):
+    ...     print buildtype
+    binarypackage
+    debian
+    sourcepackagerecipe
+    translation-templates
 
     >>> s.status()
     ['BuilderStatus.IDLE', '']

=== added file 'lib/canonical/buildd/tests/test_generate_translation_templates.py'
--- lib/canonical/buildd/tests/test_generate_translation_templates.py	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/tests/test_generate_translation_templates.py	2010-02-18 14:41:20 +0000
@@ -0,0 +1,60 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os
+from unittest import TestLoader
+
+from lp.testing import TestCase
+
+from canonical.buildd.generate_translation_templates import (
+    GenerateTranslationTemplates)
+
+from canonical.launchpad.ftests.script import run_script
+from canonical.testing.layers import ZopelessDatabaseLayer
+
+
+class MockGenerateTranslationTemplates(GenerateTranslationTemplates):
+    """A GenerateTranslationTemplates with mocked _checkout."""
+    # Records, for testing purposes, whether this object checked out a
+    # branch.
+    checked_out_branch = False
+
+    def _checkout(self, branch_url):
+        assert not self.checked_out_branch, "Checking out branch again!"
+        self.checked_out_branch = True
+
+
+class TestGenerateTranslationTemplates(TestCase):
+    """Test slave-side generate-translation-templates script."""
+    def test_getBranch_url(self):
+        # If passed a branch URL, the template generation script will
+        # check out that branch into a directory called "source-tree."
+        branch_url = 'lp://~my/translation/branch'
+
+        generator = MockGenerateTranslationTemplates(branch_url)
+        generator._getBranch()
+
+        self.assertTrue(generator.checked_out_branch)
+        self.assertTrue(generator.branch_dir.endswith('source-tree'))
+
+    def test_getBranch_dir(self):
+        # If passed a branch directory, the template generation script
+        # works directly in that directory.
+        branch_dir = '/home/me/branch'
+
+        generator = MockGenerateTranslationTemplates(branch_dir)
+        generator._getBranch()
+
+        self.assertFalse(generator.checked_out_branch)
+        self.assertEqual(branch_dir, generator.branch_dir)
+
+    def test_script(self):
+        tempdir = self.makeTemporaryDirectory()
+        (retval, out, err) = run_script(
+            'lib/canonical/buildd/generate_translation_templates.py',
+            args=[tempdir])
+        self.assertEqual(0, retval)
+
+
+def test_suite():
+    return TestLoader().loadTestsFromName(__name__)

=== added file 'lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py'
--- lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/tests/test_translationtemplatesbuildmanager.py	2010-02-18 14:41:20 +0000
@@ -0,0 +1,133 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os
+
+from unittest import TestLoader
+
+from lp.testing import TestCase
+
+from canonical.buildd.translationtemplates import (
+    TranslationTemplatesBuildManager, TranslationTemplatesBuildState)
+
+
+class FakeConfig:
+    def get(self, section, key):
+        return key
+
+
+class FakeSlave:
+    def __init__(self, tempdir):
+        self._cachepath = tempdir
+        self._config = FakeConfig()
+
+    def cachePath(self, file):
+        return os.path.join(self._cachepath, file)
+
+
+class MockBuildManager(TranslationTemplatesBuildManager):
+    def __init__(self, *args, **kwargs):
+        super(MockBuildManager, self).__init__(*args, **kwargs)
+        self.commands = []
+
+    def runSubProcess(self, path, command):
+        self.commands.append(command)
+        return 0
+
+
+class TestTranslationTemplatesBuildManagerIteration(TestCase):
+    """Run TranslationTemplatesBuildManager through its iteration steps."""
+    def setUp(self):
+        super(TestTranslationTemplatesBuildManagerIteration, self).setUp()
+        self.working_dir = self.makeTemporaryDirectory()
+        slave_dir = os.path.join(self.working_dir, 'slave')
+        home_dir = os.path.join(self.working_dir, 'home')
+        for dir in (slave_dir, home_dir):
+            os.mkdir(dir)
+        slave = FakeSlave(slave_dir)
+        buildid = '123'
+        self.buildmanager = MockBuildManager(slave, buildid)
+        self.buildmanager.home = home_dir
+        self.chrootdir = os.path.join(
+            home_dir, 'build-%s' % buildid, 'chroot-autobuild')
+
+    def getState(self):
+        """Retrieve build manager's state."""
+        return self.buildmanager._state
+
+    def test_initiate(self):
+        # Creating a BuildManager spawns no child processes.
+        self.assertEqual([], self.buildmanager.commands)
+
+        # Initiating the build executes the first command.  It leaves
+        # the build manager in the INIT state.
+        self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': 'foo'})
+        self.assertEqual(1, len(self.buildmanager.commands))
+        self.assertEqual(TranslationTemplatesBuildState.INIT, self.getState())
+
+    def test_iterate(self):
+        url = 'lp:~my/branch'
+        # The build manager's iterate() kicks off the consecutive states
+        # after INIT.
+        self.buildmanager.initiate({}, 'chroot.tar.gz', {'branch_url': url})
+
+        # UNPACK: execute unpack-chroot.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.UNPACK, self.getState())
+        self.assertEqual('unpack-chroot', self.buildmanager.commands[-1][0])
+
+        # MOUNT: Set up realistic chroot environment.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.MOUNT, self.getState())
+        self.assertEqual('mount-chroot', self.buildmanager.commands[-1][0])
+
+        # UPDATE: Get the latest versions of installed packages.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.UPDATE, self.getState())
+        expected_command = [
+            '/usr/bin/sudo',
+            '/usr/sbin/chroot', self.chrootdir,
+            'update-debian-chroot',
+            ]
+        self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])
+
+        # INSTALL: Install additional packages needed for this job into
+        # the chroot.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.INSTALL, self.getState())
+        expected_command = [
+            '/usr/bin/sudo',
+            '/usr/sbin/chroot', self.chrootdir,
+            'apt-get',
+            ]
+        self.assertEqual(expected_command, self.buildmanager.commands[-1][:4])
+
+        # GENERATE: Run the slave's payload, the script that generates
+        # templates.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.GENERATE, self.getState())
+        expected_command = [
+            '/usr/bin/sudo',
+            '/usr/sbin/chroot', self.chrootdir,
+            '/usr/bin/sudo', '-u', self.buildmanager.username,
+            'generate-translation-templates.py',
+            url,
+            ]
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+
+        # CLEANUP.
+        self.buildmanager.iterate(0)
+        self.assertEqual(
+            TranslationTemplatesBuildState.CLEANUP, self.getState())
+        self.assertEqual('remove-build', self.buildmanager.commands[-1][0])
+
+
+def test_suite():
+    return TestLoader().loadTestsFromName(__name__)

=== added file 'lib/canonical/buildd/translationtemplates.py'
--- lib/canonical/buildd/translationtemplates.py	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/translationtemplates.py	2010-02-18 14:41:20 +0000
@@ -0,0 +1,159 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os
+import pwd
+
+from canonical.buildd.slave import BuildManager
+
+
+class TranslationTemplatesBuildState(object):
+    "States this kind of build goes through."""
+    INIT = "INIT"
+    UNPACK = "UNPACK"
+    MOUNT = "MOUNT"
+    UPDATE = "UPDATE"
+    INSTALL = "INSTALL"
+    GENERATE = "GENERATE"
+    CLEANUP = "CLEANUP"
+
+
+class TranslationTemplatesBuildManager(BuildManager):
+    """Generate translation templates from branch.
+
+    This is the implementation of `TranslationTemplatesBuildJob`.  The
+    latter runs on the master server; TranslationTemplatesBuildManager
+    runs on the build slave.
+    """
+    def __init__(self, *args, **kwargs):
+        super(TranslationTemplatesBuildManager, self).__init__(
+            *args, **kwargs)
+        self._state = TranslationTemplatesBuildState.INIT
+        self.alreadyfailed = False
+
+    def initiate(self, files, chroot, extra_args):
+        """See `BuildManager`."""
+        self.branch_url = extra_args['branch_url']
+        self.username = pwd.getpwuid(os.getuid())[0]
+
+        super(TranslationTemplatesBuildManager, self).initiate(
+            files, chroot, extra_args)
+
+        self.chroot_path = os.path.join(
+            self.home, 'build-' + self._buildid, 'chroot-autobuild')
+
+    def iterate(self, success):
+        func = getattr(self, 'iterate_' + self._state, None)
+        if func is None:
+            raise ValueError("Unknown %s state: %s" % (
+                self.__class__.__name__, self._state))
+        func(success)
+
+    def iterate_INIT(self, success):
+        """Next step after initialization."""
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.UNPACK
+            self.doUnpack()
+        else:
+            if not self.alreadyfailed:
+                self._slave.builderFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.build_implementation.doCleanup()
+
+    def iterate_UNPACK(self, success):
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.MOUNT
+            self.doMounting()
+        else:
+            if not self.alreadyfailed:
+                self._slave.chrootFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+
+    def iterate_MOUNT(self, success):
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.UPDATE
+            self.doUpdate()
+        else:
+            if not self.alreadyfailed:
+                self._slave.chrootFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+
+    def iterate_UPDATE(self, success):
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.INSTALL
+            self.doInstall()
+        else:
+            if not self.alreadyfailed:
+                self._slave.chrootFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+
+    def iterate_INSTALL(self, success):
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.GENERATE
+            self.doGenerate()
+        else:
+            if not self.alreadyfailed:
+                self._slave.chrootFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+
+    def iterate_GENERATE(self, success):
+        if success == 0:
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+        else:
+            if not self.alreadyfailed:
+                self._slave.buildFail()
+                self.alreadyfailed = True
+            self._state = TranslationTemplatesBuildState.CLEANUP
+            self.doCleanup()
+
+    def iterate_CLEANUP(self, success):
+        if success == 0:
+            if not self.alreadyfailed:
+                self._slave.buildOK()
+        else:
+            if not self.alreadyfailed:
+                self._slave.builderFail()
+                self.alreadyfailed = True
+        self._slave.buildComplete()
+
+    def doInstall(self):
+        """Install packages required."""
+        required_packages = [
+            'bzr',
+            'intltool-debian',
+            ]
+        command = ['apt-get', 'install', '-y'] + required_packages
+        self.runInChroot(self.home, command, as_root=True)
+
+    def doUpdate(self):
+        """Update chroot."""
+        command = ['update-debian-chroot', self._buildid]
+        self.runInChroot(self.home, command, as_root=True)
+
+    def doGenerate(self):
+        """Generate templates."""
+        command = ['generate-translation-templates.py', self.branch_url]
+        self.runInChroot(self.home, command)
+
+    def runInChroot(self, path, command, as_root=False):
+        """Run command in chroot."""
+        chroot = ['/usr/bin/sudo', '/usr/sbin/chroot', self.chroot_path]
+        if as_root:
+            sudo = []
+        else:
+            # We have to sudo to chroot, so if the command should _not_
+            # be run as root, we then need to sudo back to who we were.
+            sudo = ['/usr/bin/sudo', '-u', self.username]
+        return self.runSubProcess(path, chroot + sudo + command)

=== modified file 'lib/lp/testing/__init__.py'
--- lib/lp/testing/__init__.py	2010-02-15 12:08:54 +0000
+++ lib/lp/testing/__init__.py	2010-02-18 14:41:20 +0000
@@ -208,6 +208,12 @@
         fixture.setUp()
         self.addCleanup(fixture.tearDown)
 
+    def makeTemporaryDirectory(self):
+        """Create a temporary directory, and return its path."""
+        tempdir = tempfile.mkdtemp()
+        self.addCleanup(lambda: shutil.rmtree(tempdir))
+        return tempdir
+
     def assertProvides(self, obj, interface):
         """Assert 'obj' correctly provides 'interface'."""
         self.assertTrue(