Merge lp:~jtv/launchpad/bug-499404 into lp:launchpad

[Jeroen T. Vermeulen]

IBuildFarmJob type for generating translation templates.

[Jeroen T. Vermeulen (jtv)]

= Bug 499404 =

This implements some new classes that will be needed for Rosetta to run jobs on the generalised build farm. The job in question is generating translation templates based on a source-code branch, and will require a lot more work before it goes operational.

Generalisation of the build farm is ongoing work, so its APIs etc. are not complete yet. What you find here is:

1. An implementation of IBuildFarmJob—a new class representing a job that can be dispatched to the build farm. Called TranslationTemplatesBuildJob, it's a derivative of BranchJob found in the Code tree.

2. A BuildFarmJobSource utility—to produce and navigate jobs of the new type. This follows the pattern of the BranchJobSource utilities from the Code hierarchy. It also creates a BuildQueue record for each job it creates; these records are matched to the records of the new type in that they both refer to the same Job record.

3. A new IBuildFarmJobBehavior implementation for dispatching the new job type to the build farm, with an adapter associating it with TranslationTemplatesBuildJob. More may be needed there in the future.

This branch does not test how the Behavior class dispatches jobs to builders; I don't think that's tested for the existing classes either and I'm not sure that part of the build-farm generalization is quite ready for use yet.

To test:
{{{
./bin/test -vv -t translationtemplatesbuild
}}}

No lint.

Jeroen

[Julian Edwards (julian-edwards)]

Hi Jeroen! This is a great start, thanks also for cleaning up the docstrings. I'll just re-iterate here what we discussed in person for future reference:

 * You need a verifyObject on the ITranslationTemplatesBuildJobSource interface
 * Remove the XXX on line 171 next to build_type.
 * The fault handling in dispatchBuildToSlave needs to be refactored into BuildFarmJobBehaviorBase, can you file a bug for that please.
 * _getChroot is not returning a chroot! I can see what you were starting to do though, returning the nominatedarchindep arch chroot for the series in question is the right thing.
 * test_getChroot should poke a dummy file into the librarian for the chroot for the pertinent distroarchseries. It should then assert that the one that _getChroot returns is the same file.
 * There's no test for score().

Cheers
J

review: Needs Fixing (code)

[Jeroen T. Vermeulen (jtv)]

All done. Filed bug 506243.

[Julian Edwards (julian-edwards)]

Land it!

review: Approve (code)

=== modified file 'lib/lp/buildmaster/interfaces/buildfarmjob.py'
--- lib/lp/buildmaster/interfaces/buildfarmjob.py	2009-12-24 14:18:35 +0000
+++ lib/lp/buildmaster/interfaces/buildfarmjob.py	2010-01-12 03:38:15 +0000
@@ -46,24 +46,24 @@
         Build a package from a bazaar branch and a recipe.
         """)
 
-    TRANSLATION = DBItem(4, """
-        TranslationJob
+    TRANSLATIONTEMPLATESBUILD = DBItem(4, """
+        TranslationTemplatesBuildJob
 
-        Perform a translation job.
+        Generate translation templates from a bazaar branch.
         """)
 
 
 class IBuildFarmJob(Interface):
-    """Operations that Soyuz build farm jobs must implement."""
+    """Operations that jobs for the build farm must implement."""
 
     def score():
         """Calculate a job score appropriate for the job type in question."""
 
     def getLogFileName():
-        """The preferred file name for the log of this Soyuz job."""
+        """The preferred file name for this job's log."""
 
     def getName():
-        """An appropriate name for this Soyuz job."""
+        """An appropriate name for this job."""
 
     def jobStarted():
         """'Job started' life cycle event, handle as appropriate."""

=== modified file 'lib/lp/code/model/branchjob.py'
--- lib/lp/code/model/branchjob.py	2010-01-07 04:57:13 +0000
+++ lib/lp/code/model/branchjob.py	2010-01-12 03:38:15 +0000
@@ -119,6 +119,12 @@
         from disk.
         """)
 
+    TRANSLATION_TEMPLATES_BUILD = DBItem(6, """
+        Generate translation templates
+
+        This job generates translations templates from a source branch.
+        """)
+
 
 class BranchJob(SQLBase):
     """Base class for jobs related to branches."""

=== modified file 'lib/lp/translations/configure.zcml'
--- lib/lp/translations/configure.zcml	2009-12-28 22:58:18 +0000
+++ lib/lp/translations/configure.zcml	2010-01-12 03:38:15 +0000
@@ -572,6 +572,35 @@
             interface="lp.translations.interfaces.poexportrequest.IPOExportRequestSet"/>
     </securedutility>
 
+    <!-- TranslationTemplateBuildJob -->
+    <class
+        class="lp.translations.model.translationtemplatesbuildjob.TranslationTemplatesBuildJob">
+        <allow
+            interface="lp.translations.interfaces.translationtemplatesbuildjob.ITranslationTemplatesBuildJob"/>
+    </class>
+    <securedutility
+        component="lp.translations.model.translationtemplatesbuildjob.TranslationTemplatesBuildJob"
+        provides="lp.translations.interfaces.translationtemplatesbuildjob.ITranslationTemplatesBuildJobSource">
+        <allow interface="lp.translations.interfaces.translationtemplatesbuildjob.ITranslationTemplatesBuildJobSource"/>
+    </securedutility>
+    <utility
+        component="lp.translations.model.translationtemplatesbuildjob.TranslationTemplatesBuildJob"
+        provides="lp.translations.interfaces.translationtemplatesbuildjob.ITranslationTemplatesBuildJob"
+        name="TRANSLATIONTEMPLATESBUILD"/>
+
+    <!-- TranslationTemplateBuildBehavior -->
+    <class
+        class="lp.translations.model.translationtemplatesbuildbehavior.TranslationTemplatesBuildBehavior">
+        <allow
+            interface="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"/>
+    </class>
+
+    <adapter
+        provides="lp.buildmaster.interfaces.buildfarmjobbehavior.IBuildFarmJobBehavior"
+        for="lp.translations.model.translationtemplatesbuildjob.TranslationTemplatesBuildJob"
+        factory="lp.translations.model.translationtemplatesbuildbehavior.TranslationTemplatesBuildBehavior"
+        />
+
   <webservice:register module="lp.translations.interfaces.webservice" />
 
 </configure>

=== added file 'lib/lp/translations/interfaces/translationtemplatesbuildjob.py'
--- lib/lp/translations/interfaces/translationtemplatesbuildjob.py	1970-01-01 00:00:00 +0000
+++ lib/lp/translations/interfaces/translationtemplatesbuildjob.py	2010-01-12 03:38:15 +0000
@@ -0,0 +1,37 @@
+# Copyright 2010 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+# pylint: disable-msg=E0213
+
+__metaclass__ = type
+
+__all__ = [
+    'ITranslationTemplatesBuildJob',
+    'ITranslationTemplatesBuildJobSource',
+    ]
+
+from zope.interface import Interface
+
+from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob
+from lp.code.interfaces.branchjob import IBranchJob
+
+
+class ITranslationTemplatesBuildJob(IBranchJob, IBuildFarmJob):
+    """Build-farm job type for generating translation templates."""
+
+
+class ITranslationTemplatesBuildJobSource(Interface):
+    """Container for `ITranslationTemplatesBuildJob`s."""
+
+    def create(branch):
+        """Create new `ITranslationTemplatesBuildJob`.
+
+        Also creates the matching `IBuildQueue` and `IJob`.
+
+        :param branch: A `Branch` that this job will check out and
+            generate templates for.
+        :return: A new `ITranslationTemplatesBuildJob`.
+        """
+
+    def getForJob(job):
+        """Find `ITranslationTemplatesBuildJob` matching given `Job`."""

=== added file 'lib/lp/translations/model/translationtemplatesbuildbehavior.py'
--- lib/lp/translations/model/translationtemplatesbuildbehavior.py	1970-01-01 00:00:00 +0000
+++ lib/lp/translations/model/translationtemplatesbuildbehavior.py	2010-01-12 03:38:15 +0000
@@ -0,0 +1,79 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""An `IBuildFarmJobBehavior` for `TranslationTemplatesBuildJob`.
+
+Dispatches translation template build jobs to build-farm slaves.
+"""
+
+__metaclass__ = type
+__all__ = [
+    'TranslationTemplatesBuildBehavior',
+    ]
+
+import socket
+import xmlrpclib
+
+from zope.component import getUtility
+from zope.interface import implements
+
+from canonical.launchpad.interfaces import ILaunchpadCelebrities
+
+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
+    IBuildFarmJobBehavior)
+from lp.buildmaster.model.buildfarmjobbehavior import (
+    BuildFarmJobBehaviorBase)
+from lp.soyuz.interfaces.builder import BuildSlaveFailure
+from lp.translations.interfaces.translationtemplatesbuildjob import (
+    ITranslationTemplatesBuildJobSource)
+
+
+class TranslationTemplatesBuildBehavior(BuildFarmJobBehaviorBase):
+    """Dispatches `TranslationTemplateBuildJob`s to slaves."""
+    implements(IBuildFarmJobBehavior)
+
+    # Identify the type of job to the slave.
+    build_type = 'translation-templates'
+
+    def dispatchBuildToSlave(self, build_queue_item, logger):
+        """See `IBuildFarmJobBehavior`."""
+        # XXX JeroenVermeulen 2009-12-24 bug=500110: This method is not
+        # covered by tests yet.  Either unify it with Soyuz code into a
+        # generalised method, or test it.
+        templatesbuildjob = self._findTranslationTemplatesBuildJob(
+            build_queue_item)
+        chroot = self._getChroot()
+        chroot_sha1 = chroot.content.sha1
+        self._builder.cacheFileOnSlave(logger, chroot)
+        buildid = templatesbuildjob.getName()
+
+        args = { 'branch_url': build_queue_item.branch.url }
+        filemap = {}
+
+        try:
+            status, info = self._builder.slave.build(
+                buildid, self.build_type, chroot_sha1, filemap, args)
+        except xmlrpclib.Fault, info:
+            # Mark builder as 'failed'.
+            logger.debug(
+                "Disabling builder: %s" % self._builder.url, exc_info=1)
+            self._builder.failbuilder(
+                "Exception (%s) when setting up to new job" % info)
+            raise BuildSlaveFailure
+        except socket.error, info:
+            error_message = "Exception (%s) when setting up new job" % info
+            self._builder.handleTimeout(logger, error_message)
+            raise BuildSlaveFailure
+
+    def _getChroot(self):
+        ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
+        return ubuntu.currentseries.nominatedarchindep.getChroot()
+
+    def _findTranslationTemplatesBuildJob(self, build_queue_item):
+        """Find the `TranslationTemplatesBuildJob` for a job.
+
+        :param build_queue_item: A `BuildQueue` entry.
+        :return: The matching `TranslationTemplatesBuildJob`.
+        """
+        jobsource = getUtility(ITranslationTemplatesBuildJobSource)
+        return jobsource.getForJob(build_queue_item.job)

=== added file 'lib/lp/translations/model/translationtemplatesbuildjob.py'
--- lib/lp/translations/model/translationtemplatesbuildjob.py	1970-01-01 00:00:00 +0000
+++ lib/lp/translations/model/translationtemplatesbuildjob.py	2010-01-12 03:38:15 +0000
@@ -0,0 +1,85 @@
+# Copyright 2010 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+__all__ = [
+    'TranslationTemplatesBuildJob',
+    ]
+
+import re
+from datetime import timedelta
+
+from zope.component import getUtility
+from zope.interface import classProvides, implements
+
+from canonical.launchpad.webapp.interfaces import (
+    DEFAULT_FLAVOR, IStoreSelector, MAIN_STORE, MASTER_FLAVOR)
+
+from lp.buildmaster.interfaces.buildfarmjob import BuildFarmJobType
+from lp.buildmaster.model.buildfarmjob import BuildFarmJob
+from lp.code.model.branchjob import BranchJob, BranchJobDerived, BranchJobType
+from lp.soyuz.model.buildqueue import BuildQueue
+from lp.translations.interfaces.translationtemplatesbuildjob import (
+    ITranslationTemplatesBuildJob, ITranslationTemplatesBuildJobSource)
+
+
+class TranslationTemplatesBuildJob(BranchJobDerived, BuildFarmJob):
+    """An `IBuildFarmJob` implementation that generates templates.
+
+    Implementation-wise, this is actually a `BranchJob`.
+    """
+    implements(ITranslationTemplatesBuildJob)
+    classProvides(ITranslationTemplatesBuildJobSource)
+
+    duration_estimate = timedelta(seconds=10)
+
+    unsafe_chars = '[^a-zA-Z0-9_+-]'
+
+    def __init__(self, branch_job):
+        super(TranslationTemplatesBuildJob, self).__init__(branch_job)
+
+    def score(self):
+        """See `IBuildFarmJob`."""
+        # Hard-code score for now; anything other than 1000 is probably
+        # inappropriate.
+        return 1000
+
+    def getLogFileName(self):
+        """See `IBuildFarmJob`."""
+        sanitized_name = re.sub(self.unsafe_chars, '_', self.getName())
+        return "translationtemplates_%s" % sanitized_name
+
+    def getName(self):
+        """See `IBuildFarmJob`."""
+        return '%s-%d' % (self.branch.name, self.job.id)
+
+    @classmethod
+    def create(cls, branch):
+        """See `ITranslationTemplatesBuildJobSource`."""
+        store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
+
+        # We don't have any JSON metadata for this BranchJob type.
+        metadata = {}
+        branch_job = BranchJob(
+            branch, BranchJobType.TRANSLATION_TEMPLATES_BUILD, metadata)
+        store.add(branch_job)
+        specific_job = TranslationTemplatesBuildJob(branch_job)
+
+        duration_estimate = cls.duration_estimate
+        build_queue_entry = BuildQueue(
+            estimated_duration=duration_estimate,
+            job_type=BuildFarmJobType.TRANSLATIONTEMPLATESBUILD,
+            job=specific_job.job.id)
+        store.add(build_queue_entry)
+
+        return specific_job
+
+    @classmethod
+    def getForJob(cls, job):
+        """See `ITranslationTemplatesBuildJobSource`."""
+        store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
+        branch_job = store.find(BranchJob, BranchJob.job == job).one()
+        if branch_job is None:
+            return None
+        else:
+            return cls(branch_job)

=== added file 'lib/lp/translations/tests/test_translationtemplatesbuildjob.py'
--- lib/lp/translations/tests/test_translationtemplatesbuildjob.py	1970-01-01 00:00:00 +0000
+++ lib/lp/translations/tests/test_translationtemplatesbuildjob.py	2010-01-12 03:38:15 +0000
@@ -0,0 +1,132 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+from unittest import TestLoader
+
+from zope.component import getUtility
+from zope.security.proxy import removeSecurityProxy
+
+from canonical.launchpad.interfaces import (
+    ILaunchpadCelebrities, ILibraryFileAliasSet)
+from canonical.launchpad.webapp.testing import verifyObject
+from canonical.testing import ZopelessDatabaseLayer
+
+from lp.testing import TestCaseWithFactory
+
+from lp.buildmaster.interfaces.buildfarmjob import IBuildFarmJob
+from lp.buildmaster.interfaces.buildfarmjobbehavior import (
+    IBuildFarmJobBehavior)
+from lp.services.job.model.job import Job
+from lp.soyuz.interfaces.buildqueue import IBuildQueueSet
+from lp.soyuz.model.buildqueue import BuildQueue
+from lp.translations.interfaces.translationtemplatesbuildjob import (
+    ITranslationTemplatesBuildJob, ITranslationTemplatesBuildJobSource)
+
+
+def get_job_id(job):
+    """Peek inside a `Job` and retrieve its id."""
+    return removeSecurityProxy(job).id
+
+
+class TestTranslationTemplatesBuildJob(TestCaseWithFactory):
+    """Test `TranslationTemplatesBuildJob`."""
+
+    layer = ZopelessDatabaseLayer
+
+    def setUp(self):
+        super(TestTranslationTemplatesBuildJob, self).setUp()
+        self.jobset = getUtility(ITranslationTemplatesBuildJobSource)
+        self.branch = self.factory.makeBranch()
+        self.specific_job = self.jobset.create(self.branch)
+
+    def test_new_TranslationTemplatesBuildJob(self):
+        # TranslationTemplateBuildJob implements IBuildFarmJob and
+        # ITranslationTemplatesBuildJob.
+        verifyObject(IBuildFarmJob, self.specific_job)
+        verifyObject(ITranslationTemplatesBuildJob, self.specific_job)
+
+        # The class also implements a utility.
+        verifyObject(ITranslationTemplatesBuildJobSource, self.jobset)
+
+        # Each of these jobs knows the branch it will operate on.
+        self.assertEqual(self.branch, self.specific_job.branch)
+
+    def test_has_Job(self):
+        # Associated with each TranslationTemplateBuildJob is a Job.
+        base_job = self.specific_job.job
+        self.assertIsInstance(base_job, Job)
+
+        # From a Job, the TranslationTemplatesBuildJobSource can find the
+        # TranslationTemplatesBuildJob back for us.
+        specific_job_for_base_job = removeSecurityProxy(
+            self.jobset.getForJob(base_job))
+        self.assertEqual(self.specific_job, specific_job_for_base_job)
+
+    def test_has_BuildQueue(self):
+        # There's also a BuildQueue item associated with the job.
+        queueset = getUtility(IBuildQueueSet)
+        job_id = get_job_id(self.specific_job.job)
+        buildqueue = queueset.get(job_id)
+
+        self.assertIsInstance(buildqueue, BuildQueue)
+        self.assertEqual(job_id, get_job_id(buildqueue.job))
+
+    def test_getName(self):
+        # Each job gets a unique name.
+        other_job = self.jobset.create(self.branch)
+        self.assertNotEqual(self.specific_job.getName(), other_job.getName())
+
+    def test_getLogFileName(self):
+        # Each job has a unique log file name.
+        other_job = self.jobset.create(self.branch)
+        self.assertNotEqual(
+            self.specific_job.getLogFileName(), other_job.getLogFileName())
+
+    def test_score(self):
+        # For now, these jobs always score themselves at 1,000.  In the
+        # future however the scoring system is to be revisited.
+        self.assertEqual(1000, self.specific_job.score())
+
+
+class TestTranslationTemplatesBuildBehavior(TestCaseWithFactory):
+    """Test `TranslationTemplatesBuildBehavior`."""
+
+    layer = ZopelessDatabaseLayer
+
+    def setUp(self):
+        super(TestTranslationTemplatesBuildBehavior, self).setUp()
+        self.jobset = getUtility(ITranslationTemplatesBuildJobSource)
+        self.branch = self.factory.makeBranch()
+        self.specific_job = self.jobset.create(self.branch)
+        self.behavior = IBuildFarmJobBehavior(self.specific_job)
+
+    def test_getChroot(self):
+        # _getChroot produces the current chroot for the current Ubuntu
+        # release, on the nominated architecture for
+        # architecture-independent builds.
+        ubuntu = getUtility(ILaunchpadCelebrities).ubuntu
+        current_ubuntu = ubuntu.currentseries
+        distroarchseries = current_ubuntu.nominatedarchindep
+
+        # Set an arbitrary chroot file.
+        fake_chroot_file = getUtility(ILibraryFileAliasSet)[1]
+        distroarchseries.addOrUpdateChroot(fake_chroot_file)
+
+        chroot = self.behavior._getChroot()
+
+        self.assertNotEqual(None, chroot)
+        self.assertEqual(fake_chroot_file, chroot)
+
+    def test_findTranslationTemplatesBuildJob(self):
+        job = self.specific_job.job
+        job_id = removeSecurityProxy(job).id
+        buildqueue = getUtility(IBuildQueueSet).get(job_id)
+        specific_job_for_buildqueue = removeSecurityProxy(
+            self.behavior._findTranslationTemplatesBuildJob(buildqueue))
+        self.assertEqual(self.specific_job, specific_job_for_buildqueue)
+
+
+def test_suite():
+    return TestLoader().loadTestsFromName(__name__)