= Bug 507681 =

This is the main work bug 507681, which in turn is part of the effort to generate translation templates on the build farm, based on source in a bzr tree.

This branch gets the branch scanner to create TranslationTemplateBuildJobs for branch changes that may require new translation templates to be generated.

You'll see occasional reference to what we call "pottery."  Pottery is a Translations module that deals with detecting various types of translation setup in a source code branch (we only support certain intltool setups at the moment) and generating templates from them.

Going through the diff step by step:

configs/testrunner/launchpad-lazr.conf
lib/canonical/config/schema-lazr.conf

A new configuration item, rosetta.generate_templates, enables or disables the generation of these jobs.  It's off by default, to avoid interfering with Soyuz operation unnecessarily before these new jobs become useful.  However the feature is enabled on the test runner.

lib/lp/code/interfaces/branchjob.py
lib/lp/code/model/branchjob.py

Two of the IRosettaUploadJobSource utility methods were published in its interface.  I changed their names to follow our naming style, and (after consultation with Henning) hopefully made the docstrings a bit clearer.

These two methods are used to control importing of templates and POFiles from a branch into a productseries' translations.  Normally those imports are triggered by a change to the branch, but the user can also request a one-time import on the productseries.

For the one-time imports, a parameter force_translations_upload is set to True.  When it's set, the importer ignores whether a productseries attached to the branch wants automatic translation imports from it and just uploads anyway.

This is strictly speaking a bug; it does the wrong thing when multiple productseries use the same branch.  I isolated the brokenness a tiny bit by not passing on the force_translations_upload parameter to the "are there any productseries using this branch" method in the utility.  Since force_translations_upload is an option on the productseries using the branch, it's safe to skip the check for productseries when the parameter is True.

In findProductSeries (née _find_product_series), I unified the two alternative queries to express that there is really only a very small difference.

lib/lp/codehosting/scanner/branch_scanner.py
lib/lp/codehosting/scanner/bzrsync.py

Here is where I registered an extra hook that runs when changes are pushed to a branch.  This is similar to what other branch jobs already do; the real logic is in the ITranslationTemplatesBuildJobSource utility for ease of testing.

lib/lp/translations/interfaces/translationtemplatesbuildjob.py
lib/lp/translations/model/translationtemplatesbuildjob.py

A bit more logic to see whether Launchpad should try to generate templates for a branch.  For now we decided not to introduce a separate setting for this; we'll simply try to generate templates for each branch that has translation series importing templates from it.

Except that pottery has a function to check whether it's worth trying this.  It makes no sense to bother a build-farm slave if a branch contains no intltool setup that pottery can work with.  So we also check that.

lib/lp/translations/pottery/detect_intltool.py

I discovered that is_intltool_structure wasn't being exported in __all__.  Never noticed a single warning about this.

lib/lp/translations/tests/test_translationtemplatesbuildjob.py

I've broken out a separate test for the TranslationTemplatesBuildJob utility.  To save having to set up intltool-compatible branch contents for every test, I faked part of the utility.  This isn't as straightforward as it might be because the TranslationTemplatesBuildJob class itself is the utility; there is no factory of objects.  So the utility is a singleton.  I "copied" it by inheriting a fake utility class from it, and giving that the option of faking behavior.

Doing it this way is a bit unæsthetic because setting the faking option affects a persistent variable, not a local one.  I defined a tearDown to make sure that the setting is undone at the end of each test.

To test:
{{{
./bin/test -vv -t translationtemplatesbuild -t rosetta_branches
}}}

There was a bit of lint left that I didn't touch:
{{{
= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  configs/testrunner/launchpad-lazr.conf
  lib/canonical/config/schema-lazr.conf
  lib/lp/code/interfaces/branchjob.py
  lib/lp/code/model/branchjob.py
  lib/lp/codehosting/scanner/branch_scanner.py
  lib/lp/codehosting/scanner/bzrsync.py
  lib/lp/translations/interfaces/translationtemplatesbuildjob.py
  lib/lp/translations/model/translationtemplatesbuildjob.py
  lib/lp/translations/pottery/detect_intltool.py
  lib/lp/translations/tests/test_translationtemplatesbuildjob.py


== Pyflakes notices ==

lib/lp/codehosting/scanner/branch_scanner.py
    17: 'zope' imported but unused


== Pylint notices ==

lib/lp/code/model/branchjob.py
    29: [F0401] Unable to import 'lazr.enum' (No module named enum)
    30: [F0401] Unable to import 'lazr.delegates' (No module named delegates)

lib/lp/codehosting/scanner/branch_scanner.py
    17: [W0611] Unused import zope

lib/lp/codehosting/scanner/bzrsync.py
    29: [F0401] Unable to import 'lazr.uri' (No module named uri)
}}}

The "Unused import zope" carries a comment saying that what it does is somehow necessary.

To Q/A this branch, turn on config.rosetta.generate_templates in the appropriate config.  Take or set up a branch attached to a productseries.  Enable translation template imports for that productseries.  Then push a change to the branch.

This should create one object each of BranchJob, Job, and BuildQueue.  After that, assuming that my branch for bug 499405 has finally received its imprimatur, comes the exciting stuff: it becomes possible to Q/A how the buildfarm management engine deals with those jobs.  The actual jobs won't do anything, and will fail to produce expected files for now, but at least this allows manual integration-testing of the infrastructure work.


Jeroen