Merge lp:~cjwatson/launchpad/refactor-cron-germinate into lp:launchpad

Proposed by Colin Watson on 2011-12-06
Status: Merged
Approved by: Jeroen T. Vermeulen on 2011-12-13
Approved revision: no longer in the source branch.
Merged at revision: 14516
Proposed branch: lp:~cjwatson/launchpad/refactor-cron-germinate
Merge into: lp:launchpad
Diff against target: 1226 lines (+950/-135)
10 files modified
cronscripts/generate-extra-overrides.py (+18/-0)
cronscripts/publishing/cron.germinate (+5/-126)
database/schema/security.cfg (+4/-0)
lib/lp/archivepublisher/config.py (+3/-0)
lib/lp/archivepublisher/scripts/generate_extra_overrides.py (+339/-0)
lib/lp/archivepublisher/tests/publisher-config.txt (+7/-0)
lib/lp/archivepublisher/tests/test_generate_extra_overrides.py (+567/-0)
lib/lp/soyuz/scripts/tests/germinate-test-data/mock-bin/germinate (+0/-5)
lib/lp/soyuz/scripts/tests/germinate-test-data/mock-lp-root/cronscripts/generate-extra-overrides.py (+5/-0)
lib/lp/soyuz/scripts/tests/test_cron_germinate.py (+2/-4)
To merge this branch: bzr merge lp:~cjwatson/launchpad/refactor-cron-germinate
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) 2011-12-06 Approve on 2011-12-07
Review via email: mp+84624@code.launchpad.net

Commit message

[r=jtv][bug=899972] Reimplement most of cron.germinate in Python using the new python-germinate package. This no longer needs to recompute the dependency expansion of common seeds over and over again, so is much faster.

Description of the change

= Summary =

cron.germinate currently takes on the order of ten minutes, and really doesn't have a particularly good excuse for taking so long. Fixing this would probably be enough to let us move back to 30-minute publisher cycles for Ubuntu (as we used to have with dak, back in the dawn of time), which would increase our velocity and make some of us very happy.

The main reason it's so slow is that it runs germinate as a separate process once for each of eight flavours (Ubuntu, Kubuntu, etc., each with its own seed collection) on each of five architectures. There are a few inefficiencies inherent in this approach, but the most significant is that it has to expand dependencies and build-dependencies of the seeds that are common to all the flavours eight times as often as necessary. Since the build-dependency chain in particular of the base system winds its way through a good fraction of main, this winds up being rather a lot of duplicated work.

== Proposed fix ==

I've been working on this problem for some time now, and have just released germinate 2.0 to support solving it properly. The most important change here is that germinate can now process multiple seed collections for a single architecture in a single instance of the core Germinator class, which allows reusing the expansions of common seeds. While I could technically have extended the command-line interface further for this, I felt that the command-line interface was already far too complicated, and decided instead to export a public, documented, and stable Python interface which can be used for this purpose.

This branch is the other half of that fix: a LaunchpadScript that uses germinate's new Python interface to do the same work as the top part of cron.germinate (i.e. everything except for maintenance-check.py). A stripped-down subset of this with just the logic (germinating seeds and producing output) runs in about three minutes on my laptop.

== Pre-implementation notes ==

The Launchpad position for a while has been that cron.germinate is owned by Ubuntu Engineering, and indeed the vast bulk of the work here was in preparing a new germinate release in Ubuntu that does what we need. However, I have talked with various Launchpad people over the last few months so that it was clear that I was working on this, and I did need some help figuring out how to approach testing. Initially I'd been planning on doing a full Soyuz publisher run so that I had Packages and Sources files to work with, but Julian said this would be too slow and pointed me towards methods like factory.makeSourcePackagePublishingHistory, which allow me to use getIndexStanza et al so that I don't have to write out RFC822 data by hand.

== Implementation details ==

I had to add a new attribute on PublisherConfig pointing to the germinate output directory; previously this was only known to the shell script.

I added a 'generate_extra_overrides' user to security.cfg, which seemed to be in line with other archivepublisher scripts. Was this the right approach? Does it require any special deployment steps?

The actual override generation was a fairly straightforward translation from shell to Python.

Predictably, the bulk of the work was in writing tests, since cron.germinate was previously untested and I didn't think continuing with untested code was likely to fly once it was in Python. The top-level shell script is still untested, but there's a lot less in it and I expect that it can eventually be removed entirely.

I ran into a couple of existing bugs that slowed me down. Firstly, I'd hoped to use the fake librarian to speed things up a bit, but ran into bug 713764, which was beyond my ability to fix (although I tried). Secondly, bug 694140 seriously confused me for a while; I added much the same workaround as is found elsewhere in Launchpad for that problem so that germinate's log output doesn't get lost.

I wanted to include a test that the cronscript could be run standalone, but I couldn't see how to ensure that a suitable PublisherConfig was in place.

== Tests ==

bin/test -vvct generate_extra_overrides

This branch makes some small changes to other archivepublisher tests, so:

bin/test -vvct archivepublisher

== Demo and Q/A ==

The deployment steps listed in bug 899972 and https://rt.admin.canonical.com/Ticket/Display.html?id=49745 need to be completed before we can land this branch.

Once that's done, we can Q/A this by doing a timed control publisher run on mawson, rolling out this change, taking a backup copy of ubuntu-germinate and ubuntu-misc, and then doing another timed publisher run and comparing the ubuntu-germinate and ubuntu-misc directories before and after. (There may be some harmless ordering changes and such.)

== Lint ==

./cronscripts/generate-extra-overrides.py
       8: '_pythonpath' imported but unused

Seems to be standard practice in cronscripts.

./lib/lp/archivepublisher/tests/publisher-config.txt
       1: narrative uses a moin header.
      26: narrative uses a moin header.
      46: narrative uses a moin header.
     114: narrative uses a moin header.
     138: narrative uses a moin header.
     163: narrative uses a moin header.
     178: want exceeds 78 characters.
     179: want exceeds 78 characters.
     180: want exceeds 78 characters.
     182: want exceeds 78 characters.

This was pre-existing lint; I only made the minimal changes necessary to add the 'germinateroot' attribute, which haven't fundamentally made the situation any worse.

To post a comment you must log in.
Colin Watson (cjwatson) wrote :

I've tested the core logic on mawson now (stripped out the database interaction and hardcoded paths to a local mirror so that I could run it in isolation). Compared to an output tree generated by the old cron.germinate with germinate 2.0, the substantive output is identical: various files are deliberately no longer generated, and the log output is different, but the actual germinate output and the more-extra.override.precise.main file are identical. It runs in about 7 minutes compared with about 14/15 minutes for the old cron.germinate, which I call a win (although these are single-run figures).

The weird part is that something is wrong that means we aren't even getting a lot of the benefit yet! It isn't actually sharing the results of germinating common seeds. Therefore, that 7/8 minutes of win must essentially come from avoiding duplicated seed parsing, avoiding duplicated Packages/Sources parsing, and a reduction in the number of files we write. I wasn't expecting that to account for so much of the runtime, but I'm not going to complain.

I will investigate further to find out why common seed output isn't being shared. That should bring the runtime down further.

Colin Watson (cjwatson) wrote :

Right. I've tracked down all the problems I could find and released germinate 2.1 to fix all of them; none of them were problems with this branch. The runtime is now down to about 4m50s on mawson; the only differences in output are (a) in places where it doesn't matter very much and (b) accountable to a pure-virtual dependency in libenchant1c2a, where germinate is entirely entitled to choose a provider at random, so I'm not worried about those.

Jeroen T. Vermeulen (jtv) wrote :

A note on the problem of ensuring that a PublisherConfig is in place: factory.makeDistribution creates one for you. Just make sure to pass it a publish_root_dir, or I think it will use default system config. This should probably be considered a bug in the factory.

Jeroen T. Vermeulen (jtv) wrote :

It looks to me like you did the right thing in creating a dedicated database user. We may ditch the per-user privileges, but we do want separate users for separate scripts.

Jeroen T. Vermeulen (jtv) wrote :

=== added file 'lib/lp/archivepublisher/scripts/generate_extra_overrides.py'
--- lib/lp/archivepublisher/scripts/generate_extra_overrides.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/scripts/generate_extra_overrides.py 2011-12-06 15:03:58 +0000

+class AtomicFile:
+ """Facilitate atomic writing of files."""
+
+ def __init__(self, filename):
+ self.filename = filename
+ self.fd = open('%s.new' % self.filename, 'w')
+
+ def __enter__(self):
+ return self.fd
+
+ def __exit__(self, exc_type, exc_value, exc_tb):
+ self.fd.close()
+ os.rename('%s.new' % self.filename, self.filename)

Does it make sense to put the file in place if the context exits with an exception?

Actually this is what I don't like about the design of python context managers: it puts you, as the author of a context manager, in the position where you may have to worry about failures in the context you manage — and yet it's easy for the author of the context to hide them from you. So maybe stupid is just better.

Jeroen T. Vermeulen (jtv) wrote :

Next bit I looked at:

=== added file 'lib/lp/archivepublisher/scripts/generate_extra_overrides.py'

+ def processOptions(self):
+ """Handle command-line options."""
+ if self.options.distribution is None:
+ raise OptionValueError("Specify a distribution.")
+
+ self.distribution = getUtility(IDistributionSet).getByName(
+ self.options.distribution)
+ if self.distribution is None:
+ raise OptionValueError(
+ "Distribution '%s' not found." % self.options.distribution)
+
+ series = None
+ wanted_status = (SeriesStatus.DEVELOPMENT,
+ SeriesStatus.FROZEN)
+ for status in wanted_status:
+ series = self.distribution.getSeriesByStatus(status)
+ if series.count() > 0:
+ break
+ else:
+ raise LaunchpadScriptFailure(
+ 'There is no DEVELOPMENT distroseries for %s' %
+ self.options.distribution)
+ self.series = series[0]

The part from “series = None” onward seems to be an isolated unit of work. I think it looks for the first distroseries in development state, or failing that, the first in frozen state. But the structure of the code makes that hard to figure out, and then only afterwards I can start wondering why you do it exactly this way.

I fervently recommend extracting that code into a sensibly-named function. (It doesn't need to be a method: AFAICS a distribution goes in, a series comes out, and that is the full extent of its interaction with the rest of the world). Come to think of it, might there already be a suitable method in Distribution somewhere?

The structure of that loop is also a bit hard to follow. It gets easier if you have a single-purpose function that you can just return from instead of using a break!

Finally, a few cosmetic tips:

 * When line-breaking a tuple, list, or dict, our “house style” is to add a line break right after the opening parenthesis, bracket, or brace. Each entry ends in a comma and a newline — including the last entry.

    wanted_status = (
        SeriesStatus.DEVELOPMENT,
        SeriesStatus.FROZEN,
        )

 * You don't really need series.count(). In fact I think you can just retrieve the first matching series as series.first(), and you'll get None if there isn't any.

 * For Python strings I find consistent double quotes a good habit, at least for free-form texts that may just as well contain an apostrophe.

Jeroen T. Vermeulen (jtv) wrote :

I'm going through the diff sequentially at the moment; the parts I'm ignoring are the parts that I have nothing to say about. When I get to the end I can give you a very brief formal review. :)

=== added file 'lib/lp/archivepublisher/scripts/generate_extra_overrides.py'

+ def getConfig(self):
+ """Set up a configuration object for this archive."""
+ for archive in self.distribution.all_distro_archives:
+ # We only work on the primary archive.
+ if archive.purpose == ArchivePurpose.PRIMARY:
+ return getPubConfig(archive)
+ else:
+ raise LaunchpadScriptFailure(
+ 'There is no PRIMARY archive for %s' %
+ self.options.distribution)

Why not just use self.distribution.main_archive?

(Also, remember to punctuate that error message!)

Jeroen T. Vermeulen (jtv) wrote :

+ def outputPath(self, flavour, series_name, arch, base):
+ return os.path.join(
+ self.config.germinateroot,
+ '%s_%s_%s_%s' % (base, flavour, series_name, arch))

Our dromedary-cased method names normally start with a verb. Read that way, “outputPath” would suggest that the method prints a path.

So consider prefixing this method's name with a verb. The unimaginative catch-all verb is “get”; personally I prefer “compose” for this kind of thing.

Jeroen T. Vermeulen (jtv) wrote :
Download full text (8.5 KiB)

+ def runGerminate(self, override_file, series_name, arch, flavours,
+ structures):

Rightly or wrongly, the “run” in that name led me to expect that this method would fire up a separate process. Maybe a very short docstring to dispel that notion?

+ germinator = Germinator(arch)
+
+ # Read archive metadata.
+ archive = TagFile(
+ series_name, self.components, arch,
+ 'file://%s' % self.config.archiveroot, cleanup=True)
+ germinator.parse_archive(archive)
+
+ for flavour in flavours:

To me, your branch is of very high quality but this is its Achilles' heel. The following loop body is massive! I can't see it all on one page, so it becomes hard even to see just how long it is. If I judged the indentations right, it makes up the entire rest of the method.

Please extract it. It's well nigh impossible to see (now or in the future, after some maintenance) whether any variables are carried across loop iterations. That sort of thing can easily trip you up, especially when it happens by accident.

Once you've extracted the loop body, given its size, it's probably still worth extracting chunks from that.

According to some, the pattern of “comment, block of code, blank line, comment, block of code, blank line” is a strong indication that you need to split things up. It shows that you have already done so in your own mind, but leaves each individual reader to figure out what the dependencies between consecutive blocks are.

Also, of course, small methods with simple purposes are easier to specify in detail and cheaper to test thoroughly.

+ self.logger.info('Germinating for %s/%s/%s',
+ flavour, series_name, arch)

We follow a different rule for line-breaking function calls than we do for function definitions.

For calls, line-break right after the opening parenthesis and then indent the arguments list by 4 spaces:

    self.logger.info(
        "Germinating for %s/%s/%s.", flavour, series_name, arch)

+ # Add this to the germinate log as well so that that can be
+ # debugged more easily. Log a separator line first.
+ self.germinate_logger.info('', extra={'progress': True})
+ self.germinate_logger.info('Germinating for %s/%s/%s',
+ flavour, series_name, arch,
+ extra={'progress': True})

What does the “extra” parameter do?

+ # Expand dependencies.
+ structure = structures[flavour]
+ germinator.plant_seeds(structure)
+ germinator.grow(structure)
+ germinator.add_extras(structure)
+ # Write output files.
+
+ # The structure file makes it possible to figure out how the
+ # other output files relate to each other.
+ structure.write(self.outputPath(
+ flavour, series_name, arch, 'structure'))
+
+ # "all" and "all.sources" list the full set of binary and source
+ # packages respectively for a given flavour/suite/architecture
+ # combination.
+ all_path = ...

Read more...

Colin Watson (cjwatson) wrote :

I think you're right that the file shouldn't be put in place if the
context exits with an exception. If I were writing this without context
managers, then rather than:

    with AtomicFile('foo') as handle:
        print >>handle, 'bar'

... I'd probably write:

    handle = open('foo.new', 'w')
    try:
        print >>handle, 'bar'
    finally:
        handle.close()
    os.rename('foo.new', 'foo')

And that's equivalent to the natural expression in (good) shell too:

    set -e
    echo bar >foo.new
    mv foo.new foo

That all suggests to me that skipping the rename on exception, and
leaving the .new file in place so that somebody with access can look at
it and find out how far the script got before it fell over, is a good
thing. I'll change the code to do that.

Jeroen T. Vermeulen (jtv) wrote :

Getting a better feel for the structure of runGerminate now. Which makes me wonder:

+ # Generate apt-ftparchive "extra overrides" for Build-Essential
+ # fields.
+ if ('build-essential' in structure.names and
+ flavour == flavours[0]):
+ writeOverrides('build-essential', 'Build-Essential', 'yes')

I don't suppose you could just do this at the end, outside of all loops?

    if 'build-essential' in structures[flavours[0]].names:
        writeOverrides('build-essential', 'Build-Essential', 'yes')

Jeroen T. Vermeulen (jtv) wrote :

Ah, I see now why you can't do that: writeOverrides is a closure. It picks up “structure” and “arch” from the surrounding method at its point of definition. There's another easy thing to break by accident! The structure should again be structures[flavours[0]], I guess.

Might it be worthwhile to separate the closure nature of writeOverrides from its functionality? If you had a separate function that takes all the necessary variables as arguments, it'd be easier to move that last paragraph out of the loop (still assuming that that makes sense at all) and it'd be easier to see that writeOverrides “borrows” some variables from runGerminate.

Colin Watson (cjwatson) wrote :

On Wed, Dec 07, 2011 at 07:20:29AM -0000, Jeroen T. Vermeulen wrote:
> The part from “series = None” onward seems to be an isolated unit of
> work. I think it looks for the first distroseries in development
> state, or failing that, the first in frozen state. But the structure
> of the code makes that hard to figure out, and then only afterwards I
> can start wondering why you do it exactly this way.
>
> I fervently recommend extracting that code into a sensibly-named
> function. (It doesn't need to be a method: AFAICS a distribution goes
> in, a series comes out, and that is the full extent of its interaction
> with the rest of the world). Come to think of it, might there already
> be a suitable method in Distribution somewhere?

Fair comment. Distribution.currentseries is nearly there: it just
sometimes returns series in statuses we don't want here; but if there's
one we can use then it will always return it, so we can just call
currentseries and then check the result.

> * For Python strings I find consistent double quotes a good habit, at
> least for free-form texts that may just as well contain an
> apostrophe.

This is an out-of-context habit of mine from Perl and shell programming:
since single and double quotes have different interpolation
characteristics there, I've trained myself to use single quotes unless
either I have an explicit reason to want interpolation or the string
contains an apostrophe and no double quotes. I realise this doesn't
make as much sense for Python, so I'll go through and amend this.

Jeroen T. Vermeulen (jtv) wrote :

Some very nice tests there! To continue:

=== added file 'lib/lp/archivepublisher/tests/test_generate_extra_overrides.py'

+def file_contents(path):
+ """Return the contents of the file at path."""
+ with open(path) as handle:
+ return handle.read()

Since you're only reading the file, there's probably no hurry to close this file handle. So you could just return “open(path).read().”

Colin Watson (cjwatson) wrote :
Download full text (4.5 KiB)

[I've not responded to your entire review point-by-point, but I believe
I've addressed all points where I haven't given an explicit response.]

On Wed, Dec 07, 2011 at 09:22:24AM -0000, Jeroen T. Vermeulen wrote:
> + def runGerminate(self, override_file, series_name, arch, flavours,
> + structures):
>
> Rightly or wrongly, the “run” in that name led me to expect that this
> method would fire up a separate process. Maybe a very short docstring
> to dispel that notion?

The name doesn't actually make a lot of sense; I think originally I was
in the mindset where this actually did fire up a separate process. I've
renamed the script instance to germinateArch (with a docstring), and the
test instance to fetchGerminatedOverrides. Does that help?

> To me, your branch is of very high quality but this is its Achilles'
> heel. The following loop body is massive! I can't see it all on one
> page, so it becomes hard even to see just how long it is. If I judged
> the indentations right, it makes up the entire rest of the method.

That's fair; I went back and forward a bit on this, but clearly landed
in the wrong place. Actually, now that I've renamed runGerminate to
germinateArch, it's easier in my mind to turn a block of it into
germinateArchFlavour in turn. (Regularised naming may be boring, but I
like it anyway.)

The methods do start ending up with quite a few parameters, but that's
probably better than the alternative.

> Once you've extracted the loop body, given its size, it's probably
> still worth extracting chunks from that.

I've done a good deal of this, although will probably end up
restructuring a bit further per your later comments.

> According to some, the pattern of “comment, block of code, blank line,
> comment, block of code, blank line” is a strong indication that you
> need to split things up. It shows that you have already done so in
> your own mind, but leaves each individual reader to figure out what
> the dependencies between consecutive blocks are.

I don't necessarily agree with this in all cases (e.g. I don't know that
the new writeGerminateOutput method would benefit much from being split
further), but I agree that the original version of this code was rather
too far in the other direction.

> + # Add this to the germinate log as well so that that can be
> + # debugged more easily. Log a separator line first.
> + self.germinate_logger.info('', extra={'progress': True})
> + self.germinate_logger.info('Germinating for %s/%s/%s',
> + flavour, series_name, arch,
> + extra={'progress': True})
>
> What does the “extra” parameter do?

That attaches a dictionary of extra attributes to the log message which
can then be picked up by the formatter. Germinate uses this for certain
messages (arguably a wart in germinate's design; I should probably look
into doing that more neatly at some point). I've moved this to its own
method so that it can have a clarifying docstring.

> That's a lot of string manipulation. It may be clearer as a regex:
>
> task_header_regex = re.compile("ta...

Read more...

Colin Watson (cjwatson) wrote :

On Wed, Dec 07, 2011 at 09:56:26AM -0000, Jeroen T. Vermeulen wrote:
> Might it be worthwhile to separate the closure nature of
> writeOverrides from its functionality? If you had a separate function
> that takes all the necessary variables as arguments, it'd be easier to
> move that last paragraph out of the loop (still assuming that that
> makes sense at all) and it'd be easier to see that writeOverrides
> “borrows” some variables from runGerminate.

Good idea. Per our conversation on IRC, I've used functools.partial for
this. I also used it in writeGerminateOutput, which IMO this approach
notably improves. What do you think?

Jeroen T. Vermeulen (jtv) wrote :

Thank you for bearing with me, making so many changes after producing an already large and difficult branch. I particularly like the way you seized on method extraction as an opportunity to document the algorithm's steps.

This is a massive step forward in terms of code maintenance alone, and I'm looking forward to seeing how it affects performance.

review: Approve
Colin Watson (cjwatson) wrote :

On Wed, Dec 07, 2011 at 10:14:27AM -0000, Jeroen T. Vermeulen wrote:
> +def file_contents(path):
> + """Return the contents of the file at path."""
> + with open(path) as handle:
> + return handle.read()
>
> Since you're only reading the file, there's probably no hurry to close this file handle. So you could just return “open(path).read().”

For the record, there was some discussion on this on IRC, and most
people seemed to think that it was better to retain the immediate close,
in particular since implementations other than cPython may not GC-close
it with any alacrity; so I've left this the way it is. (There is a lot
of variation on this in the Launchpad test suite, though ...)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'cronscripts/generate-extra-overrides.py'
2--- cronscripts/generate-extra-overrides.py 1970-01-01 00:00:00 +0000
3+++ cronscripts/generate-extra-overrides.py 2011-12-14 15:49:40 +0000
4@@ -0,0 +1,18 @@
5+#!/usr/bin/python -S
6+#
7+# Copyright 2011 Canonical Ltd. This software is licensed under the
8+# GNU Affero General Public License version 3 (see the file LICENSE).
9+
10+"""Generate extra overrides using Germinate."""
11+
12+import _pythonpath
13+
14+from lp.archivepublisher.scripts.generate_extra_overrides import (
15+ GenerateExtraOverrides,
16+ )
17+
18+
19+if __name__ == '__main__':
20+ script = GenerateExtraOverrides(
21+ "generate-extra-overrides", dbuser='generate_extra_overrides')
22+ script.lock_and_run()
23
24=== modified file 'cronscripts/publishing/cron.germinate'
25--- cronscripts/publishing/cron.germinate 2011-10-27 11:36:13 +0000
26+++ cronscripts/publishing/cron.germinate 2011-12-14 15:49:40 +0000
27@@ -12,8 +12,12 @@
28 GERMINATEROOT=$ARCHIVEROOT/../ubuntu-germinate
29
30 LAUNCHPADROOT=${TEST_LAUNCHPADROOT:-/srv/launchpad.net/codelines/current}
31+GENERATE=$LAUNCHPADROOT/cronscripts/generate-extra-overrides.py
32 MAINTAINCE_CHECK=$LAUNCHPADROOT/cronscripts/publishing/maintenance-check.py
33
34+FLAVOURS="ubuntu kubuntu kubuntu-mobile edubuntu xubuntu mythbuntu lubuntu"
35+FLAVOURS="$FLAVOURS ubuntustudio"
36+
37 ## Check to see if another germinate run is in progress
38
39 LOCKFILE=$LOCKROOT/cron.germinate.lock
40@@ -28,134 +32,9 @@
41
42 trap cleanup EXIT
43
44-suite=`$LAUNCHPADROOT/scripts/ftpmaster-tools/lp-query-distro.py development`
45-archs=`$LAUNCHPADROOT/scripts/ftpmaster-tools/lp-query-distro.py -s "$suite" archs`
46-
47-echo -n "Running germinate... "
48 cd $GERMINATEROOT
49
50-# Clean up temporary files
51-rm -f \
52- germinate.output ALL ALL.sources \
53- UBUNTU-* KUBUNTU-* EDUBUNTU-* XUBUNTU-* MYTHBUNTU-* LUBUNTU-* \
54- UBUNTUSTUDIO-*
55-rm -f all_* all.sources_*
56-rm -rf dists
57-
58-# Grab local copies of Sources and Packages files, to avoid problems in case
59-# the archive changes under our feet.
60-for component in main universe restricted multiverse; do
61- base="dists/$suite/$component"
62- mkdir -p "$base/source"
63- zcat "$ARCHIVEROOT/$base/source/Sources.gz" > "$base/source/Sources"
64- for arch in $archs; do
65- mkdir -p "$base/binary-$arch" "$base/debian-installer/binary-$arch"
66- zcat "$ARCHIVEROOT/$base/binary-$arch/Packages.gz" \
67- > "$base/binary-$arch/Packages"
68- zcat "$ARCHIVEROOT/$base/debian-installer/binary-$arch/Packages.gz" \
69- > "$base/debian-installer/binary-$arch/Packages"
70- done
71-done
72-
73-> "$MISCROOT/more-extra.override.$suite.main.new"
74-
75-germinate_components=main,universe,restricted,multiverse
76-for distro in \
77- ubuntu kubuntu kubuntu-mobile edubuntu xubuntu mythbuntu lubuntu \
78- ubuntustudio
79-do
80- DISTRO="$(echo $distro | tr a-z A-Z)"
81- germinate_suite="$distro.$suite"
82- for arch in $archs; do
83- # Run germinate
84- echo " **************** $distro/$suite/$arch ********************* " >> germinate.output
85- germinate \
86- --no-rdepends \
87- -m "file://$(pwd)/" \
88- -s "$germinate_suite" \
89- -d "$suite" \
90- -c "$germinate_components" \
91- -a $arch >> germinate.output 2>&1
92-
93- # The structure file is generally useful; keep per distro/suite/arch
94- # copies for convenience
95- cp structure structure_"$distro"_"$suite"_"$arch"
96-
97- # Keep per distro/suite/arch copies of 'all' and 'all.sources' for
98- # anastacia.
99- cp all all_"$distro"_"$suite"_"$arch"
100- cp all.sources all.sources_"$distro"_"$suite"_"$arch"
101-
102- # Keep per distro/suite/arch copies of 'minimal' and 'standard' for
103- # jessica.
104- cp minimal minimal_"$distro"_"$suite"_"$arch"
105- cp standard standard_"$distro"_"$suite"_"$arch"
106-
107- # Keep amalgamated copies of 'all' and 'all.sources', just for convenience
108- cat all >> ALL; cat all.sources >> ALL.sources
109-
110- # We need to fetch a number of seeds so that we can generate Task fields
111- # for them. This changes over time and differs from derivative to
112- # derivative, so it's best to just fetch them all.
113- taskseeds="$(cut -d: -f1 structure | xargs -n1 | sort -u)"
114-
115- for seed in $taskseeds; do
116- cp "$seed" "$seed"_"$distro"_"$suite"_"$arch"
117- done
118- echo " ********************************************************************** " >> germinate.output
119- echo "" >> germinate.output
120- echo -n "."
121-
122- ## Generate apt-ftparchive 'extra' overrides for Task: fields
123- for seed in $taskseeds; do
124- if ! grep -iq '^Task-' "$seed.seedtext"; then
125- continue
126- fi
127- # If the seed contains Task-Name header, override the normal behavior
128- if grep -iq '^Task-Name:' "$seed.seedtext"; then
129- task=$(grep '^Task-Name:' "$seed.seedtext" | cut -d: -f2)
130- elif grep -iq '^Task-Per-Derivative:' "$seed.seedtext"; then
131- task="$distro-$seed"
132- else
133- # If a seed is not per-derivative, then we only honour it for Ubuntu,
134- # and its task name is archive-global.
135- if [ "$distro" = ubuntu ]; then
136- task="$seed"
137- else
138- continue
139- fi
140- fi
141- if grep -iq '^Task-Seeds:' "$seed.seedtext"; then
142- scanseeds="$( (grep '^Task-Seeds:' "$seed.seedtext" | cut -d: -f2 | xargs -n1; echo "$seed") | sort -u )"
143- else
144- scanseeds="$seed"
145- fi
146- for scanseed in $scanseeds; do
147- egrep -v -- \
148- "^(-|Package| )" "$scanseed"_"$distro"_"$suite"_"$arch" |
149- awk '{print $1}' |
150- sed -e "s,$,/$arch Task $task," |
151- sort -u >> "$MISCROOT/more-extra.override.$suite.main.new"
152- done
153- done
154-
155- # Generate apt-ftparchive 'extra' overrides for Build-Essential: fields
156- if [ -e build-essential ] && [ "$distro" = ubuntu ]; then
157- # Keep a copy, just for convenience
158- cp build-essential build-essential_"$distro"_"$suite"_"$arch"
159- egrep -v -- \
160- "^(-|Package| )" build-essential_"$distro"_"$suite"_"$arch" |
161- awk '{print $1}' |
162- sed -e "s,$,/$arch Build-Essential yes," |
163- sort -u >> "$MISCROOT/more-extra.override.$suite.main.new"
164- fi
165- done
166-done
167-echo " done."
168-
169-mv -f \
170- "$MISCROOT/more-extra.override.$suite.main.new" \
171- "$MISCROOT/more-extra.override.$suite.main"
172+$GENERATE -d ubuntu $FLAVOURS
173
174 # Now generate the Supported extra overrides for all supported distros.
175 SUITES=`$LAUNCHPADROOT/scripts/ftpmaster-tools/lp-query-distro.py supported`
176
177=== modified file 'database/schema/security.cfg'
178--- database/schema/security.cfg 2011-12-06 21:10:57 +0000
179+++ database/schema/security.cfg 2011-12-14 15:49:40 +0000
180@@ -2266,6 +2266,10 @@
181 type=user
182 groups=archivepublisher
183
184+[generate_extra_overrides]
185+type=user
186+groups=archivepublisher
187+
188 [process_death_row]
189 type=user
190 groups=archivepublisher
191
192=== modified file 'lib/lp/archivepublisher/config.py'
193--- lib/lp/archivepublisher/config.py 2011-08-29 16:43:10 +0000
194+++ lib/lp/archivepublisher/config.py 2011-12-14 15:49:40 +0000
195@@ -74,10 +74,12 @@
196 pubconf.overrideroot = pubconf.archiveroot + '-overrides'
197 pubconf.cacheroot = pubconf.archiveroot + '-cache'
198 pubconf.miscroot = pubconf.archiveroot + '-misc'
199+ pubconf.germinateroot = pubconf.archiveroot + '-germinate'
200 else:
201 pubconf.overrideroot = None
202 pubconf.cacheroot = None
203 pubconf.miscroot = None
204+ pubconf.germinateroot = None
205
206 pubconf.poolroot = os.path.join(pubconf.archiveroot, 'pool')
207 pubconf.distsroot = os.path.join(pubconf.archiveroot, 'dists')
208@@ -106,6 +108,7 @@
209 self.cacheroot,
210 self.overrideroot,
211 self.miscroot,
212+ self.germinateroot,
213 self.temproot,
214 ]
215
216
217=== added file 'lib/lp/archivepublisher/scripts/generate_extra_overrides.py'
218--- lib/lp/archivepublisher/scripts/generate_extra_overrides.py 1970-01-01 00:00:00 +0000
219+++ lib/lp/archivepublisher/scripts/generate_extra_overrides.py 2011-12-14 15:49:40 +0000
220@@ -0,0 +1,339 @@
221+# Copyright 2011 Canonical Ltd. This software is licensed under the
222+# GNU Affero General Public License version 3 (see the file LICENSE).
223+
224+"""Generate extra overrides using Germinate."""
225+
226+__metaclass__ = type
227+__all__ = [
228+ 'GenerateExtraOverrides',
229+ ]
230+
231+from functools import partial
232+import logging
233+from optparse import OptionValueError
234+import os
235+import re
236+
237+from germinate.germinator import Germinator
238+from germinate.archive import TagFile
239+from germinate.log import GerminateFormatter
240+from germinate.seeds import SeedStructure
241+
242+from zope.component import getUtility
243+
244+from canonical.launchpad.webapp.dbpolicy import (
245+ DatabaseBlockedPolicy,
246+ SlaveOnlyDatabasePolicy,
247+ )
248+from lp.archivepublisher.config import getPubConfig
249+from lp.registry.interfaces.distribution import IDistributionSet
250+from lp.registry.interfaces.series import SeriesStatus
251+from lp.services.scripts.base import (
252+ LaunchpadScript,
253+ LaunchpadScriptFailure,
254+ )
255+from lp.services.utils import file_exists
256+
257+
258+class AtomicFile:
259+ """Facilitate atomic writing of files."""
260+
261+ def __init__(self, filename):
262+ self.filename = filename
263+ self.fd = open("%s.new" % self.filename, "w")
264+
265+ def __enter__(self):
266+ return self.fd
267+
268+ def __exit__(self, exc_type, exc_value, exc_tb):
269+ self.fd.close()
270+ if exc_type is None:
271+ os.rename("%s.new" % self.filename, self.filename)
272+
273+
274+def find_operable_series(distribution):
275+ """Find a series we can operate on in this distribution.
276+
277+ We are allowed to modify DEVELOPMENT or FROZEN series, but should leave
278+ series with any other status alone.
279+ """
280+ series = distribution.currentseries
281+ if series.status in (SeriesStatus.DEVELOPMENT, SeriesStatus.FROZEN):
282+ return series
283+ else:
284+ return None
285+
286+
287+class GenerateExtraOverrides(LaunchpadScript):
288+ """Main class for scripts/ftpmaster-tools/generate-task-overrides.py."""
289+
290+ def __init__(self, *args, **kwargs):
291+ super(GenerateExtraOverrides, self).__init__(*args, **kwargs)
292+ self.germinate_logger = None
293+
294+ def add_my_options(self):
295+ """Add a 'distribution' context option."""
296+ self.parser.add_option(
297+ "-d", "--distribution", dest="distribution",
298+ help="Context distribution name.")
299+
300+ @property
301+ def name(self):
302+ """See `LaunchpadScript`."""
303+ # Include distribution name. Clearer to admins, but also
304+ # puts runs for different distributions under separate
305+ # locks so that they can run simultaneously.
306+ return "%s-%s" % (self._name, self.options.distribution)
307+
308+ def processOptions(self):
309+ """Handle command-line options."""
310+ if self.options.distribution is None:
311+ raise OptionValueError("Specify a distribution.")
312+
313+ self.distribution = getUtility(IDistributionSet).getByName(
314+ self.options.distribution)
315+ if self.distribution is None:
316+ raise OptionValueError(
317+ "Distribution '%s' not found." % self.options.distribution)
318+
319+ self.series = find_operable_series(self.distribution)
320+ if self.series is None:
321+ raise LaunchpadScriptFailure(
322+ "There is no DEVELOPMENT distroseries for %s." %
323+ self.options.distribution)
324+
325+ # Even if DistroSeries.component_names starts including partner, we
326+ # don't want it; this applies to the primary archive only.
327+ self.components = [component
328+ for component in self.series.component_names
329+ if component != "partner"]
330+
331+ def getConfig(self):
332+ """Set up a configuration object for this archive."""
333+ archive = self.distribution.main_archive
334+ if archive:
335+ return getPubConfig(archive)
336+ else:
337+ raise LaunchpadScriptFailure(
338+ "There is no PRIMARY archive for %s." %
339+ self.options.distribution)
340+
341+ def setUpDirs(self):
342+ """Create output directories if they did not already exist."""
343+ germinateroot = self.config.germinateroot
344+ if not file_exists(germinateroot):
345+ self.logger.debug("Creating germinate root %s.", germinateroot)
346+ os.makedirs(germinateroot)
347+ miscroot = self.config.miscroot
348+ if not file_exists(miscroot):
349+ self.logger.debug("Creating misc root %s.", miscroot)
350+ os.makedirs(miscroot)
351+
352+ def addLogHandler(self):
353+ """Send germinate's log output to a separate file."""
354+ if self.germinate_logger is not None:
355+ return
356+
357+ self.germinate_logger = logging.getLogger("germinate")
358+ self.germinate_logger.setLevel(logging.INFO)
359+ log_file = os.path.join(self.config.germinateroot, "germinate.output")
360+ handler = logging.FileHandler(log_file, mode="w")
361+ handler.setFormatter(GerminateFormatter())
362+ self.germinate_logger.addHandler(handler)
363+ self.germinate_logger.propagate = False
364+
365+ def setUp(self):
366+ """Process options, and set up internal state."""
367+ self.processOptions()
368+ self.config = self.getConfig()
369+ self.setUpDirs()
370+ self.addLogHandler()
371+
372+ def makeSeedStructures(self, series_name, flavours, seed_bases=None):
373+ structures = {}
374+ for flavour in flavours:
375+ structures[flavour] = SeedStructure(
376+ "%s.%s" % (flavour, series_name), seed_bases=seed_bases)
377+ return structures
378+
379+ def logGerminateProgress(self, *args):
380+ """Log a "progress" entry to the germinate log file.
381+
382+ Germinate logs quite a bit of detailed information. To make it
383+ easier to see the structure of its operation, GerminateFormatter
384+ allows tagging some log entries as "progress" entries, which are
385+ printed without a prefix.
386+ """
387+ self.germinate_logger.info(*args, extra={"progress": True})
388+
389+ def composeOutputPath(self, flavour, series_name, arch, base):
390+ return os.path.join(
391+ self.config.germinateroot,
392+ "%s_%s_%s_%s" % (base, flavour, series_name, arch))
393+
394+ def writeGerminateOutput(self, germinator, structure, flavour,
395+ series_name, arch):
396+ """Write dependency-expanded output files.
397+
398+ These files are a reduced subset of those written by the germinate
399+ command-line program.
400+ """
401+ path = partial(self.composeOutputPath, flavour, series_name, arch)
402+
403+ # The structure file makes it possible to figure out how the other
404+ # output files relate to each other.
405+ structure.write(path("structure"))
406+
407+ # "all" and "all.sources" list the full set of binary and source
408+ # packages respectively for a given flavour/suite/architecture
409+ # combination.
410+ germinator.write_all_list(structure, path("all"))
411+ germinator.write_all_source_list(structure, path("all.sources"))
412+
413+ # Write the dependency-expanded output for each seed. Several of
414+ # these are used by archive administration tools, and others are
415+ # useful for debugging, so it's best to just write them all.
416+ for seedname in structure.names:
417+ germinator.write_full_list(structure, path(seedname), seedname)
418+
419+ def parseTaskHeaders(self, seedtext):
420+ """Parse a seed for Task headers.
421+
422+ seedtext is a file-like object. Return a dictionary of Task headers,
423+ with keys canonicalised to lower-case.
424+ """
425+ task_headers = {}
426+ task_header_regex = re.compile(
427+ r"task-(.*?):(.*)", flags=re.IGNORECASE)
428+ for line in seedtext:
429+ match = task_header_regex.match(line)
430+ if match is not None:
431+ key, value = match.groups()
432+ task_headers[key.lower()] = value.strip()
433+ return task_headers
434+
435+ def getTaskName(self, task_headers, flavour, seedname, primary_flavour):
436+ """Work out the name of the Task to be generated from this seed.
437+
438+ If there is a Task-Name header, it wins; otherwise, seeds with a
439+ Task-Per-Derivative header are honoured for all flavours and put in
440+ an appropriate namespace, while other seeds are only honoured for
441+ the first flavour and have archive-global names.
442+ """
443+ if "name" in task_headers:
444+ return task_headers["name"]
445+ elif "per-derivative" in task_headers:
446+ return "%s-%s" % (flavour, seedname)
447+ elif primary_flavour:
448+ return seedname
449+ else:
450+ return None
451+
452+ def getTaskSeeds(self, task_headers, seedname):
453+ """Return the list of seeds used to generate a task from this seed.
454+
455+ The list of packages in this task comes from this seed plus any
456+ other seeds listed in a Task-Seeds header.
457+ """
458+ scan_seeds = set([seedname])
459+ if "seeds" in task_headers:
460+ scan_seeds.update(task_headers["seeds"].split())
461+ return sorted(scan_seeds)
462+
463+ def writeOverrides(self, override_file, germinator, structure, arch,
464+ seedname, key, value):
465+ packages = germinator.get_full(structure, seedname)
466+ for package in sorted(packages):
467+ print >>override_file, "%s/%s %s %s" % (
468+ package, arch, key, value)
469+
470+ def germinateArchFlavour(self, override_file, germinator, series_name,
471+ arch, flavour, structure, primary_flavour):
472+ """Germinate seeds on a single flavour for a single architecture."""
473+ # Expand dependencies.
474+ germinator.plant_seeds(structure)
475+ germinator.grow(structure)
476+ germinator.add_extras(structure)
477+
478+ self.writeGerminateOutput(germinator, structure, flavour, series_name,
479+ arch)
480+
481+ write_overrides = partial(
482+ self.writeOverrides, override_file, germinator, structure, arch)
483+
484+ # Generate apt-ftparchive "extra overrides" for Task fields.
485+ seednames = [name for name in structure.names if name != "extra"]
486+ for seedname in seednames:
487+ with structure[seedname] as seedtext:
488+ task_headers = self.parseTaskHeaders(seedtext)
489+ if task_headers:
490+ task = self.getTaskName(
491+ task_headers, flavour, seedname, primary_flavour)
492+ if task is not None:
493+ scan_seeds = self.getTaskSeeds(task_headers, seedname)
494+ for scan_seed in scan_seeds:
495+ write_overrides(scan_seed, "Task", task)
496+
497+ # Generate apt-ftparchive "extra overrides" for Build-Essential
498+ # fields.
499+ if "build-essential" in structure.names and primary_flavour:
500+ write_overrides("build-essential", "Build-Essential", "yes")
501+
502+ def germinateArch(self, override_file, series_name, arch, flavours,
503+ structures):
504+ """Germinate seeds on all flavours for a single architecture."""
505+ germinator = Germinator(arch)
506+
507+ # Read archive metadata.
508+ archive = TagFile(
509+ series_name, self.components, arch,
510+ "file://%s" % self.config.archiveroot, cleanup=True)
511+ germinator.parse_archive(archive)
512+
513+ for flavour in flavours:
514+ self.logger.info(
515+ "Germinating for %s/%s/%s", flavour, series_name, arch)
516+ # Add this to the germinate log as well so that that can be
517+ # debugged more easily. Log a separator line first.
518+ self.logGerminateProgress("")
519+ self.logGerminateProgress(
520+ "Germinating for %s/%s/%s", flavour, series_name, arch)
521+
522+ self.germinateArchFlavour(
523+ override_file, germinator, series_name, arch, flavour,
524+ structures[flavour], flavour == flavours[0])
525+
526+ def generateExtraOverrides(self, series_name, series_architectures,
527+ flavours, seed_bases=None):
528+ structures = self.makeSeedStructures(
529+ series_name, flavours, seed_bases=seed_bases)
530+
531+ override_path = os.path.join(
532+ self.config.miscroot,
533+ "more-extra.override.%s.main" % series_name)
534+ with AtomicFile(override_path) as override_file:
535+ for arch in series_architectures:
536+ self.germinateArch(
537+ override_file, series_name, arch, flavours, structures)
538+
539+ def process(self, seed_bases=None):
540+ """Do the bulk of the work."""
541+ self.setUp()
542+
543+ series_name = self.series.name
544+ series_architectures = sorted(
545+ [arch.architecturetag for arch in self.series.architectures])
546+
547+ # This takes a while. Ensure that we do it without keeping a
548+ # database transaction open.
549+ self.txn.commit()
550+ with DatabaseBlockedPolicy():
551+ self.generateExtraOverrides(
552+ series_name, series_architectures, self.args,
553+ seed_bases=seed_bases)
554+
555+ def main(self):
556+ """See `LaunchpadScript`."""
557+ # This code has no need to alter the database.
558+ with SlaveOnlyDatabasePolicy():
559+ self.process()
560
561=== modified file 'lib/lp/archivepublisher/tests/publisher-config.txt'
562--- lib/lp/archivepublisher/tests/publisher-config.txt 2010-10-17 13:35:20 +0000
563+++ lib/lp/archivepublisher/tests/publisher-config.txt 2011-12-14 15:49:40 +0000
564@@ -14,6 +14,7 @@
565 ... 'overrideroot',
566 ... 'cacheroot',
567 ... 'miscroot',
568+ ... 'germinateroot',
569 ... 'temproot',
570 ... ]
571
572@@ -38,6 +39,7 @@
573 overrideroot: /var/tmp/archive/ubuntutest-overrides
574 cacheroot: /var/tmp/archive/ubuntutest-cache
575 miscroot: /var/tmp/archive/ubuntutest-misc
576+ germinateroot: /var/tmp/archive/ubuntutest-germinate
577 temproot: /var/tmp/archive/ubuntutest-temp
578
579
580@@ -80,6 +82,7 @@
581 overrideroot: None
582 cacheroot: None
583 miscroot: None
584+ germinateroot: None
585 temproot: /var/tmp/archive/ubuntutest-temp
586
587 There is a separate location for private PPAs that is used if the
588@@ -108,6 +111,7 @@
589 overrideroot: None
590 cacheroot: None
591 miscroot: None
592+ germinateroot: None
593 temproot: /var/tmp/archive/ubuntutest-temp
594
595
596@@ -131,6 +135,7 @@
597 overrideroot: None
598 cacheroot: None
599 miscroot: None
600+ germinateroot: None
601 temproot: /var/tmp/archive/ubuntutest-temp
602
603
604@@ -155,6 +160,7 @@
605 overrideroot: None
606 cacheroot: None
607 miscroot: None
608+ germinateroot: None
609 temproot: /var/tmp/archive/ubuntutest-temp
610
611
612@@ -177,4 +183,5 @@
613 overrideroot: /var/tmp/archive/ubuntutest-rebuildtest99/ubuntutest-overrides
614 cacheroot: /var/tmp/archive/ubuntutest-rebuildtest99/ubuntutest-cache
615 miscroot: /var/tmp/archive/ubuntutest-rebuildtest99/ubuntutest-misc
616+ germinateroot: /var/tmp/archive/ubuntutest-rebuildtest99/ubuntutest-germinate
617 temproot: /var/tmp/archive/ubuntutest-rebuildtest99/ubuntutest-temp
618
619=== added file 'lib/lp/archivepublisher/tests/test_generate_extra_overrides.py'
620--- lib/lp/archivepublisher/tests/test_generate_extra_overrides.py 1970-01-01 00:00:00 +0000
621+++ lib/lp/archivepublisher/tests/test_generate_extra_overrides.py 2011-12-14 15:49:40 +0000
622@@ -0,0 +1,567 @@
623+# Copyright 2011 Canonical Ltd. This software is licensed under the
624+# GNU Affero General Public License version 3 (see the file LICENSE).
625+
626+"""Test for the `generate-extra-overrides` script."""
627+
628+__metaclass__ = type
629+
630+import logging
631+from optparse import OptionValueError
632+import os
633+import tempfile
634+
635+from germinate import (
636+ archive,
637+ germinator,
638+ seeds,
639+ )
640+
641+import transaction
642+
643+from canonical.testing.layers import (
644+ LaunchpadZopelessLayer,
645+ ZopelessDatabaseLayer,
646+ )
647+from lp.archivepublisher.scripts.generate_extra_overrides import (
648+ AtomicFile,
649+ GenerateExtraOverrides,
650+ )
651+from lp.archivepublisher.utils import RepositoryIndexFile
652+from lp.registry.interfaces.pocket import PackagePublishingPocket
653+from lp.registry.interfaces.series import SeriesStatus
654+from lp.services.log.logger import DevNullLogger
655+from lp.services.osutils import (
656+ ensure_directory_exists,
657+ open_for_writing,
658+ )
659+from lp.services.scripts.base import LaunchpadScriptFailure
660+from lp.services.utils import file_exists
661+from lp.soyuz.enums import PackagePublishingStatus
662+from lp.testing import TestCaseWithFactory
663+from lp.testing.faketransaction import FakeTransaction
664+
665+
666+def file_contents(path):
667+ """Return the contents of the file at path."""
668+ with open(path) as handle:
669+ return handle.read()
670+
671+
672+class TestAtomicFile(TestCaseWithFactory):
673+ """Tests for the AtomicFile helper class."""
674+
675+ layer = ZopelessDatabaseLayer
676+
677+ def test_atomic_file_creates_file(self):
678+ # AtomicFile creates the named file with the requested contents.
679+ self.useTempDir()
680+ filename = self.factory.getUniqueString()
681+ text = self.factory.getUniqueString()
682+ with AtomicFile(filename) as test:
683+ test.write(text)
684+ self.assertEqual(text, file_contents(filename))
685+
686+ def test_atomic_file_removes_dot_new(self):
687+ # AtomicFile does not leave .new files lying around.
688+ self.useTempDir()
689+ filename = self.factory.getUniqueString()
690+ with AtomicFile(filename):
691+ pass
692+ self.assertFalse(file_exists("%s.new" % filename))
693+
694+
695+class TestGenerateExtraOverrides(TestCaseWithFactory):
696+ """Tests for the actual `GenerateExtraOverrides` script."""
697+
698+ layer = LaunchpadZopelessLayer
699+
700+ def setUp(self):
701+ super(TestGenerateExtraOverrides, self).setUp()
702+ self.seeddir = self.makeTemporaryDirectory()
703+ # XXX cjwatson 2011-12-06 bug=694140: Make sure germinate doesn't
704+ # lose its loggers between tests, due to Launchpad's messing with
705+ # global log state.
706+ archive._logger = logging.getLogger("germinate.archive")
707+ germinator._logger = logging.getLogger("germinate.germinator")
708+ seeds._logger = logging.getLogger("germinate.seeds")
709+
710+ def assertFilesEqual(self, expected_path, observed_path):
711+ self.assertEqual(
712+ file_contents(expected_path), file_contents(observed_path))
713+
714+ def makeDistro(self):
715+ """Create a distribution for testing.
716+
717+ The distribution will have a root directory set up, which will
718+ be cleaned up after the test. It will have an attached archive.
719+ """
720+ return self.factory.makeDistribution(
721+ publish_root_dir=unicode(self.makeTemporaryDirectory()))
722+
723+ def makeScript(self, distribution, run_setup=True, extra_args=None):
724+ """Create a script for testing."""
725+ test_args = []
726+ if distribution is not None:
727+ test_args.extend(["-d", distribution.name])
728+ if extra_args is not None:
729+ test_args.extend(extra_args)
730+ script = GenerateExtraOverrides(test_args=test_args)
731+ script.logger = DevNullLogger()
732+ script.txn = FakeTransaction()
733+ if distribution is not None and run_setup:
734+ script.setUp()
735+ else:
736+ script.distribution = distribution
737+ return script
738+
739+ def makePackage(self, component, dases, **kwargs):
740+ """Create a published source and binary package for testing."""
741+ package = self.factory.makeDistributionSourcePackage(
742+ distribution=dases[0].distroseries.distribution)
743+ spph = self.factory.makeSourcePackagePublishingHistory(
744+ distroseries=dases[0].distroseries,
745+ pocket=PackagePublishingPocket.RELEASE,
746+ status=PackagePublishingStatus.PUBLISHED,
747+ sourcepackagename=package.name, component=component)
748+ for das in dases:
749+ build = self.factory.makeBinaryPackageBuild(
750+ source_package_release=spph.sourcepackagerelease,
751+ distroarchseries=das, processor=das.default_processor)
752+ bpr = self.factory.makeBinaryPackageRelease(
753+ binarypackagename=package.name, build=build,
754+ component=component, **kwargs)
755+ lfa = self.factory.makeLibraryFileAlias(
756+ filename="%s.deb" % package.name)
757+ transaction.commit()
758+ bpr.addFile(lfa)
759+ self.factory.makeBinaryPackagePublishingHistory(
760+ binarypackagerelease=bpr, distroarchseries=das,
761+ pocket=PackagePublishingPocket.RELEASE,
762+ status=PackagePublishingStatus.PUBLISHED)
763+ return package
764+
765+ def makeIndexFiles(self, script, distroseries):
766+ """Create a limited subset of index files for testing."""
767+ ensure_directory_exists(script.config.temproot)
768+
769+ for component in distroseries.components:
770+ index_root = os.path.join(
771+ script.config.distsroot, distroseries.name, component.name)
772+
773+ source_index_root = os.path.join(index_root, "source")
774+ source_index = RepositoryIndexFile(
775+ source_index_root, script.config.temproot, "Sources")
776+ for spp in distroseries.getSourcePackagePublishing(
777+ PackagePublishingStatus.PUBLISHED,
778+ PackagePublishingPocket.RELEASE, component=component):
779+ stanza = spp.getIndexStanza().encode("utf-8") + "\n\n"
780+ source_index.write(stanza)
781+ source_index.close()
782+
783+ for arch in distroseries.architectures:
784+ package_index_root = os.path.join(
785+ index_root, "binary-%s" % arch.architecturetag)
786+ package_index = RepositoryIndexFile(
787+ package_index_root, script.config.temproot, "Packages")
788+ for bpp in distroseries.getBinaryPackagePublishing(
789+ archtag=arch.architecturetag,
790+ pocket=PackagePublishingPocket.RELEASE,
791+ component=component):
792+ stanza = bpp.getIndexStanza().encode("utf-8") + "\n\n"
793+ package_index.write(stanza)
794+ package_index.close()
795+
796+ def composeSeedPath(self, flavour, series_name, seed_name):
797+ return os.path.join(
798+ self.seeddir, "%s.%s" % (flavour, series_name), seed_name)
799+
800+ def makeSeedStructure(self, flavour, series_name, seed_names,
801+ seed_inherit=None):
802+ """Create a simple seed structure file."""
803+ if seed_inherit is None:
804+ seed_inherit = {}
805+
806+ structure_path = self.composeSeedPath(
807+ flavour, series_name, "STRUCTURE")
808+ with open_for_writing(structure_path, "w") as structure:
809+ for seed_name in seed_names:
810+ inherit = seed_inherit.get(seed_name, [])
811+ line = "%s: %s" % (seed_name, " ".join(inherit))
812+ print >>structure, line.strip()
813+
814+ def makeSeed(self, flavour, series_name, seed_name, entries,
815+ headers=None):
816+ """Create a simple seed file."""
817+ seed_path = self.composeSeedPath(flavour, series_name, seed_name)
818+ with open_for_writing(seed_path, "w") as seed:
819+ if headers is not None:
820+ for header in headers:
821+ print >>seed, header
822+ print >>seed
823+ for entry in entries:
824+ print >>seed, " * %s" % entry
825+
826+ def getTaskNameFromSeed(self, script, flavour, series_name, seed,
827+ primary_flavour):
828+ """Use script to parse a seed and return its task name."""
829+ seed_path = self.composeSeedPath(flavour, series_name, seed)
830+ with open(seed_path) as seed_text:
831+ task_headers = script.parseTaskHeaders(seed_text)
832+ return script.getTaskName(
833+ task_headers, flavour, seed, primary_flavour)
834+
835+ def getTaskSeedsFromSeed(self, script, flavour, series_name, seed):
836+ """Use script to parse a seed and return its task seed list."""
837+ seed_path = self.composeSeedPath(flavour, series_name, seed)
838+ with open(seed_path) as seed_text:
839+ task_headers = script.parseTaskHeaders(seed_text)
840+ return script.getTaskSeeds(task_headers, seed)
841+
842+ def test_name_is_consistent(self):
843+ # Script instances for the same distro get the same name.
844+ distro = self.factory.makeDistribution()
845+ self.assertEqual(
846+ GenerateExtraOverrides(test_args=["-d", distro.name]).name,
847+ GenerateExtraOverrides(test_args=["-d", distro.name]).name)
848+
849+ def test_name_is_unique_for_each_distro(self):
850+ # Script instances for different distros get different names.
851+ self.assertNotEqual(
852+ GenerateExtraOverrides(
853+ test_args=["-d", self.factory.makeDistribution().name]).name,
854+ GenerateExtraOverrides(
855+ test_args=["-d", self.factory.makeDistribution().name]).name)
856+
857+ def test_requires_distro(self):
858+ # The --distribution or -d argument is mandatory.
859+ script = self.makeScript(None)
860+ self.assertRaises(OptionValueError, script.processOptions)
861+
862+ def test_requires_real_distro(self):
863+ # An incorrect distribution name is flagged as an invalid option
864+ # value.
865+ script = self.makeScript(
866+ None, extra_args=["-d", self.factory.getUniqueString()])
867+ self.assertRaises(OptionValueError, script.processOptions)
868+
869+ def test_looks_up_distro(self):
870+ # The script looks up and keeps the distribution named on the
871+ # command line.
872+ distro = self.makeDistro()
873+ self.factory.makeDistroSeries(distro)
874+ script = self.makeScript(distro)
875+ self.assertEqual(distro, script.distribution)
876+
877+ def test_prefers_development_distro_series(self):
878+ # The script prefers a DEVELOPMENT series for the named
879+ # distribution over CURRENT and SUPPORTED series.
880+ distro = self.makeDistro()
881+ self.factory.makeDistroSeries(distro, status=SeriesStatus.SUPPORTED)
882+ self.factory.makeDistroSeries(distro, status=SeriesStatus.CURRENT)
883+ development_distroseries = self.factory.makeDistroSeries(
884+ distro, status=SeriesStatus.DEVELOPMENT)
885+ script = self.makeScript(distro)
886+ self.assertEqual(development_distroseries, script.series)
887+
888+ def test_permits_frozen_distro_series(self):
889+ # If there is no DEVELOPMENT series, a FROZEN one will do.
890+ distro = self.makeDistro()
891+ self.factory.makeDistroSeries(distro, status=SeriesStatus.SUPPORTED)
892+ self.factory.makeDistroSeries(distro, status=SeriesStatus.CURRENT)
893+ frozen_distroseries = self.factory.makeDistroSeries(
894+ distro, status=SeriesStatus.FROZEN)
895+ script = self.makeScript(distro)
896+ self.assertEqual(frozen_distroseries, script.series)
897+
898+ def test_requires_development_frozen_distro_series(self):
899+ # If there is no DEVELOPMENT or FROZEN series, the script fails.
900+ distro = self.makeDistro()
901+ self.factory.makeDistroSeries(distro, status=SeriesStatus.SUPPORTED)
902+ self.factory.makeDistroSeries(distro, status=SeriesStatus.CURRENT)
903+ script = self.makeScript(distro, run_setup=False)
904+ self.assertRaises(LaunchpadScriptFailure, script.processOptions)
905+
906+ def test_components_exclude_partner(self):
907+ # If a 'partner' component exists, it is excluded.
908+ distro = self.makeDistro()
909+ distroseries = self.factory.makeDistroSeries(distro)
910+ self.factory.makeComponentSelection(
911+ distroseries=distroseries, component="main")
912+ self.factory.makeComponentSelection(
913+ distroseries=distroseries, component="partner")
914+ script = self.makeScript(distro)
915+ self.assertEqual(["main"], script.components)
916+
917+ def test_compose_output_path_in_germinateroot(self):
918+ # Output files are written to the correct locations under
919+ # germinateroot.
920+ distro = self.makeDistro()
921+ distroseries = self.factory.makeDistroSeries(distro)
922+ script = self.makeScript(distro)
923+ flavour = self.factory.getUniqueString()
924+ arch = self.factory.getUniqueString()
925+ base = self.factory.getUniqueString()
926+ output = script.composeOutputPath(
927+ flavour, distroseries.name, arch, base)
928+ self.assertEqual(
929+ "%s/%s_%s_%s_%s" % (
930+ script.config.germinateroot, base, flavour, distroseries.name,
931+ arch),
932+ output)
933+
934+ def fetchGerminatedOverrides(self, script, series_name, arch, flavours):
935+ """Helper to call script.germinateArch and return overrides."""
936+ structures = script.makeSeedStructures(
937+ series_name, flavours, seed_bases=["file://%s" % self.seeddir])
938+
939+ override_fd, override_path = tempfile.mkstemp()
940+ with os.fdopen(override_fd, "w") as override_file:
941+ script.germinateArch(
942+ override_file, series_name, arch, flavours, structures)
943+ return file_contents(override_path).splitlines()
944+
945+ def test_germinate_output(self):
946+ # A single call to germinateArch produces output for all flavours on
947+ # one architecture.
948+ distro = self.makeDistro()
949+ distroseries = self.factory.makeDistroSeries(distribution=distro)
950+ series_name = distroseries.name
951+ component = self.factory.makeComponent()
952+ self.factory.makeComponentSelection(
953+ distroseries=distroseries, component=component)
954+ das = self.factory.makeDistroArchSeries(distroseries=distroseries)
955+ arch = das.architecturetag
956+ one = self.makePackage(component, [das])
957+ two = self.makePackage(component, [das])
958+ script = self.makeScript(distro)
959+ self.makeIndexFiles(script, distroseries)
960+
961+ flavour_one = self.factory.getUniqueString()
962+ flavour_two = self.factory.getUniqueString()
963+ seed = self.factory.getUniqueString()
964+ self.makeSeedStructure(flavour_one, series_name, [seed])
965+ self.makeSeed(flavour_one, series_name, seed, [one.name])
966+ self.makeSeedStructure(flavour_two, series_name, [seed])
967+ self.makeSeed(flavour_two, series_name, seed, [two.name])
968+
969+ overrides = self.fetchGerminatedOverrides(
970+ script, series_name, arch, [flavour_one, flavour_two])
971+ self.assertEqual([], overrides)
972+
973+ seed_dir_one = os.path.join(
974+ self.seeddir, "%s.%s" % (flavour_one, series_name))
975+ self.assertFilesEqual(
976+ os.path.join(seed_dir_one, "STRUCTURE"),
977+ script.composeOutputPath(
978+ flavour_one, series_name, arch, "structure"))
979+ self.assertTrue(file_exists(script.composeOutputPath(
980+ flavour_one, series_name, arch, "all")))
981+ self.assertTrue(file_exists(script.composeOutputPath(
982+ flavour_one, series_name, arch, "all.sources")))
983+ self.assertTrue(file_exists(script.composeOutputPath(
984+ flavour_one, series_name, arch, seed)))
985+
986+ seed_dir_two = os.path.join(
987+ self.seeddir, "%s.%s" % (flavour_two, series_name))
988+ self.assertFilesEqual(
989+ os.path.join(seed_dir_two, "STRUCTURE"),
990+ script.composeOutputPath(
991+ flavour_two, series_name, arch, "structure"))
992+ self.assertTrue(file_exists(script.composeOutputPath(
993+ flavour_two, series_name, arch, "all")))
994+ self.assertTrue(file_exists(script.composeOutputPath(
995+ flavour_two, series_name, arch, "all.sources")))
996+ self.assertTrue(file_exists(script.composeOutputPath(
997+ flavour_two, series_name, arch, seed)))
998+
999+ def test_germinate_output_task(self):
1000+ # germinateArch produces Task extra overrides.
1001+ distro = self.makeDistro()
1002+ distroseries = self.factory.makeDistroSeries(distribution=distro)
1003+ series_name = distroseries.name
1004+ component = self.factory.makeComponent()
1005+ self.factory.makeComponentSelection(
1006+ distroseries=distroseries, component=component)
1007+ das = self.factory.makeDistroArchSeries(distroseries=distroseries)
1008+ arch = das.architecturetag
1009+ one = self.makePackage(component, [das])
1010+ two = self.makePackage(component, [das], depends=one.name)
1011+ three = self.makePackage(component, [das])
1012+ self.makePackage(component, [das])
1013+ script = self.makeScript(distro)
1014+ self.makeIndexFiles(script, distroseries)
1015+
1016+ flavour = self.factory.getUniqueString()
1017+ seed_one = self.factory.getUniqueString()
1018+ seed_two = self.factory.getUniqueString()
1019+ self.makeSeedStructure(flavour, series_name, [seed_one, seed_two])
1020+ self.makeSeed(
1021+ flavour, series_name, seed_one, [two.name],
1022+ headers=["Task-Description: one"])
1023+ self.makeSeed(
1024+ flavour, series_name, seed_two, [three.name],
1025+ headers=["Task-Description: two"])
1026+
1027+ overrides = self.fetchGerminatedOverrides(
1028+ script, series_name, arch, [flavour])
1029+ expected_overrides = [
1030+ "%s/%s Task %s" % (one.name, arch, seed_one),
1031+ "%s/%s Task %s" % (two.name, arch, seed_one),
1032+ "%s/%s Task %s" % (three.name, arch, seed_two),
1033+ ]
1034+ self.assertContentEqual(expected_overrides, overrides)
1035+
1036+ def test_task_name(self):
1037+ # The Task-Name field is honoured.
1038+ series_name = self.factory.getUniqueString()
1039+ package = self.factory.getUniqueString()
1040+ script = self.makeScript(None)
1041+
1042+ flavour = self.factory.getUniqueString()
1043+ seed = self.factory.getUniqueString()
1044+ task = self.factory.getUniqueString()
1045+ self.makeSeed(
1046+ flavour, series_name, seed, [package],
1047+ headers=["Task-Name: %s" % task])
1048+
1049+ observed_task = self.getTaskNameFromSeed(
1050+ script, flavour, series_name, seed, True)
1051+ self.assertEqual(task, observed_task)
1052+
1053+ def test_task_per_derivative(self):
1054+ # The Task-Per-Derivative field is honoured.
1055+ series_name = self.factory.getUniqueString()
1056+ package = self.factory.getUniqueString()
1057+ script = self.makeScript(None)
1058+
1059+ flavour_one = self.factory.getUniqueString()
1060+ flavour_two = self.factory.getUniqueString()
1061+ seed_one = self.factory.getUniqueString()
1062+ seed_two = self.factory.getUniqueString()
1063+ self.makeSeed(
1064+ flavour_one, series_name, seed_one, [package],
1065+ headers=["Task-Description: one"])
1066+ self.makeSeed(
1067+ flavour_one, series_name, seed_two, [package],
1068+ headers=["Task-Per-Derivative: 1"])
1069+ self.makeSeed(
1070+ flavour_two, series_name, seed_one, [package],
1071+ headers=["Task-Description: one"])
1072+ self.makeSeed(
1073+ flavour_two, series_name, seed_two, [package],
1074+ headers=["Task-Per-Derivative: 1"])
1075+
1076+ observed_task_one_one = self.getTaskNameFromSeed(
1077+ script, flavour_one, series_name, seed_one, True)
1078+ observed_task_one_two = self.getTaskNameFromSeed(
1079+ script, flavour_one, series_name, seed_two, True)
1080+ observed_task_two_one = self.getTaskNameFromSeed(
1081+ script, flavour_two, series_name, seed_one, False)
1082+ observed_task_two_two = self.getTaskNameFromSeed(
1083+ script, flavour_two, series_name, seed_two, False)
1084+
1085+ # seed_one is not per-derivative, so it is honoured only for
1086+ # flavour_one and has a global name.
1087+ self.assertEqual(seed_one, observed_task_one_one)
1088+ self.assertIsNone(observed_task_two_one)
1089+
1090+ # seed_two is per-derivative, so it is honoured for both flavours
1091+ # and has the flavour name prefixed.
1092+ self.assertEqual(
1093+ "%s-%s" % (flavour_one, seed_two), observed_task_one_two)
1094+ self.assertEqual(
1095+ "%s-%s" % (flavour_two, seed_two), observed_task_two_two)
1096+
1097+ def test_task_seeds(self):
1098+ # The Task-Seeds field is honoured.
1099+ series_name = self.factory.getUniqueString()
1100+ one = self.getUniqueString()
1101+ two = self.getUniqueString()
1102+ script = self.makeScript(None)
1103+
1104+ flavour = self.factory.getUniqueString()
1105+ seed_one = self.factory.getUniqueString()
1106+ seed_two = self.factory.getUniqueString()
1107+ self.makeSeed(flavour, series_name, seed_one, [one])
1108+ self.makeSeed(
1109+ flavour, series_name, seed_two, [two],
1110+ headers=["Task-Seeds: %s" % seed_one])
1111+
1112+ task_seeds = self.getTaskSeedsFromSeed(
1113+ script, flavour, series_name, seed_two)
1114+ self.assertContentEqual([seed_one, seed_two], task_seeds)
1115+
1116+ def test_germinate_output_build_essential(self):
1117+ # germinateArch produces Build-Essential extra overrides.
1118+ distro = self.makeDistro()
1119+ distroseries = self.factory.makeDistroSeries(distribution=distro)
1120+ series_name = distroseries.name
1121+ component = self.factory.makeComponent()
1122+ self.factory.makeComponentSelection(
1123+ distroseries=distroseries, component=component)
1124+ das = self.factory.makeDistroArchSeries(distroseries=distroseries)
1125+ arch = das.architecturetag
1126+ package = self.makePackage(component, [das])
1127+ script = self.makeScript(distro)
1128+ self.makeIndexFiles(script, distroseries)
1129+
1130+ flavour = self.factory.getUniqueString()
1131+ seed = "build-essential"
1132+ self.makeSeedStructure(flavour, series_name, [seed])
1133+ self.makeSeed(flavour, series_name, seed, [package.name])
1134+
1135+ overrides = self.fetchGerminatedOverrides(
1136+ script, series_name, arch, [flavour])
1137+ self.assertContentEqual(
1138+ ["%s/%s Build-Essential yes" % (package.name, arch)], overrides)
1139+
1140+ def test_main(self):
1141+ # If run end-to-end, the script generates override files containing
1142+ # output for all architectures, and sends germinate's log output to
1143+ # a file.
1144+ distro = self.makeDistro()
1145+ distroseries = self.factory.makeDistroSeries(distribution=distro)
1146+ series_name = distroseries.name
1147+ component = self.factory.makeComponent()
1148+ self.factory.makeComponentSelection(
1149+ distroseries=distroseries, component=component)
1150+ das_one = self.factory.makeDistroArchSeries(distroseries=distroseries)
1151+ arch_one = das_one.architecturetag
1152+ das_two = self.factory.makeDistroArchSeries(distroseries=distroseries)
1153+ arch_two = das_two.architecturetag
1154+ package = self.makePackage(component, [das_one, das_two])
1155+ flavour = self.factory.getUniqueString()
1156+ script = self.makeScript(distro, extra_args=[flavour])
1157+ self.makeIndexFiles(script, distroseries)
1158+
1159+ seed = self.factory.getUniqueString()
1160+ self.makeSeedStructure(flavour, series_name, [seed])
1161+ self.makeSeed(
1162+ flavour, series_name, seed, [package.name],
1163+ headers=["Task-Description: task"])
1164+
1165+ script.process(seed_bases=["file://%s" % self.seeddir])
1166+ override_path = os.path.join(
1167+ script.config.miscroot,
1168+ "more-extra.override.%s.main" % series_name)
1169+ expected_overrides = [
1170+ "%s/%s Task %s" % (package.name, arch_one, seed),
1171+ "%s/%s Task %s" % (package.name, arch_two, seed),
1172+ ]
1173+ self.assertContentEqual(
1174+ expected_overrides, file_contents(override_path).splitlines())
1175+
1176+ log_file = os.path.join(
1177+ script.config.germinateroot, "germinate.output")
1178+ self.assertIn("Downloading file://", file_contents(log_file))
1179+
1180+ def test_run_script(self):
1181+ # The script will run stand-alone.
1182+ from canonical.launchpad.scripts.tests import run_script
1183+ distro = self.makeDistro()
1184+ self.factory.makeDistroSeries(distro)
1185+ transaction.commit()
1186+ retval, out, err = run_script(
1187+ "cronscripts/generate-extra-overrides.py",
1188+ ["-d", distro.name, "-q"])
1189+ self.assertEqual(0, retval)
1190
1191=== removed file 'lib/lp/soyuz/scripts/tests/germinate-test-data/mock-bin/germinate'
1192--- lib/lp/soyuz/scripts/tests/germinate-test-data/mock-bin/germinate 2010-10-20 13:23:05 +0000
1193+++ lib/lp/soyuz/scripts/tests/germinate-test-data/mock-bin/germinate 1970-01-01 00:00:00 +0000
1194@@ -1,5 +0,0 @@
1195-#!/bin/sh
1196-#
1197-# This is a mock germinate script that just produces enough (empty)
1198-# files so that the cron.germinate shell script can run.
1199-touch structure all all.sources minimal standard
1200
1201=== added file 'lib/lp/soyuz/scripts/tests/germinate-test-data/mock-lp-root/cronscripts/generate-extra-overrides.py'
1202--- lib/lp/soyuz/scripts/tests/germinate-test-data/mock-lp-root/cronscripts/generate-extra-overrides.py 1970-01-01 00:00:00 +0000
1203+++ lib/lp/soyuz/scripts/tests/germinate-test-data/mock-lp-root/cronscripts/generate-extra-overrides.py 2011-12-14 15:49:40 +0000
1204@@ -0,0 +1,5 @@
1205+#! /usr/bin/python
1206+#
1207+# We don't need to run generate-extra-overrides.py when testing
1208+# maintenance-check.py. We could, but it would slow down the tests and its
1209+# output is not interesting here.
1210
1211=== modified file 'lib/lp/soyuz/scripts/tests/test_cron_germinate.py'
1212--- lib/lp/soyuz/scripts/tests/test_cron_germinate.py 2011-01-19 17:49:10 +0000
1213+++ lib/lp/soyuz/scripts/tests/test_cron_germinate.py 2011-12-14 15:49:40 +0000
1214@@ -148,10 +148,8 @@
1215 os.path.join(archive_dir, "ubuntu"))
1216 fake_environ["TEST_LAUNCHPADROOT"] = os.path.abspath(
1217 os.path.join(basepath, "germinate-test-data/mock-lp-root"))
1218- # Set the PATH in the fake environment so that our mock germinate
1219- # is used. We could use the real germinate as well, but that will
1220- # slow down the tests a lot and its also not interessting for this
1221- # test as we do not use any of the germinate information.
1222+ # Set the PATH in the fake environment so that our mock lockfile is
1223+ # used.
1224 fake_environ["PATH"] = "%s:%s" % (
1225 os.path.abspath(os.path.join(
1226 basepath, "germinate-test-data/mock-bin")),