Merge lp:~cjwatson/launchpad/publish-distro-many-ppas into lp:launchpad

Proposed by Colin Watson
Status: Merged
Merged at revision: 17966
Proposed branch: lp:~cjwatson/launchpad/publish-distro-many-ppas
Merge into: lp:launchpad
Diff against target: 23 lines (+3/-2)
1 file modified
lib/lp/archivepublisher/scripts/ (+3/-2)
To merge this branch: bzr merge lp:~cjwatson/launchpad/publish-distro-many-ppas
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email:

Commit message

Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

Description of the change

Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

I'd noticed that commits were very slow on each individual archive when operating on all PPAs. This turns out to be because of accidentally quadratic behaviour: getTargetArchives materialised all archives into memory up-front, which meant that they were all alive as far as the Storm store was concerned, which meant that the pre-commit invalidate call had to walk over them all.

Turning this into an iterator makes things behave much better: commit still takes a variable amount of time depending on how many archives publish-distro inspected and skipped before it had to do any real work, but the total time spent is linear in len(PublishDistro.getPPAs()) rather than linear in len(PublishDistro.getPPAs()) * len(archives that need publishing work). It's also less annoying when running publish-distro by hand because it starts producing output more quickly.

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/archivepublisher/scripts/'
2--- lib/lp/archivepublisher/scripts/ 2016-03-21 15:40:55 +0000
3+++ lib/lp/archivepublisher/scripts/ 2016-03-23 15:20:08 +0000
4@@ -7,6 +7,7 @@
5 'PublishDistro',
6 ]
8+from itertools import ifilter
9 from optparse import OptionValueError
11 from zope.component import getUtility
12@@ -251,9 +252,9 @@
13 if self.options.partner:
14 return [distribution.getArchiveByComponent('partner')]
15 elif self.options.ppa:
16- return filter(is_ppa_public, self.getPPAs(distribution))
17+ return ifilter(is_ppa_public, self.getPPAs(distribution))
18 elif self.options.private_ppa:
19- return filter(is_ppa_private, self.getPPAs(distribution))
20+ return ifilter(is_ppa_private, self.getPPAs(distribution))
21 elif self.options.copy_archive:
22 return self.getCopyArchives(distribution)
23 else: