Merge lp:~cjwatson/launchpad/publish-distro-many-ppas into lp:launchpad

Proposed by Colin Watson
Status: Merged
Merged at revision: 17966
Proposed branch: lp:~cjwatson/launchpad/publish-distro-many-ppas
Merge into: lp:launchpad
Diff against target: 23 lines (+3/-2)
1 file modified
lib/lp/archivepublisher/scripts/ (+3/-2)
To merge this branch: bzr merge lp:~cjwatson/launchpad/publish-distro-many-ppas
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email:

Commit message

Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

Description of the change

Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

I'd noticed that commits were very slow on each individual archive when operating on all PPAs. This turns out to be because of accidentally quadratic behaviour: getTargetArchives materialised all archives into memory up-front, which meant that they were all alive as far as the Storm store was concerned, which meant that the pre-commit invalidate call had to walk over them all.

Turning this into an iterator makes things behave much better: commit still takes a variable amount of time depending on how many archives publish-distro inspected and skipped before it had to do any real work, but the total time spent is linear in len(PublishDistro.getPPAs()) rather than linear in len(PublishDistro.getPPAs()) * len(archives that need publishing work). It's also less annoying when running publish-distro by hand because it starts producing output more quickly.

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archivepublisher/scripts/'
--- lib/lp/archivepublisher/scripts/ 2016-03-21 15:40:55 +0000
+++ lib/lp/archivepublisher/scripts/ 2016-03-23 15:20:08 +0000
@@ -7,6 +7,7 @@
7 'PublishDistro',7 'PublishDistro',
8 ]8 ]
10from itertools import ifilter
10from optparse import OptionValueError11from optparse import OptionValueError
12from zope.component import getUtility13from zope.component import getUtility
@@ -251,9 +252,9 @@
251 if self.options.partner:252 if self.options.partner:
252 return [distribution.getArchiveByComponent('partner')]253 return [distribution.getArchiveByComponent('partner')]
253 elif self.options.ppa:254 elif self.options.ppa:
254 return filter(is_ppa_public, self.getPPAs(distribution))255 return ifilter(is_ppa_public, self.getPPAs(distribution))
255 elif self.options.private_ppa:256 elif self.options.private_ppa:
256 return filter(is_ppa_private, self.getPPAs(distribution))257 return ifilter(is_ppa_private, self.getPPAs(distribution))
257 elif self.options.copy_archive:258 elif self.options.copy_archive:
258 return self.getCopyArchives(distribution)259 return self.getCopyArchives(distribution)
259 else:260 else: