Merge lp:~jtv/launchpad/transitional-published into lp:launchpad

Proposed by Jeroen T. Vermeulen
Status: Superseded
Proposed branch: lp:~jtv/launchpad/transitional-published
Merge into: lp:launchpad
Diff against target: 1497 lines (+977/-161)
9 files modified
lib/lp/archivepublisher/domination.py (+233/-60)
lib/lp/archivepublisher/tests/test_dominator.py (+415/-4)
lib/lp/soyuz/doc/gina.txt (+24/-24)
lib/lp/soyuz/interfaces/publishing.py (+0/-3)
lib/lp/soyuz/model/publishing.py (+6/-10)
lib/lp/soyuz/scripts/gina/dominate.py (+82/-0)
lib/lp/soyuz/scripts/gina/handlers.py (+37/-51)
lib/lp/soyuz/scripts/tests/test_gina.py (+171/-3)
scripts/gina.py (+9/-6)
To merge this branch: bzr merge lp:~jtv/launchpad/transitional-published
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+74719@code.launchpad.net

This proposal has been superseded by a proposal from 2011-09-09.

Commit message

Convert any remaining Pending Debian SPPHs before transitional domination.

Description of the change

= Summary =

Gina is learning to run domination on its Debian imports. Yes, technically that makes Gina a dominatrix.

There is a catch: domination operates only on publication records in Published state. So far, Gina has been creating Pending records. That made sense for Ubuntu's import into Launchpad, way back when, which is what Gina was originally written for. It does not make sense now, so we changed the code and updated the legacy data in batches.

Of course that does not take care of further legacy data that is created between the update we just did in the database and the time the Gina changes roll out. But it should reduce the problem's size enough that we can afford to do the final update from inside Gina. This will only need to run once, during ‚Äútransitional domination‚ÄĚ but will do nothing if run repeatedly.

== Pre-implementation notes ==

I had wanted to do a final patch-up update later, but William's points out that there is quite a substantial risk if domination runs on partially updated data. So I inserted the change directly into the code; I have another branch waiting to remove the transitional code.

== Implementation details ==

The diff starts with some modernization of a doctest (using active_publishing_status instead of spelling out the Published and Pending statuses). This is essentially cosmetic; it does not change the meaning of the code and really isn't relevant to the branch. Buried in there however is one change where the test still assumes that Gina produces and maintains Pending publication records. Before we remove transitional domination, we may have to re-think what happens there.

== Tests ==

Two new tests verify transitional behaviour. One is very simple: an active SPPH is Pending but really ought to be Published. Gina dominates and the SPPH's version is found in the simulated Sources list. The Pending SPPH becomes Published.

Another test looks for the doom scenario that might occur when data is not properly migrated: an older release of a package is Published, but the version that is mentioned in the Sources list is still Pending. It is essential that the live version be upgraded to Published before domination, or the dominator would decide that there is no newer published version to dominate the now-obsolete published SPPH, and mark it deleted. Needless to say, the test shows doom being avoided.

{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}

== Demo and Q/A ==

= Launchpad lint =

There is some pre-existing lint, especially in the doctest, that I can't afford to do too much about. It's not just the effort (mostly a matter of running utilities/formatdoctest.py!) but also a matter of keeping the diff small. I'm piling up quite a number of differences in a series of interdependent branches, and this would increase the risk of conflicts disproportionately.

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/domination.py
  lib/lp/soyuz/scripts/gina/dominate.py
  lib/lp/soyuz/model/publishing.py
  lib/lp/soyuz/doc/gina.txt
  lib/lp/soyuz/interfaces/publishing.py
  scripts/gina.py
  lib/lp/archivepublisher/tests/test_dominator.py
  lib/lp/soyuz/scripts/tests/test_gina.py
  lib/lp/soyuz/scripts/gina/handlers.py

./lib/lp/soyuz/doc/gina.txt
     113: narrative exceeds 78 characters.
     162: want exceeds 78 characters.
     179: want exceeds 78 characters.
     189: narrative uses a moin header.
     221: want exceeds 78 characters.
     234: want exceeds 78 characters.
     240: want exceeds 78 characters.
     295: source exceeds 78 characters.
     324: narrative uses a moin header.
     342: narrative exceeds 78 characters.
     354: narrative uses a moin header.
     360: narrative exceeds 78 characters.
     361: narrative exceeds 78 characters.
     459: narrative uses a moin header.
     461: narrative exceeds 78 characters.
     462: narrative exceeds 78 characters.
     477: narrative uses a moin header.
     563: narrative exceeds 78 characters.
     600: narrative uses a moin header.
     657: narrative uses a moin header.
     746: narrative uses a moin header.
     767: narrative uses a moin header.
     780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
     381: E261 at least two spaces before inline comment
     478: E261 at least two spaces before inline comment
     511: E261 at least two spaces before inline comment
     681: E261 at least two spaces before inline comment
     767: E261 at least two spaces before inline comment
./scripts/gina.py
      26: '_pythonpath' imported but unused

To post a comment you must log in.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/archivepublisher/domination.py'
2--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
3+++ lib/lp/archivepublisher/domination.py 2011-09-09 05:48:24 +0000
4@@ -53,8 +53,6 @@
5 __all__ = ['Dominator']
6
7 from datetime import timedelta
8-import functools
9-import operator
10
11 import apt_pkg
12 from storm.expr import (
13@@ -68,7 +66,7 @@
14 flush_database_updates,
15 sqlvalues,
16 )
17-from canonical.launchpad.interfaces.lpstorm import IMasterStore
18+from canonical.launchpad.interfaces.lpstorm import IStore
19 from lp.registry.model.sourcepackagename import SourcePackageName
20 from lp.soyuz.enums import (
21 BinaryPackageFormat,
22@@ -87,17 +85,93 @@
23 apt_pkg.InitSystem()
24
25
26-def _compare_packages_by_version_and_date(get_release, p1, p2):
27- """Compare publications p1 and p2 by their version; using Debian rules.
28-
29- If the publications are for the same package, compare by datecreated
30- instead. This lets newer records win.
31- """
32- if get_release(p1).id == get_release(p2).id:
33- return cmp(p1.datecreated, p2.datecreated)
34-
35- return apt_pkg.VersionCompare(get_release(p1).version,
36- get_release(p2).version)
37+def join_spr_spn():
38+ """Join condition: SourcePackageRelease/SourcePackageName."""
39+ return (
40+ SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
41+
42+
43+def join_spph_spr():
44+ """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
45+ """
46+ # Avoid circular imports.
47+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
48+
49+ return (
50+ SourcePackageRelease.id ==
51+ SourcePackagePublishingHistory.sourcepackagereleaseID)
52+
53+
54+class SourcePublicationTraits:
55+ """Basic generalized attributes for `SourcePackagePublishingHistory`.
56+
57+ Used by `GeneralizedPublication` to hide the differences from
58+ `BinaryPackagePublishingHistory`.
59+ """
60+ @staticmethod
61+ def getPackageName(spph):
62+ """Return the name of this publication's source package."""
63+ return spph.sourcepackagerelease.sourcepackagename.name
64+
65+ @staticmethod
66+ def getPackageRelease(spph):
67+ """Return this publication's `SourcePackageRelease`."""
68+ return spph.sourcepackagerelease
69+
70+
71+class BinaryPublicationTraits:
72+ """Basic generalized attributes for `BinaryPackagePublishingHistory`.
73+
74+ Used by `GeneralizedPublication` to hide the differences from
75+ `SourcePackagePublishingHistory`.
76+ """
77+ @staticmethod
78+ def getPackageName(bpph):
79+ """Return the name of this publication's binary package."""
80+ return bpph.binarypackagerelease.binarypackagename.name
81+
82+ @staticmethod
83+ def getPackageRelease(bpph):
84+ """Return this publication's `BinaryPackageRelease`."""
85+ return bpph.binarypackagerelease
86+
87+
88+class GeneralizedPublication:
89+ """Generalize handling of publication records.
90+
91+ This allows us to write code that can be dealing with either
92+ `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
93+ without caring which. Differences are abstracted away in a traits
94+ class.
95+ """
96+ def __init__(self, is_source=True):
97+ if is_source:
98+ self.traits = SourcePublicationTraits
99+ else:
100+ self.traits = BinaryPublicationTraits
101+
102+ def getPackageName(self, pub):
103+ """Get the package's name."""
104+ return self.traits.getPackageName(pub)
105+
106+ def getPackageVersion(self, pub):
107+ """Obtain the version string for a publicaiton record."""
108+ return self.traits.getPackageRelease(pub).version
109+
110+ def compare(self, pub1, pub2):
111+ """Compare publications by version.
112+
113+ If both publications are for the same version, their creation dates
114+ break the tie.
115+ """
116+ version_comparison = apt_pkg.VersionCompare(
117+ self.getPackageVersion(pub1), self.getPackageVersion(pub2))
118+
119+ if version_comparison == 0:
120+ # Use dates as tie breaker.
121+ return cmp(pub1.datecreated, pub2.datecreated)
122+ else:
123+ return version_comparison
124
125
126 class Dominator:
127@@ -116,50 +190,95 @@
128 self.logger = logger
129 self.archive = archive
130
131- def _dominatePublications(self, pubs):
132+ def dominatePackage(self, publications, live_versions, generalization):
133+ """Dominate publications for a single package.
134+
135+ Active publications for versions in `live_versions` stay active.
136+ Any older versions are marked as superseded by the respective
137+ oldest live versions that are newer than the superseded ones.
138+
139+ Any versions that are newer than anything in `live_versions` are
140+ marked as deleted. This should not be possible in Soyuz-native
141+ archives, but it can happen during archive imports when the
142+ previous latest version of a package has disappeared from the Sources
143+ list we import.
144+
145+ :param publications: Iterable of publications for the same package,
146+ in the same archive, series, and pocket, all with status
147+ `PackagePublishingStatus.PUBLISHED`.
148+ :param live_versions: Iterable of version strings that are still
149+ considered live for this package. The given publications will
150+ remain active insofar as they represent any of these versions;
151+ older publications will be marked as superseded and newer ones
152+ as deleted.
153+ :param generalization: A `GeneralizedPublication` helper representing
154+ the kind of publications these are--source or binary.
155+ """
156+ # Go through publications from latest version to oldest. This
157+ # makes it easy to figure out which release superseded which:
158+ # the dominant is always the oldest live release that is newer
159+ # than the one being superseded.
160+ publications = sorted(
161+ publications, cmp=generalization.compare, reverse=True)
162+
163+ current_dominant = None
164+ for pub in publications:
165+ if generalization.getPackageVersion(pub) in live_versions:
166+ # This publication stays active; if any publications
167+ # that follow right after this are to be superseded,
168+ # this is the release that they are superseded by.
169+ current_dominant = pub
170+ elif current_dominant is None:
171+ # This publication is no longer live, but there is no
172+ # newer version to supersede it either. Therefore it
173+ # must be deleted.
174+ pub.requestDeletion(None)
175+ else:
176+ # This publication is superseded. This is what we're
177+ # here to do.
178+ pub.supersede(current_dominant, logger=self.logger)
179+
180+ def _dominatePublications(self, pubs, generalization):
181 """Perform dominations for the given publications.
182
183+ Keep the latest published version for each package active,
184+ superseding older versions.
185+
186 :param pubs: A dict mapping names to a list of publications. Every
187 publication must be PUBLISHED or PENDING, and the first in each
188 list will be treated as dominant (so should be the latest).
189+ :param generalization: A `GeneralizedPublication` helper representing
190+ the kind of publications these are--source or binary.
191 """
192 self.logger.debug("Dominating packages...")
193-
194- for name in pubs.keys():
195- assert pubs[name], (
196- "Empty list of publications for %s" % name)
197- for pubrec in pubs[name][1:]:
198- pubrec.supersede(pubs[name][0], logger=self.logger)
199-
200- def _sortPackages(self, pkglist, is_source=True):
201+ for name, publications in pubs.iteritems():
202+ assert publications, "Empty list of publications for %s." % name
203+ # Since this always picks the latest version as the live
204+ # one, this dominatePackage call will never result in a
205+ # deletion.
206+ latest_version = generalization.getPackageVersion(publications[0])
207+ self.dominatePackage(
208+ publications, [latest_version], generalization)
209+
210+ def _sortPackages(self, pkglist, generalization):
211 """Map out packages by name, and sort by descending version.
212
213 :param pkglist: An iterable of `SourcePackagePublishingHistory` or
214 `BinaryPackagePublishingHistory`.
215- :param is_source: Whether this call involves source package
216- publications. If so, work with `SourcePackagePublishingHistory`.
217- If not, work with `BinaryPackagepublishingHistory`.
218- :return: A dict mapping each package name (as UTF-8 encoded string)
219- to a list of publications from `pkglist`, newest first.
220+ :param generalization: A `GeneralizedPublication` helper representing
221+ the kind of publications these are--source or binary.
222+ :return: A dict mapping each package name to a list of publications
223+ from `pkglist`, newest first.
224 """
225 self.logger.debug("Sorting packages...")
226
227- if is_source:
228- get_release = operator.attrgetter("sourcepackagerelease")
229- get_name = operator.attrgetter("sourcepackagename")
230- else:
231- get_release = operator.attrgetter("binarypackagerelease")
232- get_name = operator.attrgetter("binarypackagename")
233-
234 outpkgs = {}
235 for inpkg in pkglist:
236- key = get_name(get_release(inpkg)).name.encode('utf-8')
237+ key = generalization.getPackageName(inpkg)
238 outpkgs.setdefault(key, []).append(inpkg)
239
240- sort_order = functools.partial(
241- _compare_packages_by_version_and_date, get_release)
242 for package_pubs in outpkgs.itervalues():
243- package_pubs.sort(cmp=sort_order, reverse=True)
244+ package_pubs.sort(cmp=generalization.compare, reverse=True)
245
246 return outpkgs
247
248@@ -287,6 +406,8 @@
249 # Avoid circular imports.
250 from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
251
252+ generalization = GeneralizedPublication(is_source=False)
253+
254 for distroarchseries in distroseries.architectures:
255 self.logger.debug(
256 "Performing domination across %s/%s (%s)",
257@@ -312,7 +433,7 @@
258 ),
259 group_by=BinaryPackageName.id,
260 having=Count(BinaryPackagePublishingHistory.id) > 1)
261- binaries = IMasterStore(BinaryPackagePublishingHistory).find(
262+ binaries = IStore(BinaryPackagePublishingHistory).find(
263 BinaryPackagePublishingHistory,
264 BinaryPackageRelease.id ==
265 BinaryPackagePublishingHistory.binarypackagereleaseID,
266@@ -322,7 +443,21 @@
267 BinaryPackageFormat.DDEB,
268 bpph_location_clauses)
269 self.logger.debug("Dominating binaries...")
270- self._dominatePublications(self._sortPackages(binaries, False))
271+ self._dominatePublications(
272+ self._sortPackages(binaries, generalization), generalization)
273+
274+ def _composeActiveSourcePubsCondition(self, distroseries, pocket):
275+ """Compose ORM condition for restricting relevant source pubs."""
276+ # Avoid circular imports.
277+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
278+
279+ return And(
280+ SourcePackagePublishingHistory.status ==
281+ PackagePublishingStatus.PUBLISHED,
282+ SourcePackagePublishingHistory.distroseries == distroseries,
283+ SourcePackagePublishingHistory.archive == self.archive,
284+ SourcePackagePublishingHistory.pocket == pocket,
285+ )
286
287 def dominateSources(self, distroseries, pocket):
288 """Perform domination on source package publications.
289@@ -332,38 +467,76 @@
290 """
291 # Avoid circular imports.
292 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
293+
294+ generalization = GeneralizedPublication(is_source=True)
295+
296 self.logger.debug(
297 "Performing domination across %s/%s (Source)",
298 distroseries.name, pocket.title)
299- spph_location_clauses = And(
300- SourcePackagePublishingHistory.status ==
301- PackagePublishingStatus.PUBLISHED,
302- SourcePackagePublishingHistory.distroseries == distroseries,
303- SourcePackagePublishingHistory.archive == self.archive,
304- SourcePackagePublishingHistory.pocket == pocket,
305- )
306+
307+ spph_location_clauses = self._composeActiveSourcePubsCondition(
308+ distroseries, pocket)
309+ having_multiple_active_publications = (
310+ Count(SourcePackagePublishingHistory.id) > 1)
311 candidate_source_names = Select(
312 SourcePackageName.id,
313- And(
314- SourcePackageRelease.sourcepackagenameID ==
315- SourcePackageName.id,
316- SourcePackagePublishingHistory.sourcepackagereleaseID ==
317- SourcePackageRelease.id,
318- spph_location_clauses,
319- ),
320+ And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
321 group_by=SourcePackageName.id,
322- having=Count(SourcePackagePublishingHistory.id) > 1)
323- sources = IMasterStore(SourcePackagePublishingHistory).find(
324+ having=having_multiple_active_publications)
325+ sources = IStore(SourcePackagePublishingHistory).find(
326 SourcePackagePublishingHistory,
327- SourcePackageRelease.id ==
328- SourcePackagePublishingHistory.sourcepackagereleaseID,
329+ join_spph_spr(),
330 SourcePackageRelease.sourcepackagenameID.is_in(
331 candidate_source_names),
332 spph_location_clauses)
333+
334 self.logger.debug("Dominating sources...")
335- self._dominatePublications(self._sortPackages(sources))
336+ self._dominatePublications(
337+ self._sortPackages(sources, generalization), generalization)
338 flush_database_updates()
339
340+ def findPublishedSourcePackageNames(self, distroseries, pocket):
341+ """Find names of currently published source packages."""
342+ result = IStore(SourcePackageName).find(
343+ SourcePackageName.name,
344+ join_spph_spr(),
345+ join_spr_spn(),
346+ self._composeActiveSourcePubsCondition(distroseries, pocket))
347+ return result.config(distinct=True)
348+
349+ def findPublishedSPPHs(self, distroseries, pocket, package_name):
350+ """Find currently published source publications for given package."""
351+ # Avoid circular imports.
352+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
353+
354+ return IStore(SourcePackagePublishingHistory).find(
355+ SourcePackagePublishingHistory,
356+ join_spph_spr(),
357+ join_spr_spn(),
358+ SourcePackageName.name == package_name,
359+ self._composeActiveSourcePubsCondition(distroseries, pocket))
360+
361+ def dominateRemovedSourceVersions(self, distroseries, pocket,
362+ package_name, live_versions):
363+ """Dominate source publications based on a set of "live" versions.
364+
365+ Active publications for the "live" versions will remain active. All
366+ other active publications for the same package (and the same archive,
367+ distroseries, and pocket) are marked superseded.
368+
369+ Unlike traditional domination, this allows multiple versions of a
370+ package to stay active in the same distroseries, archive, and pocket.
371+
372+ :param distroseries: `DistroSeries` to dominate.
373+ :param pocket: `PackagePublishingPocket` to dominate.
374+ :param package_name: Source package name, as text.
375+ :param live_versions: Iterable of all version strings that are to
376+ remain active.
377+ """
378+ generalization = GeneralizedPublication(is_source=True)
379+ pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
380+ self.dominatePackage(pubs, live_versions, generalization)
381+
382 def judge(self, distroseries, pocket):
383 """Judge superseded sources and binaries."""
384 # Avoid circular imports.
385
386=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
387--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
388+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-09 05:48:24 +0000
389@@ -1,4 +1,4 @@
390-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
391+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
392 # GNU Affero General Public License version 3 (see the file LICENSE).
393
394 """Tests for domination.py."""
395@@ -7,12 +7,24 @@
396
397 import datetime
398
399+import apt_pkg
400+from zope.security.proxy import removeSecurityProxy
401+
402 from canonical.database.sqlbase import flush_database_updates
403-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
404+from canonical.testing.layers import ZopelessDatabaseLayer
405+from lp.archivepublisher.domination import (
406+ Dominator,
407+ GeneralizedPublication,
408+ STAY_OF_EXECUTION,
409+ )
410 from lp.archivepublisher.publishing import Publisher
411+from lp.registry.interfaces.pocket import PackagePublishingPocket
412 from lp.registry.interfaces.series import SeriesStatus
413+from lp.services.log.logger import DevNullLogger
414 from lp.soyuz.enums import PackagePublishingStatus
415+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
416 from lp.soyuz.tests.test_publishing import TestNativePublishingBase
417+from lp.testing import TestCaseWithFactory
418
419
420 class TestDominator(TestNativePublishingBase):
421@@ -50,6 +62,8 @@
422 foo_10_source, foo_10_binaries[0])
423
424 def dominateAndCheck(self, dominant, dominated, supersededby):
425+ generalization = GeneralizedPublication(
426+ is_source=ISourcePackagePublishingHistory.providedBy(dominant))
427 dominator = Dominator(self.logger, self.ubuntutest.main_archive)
428
429 # The _dominate* test methods require a dictionary where the
430@@ -58,7 +72,7 @@
431 # and dominated, the subsequents.
432 pubs = {'foo': [dominant, dominated]}
433
434- dominator._dominatePublications(pubs)
435+ dominator._dominatePublications(pubs, generalization)
436 flush_database_updates()
437
438 # The dominant version remains correctly published.
439@@ -145,7 +159,9 @@
440 # This isn't a really good exception. It should probably be
441 # something more indicative of bad input.
442 self.assertRaises(
443- AssertionError, dominator._dominatePublications, pubs)
444+ AssertionError,
445+ dominator._dominatePublications,
446+ pubs, GeneralizedPublication(True))
447
448
449 class TestDomination(TestNativePublishingBase):
450@@ -200,3 +216,398 @@
451 TestDomination.setUp(self)
452 self.ubuntutest['breezy-autotest'].status = (
453 SeriesStatus.OBSOLETE)
454+
455+
456+def make_spphs_for_versions(factory, versions):
457+ """Create publication records for each of `versions`.
458+
459+ They records are created in the same order in which they are specified.
460+ Make the order irregular to prove that version ordering is not a
461+ coincidence of object creation order etc.
462+
463+ Versions may also be identical; each publication record will still have
464+ its own package release.
465+ """
466+ spn = factory.makeSourcePackageName()
467+ distroseries = factory.makeDistroSeries()
468+ pocket = factory.getAnyPocket()
469+ sprs = [
470+ factory.makeSourcePackageRelease(
471+ sourcepackagename=spn, version=version)
472+ for version in versions]
473+ return [
474+ factory.makeSourcePackagePublishingHistory(
475+ distroseries=distroseries, pocket=pocket,
476+ sourcepackagerelease=spr,
477+ status=PackagePublishingStatus.PUBLISHED)
478+ for spr in sprs]
479+
480+
481+def list_source_versions(spphs):
482+ """Extract the versions from `spphs` as a list, in the same order."""
483+ return [spph.sourcepackagerelease.version for spph in spphs]
484+
485+
486+class TestGeneralizedPublication(TestCaseWithFactory):
487+ """Test publication generalization helpers."""
488+
489+ layer = ZopelessDatabaseLayer
490+
491+ def alterCreationDates(self, spphs, ages):
492+ """Set `datecreated` on each of `spphs` according to `ages`.
493+
494+ :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
495+ respective creation dates will be offset by the respective ages
496+ found in `ages` (with the two being matched up in the same order).
497+ :param ages: Iterable of ages. Must provide the same number of items
498+ as `spphs`. Ages are `timedelta` objects that will be subtracted
499+ from the creation dates on the respective records in `spph`.
500+ """
501+ for spph, age in zip(spphs, ages):
502+ spph.datecreated -= age
503+
504+ def test_getPackageVersion_gets_source_version(self):
505+ spph = self.factory.makeSourcePackagePublishingHistory()
506+ self.assertEqual(
507+ spph.sourcepackagerelease.version,
508+ GeneralizedPublication(is_source=True).getPackageVersion(spph))
509+
510+ def test_getPackageVersion_gets_binary_version(self):
511+ bpph = self.factory.makeBinaryPackagePublishingHistory()
512+ self.assertEqual(
513+ bpph.binarypackagerelease.version,
514+ GeneralizedPublication(is_source=False).getPackageVersion(bpph))
515+
516+ def test_compare_sorts_versions(self):
517+ versions = [
518+ '1.1v2',
519+ '1.1v1',
520+ '1.1v3',
521+ ]
522+ spphs = make_spphs_for_versions(self.factory, versions)
523+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
524+ self.assertEqual(
525+ sorted(versions), list_source_versions(sorted_spphs))
526+
527+ def test_compare_orders_versions_by_debian_rules(self):
528+ versions = [
529+ '1.1.0',
530+ '1.10',
531+ '1.1',
532+ '1.1ubuntu0',
533+ ]
534+ spphs = make_spphs_for_versions(self.factory, versions)
535+
536+ debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
537+
538+ # Assumption: in this case, Debian version ordering is not the
539+ # same as alphabetical version ordering.
540+ self.assertNotEqual(sorted(versions), debian_sorted_versions)
541+
542+ # The compare method produces the Debian ordering.
543+ sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
544+ self.assertEqual(
545+ sorted(versions, cmp=apt_pkg.VersionCompare),
546+ list_source_versions(sorted_spphs))
547+
548+ def test_compare_breaks_tie_with_creation_date(self):
549+ # When two publications are tied for comparison because they are
550+ # for the same package release, they are ordered by creation
551+ # date.
552+ distroseries = self.factory.makeDistroSeries()
553+ pocket = self.factory.getAnyPocket()
554+ spr = self.factory.makeSourcePackageRelease()
555+ ages = [
556+ datetime.timedelta(2),
557+ datetime.timedelta(1),
558+ datetime.timedelta(3),
559+ ]
560+ spphs = [
561+ self.factory.makeSourcePackagePublishingHistory(
562+ sourcepackagerelease=spr, distroseries=distroseries,
563+ pocket=pocket)
564+ for counter in xrange(len(ages))]
565+ self.alterCreationDates(spphs, ages)
566+
567+ self.assertEqual(
568+ [spphs[2], spphs[0], spphs[1]],
569+ sorted(spphs, cmp=GeneralizedPublication().compare))
570+
571+ def test_compare_breaks_tie_for_releases_with_same_version(self):
572+ # When two publications are tied for comparison because they
573+ # belong to releases with the same version string, they are
574+ # ordered by creation date.
575+ version = "1.%d" % self.factory.getUniqueInteger()
576+ ages = [
577+ datetime.timedelta(2),
578+ datetime.timedelta(1),
579+ datetime.timedelta(3),
580+ ]
581+ distroseries = self.factory.makeDistroSeries()
582+ pocket = self.factory.getAnyPocket()
583+ spphs = [
584+ self.factory.makeSourcePackagePublishingHistory(
585+ distroseries=distroseries, pocket=pocket,
586+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
587+ version=version))
588+ for counter in xrange(len(ages))]
589+ self.alterCreationDates(spphs, ages)
590+
591+ self.assertEqual(
592+ [spphs[2], spphs[0], spphs[1]],
593+ sorted(spphs, cmp=GeneralizedPublication().compare))
594+
595+
596+class TestDominatorMethods(TestCaseWithFactory):
597+
598+ layer = ZopelessDatabaseLayer
599+
600+ def makeDominator(self, publications):
601+ """Create a `Dominator` suitable for `publications`."""
602+ if len(publications) == 0:
603+ archive = self.factory.makeArchive()
604+ else:
605+ archive = publications[0].archive
606+ return Dominator(DevNullLogger(), archive)
607+
608+ def test_dominatePackage_survives_empty_publications_list(self):
609+ # Nothing explodes when dominatePackage is called with an empty
610+ # packages list.
611+ self.makeDominator([]).dominatePackage(
612+ [], [], GeneralizedPublication(True))
613+ # The test is that we get here without error.
614+ pass
615+
616+ def test_dominatePackage_leaves_live_version_untouched(self):
617+ # dominatePackage does not supersede live versions.
618+ [pub] = make_spphs_for_versions(self.factory, ['3.1'])
619+ self.makeDominator([pub]).dominatePackage(
620+ [pub], ['3.1'], GeneralizedPublication(True))
621+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
622+
623+ def test_dominatePackage_deletes_dead_version_without_successor(self):
624+ # dominatePackage marks non-live package versions without
625+ # superseding versions as deleted.
626+ [pub] = make_spphs_for_versions(self.factory, ['1.1'])
627+ self.makeDominator([pub]).dominatePackage(
628+ [pub], [], GeneralizedPublication(True))
629+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
630+
631+ def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
632+ # When marking a package as superseded, dominatePackage
633+ # designates a newer live version as the superseding version.
634+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
635+ self.makeDominator(pubs).dominatePackage(
636+ pubs, ['1.1'], GeneralizedPublication(True))
637+ self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
638+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
639+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
640+
641+ def test_dominatePackage_only_supersedes_with_live_pub(self):
642+ # When marking a package as superseded, dominatePackage will
643+ # only pick a live version as the superseding one.
644+ pubs = make_spphs_for_versions(
645+ self.factory, ['1.0', '2.0', '3.0', '4.0'])
646+ self.makeDominator(pubs).dominatePackage(
647+ pubs, ['3.0'], GeneralizedPublication(True))
648+ self.assertEqual([
649+ pubs[2].sourcepackagerelease,
650+ pubs[2].sourcepackagerelease,
651+ None,
652+ None,
653+ ],
654+ [pub.supersededby for pub in pubs])
655+
656+ def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
657+ # When marking a package as superseded, dominatePackage picks
658+ # the oldest of the newer, live versions as the superseding one.
659+ pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
660+ self.makeDominator(pubs).dominatePackage(
661+ pubs, ['2.8', '2.9'], GeneralizedPublication(True))
662+ self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
663+
664+ def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
665+ # When marking a package as superseded, dominatePackage only
666+ # considers a newer version as the superseding one.
667+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
668+ self.makeDominator(pubs).dominatePackage(
669+ pubs, ['0.1'], GeneralizedPublication(True))
670+ self.assertEqual(None, pubs[1].supersededby)
671+ self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
672+
673+ def test_dominateRemovedSourceVersions_dominates_publications(self):
674+ # dominateRemovedSourceVersions finds the publications for a
675+ # package and calls dominatePackage on them.
676+ pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
677+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
678+
679+ self.makeDominator(pubs).dominateRemovedSourceVersions(
680+ pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
681+ self.assertEqual([
682+ PackagePublishingStatus.SUPERSEDED,
683+ PackagePublishingStatus.PUBLISHED,
684+ PackagePublishingStatus.DELETED,
685+ ],
686+ [pub.status for pub in pubs])
687+ self.assertEqual(
688+ [pubs[1].sourcepackagerelease, None, None],
689+ [pub.supersededby for pub in pubs])
690+
691+ def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
692+ # dominateRemovedSourceVersions ignores publications in other
693+ # pockets than the one specified.
694+ pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
695+ package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
696+ removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
697+ removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
698+ self.makeDominator(pubs).dominateRemovedSourceVersions(
699+ pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
700+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
701+
702+ def test_dominateRemovedSourceVersions_ignores_other_packages(self):
703+ pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
704+ other_package_name = self.factory.makeSourcePackageName().name
705+ self.makeDominator(pubs).dominateRemovedSourceVersions(
706+ pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
707+ self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
708+
709+ def test_findPublishedSourcePackageNames_finds_package(self):
710+ spph = self.factory.makeSourcePackagePublishingHistory(
711+ status=PackagePublishingStatus.PUBLISHED)
712+ dominator = self.makeDominator([spph])
713+ self.assertContentEqual(
714+ [spph.sourcepackagerelease.sourcepackagename.name],
715+ dominator.findPublishedSourcePackageNames(
716+ spph.distroseries, spph.pocket))
717+
718+ def test_findPublishedSourcePackageNames_ignores_other_states(self):
719+ series = self.factory.makeDistroSeries()
720+ pocket = PackagePublishingPocket.RELEASE
721+ spphs = dict(
722+ (status, self.factory.makeSourcePackagePublishingHistory(
723+ distroseries=series, archive=series.main_archive,
724+ pocket=pocket, status=status))
725+ for status in PackagePublishingStatus.items)
726+ published_spph = spphs[PackagePublishingStatus.PUBLISHED]
727+ dominator = self.makeDominator(spphs.values())
728+ self.assertContentEqual(
729+ [published_spph.sourcepackagerelease.sourcepackagename.name],
730+ dominator.findPublishedSourcePackageNames(series, pocket))
731+
732+ def test_findPublishedSourcePackageNames_ignores_other_archives(self):
733+ spph = self.factory.makeSourcePackagePublishingHistory(
734+ status=PackagePublishingStatus.PUBLISHED)
735+ dominator = self.makeDominator([spph])
736+ dominator.archive = self.factory.makeArchive()
737+ self.assertContentEqual(
738+ [],
739+ dominator.findPublishedSourcePackageNames(
740+ spph.distroseries, spph.pocket))
741+
742+ def test_findPublishedSourcePackageNames_ignores_other_series(self):
743+ spph = self.factory.makeSourcePackagePublishingHistory(
744+ status=PackagePublishingStatus.PUBLISHED)
745+ distro = spph.distroseries.distribution
746+ other_series = self.factory.makeDistroSeries(distribution=distro)
747+ dominator = self.makeDominator([spph])
748+ self.assertContentEqual(
749+ [],
750+ dominator.findPublishedSourcePackageNames(
751+ other_series, spph.pocket))
752+
753+ def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
754+ spph = self.factory.makeSourcePackagePublishingHistory(
755+ status=PackagePublishingStatus.PUBLISHED,
756+ pocket=PackagePublishingPocket.RELEASE)
757+ dominator = self.makeDominator([spph])
758+ self.assertContentEqual(
759+ [],
760+ dominator.findPublishedSourcePackageNames(
761+ spph.distroseries, PackagePublishingPocket.SECURITY))
762+
763+ def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
764+ series = self.factory.makeDistroSeries()
765+ pocket = PackagePublishingPocket.RELEASE
766+ package = self.factory.makeSourcePackageName()
767+ spphs = [
768+ self.factory.makeSourcePackagePublishingHistory(
769+ distroseries=series, archive=series.main_archive,
770+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
771+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
772+ sourcepackagename=package))
773+ for counter in xrange(2)]
774+ dominator = self.makeDominator(spphs)
775+ self.assertEqual(
776+ [package.name],
777+ list(dominator.findPublishedSourcePackageNames(series, pocket)))
778+
779+ def test_findPublishedSPPHs_finds_published_SPPH(self):
780+ spph = self.factory.makeSourcePackagePublishingHistory(
781+ status=PackagePublishingStatus.PUBLISHED)
782+ package_name = spph.sourcepackagerelease.sourcepackagename.name
783+ dominator = self.makeDominator([spph])
784+ self.assertContentEqual(
785+ [spph],
786+ dominator.findPublishedSPPHs(
787+ spph.distroseries, spph.pocket, package_name))
788+
789+ def test_findPublishedSPPHs_ignores_other_states(self):
790+ series = self.factory.makeDistroSeries()
791+ package = self.factory.makeSourcePackageName()
792+ pocket = PackagePublishingPocket.RELEASE
793+ spphs = dict(
794+ (status, self.factory.makeSourcePackagePublishingHistory(
795+ distroseries=series, archive=series.main_archive,
796+ pocket=pocket, status=status,
797+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
798+ sourcepackagename=package)))
799+ for status in PackagePublishingStatus.items)
800+ dominator = self.makeDominator(spphs.values())
801+ self.assertContentEqual(
802+ [spphs[PackagePublishingStatus.PUBLISHED]],
803+ dominator.findPublishedSPPHs(series, pocket, package.name))
804+
805+ def test_findPublishedSPPHs_ignores_other_archives(self):
806+ spph = self.factory.makeSourcePackagePublishingHistory(
807+ status=PackagePublishingStatus.PUBLISHED)
808+ package = spph.sourcepackagerelease.sourcepackagename
809+ dominator = self.makeDominator([spph])
810+ dominator.archive = self.factory.makeArchive()
811+ self.assertContentEqual(
812+ [],
813+ dominator.findPublishedSPPHs(
814+ spph.distroseries, spph.pocket, package.name))
815+
816+ def test_findPublishedSPPHs_ignores_other_series(self):
817+ spph = self.factory.makeSourcePackagePublishingHistory(
818+ status=PackagePublishingStatus.PUBLISHED)
819+ distro = spph.distroseries.distribution
820+ package = spph.sourcepackagerelease.sourcepackagename
821+ other_series = self.factory.makeDistroSeries(distribution=distro)
822+ dominator = self.makeDominator([spph])
823+ self.assertContentEqual(
824+ [],
825+ dominator.findPublishedSPPHs(
826+ other_series, spph.pocket, package.name))
827+
828+ def test_findPublishedSPPHs_ignores_other_pockets(self):
829+ spph = self.factory.makeSourcePackagePublishingHistory(
830+ status=PackagePublishingStatus.PUBLISHED,
831+ pocket=PackagePublishingPocket.RELEASE)
832+ package = spph.sourcepackagerelease.sourcepackagename
833+ dominator = self.makeDominator([spph])
834+ self.assertContentEqual(
835+ [],
836+ dominator.findPublishedSPPHs(
837+ spph.distroseries, PackagePublishingPocket.SECURITY,
838+ package.name))
839+
840+ def test_findPublishedSPPHs_ignores_other_packages(self):
841+ spph = self.factory.makeSourcePackagePublishingHistory(
842+ status=PackagePublishingStatus.PUBLISHED)
843+ other_package = self.factory.makeSourcePackageName()
844+ dominator = self.makeDominator([spph])
845+ self.assertContentEqual(
846+ [],
847+ dominator.findPublishedSPPHs(
848+ spph.distroseries, spph.pocket, other_package.name))
849
850=== modified file 'lib/lp/soyuz/doc/gina.txt'
851--- lib/lp/soyuz/doc/gina.txt 2011-07-29 11:35:28 +0000
852+++ lib/lp/soyuz/doc/gina.txt 2011-09-09 05:48:24 +0000
853@@ -8,6 +8,7 @@
854 Get the current counts of stuff in the database:
855
856 >>> from canonical.launchpad.database.emailaddress import EmailAddress
857+ >>> from lp.soyuz.interfaces.publishing import active_publishing_status
858 >>> from lp.soyuz.model.publishing import (
859 ... BinaryPackagePublishingHistory,
860 ... SourcePackagePublishingHistory)
861@@ -564,35 +565,34 @@
862 that's what overrides actually do.
863
864 >>> from canonical.database.sqlbase import sqlvalues
865- >>> from lp.soyuz.enums import PackagePublishingStatus
866- >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s
867- ... AND distroseries = %s
868- ... AND status in (%s, %s)""" %
869- ... sqlvalues(x11p, breezy,
870- ... PackagePublishingStatus.PUBLISHED,
871- ... PackagePublishingStatus.PENDING),
872- ... orderBy=["-datecreated"])[0]
873+ >>> x11_pub = SSPPH.select("""
874+ ... sourcepackagerelease = %s AND
875+ ... distroseries = %s AND
876+ ... status in %s
877+ ... """ % sqlvalues(
878+ ... x11p, breezy, active_publishing_status),
879+ ... orderBy=["-datecreated"])[0]
880 >>> print x11_pub.section.name
881 net
882- >>> ed_pub = SBPPH.select("""binarypackagerelease = %s
883- ... AND distroarchseries = %s
884- ... AND status in (%s, %s)""" %
885- ... sqlvalues(ed, breezy_i386,
886- ... PackagePublishingStatus.PUBLISHED,
887- ... PackagePublishingStatus.PENDING),
888- ... orderBy=["-datecreated"])[0]
889+ >>> ed_pub = SBPPH.select("""
890+ ... binarypackagerelease = %s AND
891+ ... distroarchseries = %s AND
892+ ... status in %s
893+ ... """ % sqlvalues(
894+ ... ed, breezy_i386, active_publishing_status),
895+ ... orderBy=["-datecreated"])[0]
896 >>> print ed_pub.priority
897 Extra
898 >>> n = SourcePackageName.selectOneBy(name="archive-copier")
899 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
900 ... version="0.3.6")
901- >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s
902- ... AND distroseries = %s
903- ... AND status in (%s, %s)""" %
904- ... sqlvalues(ac, breezy,
905- ... PackagePublishingStatus.PUBLISHED,
906- ... PackagePublishingStatus.PENDING),
907- ... orderBy=["-datecreated"])[0]
908+ >>> ac_pub = SSPPH.select("""
909+ ... sourcepackagerelease = %s AND
910+ ... distroseries = %s AND
911+ ... status in %s
912+ ... """ % sqlvalues(
913+ ... ac, breezy, active_publishing_status),
914+ ... orderBy=["-datecreated"])[0]
915 >>> print ac_pub.component.name
916 universe
917
918@@ -720,7 +720,7 @@
919
920 >>> transaction.commit()
921
922-There is now a number of source publications in PENDING status for the
923+There is now a number of source publications in PUBLISHED status for the
924 targetted distroseries, 'lenny'.
925
926 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
927@@ -728,7 +728,7 @@
928 12
929
930 >>> print set([pub.status.name for pub in lenny_sources])
931- set(['PENDING'])
932+ set(['PUBLISHED'])
933
934 As mentioned before, lenny/i386 is empty, no binaries were imported.
935 Also, the number of binaries published in the whole debian distribution
936
937=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
938--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
939+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-09 05:48:24 +0000
940@@ -195,9 +195,6 @@
941 the field name and value is the value string.
942 """
943
944- def supersede():
945- """Supersede this publication."""
946-
947 def requestObsolescence():
948 """Make this publication obsolete.
949
950
951=== modified file 'lib/lp/soyuz/model/publishing.py'
952--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
953+++ lib/lp/soyuz/model/publishing.py 2011-09-09 05:48:24 +0000
954@@ -114,10 +114,6 @@
955 from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
956
957
958-PENDING = PackagePublishingStatus.PENDING
959-PUBLISHED = PackagePublishingStatus.PUBLISHED
960-
961-
962 # XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py
963
964 def makePoolPath(source_name, component_name):
965@@ -327,8 +323,8 @@
966 fields = self.buildIndexStanzaFields()
967 return fields.makeOutput()
968
969- def supersede(self):
970- """See `IPublishing`."""
971+ def setSuperseded(self):
972+ """Set to SUPERSEDED status."""
973 self.status = PackagePublishingStatus.SUPERSEDED
974 self.datesuperseded = UTC_NOW
975
976@@ -742,7 +738,7 @@
977 "Should not dominate unpublished source %s" %
978 self.sourcepackagerelease.title)
979
980- super(SourcePackagePublishingHistory, self).supersede()
981+ self.setSuperseded()
982
983 if dominant is not None:
984 if logger is not None:
985@@ -1081,7 +1077,7 @@
986 return IMasterStore(BinaryPackagePublishingHistory).find(
987 BinaryPackagePublishingHistory,
988 BinaryPackagePublishingHistory.status.is_in(
989- [PUBLISHED, PENDING]),
990+ active_publishing_status),
991 BinaryPackagePublishingHistory.distroarchseriesID.is_in(
992 available_architectures),
993 binarypackagerelease=self.binarypackagerelease,
994@@ -1101,7 +1097,7 @@
995 return IMasterStore(BinaryPackagePublishingHistory).find(
996 BinaryPackagePublishingHistory,
997 BinaryPackagePublishingHistory.status.is_in(
998- [PUBLISHED, PENDING]),
999+ active_publishing_status),
1000 BinaryPackagePublishingHistory.distroarchseries ==
1001 self.distroarchseries,
1002 binarypackagerelease=self.binarypackagerelease.debug_package,
1003@@ -1126,7 +1122,7 @@
1004 self.distroarchseries.architecturetag))
1005 return
1006
1007- super(BinaryPackagePublishingHistory, self).supersede()
1008+ self.setSuperseded()
1009
1010 if dominant is not None:
1011 # DDEBs cannot themselves be dominant; they are always dominated
1012
1013=== added file 'lib/lp/soyuz/scripts/gina/dominate.py'
1014--- lib/lp/soyuz/scripts/gina/dominate.py 1970-01-01 00:00:00 +0000
1015+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:48:24 +0000
1016@@ -0,0 +1,82 @@
1017+# Copyright 2011 Canonical Ltd. This software is licensed under the
1018+# GNU Affero General Public License version 3 (see the file LICENSE).
1019+
1020+"""Retirement of packages that are removed upstream."""
1021+
1022+__metaclass__ = type
1023+__all__ = [
1024+ 'dominate_imported_source_packages',
1025+ ]
1026+
1027+from zope.component import getUtility
1028+
1029+# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
1030+# import violates import policy and elicits a warning from the test
1031+# suite. The warning helps remind us to retire this code as soon as
1032+# possible.
1033+from lp.archivepublisher.domination import (
1034+ Dominator,
1035+ GeneralizedPublication,
1036+ )
1037+from lp.registry.interfaces.distribution import IDistributionSet
1038+
1039+
1040+def dominate_imported_source_packages(logger, distro_name, series_name,
1041+ pocket, packages_map):
1042+ """Perform domination."""
1043+ series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
1044+ dominator = Dominator(logger, series.main_archive)
1045+
1046+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
1047+ # hack. Gina used to create SPPHs in Pending state. We cleaned up
1048+ # the bulk of them, and changed the code to create Published ones, but
1049+ # some new ones will have been created since.
1050+ # Update those to match what the new Gina does.
1051+ from canonical.launchpad.interfaces.lpstorm import IStore
1052+ from lp.soyuz.enums import PackagePublishingStatus
1053+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
1054+ SPPH = SourcePackagePublishingHistory
1055+ store = IStore(SPPH)
1056+ spphs = store.find(
1057+ SPPH,
1058+ SPPH.archive == series.main_archive,
1059+ SPPH.distroseries == series,
1060+ SPPH.pocket == pocket,
1061+ SPPH.status == PackagePublishingStatus.PENDING)
1062+ spphs.set(status=PackagePublishingStatus.PUBLISHED)
1063+
1064+ # Dominate packages found in the Sources list we're importing.
1065+ package_names = dominator.findPublishedSourcePackageNames(series, pocket)
1066+ for package_name in package_names:
1067+ entries = packages_map.src_map.get(package_name)
1068+
1069+ if entries is None:
1070+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
1071+ # transitional hack. The database is full of "Published"
1072+ # Debian SPPHs whose packages have actually been deleted.
1073+ # In the future such publications should simply be marked
1074+ # Deleted, but for the legacy baggage we currently carry
1075+ # around we'll just do traditional domination first: pick
1076+ # the latest Published version, and mark the rest of the
1077+ # SPPHs as superseded by that version. The latest version
1078+ # will then, finally, be marked appropriately Deleted once
1079+ # we remove this transitional hack.
1080+ # To remove the transitional hack, just let live_versions
1081+ # default to the empty list instead of doing this:
1082+ pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
1083+ generalization = GeneralizedPublication(is_source=True)
1084+ pubs_dict = dominator._sortPackages(pubs, generalization)
1085+ sorted_pubs = pubs_dict[package_name]
1086+ if len(sorted_pubs) <= 1:
1087+ # If there's only one published SPPH, the transitional
1088+ # code will just leave it Published. Don't bother; the
1089+ # migration will be costly enough as it is.
1090+ continue
1091+ live_versions = [sorted_pubs[0].sourcepackagerelease.version]
1092+ else:
1093+ live_versions = [
1094+ entry['Version']
1095+ for entry in entries if 'Version' in entry]
1096+
1097+ dominator.dominateRemovedSourceVersions(
1098+ series, pocket, package_name, live_versions)
1099
1100=== modified file 'lib/lp/soyuz/scripts/gina/handlers.py'
1101--- lib/lp/soyuz/scripts/gina/handlers.py 2011-05-20 07:43:58 +0000
1102+++ lib/lp/soyuz/scripts/gina/handlers.py 2011-09-09 05:48:24 +0000
1103@@ -35,12 +35,12 @@
1104 from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
1105 from canonical.launchpad.scripts import log
1106 from lp.archivepublisher.diskpool import poolify
1107+from lp.archiveuploader.changesfile import ChangesFile
1108 from lp.archiveuploader.tagfiles import parse_tagfile
1109 from lp.archiveuploader.utils import (
1110 determine_binary_file_type,
1111 determine_source_file_type,
1112 )
1113-from lp.archiveuploader.changesfile import ChangesFile
1114 from lp.buildmaster.enums import BuildStatus
1115 from lp.registry.interfaces.person import (
1116 IPersonSet,
1117@@ -54,7 +54,10 @@
1118 )
1119 from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
1120 from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
1121-from lp.soyuz.interfaces.publishing import IPublishingSet
1122+from lp.soyuz.interfaces.publishing import (
1123+ active_publishing_status,
1124+ IPublishingSet,
1125+ )
1126 from lp.soyuz.model.component import Component
1127 from lp.soyuz.model.files import (
1128 BinaryPackageFile,
1129@@ -723,8 +726,6 @@
1130 source_publishinghistory.status.title)
1131 return
1132
1133- # Create the Publishing entry, with status PENDING so that we
1134- # can republish this later into a Soyuz archive.
1135 entry = getUtility(IPublishingSet).newSourcePublication(
1136 distroseries=self.distroseries,
1137 sourcepackagerelease=sourcepackagerelease,
1138@@ -732,6 +733,7 @@
1139 section=section,
1140 pocket=self.pocket,
1141 archive=archive)
1142+ entry.setPublished()
1143 log.info('Source package %s (%s) published' % (
1144 entry.sourcepackagerelease.sourcepackagename.name,
1145 entry.sourcepackagerelease.version))
1146@@ -742,16 +744,14 @@
1147 from lp.soyuz.model.publishing import (
1148 SourcePackagePublishingHistory)
1149
1150- ret = SourcePackagePublishingHistory.select(
1151- """sourcepackagerelease = %s
1152- AND distroseries = %s
1153- AND archive = %s
1154- AND status in (%s, %s)""" %
1155- sqlvalues(sourcepackagerelease, self.distroseries,
1156- self.distroseries.main_archive,
1157- PackagePublishingStatus.PUBLISHED,
1158- PackagePublishingStatus.PENDING),
1159- orderBy=["-datecreated"])
1160+ ret = SourcePackagePublishingHistory.select("""
1161+ sourcepackagerelease = %s AND
1162+ distroseries = %s AND
1163+ archive = %s AND
1164+ status in %s""" % sqlvalues(
1165+ sourcepackagerelease, self.distroseries,
1166+ self.distroseries.main_archive, active_publishing_status),
1167+ orderBy=["-datecreated"])
1168 ret = list(ret)
1169 if ret:
1170 return ret[0]
1171@@ -917,14 +917,6 @@
1172 "for package %s (%s)" %
1173 (build.id, binary.package, binary.version))
1174 else:
1175-
1176- # XXX Debonzi 2005-05-16: Check it later
1177- # if bin.gpg_signing_key_owner:
1178- # key = self.getGPGKey(bin.gpg_signing_key,
1179- # *bin.gpg_signing_key_owner)
1180- # else:
1181- key = None
1182-
1183 processor = distroarchinfo['processor']
1184 build = getUtility(IBinaryPackageBuildSet).new(
1185 processor=processor.id,
1186@@ -948,8 +940,7 @@
1187 def publish(self, binarypackage, bpdata):
1188 """Create the publishing entry on db if does not exist."""
1189 # Avoid circular imports.
1190- from lp.soyuz.model.publishing import (
1191- BinaryPackagePublishingHistory)
1192+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
1193
1194 # These need to be pulled from the binary package data, not the
1195 # binary package release: the data represents data from /this
1196@@ -983,22 +974,20 @@
1197 binpkg_publishinghistory.status.title)
1198 return
1199
1200-
1201- # Create the Publishing entry with status PENDING.
1202 BinaryPackagePublishingHistory(
1203- binarypackagerelease = binarypackage.id,
1204- component = component.id,
1205- section = section.id,
1206- priority = priority,
1207- distroarchseries = self.distroarchseries.id,
1208- status = PackagePublishingStatus.PENDING,
1209- datecreated = UTC_NOW,
1210- datepublished = UTC_NOW,
1211- pocket = self.pocket,
1212- datesuperseded = None,
1213- supersededby = None,
1214- datemadepending = None,
1215- dateremoved = None,
1216+ binarypackagerelease=binarypackage.id,
1217+ component=component.id,
1218+ section=section.id,
1219+ priority=priority,
1220+ distroarchseries=self.distroarchseries.id,
1221+ status=PackagePublishingStatus.PUBLISHED,
1222+ datecreated=UTC_NOW,
1223+ datepublished=UTC_NOW,
1224+ pocket=self.pocket,
1225+ datesuperseded=None,
1226+ supersededby=None,
1227+ datemadepending=None,
1228+ dateremoved=None,
1229 archive=archive)
1230
1231 log.info('BinaryPackage %s-%s published into %s.' % (
1232@@ -1008,19 +997,16 @@
1233 def _checkPublishing(self, binarypackage):
1234 """Query for the publishing entry"""
1235 # Avoid circular imports.
1236- from lp.soyuz.model.publishing import (
1237- BinaryPackagePublishingHistory)
1238+ from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
1239
1240- ret = BinaryPackagePublishingHistory.select(
1241- """binarypackagerelease = %s
1242- AND distroarchseries = %s
1243- AND archive = %s
1244- AND status in (%s, %s)""" %
1245- sqlvalues(binarypackage, self.distroarchseries,
1246- self.distroarchseries.main_archive,
1247- PackagePublishingStatus.PUBLISHED,
1248- PackagePublishingStatus.PENDING),
1249- orderBy=["-datecreated"])
1250+ ret = BinaryPackagePublishingHistory.select("""
1251+ binarypackagerelease = %s AND
1252+ distroarchseries = %s AND
1253+ archive = %s AND
1254+ status in %s""" % sqlvalues(
1255+ binarypackage, self.distroarchseries,
1256+ self.distroarchseries.main_archive, active_publishing_status),
1257+ orderBy=["-datecreated"])
1258 ret = list(ret)
1259 if ret:
1260 return ret[0]
1261
1262=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
1263--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
1264+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:48:24 +0000
1265@@ -1,13 +1,181 @@
1266-# Copyright 2009-2010 Canonical Ltd. This software is licensed under the
1267+# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
1268 # GNU Affero General Public License version 3 (see the file LICENSE).
1269
1270 from doctest import DocTestSuite
1271-import unittest
1272+from unittest import TestLoader
1273
1274+from canonical.testing.layers import ZopelessDatabaseLayer
1275+from lp.registry.interfaces.pocket import PackagePublishingPocket
1276+from lp.services.log.logger import DevNullLogger
1277+from lp.soyuz.enums import PackagePublishingStatus
1278+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
1279 import lp.soyuz.scripts.gina.handlers
1280+from lp.soyuz.scripts.gina.handlers import (
1281+ BinaryPackagePublisher,
1282+ SourcePackagePublisher,
1283+ )
1284+from lp.soyuz.scripts.gina.packages import (
1285+ BinaryPackageData,
1286+ SourcePackageData,
1287+ )
1288+from lp.testing import TestCaseWithFactory
1289+
1290+
1291+class FakePackagesMap:
1292+ def __init__(self, src_map):
1293+ self.src_map = src_map
1294+
1295+
1296+class TestGina(TestCaseWithFactory):
1297+
1298+ layer = ZopelessDatabaseLayer
1299+
1300+ def test_dominate_imported_source_packages_dominates_imports(self):
1301+ # dominate_imported_source_packages dominates the source
1302+ # packages that Gina imports.
1303+ logger = DevNullLogger()
1304+ pub = self.factory.makeSourcePackagePublishingHistory(
1305+ status=PackagePublishingStatus.PUBLISHED)
1306+ series = pub.distroseries
1307+ spr = pub.sourcepackagerelease
1308+ package = spr.sourcepackagename
1309+ dominate_imported_source_packages(
1310+ logger, series.distribution.name, series.name, pub.pocket,
1311+ FakePackagesMap({package.name: []}))
1312+ self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
1313+
1314+ def test_dominate_imported_source_packages_dominates_deletions(self):
1315+ # dominate_imported_source_packages dominates the source
1316+ # packages that have been deleted from the Sources lists that
1317+ # Gina imports.
1318+ series = self.factory.makeDistroSeries()
1319+ pocket = PackagePublishingPocket.RELEASE
1320+ package = self.factory.makeSourcePackageName()
1321+ pubs = [
1322+ self.factory.makeSourcePackagePublishingHistory(
1323+ archive=series.main_archive, distroseries=series,
1324+ pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
1325+ sourcepackagerelease=self.factory.makeSourcePackageRelease(
1326+ sourcepackagename=package, version=version))
1327+ for version in ['1.0', '1.1', '1.1a']]
1328+ logger = DevNullLogger()
1329+ dominate_imported_source_packages(
1330+ logger, series.distribution.name, series.name, pocket,
1331+ FakePackagesMap({}))
1332+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
1333+ # "transitional" domination which supersedes older versions of
1334+ # deleted packages with the last known version. Permanent
1335+ # domination will then mark the last known version as deleted.
1336+ # For permanent domination, the expected outcome is that all
1337+ # these publications will be Deleted (but any pre-existing
1338+ # Superseded publications for older versions will remain
1339+ # Superseded).
1340+ self.assertEqual([
1341+ PackagePublishingStatus.SUPERSEDED,
1342+ PackagePublishingStatus.SUPERSEDED,
1343+ PackagePublishingStatus.PUBLISHED,
1344+ ],
1345+ [pub.status for pub in pubs])
1346+
1347+ def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
1348+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
1349+ # Gina domination, dominate_imported_source_packages turns any
1350+ # remaining Pending SPPHS into Published ones.
1351+ series = self.factory.makeDistroSeries()
1352+ spph = self.factory.makeSourcePackagePublishingHistory(
1353+ distroseries=series, archive=series.main_archive,
1354+ status=PackagePublishingStatus.PENDING)
1355+ spr = spph.sourcepackagerelease
1356+ package_name = spr.sourcepackagename.name
1357+ logger = DevNullLogger()
1358+ dominate_imported_source_packages(
1359+ logger, series.distribution.name, series.name, spph.pocket,
1360+ FakePackagesMap({package_name: [{"Version": spr.version}]}))
1361+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
1362+
1363+ def test_dominate_imported_source_packages_cleans_up_first(self):
1364+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
1365+ # Gina domination, dominate_imported_source_packages turns any
1366+ # remaining Pending SPPHS into Published ones. It does this
1367+ # *before* dominating, so no domination happens while some of
1368+ # the SPPHs are still mistakenly Pending (which would result in
1369+ # mistaken deletions).
1370+ series = self.factory.makeDistroSeries()
1371+ package = self.factory.makeSourcePackageName()
1372+ pocket = PackagePublishingPocket.RELEASE
1373+ versions = ['1.0', '1.1']
1374+ statuses_before = [
1375+ PackagePublishingStatus.PUBLISHED,
1376+ PackagePublishingStatus.PENDING,
1377+ ]
1378+ statuses_after = [
1379+ PackagePublishingStatus.SUPERSEDED,
1380+ PackagePublishingStatus.PUBLISHED,
1381+ ]
1382+ live_version = versions[-1]
1383+ sprs = [
1384+ self.factory.makeSourcePackageRelease(
1385+ sourcepackagename=package, version=version)
1386+ for version in versions]
1387+ spphs = [
1388+ self.factory.makeSourcePackagePublishingHistory(
1389+ archive=series.main_archive, distroseries=series,
1390+ sourcepackagerelease=spr, pocket=pocket, status=status)
1391+ for spr, status in zip(sprs, statuses_before)]
1392+
1393+ logger = DevNullLogger()
1394+ dominate_imported_source_packages(
1395+ logger, series.distribution.name, series.name, pocket,
1396+ FakePackagesMap({package.name: [{"Version": live_version}]}))
1397+
1398+ self.assertEqual(statuses_after, [spph.status for spph in spphs])
1399+
1400+
1401+class TestSourcePackagePublisher(TestCaseWithFactory):
1402+
1403+ layer = ZopelessDatabaseLayer
1404+
1405+ def test_publish_creates_published_publication(self):
1406+ maintainer = self.factory.makePerson()
1407+ series = self.factory.makeDistroSeries()
1408+ section = self.factory.makeSection()
1409+ pocket = PackagePublishingPocket.RELEASE
1410+ spr = self.factory.makeSourcePackageRelease()
1411+
1412+ publisher = SourcePackagePublisher(series, pocket, None)
1413+ publisher.publish(spr, SourcePackageData(
1414+ component='main', section=section.name, version='1.0',
1415+ maintainer=maintainer.preferredemail, architecture='all',
1416+ files='foo.py', binaries='foo.py'))
1417+
1418+ [spph] = series.main_archive.getPublishedSources()
1419+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
1420+
1421+
1422+class TestBinaryPackagePublisher(TestCaseWithFactory):
1423+
1424+ layer = ZopelessDatabaseLayer
1425+
1426+ def test_publish_creates_published_publication(self):
1427+ maintainer = self.factory.makePerson()
1428+ series = self.factory.makeDistroArchSeries()
1429+ section = self.factory.makeSection()
1430+ pocket = PackagePublishingPocket.RELEASE
1431+ bpr = self.factory.makeBinaryPackageRelease()
1432+
1433+ publisher = BinaryPackagePublisher(series, pocket, None)
1434+ publisher.publish(bpr, BinaryPackageData(
1435+ component='main', section=section.name, version='1.0',
1436+ maintainer=maintainer.preferredemail, architecture='all',
1437+ files='foo.py', binaries='foo.py', size=128, installed_size=1024,
1438+ md5sum='e83b5dd68079d727a494a469d40dc8db', description='test',
1439+ summary='Test!'))
1440+
1441+ [bpph] = series.main_archive.getAllPublishedBinaries()
1442+ self.assertEqual(PackagePublishingStatus.PUBLISHED, bpph.status)
1443
1444
1445 def test_suite():
1446- suite = unittest.TestSuite()
1447+ suite = TestLoader().loadTestsFromName(__name__)
1448 suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
1449 return suite
1450
1451=== modified file 'scripts/gina.py'
1452--- scripts/gina.py 2011-08-23 08:35:13 +0000
1453+++ scripts/gina.py 2011-09-09 05:48:24 +0000
1454@@ -38,6 +38,7 @@
1455 MangledArchiveError,
1456 PackagesMap,
1457 )
1458+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
1459 from lp.soyuz.scripts.gina.handlers import (
1460 DataSetupError,
1461 ImporterHandler,
1462@@ -152,6 +153,10 @@
1463 packages_map, kdb, package_root, keyrings, importer_handler)
1464 importer_handler.commit()
1465
1466+ # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
1467+ dominate_imported_source_packages(
1468+ log, distro, distroseries, pocket, packages_map)
1469+
1470 if source_only:
1471 log.info('Source only mode... done')
1472 return
1473@@ -209,9 +214,8 @@
1474 npacks = len(packages_map.src_map)
1475 log.info('%i Source Packages to be imported', npacks)
1476
1477- for list_source in sorted(
1478- packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
1479- for source in list_source:
1480+ for package in sorted(packages_map.src_map.iterkeys()):
1481+ for source in packages_map.src_map[package]:
1482 count += 1
1483 attempt_source_package_import(
1484 source, kdb, package_root, keyrings, importer_handler)
1485@@ -244,10 +248,9 @@
1486 log.info(
1487 '%i Binary Packages to be imported for %s', npacks, archtag)
1488 # Go over binarypackages importing them for this architecture
1489- for binary in sorted(packages_map.bin_map[archtag].values(),
1490- key=lambda x: x.get("Package")):
1491+ for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
1492+ binary = packages_map.bin_map[archtag][package_name]
1493 count += 1
1494- package_name = binary.get("Package", "unknown")
1495 try:
1496 try:
1497 do_one_binarypackage(binary, archtag, kdb, package_root,