Merge lp:~jtv/launchpad/transitional-published into lp:launchpad

Proposed by Jeroen T. Vermeulen
Status: Superseded
Proposed branch: lp:~jtv/launchpad/transitional-published
Merge into: lp:launchpad
Diff against target: 1497 lines (+977/-161)
9 files modified
lib/lp/archivepublisher/domination.py (+233/-60)
lib/lp/archivepublisher/tests/test_dominator.py (+415/-4)
lib/lp/soyuz/doc/gina.txt (+24/-24)
lib/lp/soyuz/interfaces/publishing.py (+0/-3)
lib/lp/soyuz/model/publishing.py (+6/-10)
lib/lp/soyuz/scripts/gina/dominate.py (+82/-0)
lib/lp/soyuz/scripts/gina/handlers.py (+37/-51)
lib/lp/soyuz/scripts/tests/test_gina.py (+171/-3)
scripts/gina.py (+9/-6)
To merge this branch: bzr merge lp:~jtv/launchpad/transitional-published
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+74719@code.launchpad.net

This proposal has been superseded by a proposal from 2011-09-09.

Commit message

Convert any remaining Pending Debian SPPHs before transitional domination.

Description of the change

= Summary =

Gina is learning to run domination on its Debian imports. Yes, technically that makes Gina a dominatrix.

There is a catch: domination operates only on publication records in Published state. So far, Gina has been creating Pending records. That made sense for Ubuntu's import into Launchpad, way back when, which is what Gina was originally written for. It does not make sense now, so we changed the code and updated the legacy data in batches.

Of course that does not take care of further legacy data that is created between the update we just did in the database and the time the Gina changes roll out. But it should reduce the problem's size enough that we can afford to do the final update from inside Gina. This will only need to run once, during ‚Äútransitional domination‚ÄĚ but will do nothing if run repeatedly.

== Pre-implementation notes ==

I had wanted to do a final patch-up update later, but William's points out that there is quite a substantial risk if domination runs on partially updated data. So I inserted the change directly into the code; I have another branch waiting to remove the transitional code.

== Implementation details ==

The diff starts with some modernization of a doctest (using active_publishing_status instead of spelling out the Published and Pending statuses). This is essentially cosmetic; it does not change the meaning of the code and really isn't relevant to the branch. Buried in there however is one change where the test still assumes that Gina produces and maintains Pending publication records. Before we remove transitional domination, we may have to re-think what happens there.

== Tests ==

Two new tests verify transitional behaviour. One is very simple: an active SPPH is Pending but really ought to be Published. Gina dominates and the SPPH's version is found in the simulated Sources list. The Pending SPPH becomes Published.

Another test looks for the doom scenario that might occur when data is not properly migrated: an older release of a package is Published, but the version that is mentioned in the Sources list is still Pending. It is essential that the live version be upgraded to Published before domination, or the dominator would decide that there is no newer published version to dominate the now-obsolete published SPPH, and mark it deleted. Needless to say, the test shows doom being avoided.

{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}

== Demo and Q/A ==

= Launchpad lint =

There is some pre-existing lint, especially in the doctest, that I can't afford to do too much about. It's not just the effort (mostly a matter of running utilities/formatdoctest.py!) but also a matter of keeping the diff small. I'm piling up quite a number of differences in a series of interdependent branches, and this would increase the risk of conflicts disproportionately.

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/domination.py
  lib/lp/soyuz/scripts/gina/dominate.py
  lib/lp/soyuz/model/publishing.py
  lib/lp/soyuz/doc/gina.txt
  lib/lp/soyuz/interfaces/publishing.py
  scripts/gina.py
  lib/lp/archivepublisher/tests/test_dominator.py
  lib/lp/soyuz/scripts/tests/test_gina.py
  lib/lp/soyuz/scripts/gina/handlers.py

./lib/lp/soyuz/doc/gina.txt
     113: narrative exceeds 78 characters.
     162: want exceeds 78 characters.
     179: want exceeds 78 characters.
     189: narrative uses a moin header.
     221: want exceeds 78 characters.
     234: want exceeds 78 characters.
     240: want exceeds 78 characters.
     295: source exceeds 78 characters.
     324: narrative uses a moin header.
     342: narrative exceeds 78 characters.
     354: narrative uses a moin header.
     360: narrative exceeds 78 characters.
     361: narrative exceeds 78 characters.
     459: narrative uses a moin header.
     461: narrative exceeds 78 characters.
     462: narrative exceeds 78 characters.
     477: narrative uses a moin header.
     563: narrative exceeds 78 characters.
     600: narrative uses a moin header.
     657: narrative uses a moin header.
     746: narrative uses a moin header.
     767: narrative uses a moin header.
     780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
     381: E261 at least two spaces before inline comment
     478: E261 at least two spaces before inline comment
     511: E261 at least two spaces before inline comment
     681: E261 at least two spaces before inline comment
     767: E261 at least two spaces before inline comment
./scripts/gina.py
      26: '_pythonpath' imported but unused

To post a comment you must log in.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py 2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py 2011-09-09 05:48:24 +0000
@@ -53,8 +53,6 @@
53__all__ = ['Dominator']53__all__ = ['Dominator']
5454
55from datetime import timedelta55from datetime import timedelta
56import functools
57import operator
5856
59import apt_pkg57import apt_pkg
60from storm.expr import (58from storm.expr import (
@@ -68,7 +66,7 @@
68 flush_database_updates,66 flush_database_updates,
69 sqlvalues,67 sqlvalues,
70 )68 )
71from canonical.launchpad.interfaces.lpstorm import IMasterStore69from canonical.launchpad.interfaces.lpstorm import IStore
72from lp.registry.model.sourcepackagename import SourcePackageName70from lp.registry.model.sourcepackagename import SourcePackageName
73from lp.soyuz.enums import (71from lp.soyuz.enums import (
74 BinaryPackageFormat,72 BinaryPackageFormat,
@@ -87,17 +85,93 @@
87apt_pkg.InitSystem()85apt_pkg.InitSystem()
8886
8987
90def _compare_packages_by_version_and_date(get_release, p1, p2):88def join_spr_spn():
91 """Compare publications p1 and p2 by their version; using Debian rules.89 """Join condition: SourcePackageRelease/SourcePackageName."""
9290 return (
93 If the publications are for the same package, compare by datecreated91 SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
94 instead. This lets newer records win.92
95 """93
96 if get_release(p1).id == get_release(p2).id:94def join_spph_spr():
97 return cmp(p1.datecreated, p2.datecreated)95 """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
9896 """
99 return apt_pkg.VersionCompare(get_release(p1).version,97 # Avoid circular imports.
100 get_release(p2).version)98 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
99
100 return (
101 SourcePackageRelease.id ==
102 SourcePackagePublishingHistory.sourcepackagereleaseID)
103
104
105class SourcePublicationTraits:
106 """Basic generalized attributes for `SourcePackagePublishingHistory`.
107
108 Used by `GeneralizedPublication` to hide the differences from
109 `BinaryPackagePublishingHistory`.
110 """
111 @staticmethod
112 def getPackageName(spph):
113 """Return the name of this publication's source package."""
114 return spph.sourcepackagerelease.sourcepackagename.name
115
116 @staticmethod
117 def getPackageRelease(spph):
118 """Return this publication's `SourcePackageRelease`."""
119 return spph.sourcepackagerelease
120
121
122class BinaryPublicationTraits:
123 """Basic generalized attributes for `BinaryPackagePublishingHistory`.
124
125 Used by `GeneralizedPublication` to hide the differences from
126 `SourcePackagePublishingHistory`.
127 """
128 @staticmethod
129 def getPackageName(bpph):
130 """Return the name of this publication's binary package."""
131 return bpph.binarypackagerelease.binarypackagename.name
132
133 @staticmethod
134 def getPackageRelease(bpph):
135 """Return this publication's `BinaryPackageRelease`."""
136 return bpph.binarypackagerelease
137
138
139class GeneralizedPublication:
140 """Generalize handling of publication records.
141
142 This allows us to write code that can be dealing with either
143 `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
144 without caring which. Differences are abstracted away in a traits
145 class.
146 """
147 def __init__(self, is_source=True):
148 if is_source:
149 self.traits = SourcePublicationTraits
150 else:
151 self.traits = BinaryPublicationTraits
152
153 def getPackageName(self, pub):
154 """Get the package's name."""
155 return self.traits.getPackageName(pub)
156
157 def getPackageVersion(self, pub):
158 """Obtain the version string for a publicaiton record."""
159 return self.traits.getPackageRelease(pub).version
160
161 def compare(self, pub1, pub2):
162 """Compare publications by version.
163
164 If both publications are for the same version, their creation dates
165 break the tie.
166 """
167 version_comparison = apt_pkg.VersionCompare(
168 self.getPackageVersion(pub1), self.getPackageVersion(pub2))
169
170 if version_comparison == 0:
171 # Use dates as tie breaker.
172 return cmp(pub1.datecreated, pub2.datecreated)
173 else:
174 return version_comparison
101175
102176
103class Dominator:177class Dominator:
@@ -116,50 +190,95 @@
116 self.logger = logger190 self.logger = logger
117 self.archive = archive191 self.archive = archive
118192
119 def _dominatePublications(self, pubs):193 def dominatePackage(self, publications, live_versions, generalization):
194 """Dominate publications for a single package.
195
196 Active publications for versions in `live_versions` stay active.
197 Any older versions are marked as superseded by the respective
198 oldest live versions that are newer than the superseded ones.
199
200 Any versions that are newer than anything in `live_versions` are
201 marked as deleted. This should not be possible in Soyuz-native
202 archives, but it can happen during archive imports when the
203 previous latest version of a package has disappeared from the Sources
204 list we import.
205
206 :param publications: Iterable of publications for the same package,
207 in the same archive, series, and pocket, all with status
208 `PackagePublishingStatus.PUBLISHED`.
209 :param live_versions: Iterable of version strings that are still
210 considered live for this package. The given publications will
211 remain active insofar as they represent any of these versions;
212 older publications will be marked as superseded and newer ones
213 as deleted.
214 :param generalization: A `GeneralizedPublication` helper representing
215 the kind of publications these are--source or binary.
216 """
217 # Go through publications from latest version to oldest. This
218 # makes it easy to figure out which release superseded which:
219 # the dominant is always the oldest live release that is newer
220 # than the one being superseded.
221 publications = sorted(
222 publications, cmp=generalization.compare, reverse=True)
223
224 current_dominant = None
225 for pub in publications:
226 if generalization.getPackageVersion(pub) in live_versions:
227 # This publication stays active; if any publications
228 # that follow right after this are to be superseded,
229 # this is the release that they are superseded by.
230 current_dominant = pub
231 elif current_dominant is None:
232 # This publication is no longer live, but there is no
233 # newer version to supersede it either. Therefore it
234 # must be deleted.
235 pub.requestDeletion(None)
236 else:
237 # This publication is superseded. This is what we're
238 # here to do.
239 pub.supersede(current_dominant, logger=self.logger)
240
241 def _dominatePublications(self, pubs, generalization):
120 """Perform dominations for the given publications.242 """Perform dominations for the given publications.
121243
244 Keep the latest published version for each package active,
245 superseding older versions.
246
122 :param pubs: A dict mapping names to a list of publications. Every247 :param pubs: A dict mapping names to a list of publications. Every
123 publication must be PUBLISHED or PENDING, and the first in each248 publication must be PUBLISHED or PENDING, and the first in each
124 list will be treated as dominant (so should be the latest).249 list will be treated as dominant (so should be the latest).
250 :param generalization: A `GeneralizedPublication` helper representing
251 the kind of publications these are--source or binary.
125 """252 """
126 self.logger.debug("Dominating packages...")253 self.logger.debug("Dominating packages...")
127254 for name, publications in pubs.iteritems():
128 for name in pubs.keys():255 assert publications, "Empty list of publications for %s." % name
129 assert pubs[name], (256 # Since this always picks the latest version as the live
130 "Empty list of publications for %s" % name)257 # one, this dominatePackage call will never result in a
131 for pubrec in pubs[name][1:]:258 # deletion.
132 pubrec.supersede(pubs[name][0], logger=self.logger)259 latest_version = generalization.getPackageVersion(publications[0])
133260 self.dominatePackage(
134 def _sortPackages(self, pkglist, is_source=True):261 publications, [latest_version], generalization)
262
263 def _sortPackages(self, pkglist, generalization):
135 """Map out packages by name, and sort by descending version.264 """Map out packages by name, and sort by descending version.
136265
137 :param pkglist: An iterable of `SourcePackagePublishingHistory` or266 :param pkglist: An iterable of `SourcePackagePublishingHistory` or
138 `BinaryPackagePublishingHistory`.267 `BinaryPackagePublishingHistory`.
139 :param is_source: Whether this call involves source package268 :param generalization: A `GeneralizedPublication` helper representing
140 publications. If so, work with `SourcePackagePublishingHistory`.269 the kind of publications these are--source or binary.
141 If not, work with `BinaryPackagepublishingHistory`.270 :return: A dict mapping each package name to a list of publications
142 :return: A dict mapping each package name (as UTF-8 encoded string)271 from `pkglist`, newest first.
143 to a list of publications from `pkglist`, newest first.
144 """272 """
145 self.logger.debug("Sorting packages...")273 self.logger.debug("Sorting packages...")
146274
147 if is_source:
148 get_release = operator.attrgetter("sourcepackagerelease")
149 get_name = operator.attrgetter("sourcepackagename")
150 else:
151 get_release = operator.attrgetter("binarypackagerelease")
152 get_name = operator.attrgetter("binarypackagename")
153
154 outpkgs = {}275 outpkgs = {}
155 for inpkg in pkglist:276 for inpkg in pkglist:
156 key = get_name(get_release(inpkg)).name.encode('utf-8')277 key = generalization.getPackageName(inpkg)
157 outpkgs.setdefault(key, []).append(inpkg)278 outpkgs.setdefault(key, []).append(inpkg)
158279
159 sort_order = functools.partial(
160 _compare_packages_by_version_and_date, get_release)
161 for package_pubs in outpkgs.itervalues():280 for package_pubs in outpkgs.itervalues():
162 package_pubs.sort(cmp=sort_order, reverse=True)281 package_pubs.sort(cmp=generalization.compare, reverse=True)
163282
164 return outpkgs283 return outpkgs
165284
@@ -287,6 +406,8 @@
287 # Avoid circular imports.406 # Avoid circular imports.
288 from lp.soyuz.model.publishing import BinaryPackagePublishingHistory407 from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
289408
409 generalization = GeneralizedPublication(is_source=False)
410
290 for distroarchseries in distroseries.architectures:411 for distroarchseries in distroseries.architectures:
291 self.logger.debug(412 self.logger.debug(
292 "Performing domination across %s/%s (%s)",413 "Performing domination across %s/%s (%s)",
@@ -312,7 +433,7 @@
312 ),433 ),
313 group_by=BinaryPackageName.id,434 group_by=BinaryPackageName.id,
314 having=Count(BinaryPackagePublishingHistory.id) > 1)435 having=Count(BinaryPackagePublishingHistory.id) > 1)
315 binaries = IMasterStore(BinaryPackagePublishingHistory).find(436 binaries = IStore(BinaryPackagePublishingHistory).find(
316 BinaryPackagePublishingHistory,437 BinaryPackagePublishingHistory,
317 BinaryPackageRelease.id ==438 BinaryPackageRelease.id ==
318 BinaryPackagePublishingHistory.binarypackagereleaseID,439 BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +443,21 @@
322 BinaryPackageFormat.DDEB,443 BinaryPackageFormat.DDEB,
323 bpph_location_clauses)444 bpph_location_clauses)
324 self.logger.debug("Dominating binaries...")445 self.logger.debug("Dominating binaries...")
325 self._dominatePublications(self._sortPackages(binaries, False))446 self._dominatePublications(
447 self._sortPackages(binaries, generalization), generalization)
448
449 def _composeActiveSourcePubsCondition(self, distroseries, pocket):
450 """Compose ORM condition for restricting relevant source pubs."""
451 # Avoid circular imports.
452 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
453
454 return And(
455 SourcePackagePublishingHistory.status ==
456 PackagePublishingStatus.PUBLISHED,
457 SourcePackagePublishingHistory.distroseries == distroseries,
458 SourcePackagePublishingHistory.archive == self.archive,
459 SourcePackagePublishingHistory.pocket == pocket,
460 )
326461
327 def dominateSources(self, distroseries, pocket):462 def dominateSources(self, distroseries, pocket):
328 """Perform domination on source package publications.463 """Perform domination on source package publications.
@@ -332,38 +467,76 @@
332 """467 """
333 # Avoid circular imports.468 # Avoid circular imports.
334 from lp.soyuz.model.publishing import SourcePackagePublishingHistory469 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
470
471 generalization = GeneralizedPublication(is_source=True)
472
335 self.logger.debug(473 self.logger.debug(
336 "Performing domination across %s/%s (Source)",474 "Performing domination across %s/%s (Source)",
337 distroseries.name, pocket.title)475 distroseries.name, pocket.title)
338 spph_location_clauses = And(476
339 SourcePackagePublishingHistory.status ==477 spph_location_clauses = self._composeActiveSourcePubsCondition(
340 PackagePublishingStatus.PUBLISHED,478 distroseries, pocket)
341 SourcePackagePublishingHistory.distroseries == distroseries,479 having_multiple_active_publications = (
342 SourcePackagePublishingHistory.archive == self.archive,480 Count(SourcePackagePublishingHistory.id) > 1)
343 SourcePackagePublishingHistory.pocket == pocket,
344 )
345 candidate_source_names = Select(481 candidate_source_names = Select(
346 SourcePackageName.id,482 SourcePackageName.id,
347 And(483 And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
348 SourcePackageRelease.sourcepackagenameID ==
349 SourcePackageName.id,
350 SourcePackagePublishingHistory.sourcepackagereleaseID ==
351 SourcePackageRelease.id,
352 spph_location_clauses,
353 ),
354 group_by=SourcePackageName.id,484 group_by=SourcePackageName.id,
355 having=Count(SourcePackagePublishingHistory.id) > 1)485 having=having_multiple_active_publications)
356 sources = IMasterStore(SourcePackagePublishingHistory).find(486 sources = IStore(SourcePackagePublishingHistory).find(
357 SourcePackagePublishingHistory,487 SourcePackagePublishingHistory,
358 SourcePackageRelease.id ==488 join_spph_spr(),
359 SourcePackagePublishingHistory.sourcepackagereleaseID,
360 SourcePackageRelease.sourcepackagenameID.is_in(489 SourcePackageRelease.sourcepackagenameID.is_in(
361 candidate_source_names),490 candidate_source_names),
362 spph_location_clauses)491 spph_location_clauses)
492
363 self.logger.debug("Dominating sources...")493 self.logger.debug("Dominating sources...")
364 self._dominatePublications(self._sortPackages(sources))494 self._dominatePublications(
495 self._sortPackages(sources, generalization), generalization)
365 flush_database_updates()496 flush_database_updates()
366497
498 def findPublishedSourcePackageNames(self, distroseries, pocket):
499 """Find names of currently published source packages."""
500 result = IStore(SourcePackageName).find(
501 SourcePackageName.name,
502 join_spph_spr(),
503 join_spr_spn(),
504 self._composeActiveSourcePubsCondition(distroseries, pocket))
505 return result.config(distinct=True)
506
507 def findPublishedSPPHs(self, distroseries, pocket, package_name):
508 """Find currently published source publications for given package."""
509 # Avoid circular imports.
510 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
511
512 return IStore(SourcePackagePublishingHistory).find(
513 SourcePackagePublishingHistory,
514 join_spph_spr(),
515 join_spr_spn(),
516 SourcePackageName.name == package_name,
517 self._composeActiveSourcePubsCondition(distroseries, pocket))
518
519 def dominateRemovedSourceVersions(self, distroseries, pocket,
520 package_name, live_versions):
521 """Dominate source publications based on a set of "live" versions.
522
523 Active publications for the "live" versions will remain active. All
524 other active publications for the same package (and the same archive,
525 distroseries, and pocket) are marked superseded.
526
527 Unlike traditional domination, this allows multiple versions of a
528 package to stay active in the same distroseries, archive, and pocket.
529
530 :param distroseries: `DistroSeries` to dominate.
531 :param pocket: `PackagePublishingPocket` to dominate.
532 :param package_name: Source package name, as text.
533 :param live_versions: Iterable of all version strings that are to
534 remain active.
535 """
536 generalization = GeneralizedPublication(is_source=True)
537 pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
538 self.dominatePackage(pubs, live_versions, generalization)
539
367 def judge(self, distroseries, pocket):540 def judge(self, distroseries, pocket):
368 """Judge superseded sources and binaries."""541 """Judge superseded sources and binaries."""
369 # Avoid circular imports.542 # Avoid circular imports.
370543
=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py 2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py 2011-09-09 05:48:24 +0000
@@ -1,4 +1,4 @@
1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Tests for domination.py."""4"""Tests for domination.py."""
@@ -7,12 +7,24 @@
77
8import datetime8import datetime
99
10import apt_pkg
11from zope.security.proxy import removeSecurityProxy
12
10from canonical.database.sqlbase import flush_database_updates13from canonical.database.sqlbase import flush_database_updates
11from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION14from canonical.testing.layers import ZopelessDatabaseLayer
15from lp.archivepublisher.domination import (
16 Dominator,
17 GeneralizedPublication,
18 STAY_OF_EXECUTION,
19 )
12from lp.archivepublisher.publishing import Publisher20from lp.archivepublisher.publishing import Publisher
21from lp.registry.interfaces.pocket import PackagePublishingPocket
13from lp.registry.interfaces.series import SeriesStatus22from lp.registry.interfaces.series import SeriesStatus
23from lp.services.log.logger import DevNullLogger
14from lp.soyuz.enums import PackagePublishingStatus24from lp.soyuz.enums import PackagePublishingStatus
25from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
15from lp.soyuz.tests.test_publishing import TestNativePublishingBase26from lp.soyuz.tests.test_publishing import TestNativePublishingBase
27from lp.testing import TestCaseWithFactory
1628
1729
18class TestDominator(TestNativePublishingBase):30class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
50 foo_10_source, foo_10_binaries[0])62 foo_10_source, foo_10_binaries[0])
5163
52 def dominateAndCheck(self, dominant, dominated, supersededby):64 def dominateAndCheck(self, dominant, dominated, supersededby):
65 generalization = GeneralizedPublication(
66 is_source=ISourcePackagePublishingHistory.providedBy(dominant))
53 dominator = Dominator(self.logger, self.ubuntutest.main_archive)67 dominator = Dominator(self.logger, self.ubuntutest.main_archive)
5468
55 # The _dominate* test methods require a dictionary where the69 # The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
58 # and dominated, the subsequents.72 # and dominated, the subsequents.
59 pubs = {'foo': [dominant, dominated]}73 pubs = {'foo': [dominant, dominated]}
6074
61 dominator._dominatePublications(pubs)75 dominator._dominatePublications(pubs, generalization)
62 flush_database_updates()76 flush_database_updates()
6377
64 # The dominant version remains correctly published.78 # The dominant version remains correctly published.
@@ -145,7 +159,9 @@
145 # This isn't a really good exception. It should probably be159 # This isn't a really good exception. It should probably be
146 # something more indicative of bad input.160 # something more indicative of bad input.
147 self.assertRaises(161 self.assertRaises(
148 AssertionError, dominator._dominatePublications, pubs)162 AssertionError,
163 dominator._dominatePublications,
164 pubs, GeneralizedPublication(True))
149165
150166
151class TestDomination(TestNativePublishingBase):167class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,398 @@
200 TestDomination.setUp(self)216 TestDomination.setUp(self)
201 self.ubuntutest['breezy-autotest'].status = (217 self.ubuntutest['breezy-autotest'].status = (
202 SeriesStatus.OBSOLETE)218 SeriesStatus.OBSOLETE)
219
220
221def make_spphs_for_versions(factory, versions):
222 """Create publication records for each of `versions`.
223
224 They records are created in the same order in which they are specified.
225 Make the order irregular to prove that version ordering is not a
226 coincidence of object creation order etc.
227
228 Versions may also be identical; each publication record will still have
229 its own package release.
230 """
231 spn = factory.makeSourcePackageName()
232 distroseries = factory.makeDistroSeries()
233 pocket = factory.getAnyPocket()
234 sprs = [
235 factory.makeSourcePackageRelease(
236 sourcepackagename=spn, version=version)
237 for version in versions]
238 return [
239 factory.makeSourcePackagePublishingHistory(
240 distroseries=distroseries, pocket=pocket,
241 sourcepackagerelease=spr,
242 status=PackagePublishingStatus.PUBLISHED)
243 for spr in sprs]
244
245
246def list_source_versions(spphs):
247 """Extract the versions from `spphs` as a list, in the same order."""
248 return [spph.sourcepackagerelease.version for spph in spphs]
249
250
251class TestGeneralizedPublication(TestCaseWithFactory):
252 """Test publication generalization helpers."""
253
254 layer = ZopelessDatabaseLayer
255
256 def alterCreationDates(self, spphs, ages):
257 """Set `datecreated` on each of `spphs` according to `ages`.
258
259 :param spphs: Iterable of `SourcePackagePublishingHistory`. Their
260 respective creation dates will be offset by the respective ages
261 found in `ages` (with the two being matched up in the same order).
262 :param ages: Iterable of ages. Must provide the same number of items
263 as `spphs`. Ages are `timedelta` objects that will be subtracted
264 from the creation dates on the respective records in `spph`.
265 """
266 for spph, age in zip(spphs, ages):
267 spph.datecreated -= age
268
269 def test_getPackageVersion_gets_source_version(self):
270 spph = self.factory.makeSourcePackagePublishingHistory()
271 self.assertEqual(
272 spph.sourcepackagerelease.version,
273 GeneralizedPublication(is_source=True).getPackageVersion(spph))
274
275 def test_getPackageVersion_gets_binary_version(self):
276 bpph = self.factory.makeBinaryPackagePublishingHistory()
277 self.assertEqual(
278 bpph.binarypackagerelease.version,
279 GeneralizedPublication(is_source=False).getPackageVersion(bpph))
280
281 def test_compare_sorts_versions(self):
282 versions = [
283 '1.1v2',
284 '1.1v1',
285 '1.1v3',
286 ]
287 spphs = make_spphs_for_versions(self.factory, versions)
288 sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
289 self.assertEqual(
290 sorted(versions), list_source_versions(sorted_spphs))
291
292 def test_compare_orders_versions_by_debian_rules(self):
293 versions = [
294 '1.1.0',
295 '1.10',
296 '1.1',
297 '1.1ubuntu0',
298 ]
299 spphs = make_spphs_for_versions(self.factory, versions)
300
301 debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
302
303 # Assumption: in this case, Debian version ordering is not the
304 # same as alphabetical version ordering.
305 self.assertNotEqual(sorted(versions), debian_sorted_versions)
306
307 # The compare method produces the Debian ordering.
308 sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
309 self.assertEqual(
310 sorted(versions, cmp=apt_pkg.VersionCompare),
311 list_source_versions(sorted_spphs))
312
313 def test_compare_breaks_tie_with_creation_date(self):
314 # When two publications are tied for comparison because they are
315 # for the same package release, they are ordered by creation
316 # date.
317 distroseries = self.factory.makeDistroSeries()
318 pocket = self.factory.getAnyPocket()
319 spr = self.factory.makeSourcePackageRelease()
320 ages = [
321 datetime.timedelta(2),
322 datetime.timedelta(1),
323 datetime.timedelta(3),
324 ]
325 spphs = [
326 self.factory.makeSourcePackagePublishingHistory(
327 sourcepackagerelease=spr, distroseries=distroseries,
328 pocket=pocket)
329 for counter in xrange(len(ages))]
330 self.alterCreationDates(spphs, ages)
331
332 self.assertEqual(
333 [spphs[2], spphs[0], spphs[1]],
334 sorted(spphs, cmp=GeneralizedPublication().compare))
335
336 def test_compare_breaks_tie_for_releases_with_same_version(self):
337 # When two publications are tied for comparison because they
338 # belong to releases with the same version string, they are
339 # ordered by creation date.
340 version = "1.%d" % self.factory.getUniqueInteger()
341 ages = [
342 datetime.timedelta(2),
343 datetime.timedelta(1),
344 datetime.timedelta(3),
345 ]
346 distroseries = self.factory.makeDistroSeries()
347 pocket = self.factory.getAnyPocket()
348 spphs = [
349 self.factory.makeSourcePackagePublishingHistory(
350 distroseries=distroseries, pocket=pocket,
351 sourcepackagerelease=self.factory.makeSourcePackageRelease(
352 version=version))
353 for counter in xrange(len(ages))]
354 self.alterCreationDates(spphs, ages)
355
356 self.assertEqual(
357 [spphs[2], spphs[0], spphs[1]],
358 sorted(spphs, cmp=GeneralizedPublication().compare))
359
360
361class TestDominatorMethods(TestCaseWithFactory):
362
363 layer = ZopelessDatabaseLayer
364
365 def makeDominator(self, publications):
366 """Create a `Dominator` suitable for `publications`."""
367 if len(publications) == 0:
368 archive = self.factory.makeArchive()
369 else:
370 archive = publications[0].archive
371 return Dominator(DevNullLogger(), archive)
372
373 def test_dominatePackage_survives_empty_publications_list(self):
374 # Nothing explodes when dominatePackage is called with an empty
375 # packages list.
376 self.makeDominator([]).dominatePackage(
377 [], [], GeneralizedPublication(True))
378 # The test is that we get here without error.
379 pass
380
381 def test_dominatePackage_leaves_live_version_untouched(self):
382 # dominatePackage does not supersede live versions.
383 [pub] = make_spphs_for_versions(self.factory, ['3.1'])
384 self.makeDominator([pub]).dominatePackage(
385 [pub], ['3.1'], GeneralizedPublication(True))
386 self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
387
388 def test_dominatePackage_deletes_dead_version_without_successor(self):
389 # dominatePackage marks non-live package versions without
390 # superseding versions as deleted.
391 [pub] = make_spphs_for_versions(self.factory, ['1.1'])
392 self.makeDominator([pub]).dominatePackage(
393 [pub], [], GeneralizedPublication(True))
394 self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
395
396 def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
397 # When marking a package as superseded, dominatePackage
398 # designates a newer live version as the superseding version.
399 pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
400 self.makeDominator(pubs).dominatePackage(
401 pubs, ['1.1'], GeneralizedPublication(True))
402 self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
403 self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
404 self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
405
406 def test_dominatePackage_only_supersedes_with_live_pub(self):
407 # When marking a package as superseded, dominatePackage will
408 # only pick a live version as the superseding one.
409 pubs = make_spphs_for_versions(
410 self.factory, ['1.0', '2.0', '3.0', '4.0'])
411 self.makeDominator(pubs).dominatePackage(
412 pubs, ['3.0'], GeneralizedPublication(True))
413 self.assertEqual([
414 pubs[2].sourcepackagerelease,
415 pubs[2].sourcepackagerelease,
416 None,
417 None,
418 ],
419 [pub.supersededby for pub in pubs])
420
421 def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
422 # When marking a package as superseded, dominatePackage picks
423 # the oldest of the newer, live versions as the superseding one.
424 pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
425 self.makeDominator(pubs).dominatePackage(
426 pubs, ['2.8', '2.9'], GeneralizedPublication(True))
427 self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
428
429 def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
430 # When marking a package as superseded, dominatePackage only
431 # considers a newer version as the superseding one.
432 pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
433 self.makeDominator(pubs).dominatePackage(
434 pubs, ['0.1'], GeneralizedPublication(True))
435 self.assertEqual(None, pubs[1].supersededby)
436 self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
437
438 def test_dominateRemovedSourceVersions_dominates_publications(self):
439 # dominateRemovedSourceVersions finds the publications for a
440 # package and calls dominatePackage on them.
441 pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
442 package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
443
444 self.makeDominator(pubs).dominateRemovedSourceVersions(
445 pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
446 self.assertEqual([
447 PackagePublishingStatus.SUPERSEDED,
448 PackagePublishingStatus.PUBLISHED,
449 PackagePublishingStatus.DELETED,
450 ],
451 [pub.status for pub in pubs])
452 self.assertEqual(
453 [pubs[1].sourcepackagerelease, None, None],
454 [pub.supersededby for pub in pubs])
455
456 def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
457 # dominateRemovedSourceVersions ignores publications in other
458 # pockets than the one specified.
459 pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
460 package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
461 removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
462 removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
463 self.makeDominator(pubs).dominateRemovedSourceVersions(
464 pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
465 self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
466
467 def test_dominateRemovedSourceVersions_ignores_other_packages(self):
468 pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
469 other_package_name = self.factory.makeSourcePackageName().name
470 self.makeDominator(pubs).dominateRemovedSourceVersions(
471 pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
472 self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
473
474 def test_findPublishedSourcePackageNames_finds_package(self):
475 spph = self.factory.makeSourcePackagePublishingHistory(
476 status=PackagePublishingStatus.PUBLISHED)
477 dominator = self.makeDominator([spph])
478 self.assertContentEqual(
479 [spph.sourcepackagerelease.sourcepackagename.name],
480 dominator.findPublishedSourcePackageNames(
481 spph.distroseries, spph.pocket))
482
483 def test_findPublishedSourcePackageNames_ignores_other_states(self):
484 series = self.factory.makeDistroSeries()
485 pocket = PackagePublishingPocket.RELEASE
486 spphs = dict(
487 (status, self.factory.makeSourcePackagePublishingHistory(
488 distroseries=series, archive=series.main_archive,
489 pocket=pocket, status=status))
490 for status in PackagePublishingStatus.items)
491 published_spph = spphs[PackagePublishingStatus.PUBLISHED]
492 dominator = self.makeDominator(spphs.values())
493 self.assertContentEqual(
494 [published_spph.sourcepackagerelease.sourcepackagename.name],
495 dominator.findPublishedSourcePackageNames(series, pocket))
496
497 def test_findPublishedSourcePackageNames_ignores_other_archives(self):
498 spph = self.factory.makeSourcePackagePublishingHistory(
499 status=PackagePublishingStatus.PUBLISHED)
500 dominator = self.makeDominator([spph])
501 dominator.archive = self.factory.makeArchive()
502 self.assertContentEqual(
503 [],
504 dominator.findPublishedSourcePackageNames(
505 spph.distroseries, spph.pocket))
506
507 def test_findPublishedSourcePackageNames_ignores_other_series(self):
508 spph = self.factory.makeSourcePackagePublishingHistory(
509 status=PackagePublishingStatus.PUBLISHED)
510 distro = spph.distroseries.distribution
511 other_series = self.factory.makeDistroSeries(distribution=distro)
512 dominator = self.makeDominator([spph])
513 self.assertContentEqual(
514 [],
515 dominator.findPublishedSourcePackageNames(
516 other_series, spph.pocket))
517
518 def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
519 spph = self.factory.makeSourcePackagePublishingHistory(
520 status=PackagePublishingStatus.PUBLISHED,
521 pocket=PackagePublishingPocket.RELEASE)
522 dominator = self.makeDominator([spph])
523 self.assertContentEqual(
524 [],
525 dominator.findPublishedSourcePackageNames(
526 spph.distroseries, PackagePublishingPocket.SECURITY))
527
528 def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
529 series = self.factory.makeDistroSeries()
530 pocket = PackagePublishingPocket.RELEASE
531 package = self.factory.makeSourcePackageName()
532 spphs = [
533 self.factory.makeSourcePackagePublishingHistory(
534 distroseries=series, archive=series.main_archive,
535 pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
536 sourcepackagerelease=self.factory.makeSourcePackageRelease(
537 sourcepackagename=package))
538 for counter in xrange(2)]
539 dominator = self.makeDominator(spphs)
540 self.assertEqual(
541 [package.name],
542 list(dominator.findPublishedSourcePackageNames(series, pocket)))
543
544 def test_findPublishedSPPHs_finds_published_SPPH(self):
545 spph = self.factory.makeSourcePackagePublishingHistory(
546 status=PackagePublishingStatus.PUBLISHED)
547 package_name = spph.sourcepackagerelease.sourcepackagename.name
548 dominator = self.makeDominator([spph])
549 self.assertContentEqual(
550 [spph],
551 dominator.findPublishedSPPHs(
552 spph.distroseries, spph.pocket, package_name))
553
554 def test_findPublishedSPPHs_ignores_other_states(self):
555 series = self.factory.makeDistroSeries()
556 package = self.factory.makeSourcePackageName()
557 pocket = PackagePublishingPocket.RELEASE
558 spphs = dict(
559 (status, self.factory.makeSourcePackagePublishingHistory(
560 distroseries=series, archive=series.main_archive,
561 pocket=pocket, status=status,
562 sourcepackagerelease=self.factory.makeSourcePackageRelease(
563 sourcepackagename=package)))
564 for status in PackagePublishingStatus.items)
565 dominator = self.makeDominator(spphs.values())
566 self.assertContentEqual(
567 [spphs[PackagePublishingStatus.PUBLISHED]],
568 dominator.findPublishedSPPHs(series, pocket, package.name))
569
570 def test_findPublishedSPPHs_ignores_other_archives(self):
571 spph = self.factory.makeSourcePackagePublishingHistory(
572 status=PackagePublishingStatus.PUBLISHED)
573 package = spph.sourcepackagerelease.sourcepackagename
574 dominator = self.makeDominator([spph])
575 dominator.archive = self.factory.makeArchive()
576 self.assertContentEqual(
577 [],
578 dominator.findPublishedSPPHs(
579 spph.distroseries, spph.pocket, package.name))
580
581 def test_findPublishedSPPHs_ignores_other_series(self):
582 spph = self.factory.makeSourcePackagePublishingHistory(
583 status=PackagePublishingStatus.PUBLISHED)
584 distro = spph.distroseries.distribution
585 package = spph.sourcepackagerelease.sourcepackagename
586 other_series = self.factory.makeDistroSeries(distribution=distro)
587 dominator = self.makeDominator([spph])
588 self.assertContentEqual(
589 [],
590 dominator.findPublishedSPPHs(
591 other_series, spph.pocket, package.name))
592
593 def test_findPublishedSPPHs_ignores_other_pockets(self):
594 spph = self.factory.makeSourcePackagePublishingHistory(
595 status=PackagePublishingStatus.PUBLISHED,
596 pocket=PackagePublishingPocket.RELEASE)
597 package = spph.sourcepackagerelease.sourcepackagename
598 dominator = self.makeDominator([spph])
599 self.assertContentEqual(
600 [],
601 dominator.findPublishedSPPHs(
602 spph.distroseries, PackagePublishingPocket.SECURITY,
603 package.name))
604
605 def test_findPublishedSPPHs_ignores_other_packages(self):
606 spph = self.factory.makeSourcePackagePublishingHistory(
607 status=PackagePublishingStatus.PUBLISHED)
608 other_package = self.factory.makeSourcePackageName()
609 dominator = self.makeDominator([spph])
610 self.assertContentEqual(
611 [],
612 dominator.findPublishedSPPHs(
613 spph.distroseries, spph.pocket, other_package.name))
203614
=== modified file 'lib/lp/soyuz/doc/gina.txt'
--- lib/lp/soyuz/doc/gina.txt 2011-07-29 11:35:28 +0000
+++ lib/lp/soyuz/doc/gina.txt 2011-09-09 05:48:24 +0000
@@ -8,6 +8,7 @@
8Get the current counts of stuff in the database:8Get the current counts of stuff in the database:
99
10 >>> from canonical.launchpad.database.emailaddress import EmailAddress10 >>> from canonical.launchpad.database.emailaddress import EmailAddress
11 >>> from lp.soyuz.interfaces.publishing import active_publishing_status
11 >>> from lp.soyuz.model.publishing import (12 >>> from lp.soyuz.model.publishing import (
12 ... BinaryPackagePublishingHistory,13 ... BinaryPackagePublishingHistory,
13 ... SourcePackagePublishingHistory)14 ... SourcePackagePublishingHistory)
@@ -564,35 +565,34 @@
564that's what overrides actually do.565that's what overrides actually do.
565566
566 >>> from canonical.database.sqlbase import sqlvalues567 >>> from canonical.database.sqlbase import sqlvalues
567 >>> from lp.soyuz.enums import PackagePublishingStatus568 >>> x11_pub = SSPPH.select("""
568 >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s569 ... sourcepackagerelease = %s AND
569 ... AND distroseries = %s570 ... distroseries = %s AND
570 ... AND status in (%s, %s)""" %571 ... status in %s
571 ... sqlvalues(x11p, breezy,572 ... """ % sqlvalues(
572 ... PackagePublishingStatus.PUBLISHED,573 ... x11p, breezy, active_publishing_status),
573 ... PackagePublishingStatus.PENDING),574 ... orderBy=["-datecreated"])[0]
574 ... orderBy=["-datecreated"])[0]
575 >>> print x11_pub.section.name575 >>> print x11_pub.section.name
576 net576 net
577 >>> ed_pub = SBPPH.select("""binarypackagerelease = %s577 >>> ed_pub = SBPPH.select("""
578 ... AND distroarchseries = %s578 ... binarypackagerelease = %s AND
579 ... AND status in (%s, %s)""" %579 ... distroarchseries = %s AND
580 ... sqlvalues(ed, breezy_i386,580 ... status in %s
581 ... PackagePublishingStatus.PUBLISHED,581 ... """ % sqlvalues(
582 ... PackagePublishingStatus.PENDING),582 ... ed, breezy_i386, active_publishing_status),
583 ... orderBy=["-datecreated"])[0]583 ... orderBy=["-datecreated"])[0]
584 >>> print ed_pub.priority584 >>> print ed_pub.priority
585 Extra585 Extra
586 >>> n = SourcePackageName.selectOneBy(name="archive-copier")586 >>> n = SourcePackageName.selectOneBy(name="archive-copier")
587 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,587 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
588 ... version="0.3.6")588 ... version="0.3.6")
589 >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s589 >>> ac_pub = SSPPH.select("""
590 ... AND distroseries = %s590 ... sourcepackagerelease = %s AND
591 ... AND status in (%s, %s)""" %591 ... distroseries = %s AND
592 ... sqlvalues(ac, breezy,592 ... status in %s
593 ... PackagePublishingStatus.PUBLISHED,593 ... """ % sqlvalues(
594 ... PackagePublishingStatus.PENDING),594 ... ac, breezy, active_publishing_status),
595 ... orderBy=["-datecreated"])[0]595 ... orderBy=["-datecreated"])[0]
596 >>> print ac_pub.component.name596 >>> print ac_pub.component.name
597 universe597 universe
598598
@@ -720,7 +720,7 @@
720720
721 >>> transaction.commit()721 >>> transaction.commit()
722722
723There is now a number of source publications in PENDING status for the723There is now a number of source publications in PUBLISHED status for the
724targetted distroseries, 'lenny'.724targetted distroseries, 'lenny'.
725725
726 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))726 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
@@ -728,7 +728,7 @@
728 12728 12
729729
730 >>> print set([pub.status.name for pub in lenny_sources])730 >>> print set([pub.status.name for pub in lenny_sources])
731 set(['PENDING'])731 set(['PUBLISHED'])
732732
733As mentioned before, lenny/i386 is empty, no binaries were imported.733As mentioned before, lenny/i386 is empty, no binaries were imported.
734Also, the number of binaries published in the whole debian distribution734Also, the number of binaries published in the whole debian distribution
735735
=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py 2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py 2011-09-09 05:48:24 +0000
@@ -195,9 +195,6 @@
195 the field name and value is the value string.195 the field name and value is the value string.
196 """196 """
197197
198 def supersede():
199 """Supersede this publication."""
200
201 def requestObsolescence():198 def requestObsolescence():
202 """Make this publication obsolete.199 """Make this publication obsolete.
203200
204201
=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py 2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py 2011-09-09 05:48:24 +0000
@@ -114,10 +114,6 @@
114from lp.soyuz.scripts.changeoverride import ArchiveOverriderError114from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
115115
116116
117PENDING = PackagePublishingStatus.PENDING
118PUBLISHED = PackagePublishingStatus.PUBLISHED
119
120
121# XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py117# XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py
122118
123def makePoolPath(source_name, component_name):119def makePoolPath(source_name, component_name):
@@ -327,8 +323,8 @@
327 fields = self.buildIndexStanzaFields()323 fields = self.buildIndexStanzaFields()
328 return fields.makeOutput()324 return fields.makeOutput()
329325
330 def supersede(self):326 def setSuperseded(self):
331 """See `IPublishing`."""327 """Set to SUPERSEDED status."""
332 self.status = PackagePublishingStatus.SUPERSEDED328 self.status = PackagePublishingStatus.SUPERSEDED
333 self.datesuperseded = UTC_NOW329 self.datesuperseded = UTC_NOW
334330
@@ -742,7 +738,7 @@
742 "Should not dominate unpublished source %s" %738 "Should not dominate unpublished source %s" %
743 self.sourcepackagerelease.title)739 self.sourcepackagerelease.title)
744740
745 super(SourcePackagePublishingHistory, self).supersede()741 self.setSuperseded()
746742
747 if dominant is not None:743 if dominant is not None:
748 if logger is not None:744 if logger is not None:
@@ -1081,7 +1077,7 @@
1081 return IMasterStore(BinaryPackagePublishingHistory).find(1077 return IMasterStore(BinaryPackagePublishingHistory).find(
1082 BinaryPackagePublishingHistory,1078 BinaryPackagePublishingHistory,
1083 BinaryPackagePublishingHistory.status.is_in(1079 BinaryPackagePublishingHistory.status.is_in(
1084 [PUBLISHED, PENDING]),1080 active_publishing_status),
1085 BinaryPackagePublishingHistory.distroarchseriesID.is_in(1081 BinaryPackagePublishingHistory.distroarchseriesID.is_in(
1086 available_architectures),1082 available_architectures),
1087 binarypackagerelease=self.binarypackagerelease,1083 binarypackagerelease=self.binarypackagerelease,
@@ -1101,7 +1097,7 @@
1101 return IMasterStore(BinaryPackagePublishingHistory).find(1097 return IMasterStore(BinaryPackagePublishingHistory).find(
1102 BinaryPackagePublishingHistory,1098 BinaryPackagePublishingHistory,
1103 BinaryPackagePublishingHistory.status.is_in(1099 BinaryPackagePublishingHistory.status.is_in(
1104 [PUBLISHED, PENDING]),1100 active_publishing_status),
1105 BinaryPackagePublishingHistory.distroarchseries ==1101 BinaryPackagePublishingHistory.distroarchseries ==
1106 self.distroarchseries,1102 self.distroarchseries,
1107 binarypackagerelease=self.binarypackagerelease.debug_package,1103 binarypackagerelease=self.binarypackagerelease.debug_package,
@@ -1126,7 +1122,7 @@
1126 self.distroarchseries.architecturetag))1122 self.distroarchseries.architecturetag))
1127 return1123 return
11281124
1129 super(BinaryPackagePublishingHistory, self).supersede()1125 self.setSuperseded()
11301126
1131 if dominant is not None:1127 if dominant is not None:
1132 # DDEBs cannot themselves be dominant; they are always dominated1128 # DDEBs cannot themselves be dominant; they are always dominated
11331129
=== added file 'lib/lp/soyuz/scripts/gina/dominate.py'
--- lib/lp/soyuz/scripts/gina/dominate.py 1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:48:24 +0000
@@ -0,0 +1,82 @@
1# Copyright 2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Retirement of packages that are removed upstream."""
5
6__metaclass__ = type
7__all__ = [
8 'dominate_imported_source_packages',
9 ]
10
11from zope.component import getUtility
12
13# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
14# import violates import policy and elicits a warning from the test
15# suite. The warning helps remind us to retire this code as soon as
16# possible.
17from lp.archivepublisher.domination import (
18 Dominator,
19 GeneralizedPublication,
20 )
21from lp.registry.interfaces.distribution import IDistributionSet
22
23
24def dominate_imported_source_packages(logger, distro_name, series_name,
25 pocket, packages_map):
26 """Perform domination."""
27 series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
28 dominator = Dominator(logger, series.main_archive)
29
30 # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
31 # hack. Gina used to create SPPHs in Pending state. We cleaned up
32 # the bulk of them, and changed the code to create Published ones, but
33 # some new ones will have been created since.
34 # Update those to match what the new Gina does.
35 from canonical.launchpad.interfaces.lpstorm import IStore
36 from lp.soyuz.enums import PackagePublishingStatus
37 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
38 SPPH = SourcePackagePublishingHistory
39 store = IStore(SPPH)
40 spphs = store.find(
41 SPPH,
42 SPPH.archive == series.main_archive,
43 SPPH.distroseries == series,
44 SPPH.pocket == pocket,
45 SPPH.status == PackagePublishingStatus.PENDING)
46 spphs.set(status=PackagePublishingStatus.PUBLISHED)
47
48 # Dominate packages found in the Sources list we're importing.
49 package_names = dominator.findPublishedSourcePackageNames(series, pocket)
50 for package_name in package_names:
51 entries = packages_map.src_map.get(package_name)
52
53 if entries is None:
54 # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
55 # transitional hack. The database is full of "Published"
56 # Debian SPPHs whose packages have actually been deleted.
57 # In the future such publications should simply be marked
58 # Deleted, but for the legacy baggage we currently carry
59 # around we'll just do traditional domination first: pick
60 # the latest Published version, and mark the rest of the
61 # SPPHs as superseded by that version. The latest version
62 # will then, finally, be marked appropriately Deleted once
63 # we remove this transitional hack.
64 # To remove the transitional hack, just let live_versions
65 # default to the empty list instead of doing this:
66 pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
67 generalization = GeneralizedPublication(is_source=True)
68 pubs_dict = dominator._sortPackages(pubs, generalization)
69 sorted_pubs = pubs_dict[package_name]
70 if len(sorted_pubs) <= 1:
71 # If there's only one published SPPH, the transitional
72 # code will just leave it Published. Don't bother; the
73 # migration will be costly enough as it is.
74 continue
75 live_versions = [sorted_pubs[0].sourcepackagerelease.version]
76 else:
77 live_versions = [
78 entry['Version']
79 for entry in entries if 'Version' in entry]
80
81 dominator.dominateRemovedSourceVersions(
82 series, pocket, package_name, live_versions)
083
=== modified file 'lib/lp/soyuz/scripts/gina/handlers.py'
--- lib/lp/soyuz/scripts/gina/handlers.py 2011-05-20 07:43:58 +0000
+++ lib/lp/soyuz/scripts/gina/handlers.py 2011-09-09 05:48:24 +0000
@@ -35,12 +35,12 @@
35from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet35from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
36from canonical.launchpad.scripts import log36from canonical.launchpad.scripts import log
37from lp.archivepublisher.diskpool import poolify37from lp.archivepublisher.diskpool import poolify
38from lp.archiveuploader.changesfile import ChangesFile
38from lp.archiveuploader.tagfiles import parse_tagfile39from lp.archiveuploader.tagfiles import parse_tagfile
39from lp.archiveuploader.utils import (40from lp.archiveuploader.utils import (
40 determine_binary_file_type,41 determine_binary_file_type,
41 determine_source_file_type,42 determine_source_file_type,
42 )43 )
43from lp.archiveuploader.changesfile import ChangesFile
44from lp.buildmaster.enums import BuildStatus44from lp.buildmaster.enums import BuildStatus
45from lp.registry.interfaces.person import (45from lp.registry.interfaces.person import (
46 IPersonSet,46 IPersonSet,
@@ -54,7 +54,10 @@
54 )54 )
55from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet55from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
56from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet56from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
57from lp.soyuz.interfaces.publishing import IPublishingSet57from lp.soyuz.interfaces.publishing import (
58 active_publishing_status,
59 IPublishingSet,
60 )
58from lp.soyuz.model.component import Component61from lp.soyuz.model.component import Component
59from lp.soyuz.model.files import (62from lp.soyuz.model.files import (
60 BinaryPackageFile,63 BinaryPackageFile,
@@ -723,8 +726,6 @@
723 source_publishinghistory.status.title)726 source_publishinghistory.status.title)
724 return727 return
725728
726 # Create the Publishing entry, with status PENDING so that we
727 # can republish this later into a Soyuz archive.
728 entry = getUtility(IPublishingSet).newSourcePublication(729 entry = getUtility(IPublishingSet).newSourcePublication(
729 distroseries=self.distroseries,730 distroseries=self.distroseries,
730 sourcepackagerelease=sourcepackagerelease,731 sourcepackagerelease=sourcepackagerelease,
@@ -732,6 +733,7 @@
732 section=section,733 section=section,
733 pocket=self.pocket,734 pocket=self.pocket,
734 archive=archive)735 archive=archive)
736 entry.setPublished()
735 log.info('Source package %s (%s) published' % (737 log.info('Source package %s (%s) published' % (
736 entry.sourcepackagerelease.sourcepackagename.name,738 entry.sourcepackagerelease.sourcepackagename.name,
737 entry.sourcepackagerelease.version))739 entry.sourcepackagerelease.version))
@@ -742,16 +744,14 @@
742 from lp.soyuz.model.publishing import (744 from lp.soyuz.model.publishing import (
743 SourcePackagePublishingHistory)745 SourcePackagePublishingHistory)
744746
745 ret = SourcePackagePublishingHistory.select(747 ret = SourcePackagePublishingHistory.select("""
746 """sourcepackagerelease = %s748 sourcepackagerelease = %s AND
747 AND distroseries = %s749 distroseries = %s AND
748 AND archive = %s750 archive = %s AND
749 AND status in (%s, %s)""" %751 status in %s""" % sqlvalues(
750 sqlvalues(sourcepackagerelease, self.distroseries,752 sourcepackagerelease, self.distroseries,
751 self.distroseries.main_archive,753 self.distroseries.main_archive, active_publishing_status),
752 PackagePublishingStatus.PUBLISHED,754 orderBy=["-datecreated"])
753 PackagePublishingStatus.PENDING),
754 orderBy=["-datecreated"])
755 ret = list(ret)755 ret = list(ret)
756 if ret:756 if ret:
757 return ret[0]757 return ret[0]
@@ -917,14 +917,6 @@
917 "for package %s (%s)" %917 "for package %s (%s)" %
918 (build.id, binary.package, binary.version))918 (build.id, binary.package, binary.version))
919 else:919 else:
920
921 # XXX Debonzi 2005-05-16: Check it later
922 # if bin.gpg_signing_key_owner:
923 # key = self.getGPGKey(bin.gpg_signing_key,
924 # *bin.gpg_signing_key_owner)
925 # else:
926 key = None
927
928 processor = distroarchinfo['processor']920 processor = distroarchinfo['processor']
929 build = getUtility(IBinaryPackageBuildSet).new(921 build = getUtility(IBinaryPackageBuildSet).new(
930 processor=processor.id,922 processor=processor.id,
@@ -948,8 +940,7 @@
948 def publish(self, binarypackage, bpdata):940 def publish(self, binarypackage, bpdata):
949 """Create the publishing entry on db if does not exist."""941 """Create the publishing entry on db if does not exist."""
950 # Avoid circular imports.942 # Avoid circular imports.
951 from lp.soyuz.model.publishing import (943 from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
952 BinaryPackagePublishingHistory)
953944
954 # These need to be pulled from the binary package data, not the945 # These need to be pulled from the binary package data, not the
955 # binary package release: the data represents data from /this946 # binary package release: the data represents data from /this
@@ -983,22 +974,20 @@
983 binpkg_publishinghistory.status.title)974 binpkg_publishinghistory.status.title)
984 return975 return
985976
986
987 # Create the Publishing entry with status PENDING.
988 BinaryPackagePublishingHistory(977 BinaryPackagePublishingHistory(
989 binarypackagerelease = binarypackage.id,978 binarypackagerelease=binarypackage.id,
990 component = component.id,979 component=component.id,
991 section = section.id,980 section=section.id,
992 priority = priority,981 priority=priority,
993 distroarchseries = self.distroarchseries.id,982 distroarchseries=self.distroarchseries.id,
994 status = PackagePublishingStatus.PENDING,983 status=PackagePublishingStatus.PUBLISHED,
995 datecreated = UTC_NOW,984 datecreated=UTC_NOW,
996 datepublished = UTC_NOW,985 datepublished=UTC_NOW,
997 pocket = self.pocket,986 pocket=self.pocket,
998 datesuperseded = None,987 datesuperseded=None,
999 supersededby = None,988 supersededby=None,
1000 datemadepending = None,989 datemadepending=None,
1001 dateremoved = None,990 dateremoved=None,
1002 archive=archive)991 archive=archive)
1003992
1004 log.info('BinaryPackage %s-%s published into %s.' % (993 log.info('BinaryPackage %s-%s published into %s.' % (
@@ -1008,19 +997,16 @@
1008 def _checkPublishing(self, binarypackage):997 def _checkPublishing(self, binarypackage):
1009 """Query for the publishing entry"""998 """Query for the publishing entry"""
1010 # Avoid circular imports.999 # Avoid circular imports.
1011 from lp.soyuz.model.publishing import (1000 from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
1012 BinaryPackagePublishingHistory)
10131001
1014 ret = BinaryPackagePublishingHistory.select(1002 ret = BinaryPackagePublishingHistory.select("""
1015 """binarypackagerelease = %s1003 binarypackagerelease = %s AND
1016 AND distroarchseries = %s1004 distroarchseries = %s AND
1017 AND archive = %s1005 archive = %s AND
1018 AND status in (%s, %s)""" %1006 status in %s""" % sqlvalues(
1019 sqlvalues(binarypackage, self.distroarchseries,1007 binarypackage, self.distroarchseries,
1020 self.distroarchseries.main_archive,1008 self.distroarchseries.main_archive, active_publishing_status),
1021 PackagePublishingStatus.PUBLISHED,1009 orderBy=["-datecreated"])
1022 PackagePublishingStatus.PENDING),
1023 orderBy=["-datecreated"])
1024 ret = list(ret)1010 ret = list(ret)
1025 if ret:1011 if ret:
1026 return ret[0]1012 return ret[0]
10271013
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:48:24 +0000
@@ -1,13 +1,181 @@
1# Copyright 2009-2010 Canonical Ltd. This software is licensed under the1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite4from doctest import DocTestSuite
5import unittest5from unittest import TestLoader
66
7from canonical.testing.layers import ZopelessDatabaseLayer
8from lp.registry.interfaces.pocket import PackagePublishingPocket
9from lp.services.log.logger import DevNullLogger
10from lp.soyuz.enums import PackagePublishingStatus
11from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
7import lp.soyuz.scripts.gina.handlers12import lp.soyuz.scripts.gina.handlers
13from lp.soyuz.scripts.gina.handlers import (
14 BinaryPackagePublisher,
15 SourcePackagePublisher,
16 )
17from lp.soyuz.scripts.gina.packages import (
18 BinaryPackageData,
19 SourcePackageData,
20 )
21from lp.testing import TestCaseWithFactory
22
23
24class FakePackagesMap:
25 def __init__(self, src_map):
26 self.src_map = src_map
27
28
29class TestGina(TestCaseWithFactory):
30
31 layer = ZopelessDatabaseLayer
32
33 def test_dominate_imported_source_packages_dominates_imports(self):
34 # dominate_imported_source_packages dominates the source
35 # packages that Gina imports.
36 logger = DevNullLogger()
37 pub = self.factory.makeSourcePackagePublishingHistory(
38 status=PackagePublishingStatus.PUBLISHED)
39 series = pub.distroseries
40 spr = pub.sourcepackagerelease
41 package = spr.sourcepackagename
42 dominate_imported_source_packages(
43 logger, series.distribution.name, series.name, pub.pocket,
44 FakePackagesMap({package.name: []}))
45 self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
46
47 def test_dominate_imported_source_packages_dominates_deletions(self):
48 # dominate_imported_source_packages dominates the source
49 # packages that have been deleted from the Sources lists that
50 # Gina imports.
51 series = self.factory.makeDistroSeries()
52 pocket = PackagePublishingPocket.RELEASE
53 package = self.factory.makeSourcePackageName()
54 pubs = [
55 self.factory.makeSourcePackagePublishingHistory(
56 archive=series.main_archive, distroseries=series,
57 pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
58 sourcepackagerelease=self.factory.makeSourcePackageRelease(
59 sourcepackagename=package, version=version))
60 for version in ['1.0', '1.1', '1.1a']]
61 logger = DevNullLogger()
62 dominate_imported_source_packages(
63 logger, series.distribution.name, series.name, pocket,
64 FakePackagesMap({}))
65 # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
66 # "transitional" domination which supersedes older versions of
67 # deleted packages with the last known version. Permanent
68 # domination will then mark the last known version as deleted.
69 # For permanent domination, the expected outcome is that all
70 # these publications will be Deleted (but any pre-existing
71 # Superseded publications for older versions will remain
72 # Superseded).
73 self.assertEqual([
74 PackagePublishingStatus.SUPERSEDED,
75 PackagePublishingStatus.SUPERSEDED,
76 PackagePublishingStatus.PUBLISHED,
77 ],
78 [pub.status for pub in pubs])
79
80 def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
81 # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
82 # Gina domination, dominate_imported_source_packages turns any
83 # remaining Pending SPPHS into Published ones.
84 series = self.factory.makeDistroSeries()
85 spph = self.factory.makeSourcePackagePublishingHistory(
86 distroseries=series, archive=series.main_archive,
87 status=PackagePublishingStatus.PENDING)
88 spr = spph.sourcepackagerelease
89 package_name = spr.sourcepackagename.name
90 logger = DevNullLogger()
91 dominate_imported_source_packages(
92 logger, series.distribution.name, series.name, spph.pocket,
93 FakePackagesMap({package_name: [{"Version": spr.version}]}))
94 self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
95
96 def test_dominate_imported_source_packages_cleans_up_first(self):
97 # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
98 # Gina domination, dominate_imported_source_packages turns any
99 # remaining Pending SPPHS into Published ones. It does this
100 # *before* dominating, so no domination happens while some of
101 # the SPPHs are still mistakenly Pending (which would result in
102 # mistaken deletions).
103 series = self.factory.makeDistroSeries()
104 package = self.factory.makeSourcePackageName()
105 pocket = PackagePublishingPocket.RELEASE
106 versions = ['1.0', '1.1']
107 statuses_before = [
108 PackagePublishingStatus.PUBLISHED,
109 PackagePublishingStatus.PENDING,
110 ]
111 statuses_after = [
112 PackagePublishingStatus.SUPERSEDED,
113 PackagePublishingStatus.PUBLISHED,
114 ]
115 live_version = versions[-1]
116 sprs = [
117 self.factory.makeSourcePackageRelease(
118 sourcepackagename=package, version=version)
119 for version in versions]
120 spphs = [
121 self.factory.makeSourcePackagePublishingHistory(
122 archive=series.main_archive, distroseries=series,
123 sourcepackagerelease=spr, pocket=pocket, status=status)
124 for spr, status in zip(sprs, statuses_before)]
125
126 logger = DevNullLogger()
127 dominate_imported_source_packages(
128 logger, series.distribution.name, series.name, pocket,
129 FakePackagesMap({package.name: [{"Version": live_version}]}))
130
131 self.assertEqual(statuses_after, [spph.status for spph in spphs])
132
133
134class TestSourcePackagePublisher(TestCaseWithFactory):
135
136 layer = ZopelessDatabaseLayer
137
138 def test_publish_creates_published_publication(self):
139 maintainer = self.factory.makePerson()
140 series = self.factory.makeDistroSeries()
141 section = self.factory.makeSection()
142 pocket = PackagePublishingPocket.RELEASE
143 spr = self.factory.makeSourcePackageRelease()
144
145 publisher = SourcePackagePublisher(series, pocket, None)
146 publisher.publish(spr, SourcePackageData(
147 component='main', section=section.name, version='1.0',
148 maintainer=maintainer.preferredemail, architecture='all',
149 files='foo.py', binaries='foo.py'))
150
151 [spph] = series.main_archive.getPublishedSources()
152 self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
153
154
155class TestBinaryPackagePublisher(TestCaseWithFactory):
156
157 layer = ZopelessDatabaseLayer
158
159 def test_publish_creates_published_publication(self):
160 maintainer = self.factory.makePerson()
161 series = self.factory.makeDistroArchSeries()
162 section = self.factory.makeSection()
163 pocket = PackagePublishingPocket.RELEASE
164 bpr = self.factory.makeBinaryPackageRelease()
165
166 publisher = BinaryPackagePublisher(series, pocket, None)
167 publisher.publish(bpr, BinaryPackageData(
168 component='main', section=section.name, version='1.0',
169 maintainer=maintainer.preferredemail, architecture='all',
170 files='foo.py', binaries='foo.py', size=128, installed_size=1024,
171 md5sum='e83b5dd68079d727a494a469d40dc8db', description='test',
172 summary='Test!'))
173
174 [bpph] = series.main_archive.getAllPublishedBinaries()
175 self.assertEqual(PackagePublishingStatus.PUBLISHED, bpph.status)
8176
9177
10def test_suite():178def test_suite():
11 suite = unittest.TestSuite()179 suite = TestLoader().loadTestsFromName(__name__)
12 suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))180 suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
13 return suite181 return suite
14182
=== modified file 'scripts/gina.py'
--- scripts/gina.py 2011-08-23 08:35:13 +0000
+++ scripts/gina.py 2011-09-09 05:48:24 +0000
@@ -38,6 +38,7 @@
38 MangledArchiveError,38 MangledArchiveError,
39 PackagesMap,39 PackagesMap,
40 )40 )
41from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
41from lp.soyuz.scripts.gina.handlers import (42from lp.soyuz.scripts.gina.handlers import (
42 DataSetupError,43 DataSetupError,
43 ImporterHandler,44 ImporterHandler,
@@ -152,6 +153,10 @@
152 packages_map, kdb, package_root, keyrings, importer_handler)153 packages_map, kdb, package_root, keyrings, importer_handler)
153 importer_handler.commit()154 importer_handler.commit()
154155
156 # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
157 dominate_imported_source_packages(
158 log, distro, distroseries, pocket, packages_map)
159
155 if source_only:160 if source_only:
156 log.info('Source only mode... done')161 log.info('Source only mode... done')
157 return162 return
@@ -209,9 +214,8 @@
209 npacks = len(packages_map.src_map)214 npacks = len(packages_map.src_map)
210 log.info('%i Source Packages to be imported', npacks)215 log.info('%i Source Packages to be imported', npacks)
211216
212 for list_source in sorted(217 for package in sorted(packages_map.src_map.iterkeys()):
213 packages_map.src_map.values(), key=lambda x: x[0].get("Package")):218 for source in packages_map.src_map[package]:
214 for source in list_source:
215 count += 1219 count += 1
216 attempt_source_package_import(220 attempt_source_package_import(
217 source, kdb, package_root, keyrings, importer_handler)221 source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
244 log.info(248 log.info(
245 '%i Binary Packages to be imported for %s', npacks, archtag)249 '%i Binary Packages to be imported for %s', npacks, archtag)
246 # Go over binarypackages importing them for this architecture250 # Go over binarypackages importing them for this architecture
247 for binary in sorted(packages_map.bin_map[archtag].values(),251 for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
248 key=lambda x: x.get("Package")):252 binary = packages_map.bin_map[archtag][package_name]
249 count += 1253 count += 1
250 package_name = binary.get("Package", "unknown")
251 try:254 try:
252 try:255 try:
253 do_one_binarypackage(binary, archtag, kdb, package_root,256 do_one_binarypackage(binary, archtag, kdb, package_root,