Merge lp:~jtv/launchpad/transitional-published into lp:launchpad

[Jeroen T. Vermeulen]

Convert any remaining Pending Debian SPPHs before transitional domination.

= Summary =

Gina is learning to run domination on its Debian imports. Yes, technically that makes Gina a dominatrix.

There is a catch: domination operates only on publication records in Published state. So far, Gina has been creating Pending records. That made sense for Ubuntu's import into Launchpad, way back when, which is what Gina was originally written for. It does not make sense now, so we changed the code and updated the legacy data in batches.

Of course that does not take care of further legacy data that is created between the update we just did in the database and the time the Gina changes roll out. But it should reduce the problem's size enough that we can afford to do the final update from inside Gina. This will only need to run once, during “transitional domination” but will do nothing if run repeatedly.

== Pre-implementation notes ==

I had wanted to do a final patch-up update later, but William's points out that there is quite a substantial risk if domination runs on partially updated data. So I inserted the change directly into the code; I have another branch waiting to remove the transitional code.

== Implementation details ==

The diff starts with some modernization of a doctest (using active_publishing_status instead of spelling out the Published and Pending statuses). This is essentially cosmetic; it does not change the meaning of the code and really isn't relevant to the branch. Buried in there however is one change where the test still assumes that Gina produces and maintains Pending publication records. Before we remove transitional domination, we may have to re-think what happens there.

== Tests ==

Two new tests verify transitional behaviour. One is very simple: an active SPPH is Pending but really ought to be Published. Gina dominates and the SPPH's version is found in the simulated Sources list. The Pending SPPH becomes Published.

Another test looks for the doom scenario that might occur when data is not properly migrated: an older release of a package is Published, but the version that is mentioned in the Sources list is still Pending. It is essential that the live version be upgraded to Published before domination, or the dominator would decide that there is no newer published version to dominate the now-obsolete published SPPH, and mark it deleted. Needless to say, the test shows doom being avoided.

{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}

== Demo and Q/A ==

= Launchpad lint =

There is some pre-existing lint, especially in the doctest, that I can't afford to do too much about. It's not just the effort (mostly a matter of running utilities/formatdoctest.py!) but also a matter of keeping the diff small. I'm piling up quite a number of differences in a series of interdependent branches, and this would increase the risk of conflicts disproportionately.

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/domination.py
  lib/lp/soyuz/scripts/gina/dominate.py
  lib/lp/soyuz/model/publishing.py
  lib/lp/soyuz/doc/gina.txt
  lib/lp/soyuz/interfaces/publishing.py
  scripts/gina.py
  lib/lp/archivepublisher/tests/test_dominator.py
  lib/lp/soyuz/scripts/tests/test_gina.py
  lib/lp/soyuz/scripts/gina/handlers.py

./lib/lp/soyuz/doc/gina.txt
     113: narrative exceeds 78 characters.
     162: want exceeds 78 characters.
     179: want exceeds 78 characters.
     189: narrative uses a moin header.
     221: want exceeds 78 characters.
     234: want exceeds 78 characters.
     240: want exceeds 78 characters.
     295: source exceeds 78 characters.
     324: narrative uses a moin header.
     342: narrative exceeds 78 characters.
     354: narrative uses a moin header.
     360: narrative exceeds 78 characters.
     361: narrative exceeds 78 characters.
     459: narrative uses a moin header.
     461: narrative exceeds 78 characters.
     462: narrative exceeds 78 characters.
     477: narrative uses a moin header.
     563: narrative exceeds 78 characters.
     600: narrative uses a moin header.
     657: narrative uses a moin header.
     746: narrative uses a moin header.
     767: narrative uses a moin header.
     780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
     381: E261 at least two spaces before inline comment
     478: E261 at least two spaces before inline comment
     511: E261 at least two spaces before inline comment
     681: E261 at least two spaces before inline comment
     767: E261 at least two spaces before inline comment
./scripts/gina.py
      26: '_pythonpath' imported but unused

=== modified file 'lib/lp/archivepublisher/domination.py'
--- lib/lp/archivepublisher/domination.py	2011-08-30 06:37:55 +0000
+++ lib/lp/archivepublisher/domination.py	2011-09-09 05:48:24 +0000
@@ -53,8 +53,6 @@
 __all__ = ['Dominator']
 
 from datetime import timedelta
-import functools
-import operator
 
 import apt_pkg
 from storm.expr import (
@@ -68,7 +66,7 @@
     flush_database_updates,
     sqlvalues,
     )
-from canonical.launchpad.interfaces.lpstorm import IMasterStore
+from canonical.launchpad.interfaces.lpstorm import IStore
 from lp.registry.model.sourcepackagename import SourcePackageName
 from lp.soyuz.enums import (
     BinaryPackageFormat,
@@ -87,17 +85,93 @@
 apt_pkg.InitSystem()
 
 
-def _compare_packages_by_version_and_date(get_release, p1, p2):
-    """Compare publications p1 and p2 by their version; using Debian rules.
-
-    If the publications are for the same package, compare by datecreated
-    instead. This lets newer records win.
-    """
-    if get_release(p1).id == get_release(p2).id:
-        return cmp(p1.datecreated, p2.datecreated)
-
-    return apt_pkg.VersionCompare(get_release(p1).version,
-                                  get_release(p2).version)
+def join_spr_spn():
+    """Join condition: SourcePackageRelease/SourcePackageName."""
+    return (
+        SourcePackageName.id == SourcePackageRelease.sourcepackagenameID)
+
+
+def join_spph_spr():
+    """Join condition: SourcePackageRelease/SourcePackagePublishingHistory.
+    """
+    # Avoid circular imports.
+    from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+    return (
+        SourcePackageRelease.id ==
+            SourcePackagePublishingHistory.sourcepackagereleaseID)
+
+
+class SourcePublicationTraits:
+    """Basic generalized attributes for `SourcePackagePublishingHistory`.
+
+    Used by `GeneralizedPublication` to hide the differences from
+    `BinaryPackagePublishingHistory`.
+    """
+    @staticmethod
+    def getPackageName(spph):
+        """Return the name of this publication's source package."""
+        return spph.sourcepackagerelease.sourcepackagename.name
+
+    @staticmethod
+    def getPackageRelease(spph):
+        """Return this publication's `SourcePackageRelease`."""
+        return spph.sourcepackagerelease
+
+
+class BinaryPublicationTraits:
+    """Basic generalized attributes for `BinaryPackagePublishingHistory`.
+
+    Used by `GeneralizedPublication` to hide the differences from
+    `SourcePackagePublishingHistory`.
+    """
+    @staticmethod
+    def getPackageName(bpph):
+        """Return the name of this publication's binary package."""
+        return bpph.binarypackagerelease.binarypackagename.name
+
+    @staticmethod
+    def getPackageRelease(bpph):
+        """Return this publication's `BinaryPackageRelease`."""
+        return bpph.binarypackagerelease
+
+
+class GeneralizedPublication:
+    """Generalize handling of publication records.
+
+    This allows us to write code that can be dealing with either
+    `SourcePackagePublishingHistory`s or `BinaryPackagePublishingHistory`s
+    without caring which.  Differences are abstracted away in a traits
+    class.
+    """
+    def __init__(self, is_source=True):
+        if is_source:
+            self.traits = SourcePublicationTraits
+        else:
+            self.traits = BinaryPublicationTraits
+
+    def getPackageName(self, pub):
+        """Get the package's name."""
+        return self.traits.getPackageName(pub)
+
+    def getPackageVersion(self, pub):
+        """Obtain the version string for a publicaiton record."""
+        return self.traits.getPackageRelease(pub).version
+
+    def compare(self, pub1, pub2):
+        """Compare publications by version.
+
+        If both publications are for the same version, their creation dates
+        break the tie.
+        """
+        version_comparison = apt_pkg.VersionCompare(
+            self.getPackageVersion(pub1), self.getPackageVersion(pub2))
+
+        if version_comparison == 0:
+            # Use dates as tie breaker.
+            return cmp(pub1.datecreated, pub2.datecreated)
+        else:
+            return version_comparison
 
 
 class Dominator:
@@ -116,50 +190,95 @@
         self.logger = logger
         self.archive = archive
 
-    def _dominatePublications(self, pubs):
+    def dominatePackage(self, publications, live_versions, generalization):
+        """Dominate publications for a single package.
+
+        Active publications for versions in `live_versions` stay active.
+        Any older versions are marked as superseded by the respective
+        oldest live versions that are newer than the superseded ones.
+
+        Any versions that are newer than anything in `live_versions` are
+        marked as deleted.  This should not be possible in Soyuz-native
+        archives, but it can happen during archive imports when the
+        previous latest version of a package has disappeared from the Sources
+        list we import.
+
+        :param publications: Iterable of publications for the same package,
+            in the same archive, series, and pocket, all with status
+            `PackagePublishingStatus.PUBLISHED`.
+        :param live_versions: Iterable of version strings that are still
+            considered live for this package.  The given publications will
+            remain active insofar as they represent any of these versions;
+            older publications will be marked as superseded and newer ones
+            as deleted.
+        :param generalization: A `GeneralizedPublication` helper representing
+            the kind of publications these are--source or binary.
+        """
+        # Go through publications from latest version to oldest.  This
+        # makes it easy to figure out which release superseded which:
+        # the dominant is always the oldest live release that is newer
+        # than the one being superseded.
+        publications = sorted(
+            publications, cmp=generalization.compare, reverse=True)
+
+        current_dominant = None
+        for pub in publications:
+            if generalization.getPackageVersion(pub) in live_versions:
+                # This publication stays active; if any publications
+                # that follow right after this are to be superseded,
+                # this is the release that they are superseded by.
+                current_dominant = pub
+            elif current_dominant is None:
+                # This publication is no longer live, but there is no
+                # newer version to supersede it either.  Therefore it
+                # must be deleted.
+                pub.requestDeletion(None)
+            else:
+                # This publication is superseded.  This is what we're
+                # here to do.
+                pub.supersede(current_dominant, logger=self.logger)
+
+    def _dominatePublications(self, pubs, generalization):
         """Perform dominations for the given publications.
 
+        Keep the latest published version for each package active,
+        superseding older versions.
+
         :param pubs: A dict mapping names to a list of publications. Every
             publication must be PUBLISHED or PENDING, and the first in each
             list will be treated as dominant (so should be the latest).
+        :param generalization: A `GeneralizedPublication` helper representing
+            the kind of publications these are--source or binary.
         """
         self.logger.debug("Dominating packages...")
-
-        for name in pubs.keys():
-            assert pubs[name], (
-                "Empty list of publications for %s" % name)
-            for pubrec in pubs[name][1:]:
-                pubrec.supersede(pubs[name][0], logger=self.logger)
-
-    def _sortPackages(self, pkglist, is_source=True):
+        for name, publications in pubs.iteritems():
+            assert publications, "Empty list of publications for %s." % name
+            # Since this always picks the latest version as the live
+            # one, this dominatePackage call will never result in a
+            # deletion.
+            latest_version = generalization.getPackageVersion(publications[0])
+            self.dominatePackage(
+                publications, [latest_version], generalization)
+
+    def _sortPackages(self, pkglist, generalization):
         """Map out packages by name, and sort by descending version.
 
         :param pkglist: An iterable of `SourcePackagePublishingHistory` or
             `BinaryPackagePublishingHistory`.
-        :param is_source: Whether this call involves source package
-            publications.  If so, work with `SourcePackagePublishingHistory`.
-            If not, work with `BinaryPackagepublishingHistory`.
-        :return: A dict mapping each package name (as UTF-8 encoded string)
-            to a list of publications from `pkglist`, newest first.
+        :param generalization: A `GeneralizedPublication` helper representing
+            the kind of publications these are--source or binary.
+        :return: A dict mapping each package name to a list of publications
+            from `pkglist`, newest first.
         """
         self.logger.debug("Sorting packages...")
 
-        if is_source:
-            get_release = operator.attrgetter("sourcepackagerelease")
-            get_name = operator.attrgetter("sourcepackagename")
-        else:
-            get_release = operator.attrgetter("binarypackagerelease")
-            get_name = operator.attrgetter("binarypackagename")
-
         outpkgs = {}
         for inpkg in pkglist:
-            key = get_name(get_release(inpkg)).name.encode('utf-8')
+            key = generalization.getPackageName(inpkg)
             outpkgs.setdefault(key, []).append(inpkg)
 
-        sort_order = functools.partial(
-            _compare_packages_by_version_and_date, get_release)
         for package_pubs in outpkgs.itervalues():
-            package_pubs.sort(cmp=sort_order, reverse=True)
+            package_pubs.sort(cmp=generalization.compare, reverse=True)
 
         return outpkgs
 
@@ -287,6 +406,8 @@
         # Avoid circular imports.
         from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
 
+        generalization = GeneralizedPublication(is_source=False)
+
         for distroarchseries in distroseries.architectures:
             self.logger.debug(
                 "Performing domination across %s/%s (%s)",
@@ -312,7 +433,7 @@
                 ),
                 group_by=BinaryPackageName.id,
                 having=Count(BinaryPackagePublishingHistory.id) > 1)
-            binaries = IMasterStore(BinaryPackagePublishingHistory).find(
+            binaries = IStore(BinaryPackagePublishingHistory).find(
                 BinaryPackagePublishingHistory,
                 BinaryPackageRelease.id ==
                     BinaryPackagePublishingHistory.binarypackagereleaseID,
@@ -322,7 +443,21 @@
                     BinaryPackageFormat.DDEB,
                 bpph_location_clauses)
             self.logger.debug("Dominating binaries...")
-            self._dominatePublications(self._sortPackages(binaries, False))
+            self._dominatePublications(
+                self._sortPackages(binaries, generalization), generalization)
+
+    def _composeActiveSourcePubsCondition(self, distroseries, pocket):
+        """Compose ORM condition for restricting relevant source pubs."""
+        # Avoid circular imports.
+        from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+        return And(
+            SourcePackagePublishingHistory.status ==
+                PackagePublishingStatus.PUBLISHED,
+            SourcePackagePublishingHistory.distroseries == distroseries,
+            SourcePackagePublishingHistory.archive == self.archive,
+            SourcePackagePublishingHistory.pocket == pocket,
+            )
 
     def dominateSources(self, distroseries, pocket):
         """Perform domination on source package publications.
@@ -332,38 +467,76 @@
         """
         # Avoid circular imports.
         from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+        generalization = GeneralizedPublication(is_source=True)
+
         self.logger.debug(
             "Performing domination across %s/%s (Source)",
             distroseries.name, pocket.title)
-        spph_location_clauses = And(
-            SourcePackagePublishingHistory.status ==
-                PackagePublishingStatus.PUBLISHED,
-            SourcePackagePublishingHistory.distroseries == distroseries,
-            SourcePackagePublishingHistory.archive == self.archive,
-            SourcePackagePublishingHistory.pocket == pocket,
-            )
+
+        spph_location_clauses = self._composeActiveSourcePubsCondition(
+            distroseries, pocket)
+        having_multiple_active_publications = (
+            Count(SourcePackagePublishingHistory.id) > 1)
         candidate_source_names = Select(
             SourcePackageName.id,
-            And(
-                SourcePackageRelease.sourcepackagenameID ==
-                    SourcePackageName.id,
-                SourcePackagePublishingHistory.sourcepackagereleaseID ==
-                    SourcePackageRelease.id,
-                spph_location_clauses,
-            ),
+            And(join_spph_spr(), join_spr_spn(), spph_location_clauses),
             group_by=SourcePackageName.id,
-            having=Count(SourcePackagePublishingHistory.id) > 1)
-        sources = IMasterStore(SourcePackagePublishingHistory).find(
+            having=having_multiple_active_publications)
+        sources = IStore(SourcePackagePublishingHistory).find(
             SourcePackagePublishingHistory,
-            SourcePackageRelease.id ==
-                SourcePackagePublishingHistory.sourcepackagereleaseID,
+            join_spph_spr(),
             SourcePackageRelease.sourcepackagenameID.is_in(
                 candidate_source_names),
             spph_location_clauses)
+
         self.logger.debug("Dominating sources...")
-        self._dominatePublications(self._sortPackages(sources))
+        self._dominatePublications(
+            self._sortPackages(sources, generalization), generalization)
         flush_database_updates()
 
+    def findPublishedSourcePackageNames(self, distroseries, pocket):
+        """Find names of currently published source packages."""
+        result = IStore(SourcePackageName).find(
+            SourcePackageName.name,
+            join_spph_spr(),
+            join_spr_spn(),
+            self._composeActiveSourcePubsCondition(distroseries, pocket))
+        return result.config(distinct=True)
+
+    def findPublishedSPPHs(self, distroseries, pocket, package_name):
+        """Find currently published source publications for given package."""
+        # Avoid circular imports.
+        from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+
+        return IStore(SourcePackagePublishingHistory).find(
+            SourcePackagePublishingHistory,
+            join_spph_spr(),
+            join_spr_spn(),
+            SourcePackageName.name == package_name,
+            self._composeActiveSourcePubsCondition(distroseries, pocket))
+
+    def dominateRemovedSourceVersions(self, distroseries, pocket,
+                                      package_name, live_versions):
+        """Dominate source publications based on a set of "live" versions.
+
+        Active publications for the "live" versions will remain active.  All
+        other active publications for the same package (and the same archive,
+        distroseries, and pocket) are marked superseded.
+
+        Unlike traditional domination, this allows multiple versions of a
+        package to stay active in the same distroseries, archive, and pocket.
+
+        :param distroseries: `DistroSeries` to dominate.
+        :param pocket: `PackagePublishingPocket` to dominate.
+        :param package_name: Source package name, as text.
+        :param live_versions: Iterable of all version strings that are to
+            remain active.
+        """
+        generalization = GeneralizedPublication(is_source=True)
+        pubs = self.findPublishedSPPHs(distroseries, pocket, package_name)
+        self.dominatePackage(pubs, live_versions, generalization)
+
     def judge(self, distroseries, pocket):
         """Judge superseded sources and binaries."""
         # Avoid circular imports.

=== modified file 'lib/lp/archivepublisher/tests/test_dominator.py'
--- lib/lp/archivepublisher/tests/test_dominator.py	2011-02-04 05:11:00 +0000
+++ lib/lp/archivepublisher/tests/test_dominator.py	2011-09-09 05:48:24 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2010 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Tests for domination.py."""
@@ -7,12 +7,24 @@
 
 import datetime
 
+import apt_pkg
+from zope.security.proxy import removeSecurityProxy
+
 from canonical.database.sqlbase import flush_database_updates
-from lp.archivepublisher.domination import Dominator, STAY_OF_EXECUTION
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.domination import (
+    Dominator,
+    GeneralizedPublication,
+    STAY_OF_EXECUTION,
+    )
 from lp.archivepublisher.publishing import Publisher
+from lp.registry.interfaces.pocket import PackagePublishingPocket
 from lp.registry.interfaces.series import SeriesStatus
+from lp.services.log.logger import DevNullLogger
 from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.interfaces.publishing import ISourcePackagePublishingHistory
 from lp.soyuz.tests.test_publishing import TestNativePublishingBase
+from lp.testing import TestCaseWithFactory
 
 
 class TestDominator(TestNativePublishingBase):
@@ -50,6 +62,8 @@
                 foo_10_source, foo_10_binaries[0])
 
     def dominateAndCheck(self, dominant, dominated, supersededby):
+        generalization = GeneralizedPublication(
+            is_source=ISourcePackagePublishingHistory.providedBy(dominant))
         dominator = Dominator(self.logger, self.ubuntutest.main_archive)
 
         # The _dominate* test methods require a dictionary where the
@@ -58,7 +72,7 @@
         # and dominated, the subsequents.
         pubs = {'foo': [dominant, dominated]}
 
-        dominator._dominatePublications(pubs)
+        dominator._dominatePublications(pubs, generalization)
         flush_database_updates()
 
         # The dominant version remains correctly published.
@@ -145,7 +159,9 @@
         # This isn't a really good exception. It should probably be
         # something more indicative of bad input.
         self.assertRaises(
-            AssertionError, dominator._dominatePublications, pubs)
+            AssertionError,
+            dominator._dominatePublications,
+            pubs, GeneralizedPublication(True))
 
 
 class TestDomination(TestNativePublishingBase):
@@ -200,3 +216,398 @@
         TestDomination.setUp(self)
         self.ubuntutest['breezy-autotest'].status = (
             SeriesStatus.OBSOLETE)
+
+
+def make_spphs_for_versions(factory, versions):
+    """Create publication records for each of `versions`.
+
+    They records are created in the same order in which they are specified.
+    Make the order irregular to prove that version ordering is not a
+    coincidence of object creation order etc.
+
+    Versions may also be identical; each publication record will still have
+    its own package release.
+    """
+    spn = factory.makeSourcePackageName()
+    distroseries = factory.makeDistroSeries()
+    pocket = factory.getAnyPocket()
+    sprs = [
+        factory.makeSourcePackageRelease(
+            sourcepackagename=spn, version=version)
+        for version in versions]
+    return [
+        factory.makeSourcePackagePublishingHistory(
+            distroseries=distroseries, pocket=pocket,
+            sourcepackagerelease=spr,
+            status=PackagePublishingStatus.PUBLISHED)
+        for spr in sprs]
+
+
+def list_source_versions(spphs):
+    """Extract the versions from `spphs` as a list, in the same order."""
+    return [spph.sourcepackagerelease.version for spph in spphs]
+
+
+class TestGeneralizedPublication(TestCaseWithFactory):
+    """Test publication generalization helpers."""
+
+    layer = ZopelessDatabaseLayer
+
+    def alterCreationDates(self, spphs, ages):
+        """Set `datecreated` on each of `spphs` according to `ages`.
+
+        :param spphs: Iterable of `SourcePackagePublishingHistory`.  Their
+            respective creation dates will be offset by the respective ages
+            found in `ages` (with the two being matched up in the same order).
+        :param ages: Iterable of ages.  Must provide the same number of items
+            as `spphs`.  Ages are `timedelta` objects that will be subtracted
+            from the creation dates on the respective records in `spph`.
+        """
+        for spph, age in zip(spphs, ages):
+            spph.datecreated -= age
+
+    def test_getPackageVersion_gets_source_version(self):
+        spph = self.factory.makeSourcePackagePublishingHistory()
+        self.assertEqual(
+            spph.sourcepackagerelease.version,
+            GeneralizedPublication(is_source=True).getPackageVersion(spph))
+
+    def test_getPackageVersion_gets_binary_version(self):
+        bpph = self.factory.makeBinaryPackagePublishingHistory()
+        self.assertEqual(
+            bpph.binarypackagerelease.version,
+            GeneralizedPublication(is_source=False).getPackageVersion(bpph))
+
+    def test_compare_sorts_versions(self):
+        versions = [
+            '1.1v2',
+            '1.1v1',
+            '1.1v3',
+            ]
+        spphs = make_spphs_for_versions(self.factory, versions)
+        sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+        self.assertEqual(
+            sorted(versions), list_source_versions(sorted_spphs))
+
+    def test_compare_orders_versions_by_debian_rules(self):
+        versions = [
+            '1.1.0',
+            '1.10',
+            '1.1',
+            '1.1ubuntu0',
+            ]
+        spphs = make_spphs_for_versions(self.factory, versions)
+
+        debian_sorted_versions = sorted(versions, cmp=apt_pkg.VersionCompare)
+
+        # Assumption: in this case, Debian version ordering is not the
+        # same as alphabetical version ordering.
+        self.assertNotEqual(sorted(versions), debian_sorted_versions)
+
+        # The compare method produces the Debian ordering.
+        sorted_spphs = sorted(spphs, cmp=GeneralizedPublication().compare)
+        self.assertEqual(
+            sorted(versions, cmp=apt_pkg.VersionCompare),
+            list_source_versions(sorted_spphs))
+
+    def test_compare_breaks_tie_with_creation_date(self):
+        # When two publications are tied for comparison because they are
+        # for the same package release, they are ordered by creation
+        # date.
+        distroseries = self.factory.makeDistroSeries()
+        pocket = self.factory.getAnyPocket()
+        spr = self.factory.makeSourcePackageRelease()
+        ages = [
+            datetime.timedelta(2),
+            datetime.timedelta(1),
+            datetime.timedelta(3),
+            ]
+        spphs = [
+            self.factory.makeSourcePackagePublishingHistory(
+                sourcepackagerelease=spr, distroseries=distroseries,
+                pocket=pocket)
+            for counter in xrange(len(ages))]
+        self.alterCreationDates(spphs, ages)
+
+        self.assertEqual(
+            [spphs[2], spphs[0], spphs[1]],
+            sorted(spphs, cmp=GeneralizedPublication().compare))
+
+    def test_compare_breaks_tie_for_releases_with_same_version(self):
+        # When two publications are tied for comparison because they
+        # belong to releases with the same version string, they are
+        # ordered by creation date.
+        version = "1.%d" % self.factory.getUniqueInteger()
+        ages = [
+            datetime.timedelta(2),
+            datetime.timedelta(1),
+            datetime.timedelta(3),
+            ]
+        distroseries = self.factory.makeDistroSeries()
+        pocket = self.factory.getAnyPocket()
+        spphs = [
+            self.factory.makeSourcePackagePublishingHistory(
+                distroseries=distroseries, pocket=pocket,
+                sourcepackagerelease=self.factory.makeSourcePackageRelease(
+                    version=version))
+            for counter in xrange(len(ages))]
+        self.alterCreationDates(spphs, ages)
+
+        self.assertEqual(
+            [spphs[2], spphs[0], spphs[1]],
+            sorted(spphs, cmp=GeneralizedPublication().compare))
+
+
+class TestDominatorMethods(TestCaseWithFactory):
+
+    layer = ZopelessDatabaseLayer
+
+    def makeDominator(self, publications):
+        """Create a `Dominator` suitable for `publications`."""
+        if len(publications) == 0:
+            archive = self.factory.makeArchive()
+        else:
+            archive = publications[0].archive
+        return Dominator(DevNullLogger(), archive)
+
+    def test_dominatePackage_survives_empty_publications_list(self):
+        # Nothing explodes when dominatePackage is called with an empty
+        # packages list.
+        self.makeDominator([]).dominatePackage(
+            [], [], GeneralizedPublication(True))
+        # The test is that we get here without error.
+        pass
+
+    def test_dominatePackage_leaves_live_version_untouched(self):
+        # dominatePackage does not supersede live versions.
+        [pub] = make_spphs_for_versions(self.factory, ['3.1'])
+        self.makeDominator([pub]).dominatePackage(
+            [pub], ['3.1'], GeneralizedPublication(True))
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, pub.status)
+
+    def test_dominatePackage_deletes_dead_version_without_successor(self):
+        # dominatePackage marks non-live package versions without
+        # superseding versions as deleted.
+        [pub] = make_spphs_for_versions(self.factory, ['1.1'])
+        self.makeDominator([pub]).dominatePackage(
+            [pub], [], GeneralizedPublication(True))
+        self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+    def test_dominatePackage_supersedes_older_pub_with_newer_live_pub(self):
+        # When marking a package as superseded, dominatePackage
+        # designates a newer live version as the superseding version.
+        pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+        self.makeDominator(pubs).dominatePackage(
+            pubs, ['1.1'], GeneralizedPublication(True))
+        self.assertEqual(PackagePublishingStatus.SUPERSEDED, pubs[0].status)
+        self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+    def test_dominatePackage_only_supersedes_with_live_pub(self):
+        # When marking a package as superseded, dominatePackage will
+        # only pick a live version as the superseding one.
+        pubs = make_spphs_for_versions(
+            self.factory, ['1.0', '2.0', '3.0', '4.0'])
+        self.makeDominator(pubs).dominatePackage(
+            pubs, ['3.0'], GeneralizedPublication(True))
+        self.assertEqual([
+                pubs[2].sourcepackagerelease,
+                pubs[2].sourcepackagerelease,
+                None,
+                None,
+                ],
+            [pub.supersededby for pub in pubs])
+
+    def test_dominatePackage_supersedes_with_oldest_newer_live_pub(self):
+        # When marking a package as superseded, dominatePackage picks
+        # the oldest of the newer, live versions as the superseding one.
+        pubs = make_spphs_for_versions(self.factory, ['2.7', '2.8', '2.9'])
+        self.makeDominator(pubs).dominatePackage(
+            pubs, ['2.8', '2.9'], GeneralizedPublication(True))
+        self.assertEqual(pubs[1].sourcepackagerelease, pubs[0].supersededby)
+
+    def test_dominatePackage_only_supersedes_with_newer_live_pub(self):
+        # When marking a package as superseded, dominatePackage only
+        # considers a newer version as the superseding one.
+        pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2'])
+        self.makeDominator(pubs).dominatePackage(
+            pubs, ['0.1'], GeneralizedPublication(True))
+        self.assertEqual(None, pubs[1].supersededby)
+        self.assertEqual(PackagePublishingStatus.DELETED, pubs[1].status)
+
+    def test_dominateRemovedSourceVersions_dominates_publications(self):
+        # dominateRemovedSourceVersions finds the publications for a
+        # package and calls dominatePackage on them.
+        pubs = make_spphs_for_versions(self.factory, ['0.1', '0.2', '0.3'])
+        package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+
+        self.makeDominator(pubs).dominateRemovedSourceVersions(
+            pubs[0].distroseries, pubs[0].pocket, package_name, ['0.2'])
+        self.assertEqual([
+                PackagePublishingStatus.SUPERSEDED,
+                PackagePublishingStatus.PUBLISHED,
+                PackagePublishingStatus.DELETED,
+                ],
+            [pub.status for pub in pubs])
+        self.assertEqual(
+            [pubs[1].sourcepackagerelease, None, None],
+            [pub.supersededby for pub in pubs])
+
+    def test_dominateRemovedSourceVersions_ignores_other_pockets(self):
+        # dominateRemovedSourceVersions ignores publications in other
+        # pockets than the one specified.
+        pubs = make_spphs_for_versions(self.factory, ['2.3', '2.4'])
+        package_name = pubs[0].sourcepackagerelease.sourcepackagename.name
+        removeSecurityProxy(pubs[0]).pocket = PackagePublishingPocket.UPDATES
+        removeSecurityProxy(pubs[1]).pocket = PackagePublishingPocket.PROPOSED
+        self.makeDominator(pubs).dominateRemovedSourceVersions(
+            pubs[0].distroseries, pubs[0].pocket, package_name, ['2.3'])
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[1].status)
+
+    def test_dominateRemovedSourceVersions_ignores_other_packages(self):
+        pubs = make_spphs_for_versions(self.factory, ['1.0', '1.1'])
+        other_package_name = self.factory.makeSourcePackageName().name
+        self.makeDominator(pubs).dominateRemovedSourceVersions(
+            pubs[0].distroseries, pubs[0].pocket, other_package_name, ['1.1'])
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, pubs[0].status)
+
+    def test_findPublishedSourcePackageNames_finds_package(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [spph.sourcepackagerelease.sourcepackagename.name],
+            dominator.findPublishedSourcePackageNames(
+                spph.distroseries, spph.pocket))
+
+    def test_findPublishedSourcePackageNames_ignores_other_states(self):
+        series = self.factory.makeDistroSeries()
+        pocket = PackagePublishingPocket.RELEASE
+        spphs = dict(
+            (status, self.factory.makeSourcePackagePublishingHistory(
+                distroseries=series, archive=series.main_archive,
+                pocket=pocket, status=status))
+            for status in PackagePublishingStatus.items)
+        published_spph = spphs[PackagePublishingStatus.PUBLISHED]
+        dominator = self.makeDominator(spphs.values())
+        self.assertContentEqual(
+            [published_spph.sourcepackagerelease.sourcepackagename.name],
+            dominator.findPublishedSourcePackageNames(series, pocket))
+
+    def test_findPublishedSourcePackageNames_ignores_other_archives(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        dominator = self.makeDominator([spph])
+        dominator.archive = self.factory.makeArchive()
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSourcePackageNames(
+                spph.distroseries, spph.pocket))
+
+    def test_findPublishedSourcePackageNames_ignores_other_series(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        distro = spph.distroseries.distribution
+        other_series = self.factory.makeDistroSeries(distribution=distro)
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSourcePackageNames(
+                other_series, spph.pocket))
+
+    def test_findPublishedSourcePackageNames_ignores_other_pockets(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED,
+            pocket=PackagePublishingPocket.RELEASE)
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSourcePackageNames(
+                spph.distroseries, PackagePublishingPocket.SECURITY))
+
+    def test_findPublishedSourcePackageNames_does_not_return_duplicates(self):
+        series = self.factory.makeDistroSeries()
+        pocket = PackagePublishingPocket.RELEASE
+        package = self.factory.makeSourcePackageName()
+        spphs = [
+            self.factory.makeSourcePackagePublishingHistory(
+                distroseries=series, archive=series.main_archive,
+                pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+                sourcepackagerelease=self.factory.makeSourcePackageRelease(
+                    sourcepackagename=package))
+            for counter in xrange(2)]
+        dominator = self.makeDominator(spphs)
+        self.assertEqual(
+            [package.name],
+            list(dominator.findPublishedSourcePackageNames(series, pocket)))
+
+    def test_findPublishedSPPHs_finds_published_SPPH(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        package_name = spph.sourcepackagerelease.sourcepackagename.name
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [spph],
+            dominator.findPublishedSPPHs(
+                spph.distroseries, spph.pocket, package_name))
+
+    def test_findPublishedSPPHs_ignores_other_states(self):
+        series = self.factory.makeDistroSeries()
+        package = self.factory.makeSourcePackageName()
+        pocket = PackagePublishingPocket.RELEASE
+        spphs = dict(
+            (status, self.factory.makeSourcePackagePublishingHistory(
+                distroseries=series, archive=series.main_archive,
+                pocket=pocket, status=status,
+                sourcepackagerelease=self.factory.makeSourcePackageRelease(
+                    sourcepackagename=package)))
+            for status in PackagePublishingStatus.items)
+        dominator = self.makeDominator(spphs.values())
+        self.assertContentEqual(
+            [spphs[PackagePublishingStatus.PUBLISHED]],
+            dominator.findPublishedSPPHs(series, pocket, package.name))
+
+    def test_findPublishedSPPHs_ignores_other_archives(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        package = spph.sourcepackagerelease.sourcepackagename
+        dominator = self.makeDominator([spph])
+        dominator.archive = self.factory.makeArchive()
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSPPHs(
+                spph.distroseries, spph.pocket, package.name))
+
+    def test_findPublishedSPPHs_ignores_other_series(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        distro = spph.distroseries.distribution
+        package = spph.sourcepackagerelease.sourcepackagename
+        other_series = self.factory.makeDistroSeries(distribution=distro)
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSPPHs(
+                other_series, spph.pocket, package.name))
+
+    def test_findPublishedSPPHs_ignores_other_pockets(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED,
+            pocket=PackagePublishingPocket.RELEASE)
+        package = spph.sourcepackagerelease.sourcepackagename
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSPPHs(
+                spph.distroseries, PackagePublishingPocket.SECURITY,
+                package.name))
+
+    def test_findPublishedSPPHs_ignores_other_packages(self):
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        other_package = self.factory.makeSourcePackageName()
+        dominator = self.makeDominator([spph])
+        self.assertContentEqual(
+            [],
+            dominator.findPublishedSPPHs(
+                spph.distroseries, spph.pocket, other_package.name))

=== modified file 'lib/lp/soyuz/doc/gina.txt'
--- lib/lp/soyuz/doc/gina.txt	2011-07-29 11:35:28 +0000
+++ lib/lp/soyuz/doc/gina.txt	2011-09-09 05:48:24 +0000
@@ -8,6 +8,7 @@
 Get the current counts of stuff in the database:
 
     >>> from canonical.launchpad.database.emailaddress import EmailAddress
+    >>> from lp.soyuz.interfaces.publishing import active_publishing_status
     >>> from lp.soyuz.model.publishing import (
     ...     BinaryPackagePublishingHistory,
     ...     SourcePackagePublishingHistory)
@@ -564,35 +565,34 @@
 that's what overrides actually do.
 
     >>> from canonical.database.sqlbase import sqlvalues
-    >>> from lp.soyuz.enums import PackagePublishingStatus
-    >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s
-    ...                           AND distroseries = %s
-    ...                           AND status in (%s, %s)""" %
-    ...                         sqlvalues(x11p, breezy,
-    ...                          PackagePublishingStatus.PUBLISHED,
-    ...                          PackagePublishingStatus.PENDING),
-    ...                          orderBy=["-datecreated"])[0]
+    >>> x11_pub = SSPPH.select("""
+    ...     sourcepackagerelease = %s AND
+    ...     distroseries = %s AND
+    ...     status in %s
+    ...     """ % sqlvalues(
+    ...         x11p, breezy, active_publishing_status),
+    ...     orderBy=["-datecreated"])[0]
     >>> print x11_pub.section.name
     net
-    >>> ed_pub = SBPPH.select("""binarypackagerelease = %s
-    ...                           AND distroarchseries = %s
-    ...                           AND status in (%s, %s)""" %
-    ...                         sqlvalues(ed, breezy_i386,
-    ...                          PackagePublishingStatus.PUBLISHED,
-    ...                          PackagePublishingStatus.PENDING),
-    ...                          orderBy=["-datecreated"])[0]
+    >>> ed_pub = SBPPH.select("""
+    ...     binarypackagerelease = %s AND
+    ...     distroarchseries = %s AND
+    ...     status in %s
+    ...     """ % sqlvalues(
+    ...         ed, breezy_i386, active_publishing_status),
+    ...     orderBy=["-datecreated"])[0]
     >>> print ed_pub.priority
     Extra
     >>> n = SourcePackageName.selectOneBy(name="archive-copier")
     >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
     ...         version="0.3.6")
-    >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s
-    ...                          AND distroseries = %s
-    ...                          AND status in (%s, %s)""" %
-    ...                        sqlvalues(ac, breezy,
-    ...                         PackagePublishingStatus.PUBLISHED,
-    ...                         PackagePublishingStatus.PENDING),
-    ...                         orderBy=["-datecreated"])[0]
+    >>> ac_pub = SSPPH.select("""
+    ...     sourcepackagerelease = %s AND
+    ...     distroseries = %s AND
+    ...     status in %s
+    ...     """ % sqlvalues(
+    ...         ac, breezy, active_publishing_status),
+    ...     orderBy=["-datecreated"])[0]
     >>> print ac_pub.component.name
     universe
 
@@ -720,7 +720,7 @@
 
     >>> transaction.commit()
 
-There is now a number of source publications in PENDING status for the
+There is now a number of source publications in PUBLISHED status for the
 targetted distroseries, 'lenny'.
 
     >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
@@ -728,7 +728,7 @@
     12
 
     >>> print set([pub.status.name for pub in lenny_sources])
-    set(['PENDING'])
+    set(['PUBLISHED'])
 
 As mentioned before, lenny/i386 is empty, no binaries were imported.
 Also, the number of binaries published in the whole debian distribution

=== modified file 'lib/lp/soyuz/interfaces/publishing.py'
--- lib/lp/soyuz/interfaces/publishing.py	2011-09-02 04:51:25 +0000
+++ lib/lp/soyuz/interfaces/publishing.py	2011-09-09 05:48:24 +0000
@@ -195,9 +195,6 @@
         the field name and value is the value string.
         """
 
-    def supersede():
-        """Supersede this publication."""
-
     def requestObsolescence():
         """Make this publication obsolete.
 

=== modified file 'lib/lp/soyuz/model/publishing.py'
--- lib/lp/soyuz/model/publishing.py	2011-08-31 04:40:44 +0000
+++ lib/lp/soyuz/model/publishing.py	2011-09-09 05:48:24 +0000
@@ -114,10 +114,6 @@
 from lp.soyuz.scripts.changeoverride import ArchiveOverriderError
 
 
-PENDING = PackagePublishingStatus.PENDING
-PUBLISHED = PackagePublishingStatus.PUBLISHED
-
-
 # XXX cprov 2006-08-18: move it away, perhaps archivepublisher/pool.py
 
 def makePoolPath(source_name, component_name):
@@ -327,8 +323,8 @@
         fields = self.buildIndexStanzaFields()
         return fields.makeOutput()
 
-    def supersede(self):
-        """See `IPublishing`."""
+    def setSuperseded(self):
+        """Set to SUPERSEDED status."""
         self.status = PackagePublishingStatus.SUPERSEDED
         self.datesuperseded = UTC_NOW
 
@@ -742,7 +738,7 @@
             "Should not dominate unpublished source %s" %
             self.sourcepackagerelease.title)
 
-        super(SourcePackagePublishingHistory, self).supersede()
+        self.setSuperseded()
 
         if dominant is not None:
             if logger is not None:
@@ -1081,7 +1077,7 @@
         return IMasterStore(BinaryPackagePublishingHistory).find(
                 BinaryPackagePublishingHistory,
                 BinaryPackagePublishingHistory.status.is_in(
-                    [PUBLISHED, PENDING]),
+                    active_publishing_status),
                 BinaryPackagePublishingHistory.distroarchseriesID.is_in(
                     available_architectures),
                 binarypackagerelease=self.binarypackagerelease,
@@ -1101,7 +1097,7 @@
         return IMasterStore(BinaryPackagePublishingHistory).find(
                 BinaryPackagePublishingHistory,
                 BinaryPackagePublishingHistory.status.is_in(
-                    [PUBLISHED, PENDING]),
+                    active_publishing_status),
                 BinaryPackagePublishingHistory.distroarchseries ==
                     self.distroarchseries,
                 binarypackagerelease=self.binarypackagerelease.debug_package,
@@ -1126,7 +1122,7 @@
                 self.distroarchseries.architecturetag))
             return
 
-        super(BinaryPackagePublishingHistory, self).supersede()
+        self.setSuperseded()
 
         if dominant is not None:
             # DDEBs cannot themselves be dominant; they are always dominated

=== added file 'lib/lp/soyuz/scripts/gina/dominate.py'
--- lib/lp/soyuz/scripts/gina/dominate.py	1970-01-01 00:00:00 +0000
+++ lib/lp/soyuz/scripts/gina/dominate.py	2011-09-09 05:48:24 +0000
@@ -0,0 +1,82 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Retirement of packages that are removed upstream."""
+
+__metaclass__ = type
+__all__ = [
+    'dominate_imported_source_packages',
+    ]
+
+from zope.component import getUtility
+
+# XXX JeroenVermeulen 2011-09-08, bug=844550: The GeneralizedPublication
+# import violates import policy and elicits a warning from the test
+# suite.  The warning helps remind us to retire this code as soon as
+# possible.
+from lp.archivepublisher.domination import (
+    Dominator,
+    GeneralizedPublication,
+    )
+from lp.registry.interfaces.distribution import IDistributionSet
+
+
+def dominate_imported_source_packages(logger, distro_name, series_name,
+                                      pocket, packages_map):
+    """Perform domination."""
+    series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
+    dominator = Dominator(logger, series.main_archive)
+
+    # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
+    # hack.  Gina used to create SPPHs in Pending state.  We cleaned up
+    # the bulk of them, and changed the code to create Published ones, but
+    # some new ones will have been created since.
+    # Update those to match what the new Gina does.
+    from canonical.launchpad.interfaces.lpstorm import IStore
+    from lp.soyuz.enums import PackagePublishingStatus
+    from lp.soyuz.model.publishing import SourcePackagePublishingHistory
+    SPPH = SourcePackagePublishingHistory
+    store = IStore(SPPH)
+    spphs = store.find(
+        SPPH,
+        SPPH.archive == series.main_archive,
+        SPPH.distroseries == series,
+        SPPH.pocket == pocket,
+        SPPH.status == PackagePublishingStatus.PENDING)
+    spphs.set(status=PackagePublishingStatus.PUBLISHED)
+
+    # Dominate packages found in the Sources list we're importing.
+    package_names = dominator.findPublishedSourcePackageNames(series, pocket)
+    for package_name in package_names:
+        entries = packages_map.src_map.get(package_name)
+
+        if entries is None:
+            # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a
+            # transitional hack.  The database is full of "Published"
+            # Debian SPPHs whose packages have actually been deleted.
+            # In the future such publications should simply be marked
+            # Deleted, but for the legacy baggage we currently carry
+            # around we'll just do traditional domination first: pick
+            # the latest Published version, and mark the rest of the
+            # SPPHs as superseded by that version.  The latest version
+            # will then, finally, be marked appropriately Deleted once
+            # we remove this transitional hack.
+            # To remove the transitional hack, just let live_versions
+            # default to the empty list instead of doing this:
+            pubs = dominator.findPublishedSPPHs(series, pocket, package_name)
+            generalization = GeneralizedPublication(is_source=True)
+            pubs_dict = dominator._sortPackages(pubs, generalization)
+            sorted_pubs = pubs_dict[package_name]
+            if len(sorted_pubs) <= 1:
+                # If there's only one published SPPH, the transitional
+                # code will just leave it Published.  Don't bother; the
+                # migration will be costly enough as it is.
+                continue
+            live_versions = [sorted_pubs[0].sourcepackagerelease.version]
+        else:
+            live_versions = [
+                entry['Version']
+                for entry in entries if 'Version' in entry]
+
+        dominator.dominateRemovedSourceVersions(
+            series, pocket, package_name, live_versions)

=== modified file 'lib/lp/soyuz/scripts/gina/handlers.py'
--- lib/lp/soyuz/scripts/gina/handlers.py	2011-05-20 07:43:58 +0000
+++ lib/lp/soyuz/scripts/gina/handlers.py	2011-09-09 05:48:24 +0000
@@ -35,12 +35,12 @@
 from canonical.launchpad.interfaces.librarian import ILibraryFileAliasSet
 from canonical.launchpad.scripts import log
 from lp.archivepublisher.diskpool import poolify
+from lp.archiveuploader.changesfile import ChangesFile
 from lp.archiveuploader.tagfiles import parse_tagfile
 from lp.archiveuploader.utils import (
     determine_binary_file_type,
     determine_source_file_type,
     )
-from lp.archiveuploader.changesfile import ChangesFile
 from lp.buildmaster.enums import BuildStatus
 from lp.registry.interfaces.person import (
     IPersonSet,
@@ -54,7 +54,10 @@
     )
 from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
 from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
-from lp.soyuz.interfaces.publishing import IPublishingSet
+from lp.soyuz.interfaces.publishing import (
+    active_publishing_status,
+    IPublishingSet,
+    )
 from lp.soyuz.model.component import Component
 from lp.soyuz.model.files import (
     BinaryPackageFile,
@@ -723,8 +726,6 @@
                          source_publishinghistory.status.title)
                 return
 
-        # Create the Publishing entry, with status PENDING so that we
-        # can republish this later into a Soyuz archive.
         entry = getUtility(IPublishingSet).newSourcePublication(
             distroseries=self.distroseries,
             sourcepackagerelease=sourcepackagerelease,
@@ -732,6 +733,7 @@
             section=section,
             pocket=self.pocket,
             archive=archive)
+        entry.setPublished()
         log.info('Source package %s (%s) published' % (
             entry.sourcepackagerelease.sourcepackagename.name,
             entry.sourcepackagerelease.version))
@@ -742,16 +744,14 @@
         from lp.soyuz.model.publishing import (
             SourcePackagePublishingHistory)
 
-        ret = SourcePackagePublishingHistory.select(
-                """sourcepackagerelease = %s
-                   AND distroseries = %s
-                   AND archive = %s
-                   AND status in (%s, %s)""" %
-                sqlvalues(sourcepackagerelease, self.distroseries,
-                          self.distroseries.main_archive,
-                          PackagePublishingStatus.PUBLISHED,
-                          PackagePublishingStatus.PENDING),
-                orderBy=["-datecreated"])
+        ret = SourcePackagePublishingHistory.select("""
+            sourcepackagerelease = %s AND
+            distroseries = %s AND
+            archive = %s AND
+            status in %s""" % sqlvalues(
+                sourcepackagerelease, self.distroseries,
+                self.distroseries.main_archive, active_publishing_status),
+            orderBy=["-datecreated"])
         ret = list(ret)
         if ret:
             return ret[0]
@@ -917,14 +917,6 @@
                         "for package %s (%s)" %
                         (build.id, binary.package, binary.version))
         else:
-
-            # XXX Debonzi 2005-05-16: Check it later
-            #         if bin.gpg_signing_key_owner:
-            #             key = self.getGPGKey(bin.gpg_signing_key,
-            #                                  *bin.gpg_signing_key_owner)
-            #         else:
-            key = None
-
             processor = distroarchinfo['processor']
             build = getUtility(IBinaryPackageBuildSet).new(
                         processor=processor.id,
@@ -948,8 +940,7 @@
     def publish(self, binarypackage, bpdata):
         """Create the publishing entry on db if does not exist."""
         # Avoid circular imports.
-        from lp.soyuz.model.publishing import (
-            BinaryPackagePublishingHistory)
+        from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
 
         # These need to be pulled from the binary package data, not the
         # binary package release: the data represents data from /this
@@ -983,22 +974,20 @@
                          binpkg_publishinghistory.status.title)
                 return
 
-
-        # Create the Publishing entry with status PENDING.
         BinaryPackagePublishingHistory(
-            binarypackagerelease = binarypackage.id,
-            component = component.id,
-            section = section.id,
-            priority = priority,
-            distroarchseries = self.distroarchseries.id,
-            status = PackagePublishingStatus.PENDING,
-            datecreated = UTC_NOW,
-            datepublished = UTC_NOW,
-            pocket = self.pocket,
-            datesuperseded = None,
-            supersededby = None,
-            datemadepending = None,
-            dateremoved = None,
+            binarypackagerelease=binarypackage.id,
+            component=component.id,
+            section=section.id,
+            priority=priority,
+            distroarchseries=self.distroarchseries.id,
+            status=PackagePublishingStatus.PUBLISHED,
+            datecreated=UTC_NOW,
+            datepublished=UTC_NOW,
+            pocket=self.pocket,
+            datesuperseded=None,
+            supersededby=None,
+            datemadepending=None,
+            dateremoved=None,
             archive=archive)
 
         log.info('BinaryPackage %s-%s published into %s.' % (
@@ -1008,19 +997,16 @@
     def _checkPublishing(self, binarypackage):
         """Query for the publishing entry"""
         # Avoid circular imports.
-        from lp.soyuz.model.publishing import (
-            BinaryPackagePublishingHistory)
+        from lp.soyuz.model.publishing import BinaryPackagePublishingHistory
 
-        ret = BinaryPackagePublishingHistory.select(
-                """binarypackagerelease = %s
-                   AND distroarchseries = %s
-                   AND archive = %s
-                   AND status in (%s, %s)""" %
-                sqlvalues(binarypackage, self.distroarchseries,
-                          self.distroarchseries.main_archive,
-                          PackagePublishingStatus.PUBLISHED,
-                          PackagePublishingStatus.PENDING),
-                orderBy=["-datecreated"])
+        ret = BinaryPackagePublishingHistory.select("""
+            binarypackagerelease = %s AND
+            distroarchseries = %s AND
+            archive = %s AND
+            status in %s""" % sqlvalues(
+                binarypackage, self.distroarchseries,
+                self.distroarchseries.main_archive, active_publishing_status),
+            orderBy=["-datecreated"])
         ret = list(ret)
         if ret:
             return ret[0]

=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py	2010-08-20 20:31:18 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py	2011-09-09 05:48:24 +0000
@@ -1,13 +1,181 @@
-# Copyright 2009-2010 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 from doctest import DocTestSuite
-import unittest
+from unittest import TestLoader
 
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.services.log.logger import DevNullLogger
+from lp.soyuz.enums import PackagePublishingStatus
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
 import lp.soyuz.scripts.gina.handlers
+from lp.soyuz.scripts.gina.handlers import (
+    BinaryPackagePublisher,
+    SourcePackagePublisher,
+    )
+from lp.soyuz.scripts.gina.packages import (
+    BinaryPackageData,
+    SourcePackageData,
+    )
+from lp.testing import TestCaseWithFactory
+
+
+class FakePackagesMap:
+    def __init__(self, src_map):
+        self.src_map = src_map
+
+
+class TestGina(TestCaseWithFactory):
+
+    layer = ZopelessDatabaseLayer
+
+    def test_dominate_imported_source_packages_dominates_imports(self):
+        # dominate_imported_source_packages dominates the source
+        # packages that Gina imports.
+        logger = DevNullLogger()
+        pub = self.factory.makeSourcePackagePublishingHistory(
+            status=PackagePublishingStatus.PUBLISHED)
+        series = pub.distroseries
+        spr = pub.sourcepackagerelease
+        package = spr.sourcepackagename
+        dominate_imported_source_packages(
+            logger, series.distribution.name, series.name, pub.pocket,
+            FakePackagesMap({package.name: []}))
+        self.assertEqual(PackagePublishingStatus.DELETED, pub.status)
+
+    def test_dominate_imported_source_packages_dominates_deletions(self):
+        # dominate_imported_source_packages dominates the source
+        # packages that have been deleted from the Sources lists that
+        # Gina imports.
+        series = self.factory.makeDistroSeries()
+        pocket = PackagePublishingPocket.RELEASE
+        package = self.factory.makeSourcePackageName()
+        pubs = [
+            self.factory.makeSourcePackagePublishingHistory(
+                archive=series.main_archive, distroseries=series,
+                pocket=pocket, status=PackagePublishingStatus.PUBLISHED,
+                sourcepackagerelease=self.factory.makeSourcePackageRelease(
+                    sourcepackagename=package, version=version))
+            for version in ['1.0', '1.1', '1.1a']]
+        logger = DevNullLogger()
+        dominate_imported_source_packages(
+            logger, series.distribution.name, series.name, pocket,
+            FakePackagesMap({}))
+        # XXX JeroenVermeulen 2011-09-08, bug=844550: This is
+        # "transitional" domination which supersedes older versions of
+        # deleted packages with the last known version.  Permanent
+        # domination will then mark the last known version as deleted.
+        # For permanent domination, the expected outcome is that all
+        # these publications will be Deleted (but any pre-existing
+        # Superseded publications for older versions will remain
+        # Superseded).
+        self.assertEqual([
+            PackagePublishingStatus.SUPERSEDED,
+            PackagePublishingStatus.SUPERSEDED,
+            PackagePublishingStatus.PUBLISHED,
+            ],
+            [pub.status for pub in pubs])
+
+    def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
+        # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
+        # Gina domination, dominate_imported_source_packages turns any
+        # remaining Pending SPPHS into Published ones.
+        series = self.factory.makeDistroSeries()
+        spph = self.factory.makeSourcePackagePublishingHistory(
+            distroseries=series, archive=series.main_archive,
+            status=PackagePublishingStatus.PENDING)
+        spr = spph.sourcepackagerelease
+        package_name = spr.sourcepackagename.name
+        logger = DevNullLogger()
+        dominate_imported_source_packages(
+            logger, series.distribution.name, series.name, spph.pocket,
+            FakePackagesMap({package_name: [{"Version": spr.version}]}))
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
+
+    def test_dominate_imported_source_packages_cleans_up_first(self):
+        # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
+        # Gina domination, dominate_imported_source_packages turns any
+        # remaining Pending SPPHS into Published ones.  It does this
+        # *before* dominating, so no domination happens while some of
+        # the SPPHs are still mistakenly Pending (which would result in
+        # mistaken deletions).
+        series = self.factory.makeDistroSeries()
+        package = self.factory.makeSourcePackageName()
+        pocket = PackagePublishingPocket.RELEASE
+        versions = ['1.0', '1.1']
+        statuses_before = [
+            PackagePublishingStatus.PUBLISHED,
+            PackagePublishingStatus.PENDING,
+            ]
+        statuses_after = [
+            PackagePublishingStatus.SUPERSEDED,
+            PackagePublishingStatus.PUBLISHED,
+            ]
+        live_version = versions[-1]
+        sprs = [
+            self.factory.makeSourcePackageRelease(
+                sourcepackagename=package, version=version)
+            for version in versions]
+        spphs = [
+            self.factory.makeSourcePackagePublishingHistory(
+                archive=series.main_archive, distroseries=series,
+                sourcepackagerelease=spr, pocket=pocket, status=status)
+            for spr, status in zip(sprs, statuses_before)]
+
+        logger = DevNullLogger()
+        dominate_imported_source_packages(
+            logger, series.distribution.name, series.name, pocket,
+            FakePackagesMap({package.name: [{"Version": live_version}]}))
+
+        self.assertEqual(statuses_after, [spph.status for spph in spphs])
+
+
+class TestSourcePackagePublisher(TestCaseWithFactory):
+
+    layer = ZopelessDatabaseLayer
+
+    def test_publish_creates_published_publication(self):
+        maintainer = self.factory.makePerson()
+        series = self.factory.makeDistroSeries()
+        section = self.factory.makeSection()
+        pocket = PackagePublishingPocket.RELEASE
+        spr = self.factory.makeSourcePackageRelease()
+
+        publisher = SourcePackagePublisher(series, pocket, None)
+        publisher.publish(spr, SourcePackageData(
+            component='main', section=section.name, version='1.0',
+            maintainer=maintainer.preferredemail, architecture='all',
+            files='foo.py', binaries='foo.py'))
+
+        [spph] = series.main_archive.getPublishedSources()
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
+
+
+class TestBinaryPackagePublisher(TestCaseWithFactory):
+
+    layer = ZopelessDatabaseLayer
+
+    def test_publish_creates_published_publication(self):
+        maintainer = self.factory.makePerson()
+        series = self.factory.makeDistroArchSeries()
+        section = self.factory.makeSection()
+        pocket = PackagePublishingPocket.RELEASE
+        bpr = self.factory.makeBinaryPackageRelease()
+
+        publisher = BinaryPackagePublisher(series, pocket, None)
+        publisher.publish(bpr, BinaryPackageData(
+            component='main', section=section.name, version='1.0',
+            maintainer=maintainer.preferredemail, architecture='all',
+            files='foo.py', binaries='foo.py', size=128, installed_size=1024,
+            md5sum='e83b5dd68079d727a494a469d40dc8db', description='test',
+            summary='Test!'))
+
+        [bpph] = series.main_archive.getAllPublishedBinaries()
+        self.assertEqual(PackagePublishingStatus.PUBLISHED, bpph.status)
 
 
 def test_suite():
-    suite = unittest.TestSuite()
+    suite = TestLoader().loadTestsFromName(__name__)
     suite.addTest(DocTestSuite(lp.soyuz.scripts.gina.handlers))
     return suite

=== modified file 'scripts/gina.py'
--- scripts/gina.py	2011-08-23 08:35:13 +0000
+++ scripts/gina.py	2011-09-09 05:48:24 +0000
@@ -38,6 +38,7 @@
     MangledArchiveError,
     PackagesMap,
     )
+from lp.soyuz.scripts.gina.dominate import dominate_imported_source_packages
 from lp.soyuz.scripts.gina.handlers import (
     DataSetupError,
     ImporterHandler,
@@ -152,6 +153,10 @@
         packages_map, kdb, package_root, keyrings, importer_handler)
     importer_handler.commit()
 
+    # XXX JeroenVermeulen 2011-09-07 bug=843728: Dominate binaries as well.
+    dominate_imported_source_packages(
+        log, distro, distroseries, pocket, packages_map)
+
     if source_only:
         log.info('Source only mode... done')
         return
@@ -209,9 +214,8 @@
     npacks = len(packages_map.src_map)
     log.info('%i Source Packages to be imported', npacks)
 
-    for list_source in sorted(
-        packages_map.src_map.values(), key=lambda x: x[0].get("Package")):
-        for source in list_source:
+    for package in sorted(packages_map.src_map.iterkeys()):
+        for source in packages_map.src_map[package]:
             count += 1
             attempt_source_package_import(
                 source, kdb, package_root, keyrings, importer_handler)
@@ -244,10 +248,9 @@
         log.info(
             '%i Binary Packages to be imported for %s', npacks, archtag)
         # Go over binarypackages importing them for this architecture
-        for binary in sorted(packages_map.bin_map[archtag].values(),
-                             key=lambda x: x.get("Package")):
+        for package_name in sorted(packages_map.bin_map[archtag].iterkeys()):
+            binary = packages_map.bin_map[archtag][package_name]
             count += 1
-            package_name = binary.get("Package", "unknown")
             try:
                 try:
                     do_one_binarypackage(binary, archtag, kdb, package_root,