Merge lp:~jtv/launchpad/transitional-published into lp:launchpad

Proposed by Jeroen T. Vermeulen
Status: Merged
Approved by: William Grant
Approved revision: no longer in the source branch.
Merged at revision: 13911
Proposed branch: lp:~jtv/launchpad/transitional-published
Merge into: lp:launchpad
Prerequisite: lp:~jtv/launchpad/bug-844577
Diff against target: 178 lines (+95/-24)
3 files modified
lib/lp/soyuz/doc/gina.txt (+24/-24)
lib/lp/soyuz/scripts/gina/dominate.py (+18/-0)
lib/lp/soyuz/scripts/tests/test_gina.py (+53/-0)
To merge this branch: bzr merge lp:~jtv/launchpad/transitional-published
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email: mp+74720@code.launchpad.net

This proposal supersedes a proposal from 2011-09-09.

Commit message

Convert any remaining Pending Debian SPPHs before transitional domination.

Description of the change

= Summary =

Gina is learning to run domination on its Debian imports. Yes, technically that makes Gina a dominatrix.

There is a catch: domination operates only on publication records in Published state. So far, Gina has been creating Pending records. That made sense for Ubuntu's import into Launchpad, way back when, which is what Gina was originally written for. It does not make sense now, so we changed the code and updated the legacy data in batches.

Of course that does not take care of further legacy data that is created between the update we just did in the database and the time the Gina changes roll out. But it should reduce the problem's size enough that we can afford to do the final update from inside Gina. This will only need to run once, during ‚Äútransitional domination‚ÄĚ but will do nothing if run repeatedly.

== Pre-implementation notes ==

I had wanted to do a final patch-up update later, but William's points out that there is quite a substantial risk if domination runs on partially updated data. So I inserted the change directly into the code; I have another branch waiting to remove the transitional code.

== Implementation details ==

The diff starts with some modernization of a doctest (using active_publishing_status instead of spelling out the Published and Pending statuses). This is essentially cosmetic; it does not change the meaning of the code and really isn't relevant to the branch. Buried in there however is one change where the test still assumes that Gina produces and maintains Pending publication records. Before we remove transitional domination, we may have to re-think what happens there.

== Tests ==

Two new tests verify transitional behaviour. One is very simple: an active SPPH is Pending but really ought to be Published. Gina dominates and the SPPH's version is found in the simulated Sources list. The Pending SPPH becomes Published.

Another test looks for the doom scenario that might occur when data is not properly migrated: an older release of a package is Published, but the version that is mentioned in the Sources list is still Pending. It is essential that the live version be upgraded to Published before domination, or the dominator would decide that there is no newer published version to dominate the now-obsolete published SPPH, and mark it deleted. Needless to say, the test shows doom being avoided.

{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}

== Demo and Q/A ==

= Launchpad lint =

There is some pre-existing lint, especially in the doctest, that I can't afford to do too much about. It's not just the effort (mostly a matter of running utilities/formatdoctest.py!) but also a matter of keeping the diff small. I'm piling up quite a number of differences in a series of interdependent branches, and this would increase the risk of conflicts disproportionately.

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/domination.py
  lib/lp/soyuz/scripts/gina/dominate.py
  lib/lp/soyuz/model/publishing.py
  lib/lp/soyuz/doc/gina.txt
  lib/lp/soyuz/interfaces/publishing.py
  scripts/gina.py
  lib/lp/archivepublisher/tests/test_dominator.py
  lib/lp/soyuz/scripts/tests/test_gina.py
  lib/lp/soyuz/scripts/gina/handlers.py

./lib/lp/soyuz/doc/gina.txt
     113: narrative exceeds 78 characters.
     162: want exceeds 78 characters.
     179: want exceeds 78 characters.
     189: narrative uses a moin header.
     221: want exceeds 78 characters.
     234: want exceeds 78 characters.
     240: want exceeds 78 characters.
     295: source exceeds 78 characters.
     324: narrative uses a moin header.
     342: narrative exceeds 78 characters.
     354: narrative uses a moin header.
     360: narrative exceeds 78 characters.
     361: narrative exceeds 78 characters.
     459: narrative uses a moin header.
     461: narrative exceeds 78 characters.
     462: narrative exceeds 78 characters.
     477: narrative uses a moin header.
     563: narrative exceeds 78 characters.
     600: narrative uses a moin header.
     657: narrative uses a moin header.
     746: narrative uses a moin header.
     767: narrative uses a moin header.
     780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
     381: E261 at least two spaces before inline comment
     478: E261 at least two spaces before inline comment
     511: E261 at least two spaces before inline comment
     681: E261 at least two spaces before inline comment
     767: E261 at least two spaces before inline comment
./scripts/gina.py
      26: '_pythonpath' imported but unused

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/soyuz/doc/gina.txt'
--- lib/lp/soyuz/doc/gina.txt 2011-07-29 11:35:28 +0000
+++ lib/lp/soyuz/doc/gina.txt 2011-09-09 05:51:26 +0000
@@ -8,6 +8,7 @@
8Get the current counts of stuff in the database:8Get the current counts of stuff in the database:
99
10 >>> from canonical.launchpad.database.emailaddress import EmailAddress10 >>> from canonical.launchpad.database.emailaddress import EmailAddress
11 >>> from lp.soyuz.interfaces.publishing import active_publishing_status
11 >>> from lp.soyuz.model.publishing import (12 >>> from lp.soyuz.model.publishing import (
12 ... BinaryPackagePublishingHistory,13 ... BinaryPackagePublishingHistory,
13 ... SourcePackagePublishingHistory)14 ... SourcePackagePublishingHistory)
@@ -564,35 +565,34 @@
564that's what overrides actually do.565that's what overrides actually do.
565566
566 >>> from canonical.database.sqlbase import sqlvalues567 >>> from canonical.database.sqlbase import sqlvalues
567 >>> from lp.soyuz.enums import PackagePublishingStatus568 >>> x11_pub = SSPPH.select("""
568 >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s569 ... sourcepackagerelease = %s AND
569 ... AND distroseries = %s570 ... distroseries = %s AND
570 ... AND status in (%s, %s)""" %571 ... status in %s
571 ... sqlvalues(x11p, breezy,572 ... """ % sqlvalues(
572 ... PackagePublishingStatus.PUBLISHED,573 ... x11p, breezy, active_publishing_status),
573 ... PackagePublishingStatus.PENDING),574 ... orderBy=["-datecreated"])[0]
574 ... orderBy=["-datecreated"])[0]
575 >>> print x11_pub.section.name575 >>> print x11_pub.section.name
576 net576 net
577 >>> ed_pub = SBPPH.select("""binarypackagerelease = %s577 >>> ed_pub = SBPPH.select("""
578 ... AND distroarchseries = %s578 ... binarypackagerelease = %s AND
579 ... AND status in (%s, %s)""" %579 ... distroarchseries = %s AND
580 ... sqlvalues(ed, breezy_i386,580 ... status in %s
581 ... PackagePublishingStatus.PUBLISHED,581 ... """ % sqlvalues(
582 ... PackagePublishingStatus.PENDING),582 ... ed, breezy_i386, active_publishing_status),
583 ... orderBy=["-datecreated"])[0]583 ... orderBy=["-datecreated"])[0]
584 >>> print ed_pub.priority584 >>> print ed_pub.priority
585 Extra585 Extra
586 >>> n = SourcePackageName.selectOneBy(name="archive-copier")586 >>> n = SourcePackageName.selectOneBy(name="archive-copier")
587 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,587 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
588 ... version="0.3.6")588 ... version="0.3.6")
589 >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s589 >>> ac_pub = SSPPH.select("""
590 ... AND distroseries = %s590 ... sourcepackagerelease = %s AND
591 ... AND status in (%s, %s)""" %591 ... distroseries = %s AND
592 ... sqlvalues(ac, breezy,592 ... status in %s
593 ... PackagePublishingStatus.PUBLISHED,593 ... """ % sqlvalues(
594 ... PackagePublishingStatus.PENDING),594 ... ac, breezy, active_publishing_status),
595 ... orderBy=["-datecreated"])[0]595 ... orderBy=["-datecreated"])[0]
596 >>> print ac_pub.component.name596 >>> print ac_pub.component.name
597 universe597 universe
598598
@@ -720,7 +720,7 @@
720720
721 >>> transaction.commit()721 >>> transaction.commit()
722722
723There is now a number of source publications in PENDING status for the723There is now a number of source publications in PUBLISHED status for the
724targetted distroseries, 'lenny'.724targetted distroseries, 'lenny'.
725725
726 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))726 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
@@ -728,7 +728,7 @@
728 12728 12
729729
730 >>> print set([pub.status.name for pub in lenny_sources])730 >>> print set([pub.status.name for pub in lenny_sources])
731 set(['PENDING'])731 set(['PUBLISHED'])
732732
733As mentioned before, lenny/i386 is empty, no binaries were imported.733As mentioned before, lenny/i386 is empty, no binaries were imported.
734Also, the number of binaries published in the whole debian distribution734Also, the number of binaries published in the whole debian distribution
735735
=== modified file 'lib/lp/soyuz/scripts/gina/dominate.py'
--- lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:51:25 +0000
+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:51:26 +0000
@@ -27,6 +27,24 @@
27 series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)27 series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
28 dominator = Dominator(logger, series.main_archive)28 dominator = Dominator(logger, series.main_archive)
2929
30 # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
31 # hack. Gina used to create SPPHs in Pending state. We cleaned up
32 # the bulk of them, and changed the code to create Published ones, but
33 # some new ones will have been created since.
34 # Update those to match what the new Gina does.
35 from canonical.launchpad.interfaces.lpstorm import IStore
36 from lp.soyuz.enums import PackagePublishingStatus
37 from lp.soyuz.model.publishing import SourcePackagePublishingHistory
38 SPPH = SourcePackagePublishingHistory
39 store = IStore(SPPH)
40 spphs = store.find(
41 SPPH,
42 SPPH.archive == series.main_archive,
43 SPPH.distroseries == series,
44 SPPH.pocket == pocket,
45 SPPH.status == PackagePublishingStatus.PENDING)
46 spphs.set(status=PackagePublishingStatus.PUBLISHED)
47
30 # Dominate packages found in the Sources list we're importing.48 # Dominate packages found in the Sources list we're importing.
31 package_names = dominator.findPublishedSourcePackageNames(series, pocket)49 package_names = dominator.findPublishedSourcePackageNames(series, pocket)
32 for package_name in package_names:50 for package_name in package_names:
3351
=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
--- lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:51:25 +0000
+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:51:26 +0000
@@ -77,6 +77,59 @@
77 ],77 ],
78 [pub.status for pub in pubs])78 [pub.status for pub in pubs])
7979
80 def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
81 # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
82 # Gina domination, dominate_imported_source_packages turns any
83 # remaining Pending SPPHS into Published ones.
84 series = self.factory.makeDistroSeries()
85 spph = self.factory.makeSourcePackagePublishingHistory(
86 distroseries=series, archive=series.main_archive,
87 status=PackagePublishingStatus.PENDING)
88 spr = spph.sourcepackagerelease
89 package_name = spr.sourcepackagename.name
90 logger = DevNullLogger()
91 dominate_imported_source_packages(
92 logger, series.distribution.name, series.name, spph.pocket,
93 FakePackagesMap({package_name: [{"Version": spr.version}]}))
94 self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
95
96 def test_dominate_imported_source_packages_cleans_up_first(self):
97 # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
98 # Gina domination, dominate_imported_source_packages turns any
99 # remaining Pending SPPHS into Published ones. It does this
100 # *before* dominating, so no domination happens while some of
101 # the SPPHs are still mistakenly Pending (which would result in
102 # mistaken deletions).
103 series = self.factory.makeDistroSeries()
104 package = self.factory.makeSourcePackageName()
105 pocket = PackagePublishingPocket.RELEASE
106 versions = ['1.0', '1.1']
107 statuses_before = [
108 PackagePublishingStatus.PUBLISHED,
109 PackagePublishingStatus.PENDING,
110 ]
111 statuses_after = [
112 PackagePublishingStatus.SUPERSEDED,
113 PackagePublishingStatus.PUBLISHED,
114 ]
115 live_version = versions[-1]
116 sprs = [
117 self.factory.makeSourcePackageRelease(
118 sourcepackagename=package, version=version)
119 for version in versions]
120 spphs = [
121 self.factory.makeSourcePackagePublishingHistory(
122 archive=series.main_archive, distroseries=series,
123 sourcepackagerelease=spr, pocket=pocket, status=status)
124 for spr, status in zip(sprs, statuses_before)]
125
126 logger = DevNullLogger()
127 dominate_imported_source_packages(
128 logger, series.distribution.name, series.name, pocket,
129 FakePackagesMap({package.name: [{"Version": live_version}]}))
130
131 self.assertEqual(statuses_after, [spph.status for spph in spphs])
132
80133
81class TestSourcePackagePublisher(TestCaseWithFactory):134class TestSourcePackagePublisher(TestCaseWithFactory):
82135