Merge lp:~jtv/launchpad/transitional-published into lp:launchpad

Proposed by Jeroen T. Vermeulen
Status: Merged
Approved by: William Grant
Approved revision: no longer in the source branch.
Merged at revision: 13911
Proposed branch: lp:~jtv/launchpad/transitional-published
Merge into: lp:launchpad
Prerequisite: lp:~jtv/launchpad/bug-844577
Diff against target: 178 lines (+95/-24)
3 files modified
lib/lp/soyuz/doc/gina.txt (+24/-24)
lib/lp/soyuz/scripts/gina/dominate.py (+18/-0)
lib/lp/soyuz/scripts/tests/test_gina.py (+53/-0)
To merge this branch: bzr merge lp:~jtv/launchpad/transitional-published
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email: mp+74720@code.launchpad.net

This proposal supersedes a proposal from 2011-09-09.

Commit message

Convert any remaining Pending Debian SPPHs before transitional domination.

Description of the change

= Summary =

Gina is learning to run domination on its Debian imports. Yes, technically that makes Gina a dominatrix.

There is a catch: domination operates only on publication records in Published state. So far, Gina has been creating Pending records. That made sense for Ubuntu's import into Launchpad, way back when, which is what Gina was originally written for. It does not make sense now, so we changed the code and updated the legacy data in batches.

Of course that does not take care of further legacy data that is created between the update we just did in the database and the time the Gina changes roll out. But it should reduce the problem's size enough that we can afford to do the final update from inside Gina. This will only need to run once, during ‚Äútransitional domination‚ÄĚ but will do nothing if run repeatedly.

== Pre-implementation notes ==

I had wanted to do a final patch-up update later, but William's points out that there is quite a substantial risk if domination runs on partially updated data. So I inserted the change directly into the code; I have another branch waiting to remove the transitional code.

== Implementation details ==

The diff starts with some modernization of a doctest (using active_publishing_status instead of spelling out the Published and Pending statuses). This is essentially cosmetic; it does not change the meaning of the code and really isn't relevant to the branch. Buried in there however is one change where the test still assumes that Gina produces and maintains Pending publication records. Before we remove transitional domination, we may have to re-think what happens there.

== Tests ==

Two new tests verify transitional behaviour. One is very simple: an active SPPH is Pending but really ought to be Published. Gina dominates and the SPPH's version is found in the simulated Sources list. The Pending SPPH becomes Published.

Another test looks for the doom scenario that might occur when data is not properly migrated: an older release of a package is Published, but the version that is mentioned in the Sources list is still Pending. It is essential that the live version be upgraded to Published before domination, or the dominator would decide that there is no newer published version to dominate the now-obsolete published SPPH, and mark it deleted. Needless to say, the test shows doom being avoided.

{{{
./bin/test -vvc lp.soyuz.scripts.tests.test_gina
}}}

== Demo and Q/A ==

= Launchpad lint =

There is some pre-existing lint, especially in the doctest, that I can't afford to do too much about. It's not just the effort (mostly a matter of running utilities/formatdoctest.py!) but also a matter of keeping the diff small. I'm piling up quite a number of differences in a series of interdependent branches, and this would increase the risk of conflicts disproportionately.

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/archivepublisher/domination.py
  lib/lp/soyuz/scripts/gina/dominate.py
  lib/lp/soyuz/model/publishing.py
  lib/lp/soyuz/doc/gina.txt
  lib/lp/soyuz/interfaces/publishing.py
  scripts/gina.py
  lib/lp/archivepublisher/tests/test_dominator.py
  lib/lp/soyuz/scripts/tests/test_gina.py
  lib/lp/soyuz/scripts/gina/handlers.py

./lib/lp/soyuz/doc/gina.txt
     113: narrative exceeds 78 characters.
     162: want exceeds 78 characters.
     179: want exceeds 78 characters.
     189: narrative uses a moin header.
     221: want exceeds 78 characters.
     234: want exceeds 78 characters.
     240: want exceeds 78 characters.
     295: source exceeds 78 characters.
     324: narrative uses a moin header.
     342: narrative exceeds 78 characters.
     354: narrative uses a moin header.
     360: narrative exceeds 78 characters.
     361: narrative exceeds 78 characters.
     459: narrative uses a moin header.
     461: narrative exceeds 78 characters.
     462: narrative exceeds 78 characters.
     477: narrative uses a moin header.
     563: narrative exceeds 78 characters.
     600: narrative uses a moin header.
     657: narrative uses a moin header.
     746: narrative uses a moin header.
     767: narrative uses a moin header.
     780: narrative uses a moin header.
./lib/lp/soyuz/interfaces/publishing.py
     381: E261 at least two spaces before inline comment
     478: E261 at least two spaces before inline comment
     511: E261 at least two spaces before inline comment
     681: E261 at least two spaces before inline comment
     767: E261 at least two spaces before inline comment
./scripts/gina.py
      26: '_pythonpath' imported but unused

To post a comment you must log in.
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/soyuz/doc/gina.txt'
2--- lib/lp/soyuz/doc/gina.txt 2011-07-29 11:35:28 +0000
3+++ lib/lp/soyuz/doc/gina.txt 2011-09-09 05:51:26 +0000
4@@ -8,6 +8,7 @@
5 Get the current counts of stuff in the database:
6
7 >>> from canonical.launchpad.database.emailaddress import EmailAddress
8+ >>> from lp.soyuz.interfaces.publishing import active_publishing_status
9 >>> from lp.soyuz.model.publishing import (
10 ... BinaryPackagePublishingHistory,
11 ... SourcePackagePublishingHistory)
12@@ -564,35 +565,34 @@
13 that's what overrides actually do.
14
15 >>> from canonical.database.sqlbase import sqlvalues
16- >>> from lp.soyuz.enums import PackagePublishingStatus
17- >>> x11_pub = SSPPH.select("""sourcepackagerelease = %s
18- ... AND distroseries = %s
19- ... AND status in (%s, %s)""" %
20- ... sqlvalues(x11p, breezy,
21- ... PackagePublishingStatus.PUBLISHED,
22- ... PackagePublishingStatus.PENDING),
23- ... orderBy=["-datecreated"])[0]
24+ >>> x11_pub = SSPPH.select("""
25+ ... sourcepackagerelease = %s AND
26+ ... distroseries = %s AND
27+ ... status in %s
28+ ... """ % sqlvalues(
29+ ... x11p, breezy, active_publishing_status),
30+ ... orderBy=["-datecreated"])[0]
31 >>> print x11_pub.section.name
32 net
33- >>> ed_pub = SBPPH.select("""binarypackagerelease = %s
34- ... AND distroarchseries = %s
35- ... AND status in (%s, %s)""" %
36- ... sqlvalues(ed, breezy_i386,
37- ... PackagePublishingStatus.PUBLISHED,
38- ... PackagePublishingStatus.PENDING),
39- ... orderBy=["-datecreated"])[0]
40+ >>> ed_pub = SBPPH.select("""
41+ ... binarypackagerelease = %s AND
42+ ... distroarchseries = %s AND
43+ ... status in %s
44+ ... """ % sqlvalues(
45+ ... ed, breezy_i386, active_publishing_status),
46+ ... orderBy=["-datecreated"])[0]
47 >>> print ed_pub.priority
48 Extra
49 >>> n = SourcePackageName.selectOneBy(name="archive-copier")
50 >>> ac = SourcePackageRelease.selectOneBy(sourcepackagenameID=n.id,
51 ... version="0.3.6")
52- >>> ac_pub = SSPPH.select("""sourcepackagerelease = %s
53- ... AND distroseries = %s
54- ... AND status in (%s, %s)""" %
55- ... sqlvalues(ac, breezy,
56- ... PackagePublishingStatus.PUBLISHED,
57- ... PackagePublishingStatus.PENDING),
58- ... orderBy=["-datecreated"])[0]
59+ >>> ac_pub = SSPPH.select("""
60+ ... sourcepackagerelease = %s AND
61+ ... distroseries = %s AND
62+ ... status in %s
63+ ... """ % sqlvalues(
64+ ... ac, breezy, active_publishing_status),
65+ ... orderBy=["-datecreated"])[0]
66 >>> print ac_pub.component.name
67 universe
68
69@@ -720,7 +720,7 @@
70
71 >>> transaction.commit()
72
73-There is now a number of source publications in PENDING status for the
74+There is now a number of source publications in PUBLISHED status for the
75 targetted distroseries, 'lenny'.
76
77 >>> lenny_sources = SSPPH.select("distroseries = %s" % sqlvalues(lenny))
78@@ -728,7 +728,7 @@
79 12
80
81 >>> print set([pub.status.name for pub in lenny_sources])
82- set(['PENDING'])
83+ set(['PUBLISHED'])
84
85 As mentioned before, lenny/i386 is empty, no binaries were imported.
86 Also, the number of binaries published in the whole debian distribution
87
88=== modified file 'lib/lp/soyuz/scripts/gina/dominate.py'
89--- lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:51:25 +0000
90+++ lib/lp/soyuz/scripts/gina/dominate.py 2011-09-09 05:51:26 +0000
91@@ -27,6 +27,24 @@
92 series = getUtility(IDistributionSet)[distro_name].getSeries(series_name)
93 dominator = Dominator(logger, series.main_archive)
94
95+ # XXX JeroenVermeulen 2011-09-08, bug=844550: This is a transitional
96+ # hack. Gina used to create SPPHs in Pending state. We cleaned up
97+ # the bulk of them, and changed the code to create Published ones, but
98+ # some new ones will have been created since.
99+ # Update those to match what the new Gina does.
100+ from canonical.launchpad.interfaces.lpstorm import IStore
101+ from lp.soyuz.enums import PackagePublishingStatus
102+ from lp.soyuz.model.publishing import SourcePackagePublishingHistory
103+ SPPH = SourcePackagePublishingHistory
104+ store = IStore(SPPH)
105+ spphs = store.find(
106+ SPPH,
107+ SPPH.archive == series.main_archive,
108+ SPPH.distroseries == series,
109+ SPPH.pocket == pocket,
110+ SPPH.status == PackagePublishingStatus.PENDING)
111+ spphs.set(status=PackagePublishingStatus.PUBLISHED)
112+
113 # Dominate packages found in the Sources list we're importing.
114 package_names = dominator.findPublishedSourcePackageNames(series, pocket)
115 for package_name in package_names:
116
117=== modified file 'lib/lp/soyuz/scripts/tests/test_gina.py'
118--- lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:51:25 +0000
119+++ lib/lp/soyuz/scripts/tests/test_gina.py 2011-09-09 05:51:26 +0000
120@@ -77,6 +77,59 @@
121 ],
122 [pub.status for pub in pubs])
123
124+ def test_dominate_imported_source_packages_cleans_up_pending_spphs(self):
125+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
126+ # Gina domination, dominate_imported_source_packages turns any
127+ # remaining Pending SPPHS into Published ones.
128+ series = self.factory.makeDistroSeries()
129+ spph = self.factory.makeSourcePackagePublishingHistory(
130+ distroseries=series, archive=series.main_archive,
131+ status=PackagePublishingStatus.PENDING)
132+ spr = spph.sourcepackagerelease
133+ package_name = spr.sourcepackagename.name
134+ logger = DevNullLogger()
135+ dominate_imported_source_packages(
136+ logger, series.distribution.name, series.name, spph.pocket,
137+ FakePackagesMap({package_name: [{"Version": spr.version}]}))
138+ self.assertEqual(PackagePublishingStatus.PUBLISHED, spph.status)
139+
140+ def test_dominate_imported_source_packages_cleans_up_first(self):
141+ # XXX JeroenVermeulen 2011-09-08, bug=844550: For transition to
142+ # Gina domination, dominate_imported_source_packages turns any
143+ # remaining Pending SPPHS into Published ones. It does this
144+ # *before* dominating, so no domination happens while some of
145+ # the SPPHs are still mistakenly Pending (which would result in
146+ # mistaken deletions).
147+ series = self.factory.makeDistroSeries()
148+ package = self.factory.makeSourcePackageName()
149+ pocket = PackagePublishingPocket.RELEASE
150+ versions = ['1.0', '1.1']
151+ statuses_before = [
152+ PackagePublishingStatus.PUBLISHED,
153+ PackagePublishingStatus.PENDING,
154+ ]
155+ statuses_after = [
156+ PackagePublishingStatus.SUPERSEDED,
157+ PackagePublishingStatus.PUBLISHED,
158+ ]
159+ live_version = versions[-1]
160+ sprs = [
161+ self.factory.makeSourcePackageRelease(
162+ sourcepackagename=package, version=version)
163+ for version in versions]
164+ spphs = [
165+ self.factory.makeSourcePackagePublishingHistory(
166+ archive=series.main_archive, distroseries=series,
167+ sourcepackagerelease=spr, pocket=pocket, status=status)
168+ for spr, status in zip(sprs, statuses_before)]
169+
170+ logger = DevNullLogger()
171+ dominate_imported_source_packages(
172+ logger, series.distribution.name, series.name, pocket,
173+ FakePackagesMap({package.name: [{"Version": live_version}]}))
174+
175+ self.assertEqual(statuses_after, [spph.status for spph in spphs])
176+
177
178 class TestSourcePackagePublisher(TestCaseWithFactory):
179