Launchpad itself

Merge lp:~cjwatson/launchpad/packageset-score into lp:launchpad

  1. packageset-score
  2. Merge into devel
Proposed by Colin Watson on 2012-05-16
Status: Merged
Approved by: Francesco Banconi on 2012-05-16
Approved revision: no longer in the source branch.
Merged at revision: 15264
Proposed branch: lp:~cjwatson/launchpad/packageset-score
Merge into: lp:launchpad
Diff against target: 815 lines (+262/-362)
10 files modified
database/schema/patch-2209-18-0.sql (+2/-0)
lib/lp/buildmaster/tests/test_buildqueue.py (+17/-1)
lib/lp/security.py (+5/-1)
lib/lp/soyuz/configure.zcml (+6/-1)
lib/lp/soyuz/doc/buildd-scoring.txt (+0/-262)
lib/lp/soyuz/interfaces/packageset.py (+11/-2)
lib/lp/soyuz/model/buildpackagejob.py (+12/-6)
lib/lp/soyuz/model/packageset.py (+3/-1)
lib/lp/soyuz/tests/test_buildpackagejob.py (+206/-83)
lib/lp/soyuz/tests/test_doc.py (+0/-5)
To merge this branch: bzr merge lp:~cjwatson/launchpad/packageset-score
Reviewer Review Type Date Requested Status
Raphaël Badin (community) 2012-05-16 Approve on 2012-05-16
Francesco Banconi (community) code* 2012-05-16 Approve on 2012-05-16
Review via email: mp+105915@code.launchpad.net

Commit Message

Allow members of launchpad-buildd-admins to set a build score bonus for a package set.

Description of the Change

== Summary ==

Make it possible to adjust build scores by packagesets, so that we can favour builds that are more likely to be needed to build images. See bug 990219.

== Proposed fix ==

In a previous branch, I added a score column on Packageset. This adds the code to make use of it, and a webservice method restricted to launchpad-buildd-admins to set it.

== Pre-implementation notes ==

I discussed this briefly with William Grant on IRC; the notes are in the bug. I discussed the permission handling with Curtis Hovey.

== Implementation details ==

The main awkwardness was where to put the webservice method to set the score, since nothing else in Packageset is restricted to launchpad-buildd-admins. In the end, at Curtis' suggestion, I created a new interface requiring launchpad.Moderate, and made that be AdminByBuilddAdmins. This might get more complicated in future if Packageset needs to grow new methods with other restrictions, but we can cross that bridge if and when we come to it.

If a package is in multiple package sets, it gets the maximum of their scores, not (say) the sum. The latter seems likely to be overkill.

As usual, I had to spend some time reducing LoC. I did some refactoring of test_buildpackagejob, which helped, but wasn't enough on its own. Eventually I noticed that the buildd-scoring doctest covers substantially the same ground as test_buildpackagejob, so I destroyed it and made sure that everything previously in it is now covered by unit tests.

== Tests ==

bin/test -vvct TestBuildPackageJobScore -t TestBuildQueueManual

== Demo and Q/A ==

Add hello to some packageset on dogfood and set that packageset's score to something non-zero. Upload hello to quantal on dogfood with urgency=low. Its initial score should be the RELEASE/main/low default of 2505 plus the packageset score.

Double-check, for good measure, that a user not in launchpad-buildd-admins can't set a packageset score even if they can upload packages in that packageset. However, they should be able to see the score in the API.

== Lint ==

Just one:

./database/schema/patch-2209-18-0.sql
       8: Line exceeds 80 characters.

This is a COMMENT statement, where the convention appears to be to not wrap.

To post a comment you must log in.
Francesco Banconi (frankban) wrote :

This branch looks great Colin, thank you!
I am a little confused about how to actually demo your changes, and I think that's mostly my fault due to my my lack of experience with the API.
However, the tests pass, and I liked a lot how you reduced LoC by removing a doctests file and refactoring `test_buildpackagejob`.
Approved, and waiting for Raphaël suggestions.

review: Approve (code*)
Raphaël Badin (rvb) wrote :

Great work! As Francesco said, congrats for turning the doc tests into proper unit tests.

[0]

551 + for sourcename in (
552 + "gedit",
553 + "firefox",
554 + "cobblers",
555 + "thunderpants",
556 + "apg",
557 + "vim",
558 + "gcc",
559 + "bison",
560 + "flex",
561 + "postgres",
562 + ):

You've made this code much more compact and readable but maybe you can go one step further and avoid having the list defined in the loop statement: please consider using this construct:
sourcenames = [
    "gedit",
    "firefox",
    [...]
    "cobblers",
    "thunderpants"
    ]
for sourcename in sourcenames:
    [...]

[1]

614 + # 1500 (RELEASE) + 1000 (main) + 5 (low) = 2505.
615 + job = self.makeBuildJob(component="main", urgency="low")
616 + self.assertEqual(2505, job.score())

Maybe you could use:
self.assertEqual(
    SCORE_BY_POCKET[PackagePublishingPocket.RELEASE] + SCORE_BY_COMPONENT['main'] +
        SCORE_BY_URGENCY[SourcePackageUrgency.LOW],
    job.score())
instead of putting the integer value (2505). This would help document the code in the other tests similar to this one where no comment is present and it also would avoid hardcoding the integer values in the tests. I know these values are probably not gonna change soon but I think it's still a healthy way to write more explicit tests

[2]

I think you're missing a test to make sure that 'score' can be read through the api. Strickly speaking, it's tested in test_score_packageset_allows_buildd_admin but please consider adding a simple test for that. My idea is that if by any chance someone removes that field on the API, the name of the failing tests should make the problem obvious to solve. Right now, the failure will sort of imply that something is wrong with the permissions.

review: Approve
Colin Watson (cjwatson) wrote :

Thanks for your review and suggestions. I believe I've implemented all
of these now, along with some further changes to speed up the webservice
tests.

If you're happy with this, could you set the MP status to Approved and
then I can land it? (I can't do that myself as I'm not in ~launchpad.)

Julian Edwards (julian-edwards) wrote :

Great, thanks for removing the doctest, that's awesome!

However, the new PackageSet.score is a terrible name :( The comparable field on IArchive is called relative_build_score, we could have been consistent with that.

I have filed bug 1000570 to track this.

Colin Watson (cjwatson) wrote :

The property name (aside from the DB column name) is now fixed courtesy of a later merge from this branch.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'database/schema/patch-2209-18-0.sql'
2--- database/schema/patch-2209-18-0.sql 2012-05-08 18:55:36 +0000
3+++ database/schema/patch-2209-18-0.sql 2012-05-16 17:33:19 +0000
4@@ -5,4 +5,6 @@
5
6 ALTER TABLE Packageset ADD COLUMN score INTEGER DEFAULT 0 NOT NULL;
7
8+COMMENT ON COLUMN Packageset.score IS 'Build score bonus for packages in this package set.';
9+
10 INSERT INTO LaunchpadDatabaseRevision VALUES (2209, 18, 0);
11
12=== modified file 'lib/lp/buildmaster/tests/test_buildqueue.py'
13--- lib/lp/buildmaster/tests/test_buildqueue.py 2012-01-01 02:58:52 +0000
14+++ lib/lp/buildmaster/tests/test_buildqueue.py 2012-05-16 17:33:19 +0000
15@@ -1,4 +1,4 @@
16-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
17+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
18 # GNU Affero General Public License version 3 (see the file LICENSE).
19 # pylint: disable-msg=C0324
20
21@@ -1412,3 +1412,19 @@
22 assign_to_builder(self, 'xxr-daptup', 1, None)
23 postgres_build, postgres_job = find_job(self, 'postgres', '386')
24 check_estimate(self, postgres_job, 120)
25+
26+
27+class TestBuildQueueManual(TestCaseWithFactory):
28+ layer = ZopelessDatabaseLayer
29+
30+ def _makeBuildQueue(self):
31+ """Produce a `BuildQueue` object to test."""
32+ return self.factory.makeSourcePackageRecipeBuildJob()
33+
34+ def test_manualScore_prevents_rescoring(self):
35+ # Manually-set scores are fixed.
36+ buildqueue = self._makeBuildQueue()
37+ initial_score = buildqueue.lastscore
38+ buildqueue.manualScore(initial_score + 5000)
39+ buildqueue.score()
40+ self.assertEqual(initial_score + 5000, buildqueue.lastscore)
41
42=== modified file 'lib/lp/security.py'
43--- lib/lp/security.py 2012-05-14 23:59:33 +0000
44+++ lib/lp/security.py 2012-05-16 17:33:19 +0000
45@@ -12,7 +12,6 @@
46 ]
47
48 from zope.component import (
49- getAdapter,
50 getUtility,
51 queryAdapter,
52 )
53@@ -2654,6 +2653,11 @@
54 return user.isOwner(self.obj) or user.in_admin
55
56
57+class ModeratePackageset(AdminByBuilddAdmin):
58+ permission = 'launchpad.Moderate'
59+ usedfor = IPackageset
60+
61+
62 class EditPackagesetSet(AuthorizationBase):
63 permission = 'launchpad.Edit'
64 usedfor = IPackagesetSet
65
66=== modified file 'lib/lp/soyuz/configure.zcml'
67--- lib/lp/soyuz/configure.zcml 2012-05-04 13:21:43 +0000
68+++ lib/lp/soyuz/configure.zcml 2012-05-16 17:33:19 +0000
69@@ -1,4 +1,4 @@
70-<!-- Copyright 2009-2011 Canonical Ltd. This software is licensed under the
71+<!-- Copyright 2009-2012 Canonical Ltd. This software is licensed under the
72 GNU Affero General Public License version 3 (see the file LICENSE).
73 -->
74
75@@ -784,9 +784,14 @@
76 class="lp.soyuz.model.packageset.Packageset">
77 <allow
78 interface="lp.soyuz.interfaces.packageset.IPackagesetViewOnly"/>
79+ <allow
80+ interface="lp.soyuz.interfaces.packageset.IPackagesetRestricted"/>
81 <require
82 permission="launchpad.Edit"
83 interface="lp.soyuz.interfaces.packageset.IPackagesetEdit"/>
84+ <require
85+ permission="launchpad.Moderate"
86+ set_schema="lp.soyuz.interfaces.packageset.IPackagesetRestricted"/>
87 </class>
88 <class
89 class="lp.soyuz.model.packageset.PackagesetSet">
90
91=== removed file 'lib/lp/soyuz/doc/buildd-scoring.txt'
92--- lib/lp/soyuz/doc/buildd-scoring.txt 2012-01-20 15:42:44 +0000
93+++ lib/lp/soyuz/doc/buildd-scoring.txt 1970-01-01 00:00:00 +0000
94@@ -1,262 +0,0 @@
95-Buildd Scoring
96-==============
97-
98-Some tests for build jobs scoring implementation, which envolves the
99-analysis of each job pending in the queue. The actions to be performed are
100-described in <https://launchpad.canonical.com/AutoBuildManagement>.
101-A summary:
102-
103- * ETA to build (smaller == more points)
104- * Time spent in build queue (longer == more points)
105- * urgency
106- * priority/seed/component (BASE|DESKTOP|SUPPORTED) [PEND]
107- * Overarching policy (SECURITY/UPDATES/RELEASE) [PEND]
108- * Per-archive score delta.
109-
110- >>> import datetime
111- >>> import pytz
112- >>> LOCAL_NOW = datetime.datetime.now(pytz.timezone('UTC'))
113-
114-Let's create a 'mock' class which emulate the real behaviour of
115-BuildQueue entries.
116-
117- >>> from lp.registry.interfaces.distribution import IDistributionSet
118-
119- >>> ubuntu = getUtility(IDistributionSet)['ubuntu']
120- >>> hoary = ubuntu['hoary']
121- >>> hoary386 = hoary['i386']
122- >>> hoary386.title
123- u'The Hoary Hedgehog Release for i386 (x86)'
124-
125- >>> from lp.registry.interfaces.pocket import PackagePublishingPocket
126- >>> from lp.registry.interfaces.sourcepackage import SourcePackageUrgency
127- >>> from lp.soyuz.enums import PackagePublishingStatus
128- >>> from lp.soyuz.tests.test_publishing import (
129- ... SoyuzTestPublisher)
130- >>> from lp.testing.dbuser import (
131- ... lp_dbuser,
132- ... switch_dbuser,
133- ... )
134-
135- >>> test_publisher = SoyuzTestPublisher()
136-
137- >>> with lp_dbuser():
138- ... test_publisher.prepareBreezyAutotest()
139-
140- >>> version = 1
141-
142- >>> def setUpBuildQueueEntry(
143- ... component_name='main', urgency=SourcePackageUrgency.HIGH,
144- ... pocket=PackagePublishingPocket.RELEASE,
145- ... date_created=LOCAL_NOW, manual=False, archive=None):
146- ... global version
147- ... with lp_dbuser():
148- ... pub = test_publisher.getPubSource(
149- ... sourcename='test-build', version=str(version),
150- ... distroseries=hoary, component=component_name,
151- ... urgency=urgency, pocket=pocket,
152- ... status=PackagePublishingStatus.PUBLISHED, archive=archive)
153- ... version += 1
154- ... build = pub.sourcepackagerelease.createBuild(
155- ... hoary386, pub.pocket, pub.archive)
156- ...
157- ... build_queue = build.queueBuild()
158- ... from zope.security.proxy import removeSecurityProxy
159- ... naked_build_queue = removeSecurityProxy(build_queue)
160- ... naked_build_queue.job.date_created = date_created
161- ... naked_build_queue.manual = manual
162- ...
163- ... return build_queue
164-
165-
166- * 1500 points for pocket 'RELEASE',
167- * 1000 points for component 'main',
168- * 15 points for urgency HIGH.
169- * nothing for queue_time
170-
171- >>> bq0 = setUpBuildQueueEntry()
172-
173- >>> bq0.score()
174- >>> bq0.lastscore
175- 2515
176-
177-If the archive is private, its score is boosted by 10000:
178-
179- >>> switch_dbuser('launchpad')
180- >>> private_ppa = factory.makeArchive()
181- >>> private_ppa.buildd_secret = "secret"
182- >>> private_ppa.private = True
183- >>> bq1 = setUpBuildQueueEntry(archive=private_ppa)
184- >>> bq1.score()
185- >>> bq1.lastscore
186- 12515
187-
188-The archive can also have a delta applied to all its build scores. Setting
189-IArchive.relative_build_score to boost by 100 changes the lastscore value
190-appropriately.
191-
192- >>> private_ppa.relative_build_score = 100
193- >>> bq1.score()
194- >>> bq1.lastscore
195- 12615
196-
197-The delta can also be negative.
198-
199- >>> private_ppa.relative_build_score = -100
200- >>> bq1.score()
201- >>> bq1.lastscore
202- 12415
203-
204- >>> private_ppa.relative_build_score = 0
205- >>> switch_dbuser(test_dbuser)
206-
207-
208- * 1500 points for pocket 'RELEASE',
209- * 1000 points for main components
210- * 5 point for priority LOW
211- * nothing for queue_time
212-
213- >>> time1 = LOCAL_NOW - datetime.timedelta(seconds=290)
214- >>> bq1 = setUpBuildQueueEntry(
215- ... urgency=SourcePackageUrgency.LOW, date_created=time1)
216-
217- >>> bq1.score()
218- >>> bq1.lastscore
219- 2505
220-
221- * 1500 points for pocket 'RELEASE',
222- * 250 points for universe component universe
223- * 15 points for priority HIGH
224- * 5 points for queue_time ( > 300 seconds)
225-
226- >>> time2 = LOCAL_NOW - datetime.timedelta(seconds=310)
227- >>> bq2 = setUpBuildQueueEntry(
228- ... component_name='universe', urgency=SourcePackageUrgency.HIGH,
229- ... date_created=time2)
230-
231- >>> bq2.score()
232-
233- >>> bq2.lastscore
234- 1770
235-
236- * 1500 points for pocket 'RELEASE',
237- * nothing for component multiverse
238- * 10 points for MEDIUM priority
239- * 10 points for queue_time ( > 900 seconds)
240-
241- >>> time3 = LOCAL_NOW - datetime.timedelta(seconds=1000)
242- >>> bq3 = setUpBuildQueueEntry(
243- ... component_name='multiverse', urgency=SourcePackageUrgency.MEDIUM,
244- ... date_created=time3)
245-
246- >>> bq3.score()
247-
248- >>> bq3.lastscore
249- 1520
250-
251- * 1500 points for pocket 'RELEASE',
252- * 1000 points for main component
253- * 20 points for EMERGENCY priority
254- * 15 points for queue_time ( > 1800 seconds)
255-
256- >>> time4 = LOCAL_NOW - datetime.timedelta(seconds=1801)
257- >>> bq4 = setUpBuildQueueEntry(
258- ... component_name='main', urgency=SourcePackageUrgency.EMERGENCY,
259- ... date_created=time4)
260-
261- >>> bq4.score()
262- >>> bq4.lastscore
263- 2535
264-
265- * 1500 points for pocket 'RELEASE',
266- * 750 points for restricted component
267- * 5 points for LOW priority
268- * 20 points for queue_time ( > 3600 seconds)
269-
270- >>> time5 = LOCAL_NOW - datetime.timedelta(seconds=4000)
271- >>> bq5 = setUpBuildQueueEntry(
272- ... component_name='restricted', urgency=SourcePackageUrgency.LOW,
273- ... date_created=time5)
274-
275- >>> bq5.score()
276- >>> bq5.lastscore
277- 2275
278-
279-By setting manual attribute of a BuildQueue entry we prevent it to be
280-rescored, which allows us to set an arbitrary value on it.
281-
282- >>> time6 = LOCAL_NOW
283- >>> bq6 = setUpBuildQueueEntry(
284- ... urgency=SourcePackageUrgency.LOW, date_created=time6,
285- ... manual=True)
286-
287- >>> bq6.lastscore = 5000
288-
289- >>> bq6.score()
290-
291- >>> bq6.lastscore
292- 5000
293-
294-Let's see how the score varies for different publishing pockets.
295-
296-We will start with the lowest priority pocket: backports.
297-
298- >>> bq7 = setUpBuildQueueEntry(
299- ... pocket=PackagePublishingPocket.BACKPORTS)
300- >>> bq7.score()
301- >>> bq7.lastscore
302- 1015
303-
304-The score will increase by 3000 for the next ranked pocket: release.
305-
306- >>> bq8 = setUpBuildQueueEntry(
307- ... pocket=PackagePublishingPocket.RELEASE)
308- >>> bq8.score()
309- >>> bq8.lastscore
310- 2515
311-
312-Going to the next ranked pocket (PROPOSED or UPDATES) there will be a
313-score increase of 1500. The reason why PROPOSED and UPDATES have the
314-same priority is because sources in both pockets are submitted to the
315-same policy and should reach their audience as soon as possible (see
316-more information about this decision in bug #372491).
317-
318- >>> bq9 = setUpBuildQueueEntry(
319- ... pocket=PackagePublishingPocket.PROPOSED)
320- >>> bq9.score()
321- >>> bq9.lastscore
322- 4015
323-
324- >>> bqa = setUpBuildQueueEntry(
325- ... pocket=PackagePublishingPocket.UPDATES)
326- >>> bqa.score()
327- >>> bqa.lastscore
328- 4015
329-
330-Placing the build in the SECURITY pocket will push its score
331-up by another 1500.
332-
333- >>> bqb = setUpBuildQueueEntry(
334- ... pocket=PackagePublishingPocket.SECURITY)
335- >>> bqb.score()
336- >>> bqb.lastscore
337- 5515
338-
339-Builds in COPY archives have a score below zero, so they will only
340-be considered when there is nothing else to build. Even language-packs
341-and build retries will be built before them.
342-
343- >>> switch_dbuser('launchpad')
344- >>> from lp.soyuz.enums import ArchivePurpose
345- >>> from lp.soyuz.interfaces.archive import IArchiveSet
346- >>> copy = getUtility(IArchiveSet).new(
347- ... owner=ubuntu.owner, purpose=ArchivePurpose.COPY,
348- ... name='test-rebuild')
349-
350- >>> bqc = setUpBuildQueueEntry(archive=copy)
351- >>> from lp.soyuz.interfaces.binarypackagebuild import (
352- ... IBinaryPackageBuildSet)
353- >>> build = getUtility(IBinaryPackageBuildSet).getByQueueEntry(bqc)
354- >>> bqc.score()
355- >>> bqc.lastscore
356- -85
357
358=== modified file 'lib/lp/soyuz/interfaces/packageset.py'
359--- lib/lp/soyuz/interfaces/packageset.py 2011-12-24 16:54:44 +0000
360+++ lib/lp/soyuz/interfaces/packageset.py 2012-05-16 17:33:19 +0000
361@@ -1,4 +1,4 @@
362-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
363+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
364 # GNU Affero General Public License version 3 (see the file LICENSE).
365
366 # pylint: disable-msg=E0211,E0213
367@@ -348,7 +348,16 @@
368 """
369
370
371-class IPackageset(IPackagesetViewOnly, IPackagesetEdit):
372+class IPackagesetRestricted(Interface):
373+ """A writeable interface for restricted attributes of package sets."""
374+ export_as_webservice_entry(publish_web_link=False)
375+
376+ score = exported(Int(
377+ title=_("Build score"), required=True, readonly=False,
378+ description=_("Build score bonus for packages in this package set.")))
379+
380+
381+class IPackageset(IPackagesetViewOnly, IPackagesetEdit, IPackagesetRestricted):
382 """An interface for package sets."""
383 export_as_webservice_entry(publish_web_link=False)
384
385
386=== modified file 'lib/lp/soyuz/model/buildpackagejob.py'
387--- lib/lp/soyuz/model/buildpackagejob.py 2012-03-16 01:25:51 +0000
388+++ lib/lp/soyuz/model/buildpackagejob.py 2012-05-16 17:33:19 +0000
389@@ -1,4 +1,4 @@
390-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
391+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
392 # GNU Affero General Public License version 3 (see the file LICENSE).
393
394 __metaclass__ = type
395@@ -38,7 +38,9 @@
396 SCORE_BY_POCKET,
397 SCORE_BY_URGENCY,
398 )
399+from lp.soyuz.interfaces.packageset import IPackagesetSet
400 from lp.soyuz.model.buildfarmbuildjob import BuildFarmBuildJob
401+from lp.soyuz.model.packageset import Packageset
402
403
404 class BuildPackageJob(BuildFarmJobOldDerived, Storm):
405@@ -95,18 +97,22 @@
406 score = 0
407
408 # Calculates the urgency-related part of the score.
409- urgency = SCORE_BY_URGENCY[
410- self.build.source_package_release.urgency]
411- score += urgency
412+ score += SCORE_BY_URGENCY[self.build.source_package_release.urgency]
413
414 # Calculates the pocket-related part of the score.
415- score_pocket = SCORE_BY_POCKET[self.build.pocket]
416- score += score_pocket
417+ score += SCORE_BY_POCKET[self.build.pocket]
418
419 # Calculates the component-related part of the score.
420 score += SCORE_BY_COMPONENT.get(
421 self.build.current_component.name, 0)
422
423+ # Calculates the package-set-related part of the score.
424+ package_sets = getUtility(IPackagesetSet).setsIncludingSource(
425+ self.build.source_package_release.name,
426+ distroseries=self.build.distro_series)
427+ if not package_sets.is_empty():
428+ score += package_sets.max(Packageset.score)
429+
430 # Calculates the build queue time component of the score.
431 right_now = datetime.now(pytz.timezone('UTC'))
432 eta = right_now - self.job.date_created
433
434=== modified file 'lib/lp/soyuz/model/packageset.py'
435--- lib/lp/soyuz/model/packageset.py 2012-01-01 02:58:52 +0000
436+++ lib/lp/soyuz/model/packageset.py 2012-05-16 17:33:19 +0000
437@@ -1,4 +1,4 @@
438-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
439+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
440 # GNU Affero General Public License version 3 (see the file LICENSE).
441
442 __metaclass__ = type
443@@ -70,6 +70,8 @@
444 packagesetgroup_id = Int(name='packagesetgroup', allow_none=False)
445 packagesetgroup = Reference(packagesetgroup_id, 'PackagesetGroup.id')
446
447+ score = Int(allow_none=False)
448+
449 def add(self, data):
450 """See `IPackageset`."""
451 handlers = (
452
453=== modified file 'lib/lp/soyuz/tests/test_buildpackagejob.py'
454--- lib/lp/soyuz/tests/test_buildpackagejob.py 2012-01-01 02:58:52 +0000
455+++ lib/lp/soyuz/tests/test_buildpackagejob.py 2012-05-16 17:33:19 +0000
456@@ -1,34 +1,54 @@
457-# Copyright 2009 Canonical Ltd. This software is licensed under the
458+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
459 # GNU Affero General Public License version 3 (see the file LICENSE).
460
461 """Test BuildQueue features."""
462
463-from datetime import timedelta
464+from datetime import (
465+ datetime,
466+ timedelta,
467+ )
468
469+import pytz
470+from simplejson import dumps
471 from zope.component import getUtility
472 from zope.security.proxy import removeSecurityProxy
473
474 from lp.buildmaster.enums import BuildStatus
475 from lp.buildmaster.interfaces.builder import IBuilderSet
476+from lp.registry.interfaces.person import IPersonSet
477+from lp.registry.interfaces.pocket import PackagePublishingPocket
478+from lp.registry.interfaces.sourcepackage import SourcePackageUrgency
479 from lp.services.webapp.interfaces import (
480 DEFAULT_FLAVOR,
481 IStoreSelector,
482 MAIN_STORE,
483+ OAuthPermission,
484 )
485 from lp.soyuz.enums import (
486 ArchivePurpose,
487 PackagePublishingStatus,
488 )
489 from lp.soyuz.interfaces.buildfarmbuildjob import IBuildFarmBuildJob
490-from lp.soyuz.interfaces.buildpackagejob import IBuildPackageJob
491+from lp.soyuz.interfaces.buildpackagejob import (
492+ COPY_ARCHIVE_SCORE_PENALTY,
493+ IBuildPackageJob,
494+ PRIVATE_ARCHIVE_SCORE_BONUS,
495+ SCORE_BY_COMPONENT,
496+ SCORE_BY_POCKET,
497+ SCORE_BY_URGENCY,
498+ )
499 from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
500 from lp.soyuz.model.processor import ProcessorFamilySet
501 from lp.soyuz.tests.test_publishing import SoyuzTestPublisher
502-from lp.testing import TestCaseWithFactory
503+from lp.testing import (
504+ api_url,
505+ TestCaseWithFactory,
506+ )
507 from lp.testing.layers import (
508 DatabaseFunctionalLayer,
509 LaunchpadZopelessLayer,
510 )
511+from lp.testing.pages import webservice_for_person
512
513
514 def find_job(test, name, processor='386'):
515@@ -147,60 +167,26 @@
516 self.non_ppa.require_virtualized = False
517
518 self.builds = []
519- self.builds.extend(
520- self.publisher.getPubSource(
521- sourcename="gedit", status=PackagePublishingStatus.PUBLISHED,
522- archive=self.non_ppa,
523- architecturehintlist='any').createMissingBuilds())
524- self.builds.extend(
525- self.publisher.getPubSource(
526- sourcename="firefox",
527- status=PackagePublishingStatus.PUBLISHED,
528- archive=self.non_ppa,
529- architecturehintlist='any').createMissingBuilds())
530- self.builds.extend(
531- self.publisher.getPubSource(
532- sourcename="cobblers",
533- status=PackagePublishingStatus.PUBLISHED,
534- archive=self.non_ppa,
535- architecturehintlist='any').createMissingBuilds())
536- self.builds.extend(
537- self.publisher.getPubSource(
538- sourcename="thunderpants",
539- status=PackagePublishingStatus.PUBLISHED,
540- archive=self.non_ppa,
541- architecturehintlist='any').createMissingBuilds())
542- self.builds.extend(
543- self.publisher.getPubSource(
544- sourcename="apg", status=PackagePublishingStatus.PUBLISHED,
545- archive=self.non_ppa,
546- architecturehintlist='any').createMissingBuilds())
547- self.builds.extend(
548- self.publisher.getPubSource(
549- sourcename="vim", status=PackagePublishingStatus.PUBLISHED,
550- archive=self.non_ppa,
551- architecturehintlist='any').createMissingBuilds())
552- self.builds.extend(
553- self.publisher.getPubSource(
554- sourcename="gcc", status=PackagePublishingStatus.PUBLISHED,
555- archive=self.non_ppa,
556- architecturehintlist='any').createMissingBuilds())
557- self.builds.extend(
558- self.publisher.getPubSource(
559- sourcename="bison", status=PackagePublishingStatus.PUBLISHED,
560- archive=self.non_ppa,
561- architecturehintlist='any').createMissingBuilds())
562- self.builds.extend(
563- self.publisher.getPubSource(
564- sourcename="flex", status=PackagePublishingStatus.PUBLISHED,
565- archive=self.non_ppa,
566- architecturehintlist='any').createMissingBuilds())
567- self.builds.extend(
568- self.publisher.getPubSource(
569- sourcename="postgres",
570- status=PackagePublishingStatus.PUBLISHED,
571- archive=self.non_ppa,
572- architecturehintlist='any').createMissingBuilds())
573+ sourcenames = [
574+ "gedit",
575+ "firefox",
576+ "cobblers",
577+ "thunderpants",
578+ "apg",
579+ "vim",
580+ "gcc",
581+ "bison",
582+ "flex",
583+ "postgres",
584+ ]
585+ for sourcename in sourcenames:
586+ self.builds.extend(
587+ self.publisher.getPubSource(
588+ sourcename=sourcename,
589+ status=PackagePublishingStatus.PUBLISHED,
590+ archive=self.non_ppa,
591+ architecturehintlist='any').createMissingBuilds())
592+
593 # We want the builds to have a lot of variety when it comes to score
594 # and estimated duration etc. so that the queries under test get
595 # exercised properly.
596@@ -257,9 +243,38 @@
597
598 layer = DatabaseFunctionalLayer
599
600+ def makeBuildJob(self, purpose=None, private=False, component="main",
601+ urgency="high", pocket="RELEASE", age=None):
602+ if purpose is not None or private:
603+ archive = self.factory.makeArchive(
604+ purpose=purpose, private=private)
605+ else:
606+ archive = None
607+ spph = self.factory.makeSourcePackagePublishingHistory(
608+ archive=archive, component=component, urgency=urgency)
609+ naked_spph = removeSecurityProxy(spph) # needed for private archives
610+ build = self.factory.makeBinaryPackageBuild(
611+ source_package_release=naked_spph.sourcepackagerelease,
612+ pocket=pocket)
613+ job = removeSecurityProxy(build).makeJob()
614+ if age is not None:
615+ removeSecurityProxy(job).job.date_created = (
616+ datetime.now(pytz.timezone("UTC")) - timedelta(seconds=age))
617+ return job
618+
619+ # The defaults for pocket, component, and urgency here match those in
620+ # makeBuildJob.
621+ def assertCorrectScore(self, job, pocket="RELEASE", component="main",
622+ urgency="high", other_bonus=0):
623+ self.assertEqual(
624+ (SCORE_BY_POCKET[PackagePublishingPocket.items[pocket.upper()]] +
625+ SCORE_BY_COMPONENT[component] +
626+ SCORE_BY_URGENCY[SourcePackageUrgency.items[urgency.upper()]] +
627+ other_bonus), job.score())
628+
629 def test_score_unusual_component(self):
630 spph = self.factory.makeSourcePackagePublishingHistory(
631- component='unusual')
632+ component="unusual")
633 build = self.factory.makeBinaryPackageBuild(
634 source_package_release=spph.sourcepackagerelease)
635 build.queueBuild()
636@@ -268,31 +283,139 @@
637 job.score()
638
639 def test_main_release_low_score(self):
640- spph = self.factory.makeSourcePackagePublishingHistory(
641- component='main', urgency='low')
642- build = self.factory.makeBinaryPackageBuild(
643- source_package_release=spph.sourcepackagerelease,
644- pocket='RELEASE')
645- job = build.makeJob()
646- self.assertEquals(2505, job.score())
647+ # 1500 (RELEASE) + 1000 (main) + 5 (low) = 2505.
648+ job = self.makeBuildJob(component="main", urgency="low")
649+ self.assertCorrectScore(job, "RELEASE", "main", "low")
650
651 def test_copy_archive_main_release_low_score(self):
652- copy_archive = self.factory.makeArchive(purpose='COPY')
653- spph = self.factory.makeSourcePackagePublishingHistory(
654- archive=copy_archive, component='main', urgency='low')
655- build = self.factory.makeBinaryPackageBuild(
656- source_package_release=spph.sourcepackagerelease,
657- pocket='RELEASE')
658- job = build.makeJob()
659- self.assertEquals(-95, job.score())
660+ # 1500 (RELEASE) + 1000 (main) + 5 (low) - 2600 (copy archive) = -95.
661+ # With this penalty, even language-packs and build retries will be
662+ # built before copy archives.
663+ job = self.makeBuildJob(
664+ purpose="COPY", component="main", urgency="low")
665+ self.assertCorrectScore(
666+ job, "RELEASE", "main", "low", -COPY_ARCHIVE_SCORE_PENALTY)
667
668 def test_copy_archive_relative_score_is_applied(self):
669- copy_archive = self.factory.makeArchive(purpose='COPY')
670- removeSecurityProxy(copy_archive).relative_build_score = 2600
671- spph = self.factory.makeSourcePackagePublishingHistory(
672- archive=copy_archive, component='main', urgency='low')
673- build = self.factory.makeBinaryPackageBuild(
674- source_package_release=spph.sourcepackagerelease,
675- pocket='RELEASE')
676- job = build.makeJob()
677- self.assertEquals(2505, job.score())
678+ # Per-archive relative build scores are applied, in this case
679+ # exactly offsetting the copy-archive penalty.
680+ job = self.makeBuildJob(
681+ purpose="COPY", component="main", urgency="low")
682+ removeSecurityProxy(job.build.archive).relative_build_score = 2600
683+ self.assertCorrectScore(
684+ job, "RELEASE", "main", "low", -COPY_ARCHIVE_SCORE_PENALTY + 2600)
685+
686+ def test_archive_negative_relative_score_is_applied(self):
687+ # Negative per-archive relative build scores are allowed.
688+ job = self.makeBuildJob(component="main", urgency="low")
689+ removeSecurityProxy(job.build.archive).relative_build_score = -100
690+ self.assertCorrectScore(job, "RELEASE", "main", "low", -100)
691+
692+ def test_private_archive_bonus_is_applied(self):
693+ # Private archives get a bonus of 10000.
694+ job = self.makeBuildJob(private=True, component="main", urgency="high")
695+ self.assertCorrectScore(
696+ job, "RELEASE", "main", "high", PRIVATE_ARCHIVE_SCORE_BONUS)
697+
698+ def test_main_release_low_recent_score(self):
699+ # Builds created less than five minutes ago get no bonus.
700+ job = self.makeBuildJob(component="main", urgency="low", age=290)
701+ self.assertCorrectScore(job, "RELEASE", "main", "low")
702+
703+ def test_universe_release_high_five_minutes_score(self):
704+ # 1500 (RELEASE) + 250 (universe) + 15 (high) + 5 (>300s) = 1770.
705+ job = self.makeBuildJob(component="universe", urgency="high", age=310)
706+ self.assertCorrectScore(job, "RELEASE", "universe", "high", 5)
707+
708+ def test_multiverse_release_medium_fifteen_minutes_score(self):
709+ # 1500 (RELEASE) + 0 (multiverse) + 10 (medium) + 10 (>900s) = 1520.
710+ job = self.makeBuildJob(
711+ component="multiverse", urgency="medium", age=1000)
712+ self.assertCorrectScore(job, "RELEASE", "multiverse", "medium", 10)
713+
714+ def test_main_release_emergency_thirty_minutes_score(self):
715+ # 1500 (RELEASE) + 1000 (main) + 20 (emergency) + 15 (>1800s) = 2535.
716+ job = self.makeBuildJob(
717+ component="main", urgency="emergency", age=1801)
718+ self.assertCorrectScore(job, "RELEASE", "main", "emergency", 15)
719+
720+ def test_restricted_release_low_one_hour_score(self):
721+ # 1500 (RELEASE) + 750 (restricted) + 5 (low) + 20 (>3600s) = 2275.
722+ job = self.makeBuildJob(
723+ component="restricted", urgency="low", age=4000)
724+ self.assertCorrectScore(job, "RELEASE", "restricted", "low", 20)
725+
726+ def test_backports_score(self):
727+ # BACKPORTS is the lowest-priority pocket.
728+ job = self.makeBuildJob(pocket="BACKPORTS")
729+ self.assertCorrectScore(job, "BACKPORTS")
730+
731+ def test_release_score(self):
732+ # RELEASE ranks next above BACKPORTS.
733+ job = self.makeBuildJob(pocket="RELEASE")
734+ self.assertCorrectScore(job, "RELEASE")
735+
736+ def test_proposed_updates_score(self):
737+ # PROPOSED and UPDATES both rank next above RELEASE. The reason why
738+ # PROPOSED and UPDATES have the same priority is because sources in
739+ # both pockets are submitted to the same policy and should reach
740+ # their audience as soon as possible (see more information about
741+ # this decision in bug #372491).
742+ proposed_job = self.makeBuildJob(pocket="PROPOSED")
743+ self.assertCorrectScore(proposed_job, "PROPOSED")
744+ updates_job = self.makeBuildJob(pocket="UPDATES")
745+ self.assertCorrectScore(updates_job, "UPDATES")
746+
747+ def test_security_updates_score(self):
748+ # SECURITY is the top-ranked pocket.
749+ job = self.makeBuildJob(pocket="SECURITY")
750+ self.assertCorrectScore(job, "SECURITY")
751+
752+ def test_score_packageset(self):
753+ job = self.makeBuildJob(component="main", urgency="low")
754+ packageset = self.factory.makePackageset(
755+ distroseries=job.build.distro_series)
756+ removeSecurityProxy(packageset).add(
757+ [job.build.source_package_release.sourcepackagename])
758+ removeSecurityProxy(packageset).score = 100
759+ self.assertCorrectScore(job, "RELEASE", "main", "low", 100)
760+
761+ def test_score_packageset_readable(self):
762+ # A packageset's build score is readable by anyone.
763+ packageset = self.factory.makePackageset()
764+ removeSecurityProxy(packageset).score = 100
765+ webservice = webservice_for_person(
766+ self.factory.makePerson(), permission=OAuthPermission.WRITE_PUBLIC)
767+ entry = webservice.get(
768+ api_url(packageset), api_version="devel").jsonBody()
769+ self.assertEqual(100, entry["score"])
770+
771+ def test_score_packageset_forbids_non_buildd_admin(self):
772+ # Being the owner of a packageset is not enough to allow changing
773+ # its build score, since this affects a site-wide resource.
774+ person = self.factory.makePerson()
775+ packageset = self.factory.makePackageset(owner=person)
776+ webservice = webservice_for_person(
777+ person, permission=OAuthPermission.WRITE_PUBLIC)
778+ entry = webservice.get(
779+ api_url(packageset), api_version="devel").jsonBody()
780+ response = webservice.patch(
781+ entry["self_link"], "application/json", dumps(dict(score=100)))
782+ self.assertEqual(401, response.status)
783+ new_entry = webservice.get(
784+ api_url(packageset), api_version="devel").jsonBody()
785+ self.assertEqual(0, new_entry["score"])
786+
787+ def test_score_packageset_allows_buildd_admin(self):
788+ buildd_admins = getUtility(IPersonSet).getByName(
789+ "launchpad-buildd-admins")
790+ buildd_admin = self.factory.makePerson(member_of=[buildd_admins])
791+ packageset = self.factory.makePackageset()
792+ webservice = webservice_for_person(
793+ buildd_admin, permission=OAuthPermission.WRITE_PUBLIC)
794+ entry = webservice.get(
795+ api_url(packageset), api_version="devel").jsonBody()
796+ response = webservice.patch(
797+ entry["self_link"], "application/json", dumps(dict(score=100)))
798+ self.assertEqual(209, response.status)
799+ self.assertEqual(100, response.jsonBody()["score"])
800
801=== modified file 'lib/lp/soyuz/tests/test_doc.py'
802--- lib/lp/soyuz/tests/test_doc.py 2012-03-27 13:41:38 +0000
803+++ lib/lp/soyuz/tests/test_doc.py 2012-05-16 17:33:19 +0000
804@@ -132,11 +132,6 @@
805
806
807 special = {
808- 'buildd-scoring.txt': LayeredDocFileSuite(
809- '../doc/buildd-scoring.txt',
810- setUp=builddmasterSetUp,
811- layer=LaunchpadZopelessLayer,
812- ),
813 'package-cache.txt': LayeredDocFileSuite(
814 '../doc/package-cache.txt',
815 setUp=statisticianSetUp, tearDown=statisticianTearDown,