Merge lp:~cjwatson/launchpad/queue-api into lp:launchpad

Proposed by Colin Watson
Status: Rejected
Rejected by: Colin Watson
Proposed branch: lp:~cjwatson/launchpad/queue-api
Merge into: lp:launchpad
Diff against target: 1647 lines (+802/-187)
16 files modified
lib/lp/archiveuploader/tests/nascentupload-ddebs.txt (+2/-1)
lib/lp/registry/interfaces/distroseries.py (+7/-0)
lib/lp/soyuz/browser/queue.py (+7/-3)
lib/lp/soyuz/configure.zcml (+6/-0)
lib/lp/soyuz/doc/distroseriesqueue.txt (+22/-26)
lib/lp/soyuz/interfaces/archive.py (+13/-1)
lib/lp/soyuz/interfaces/binarypackagerelease.py (+8/-1)
lib/lp/soyuz/interfaces/queue.py (+133/-31)
lib/lp/soyuz/model/binarypackagerelease.py (+13/-2)
lib/lp/soyuz/model/queue.py (+241/-36)
lib/lp/soyuz/scripts/queue.py (+3/-3)
lib/lp/soyuz/stories/webservice/xx-packageupload.txt (+13/-0)
lib/lp/soyuz/tests/test_distroseriesqueue_ddtp_tarball.py (+0/-28)
lib/lp/soyuz/tests/test_distroseriesqueue_dist_upgrader.py (+4/-27)
lib/lp/soyuz/tests/test_packageupload.py (+327/-26)
lib/lp/testing/factory.py (+3/-2)
To merge this branch: bzr merge lp:~cjwatson/launchpad/queue-api
Reviewer Review Type Date Requested Status
Launchpad code reviewers Pending
Review via email: mp+108967@code.launchpad.net

Commit message

Export enough of PackageUpload and DistroSeries.getPackageUploads to allow implementing an API version of the queue tool.

Description of the change

== Summary ==

Implement the next stage of https://code.launchpad.net/~cjwatson/launchpad/queue-api-accept-reject/+merge/107894, by exporting substantially more of PackageUpload.

== Proposed fix ==

Export enough of PackageUpload and DistroSeries.getPackageUploads to allow implementing an API client that replaces scripts/ftpmaster-tools/queue.

== Pre-implementation notes ==

I've gone round a few times with various people, particularly William Grant, on the exact way to export all of this stuff, because I gather that we want to avoid exposing the current data model in order that it can be rearranged in the future. This has led to the following design choices:

 * Everything is on devel. The only clients for this should be tools such as those in lp:ubuntu-archive-tools, which can be kept up to date if there's a need to change these interfaces.
 * Even though some of the underlying methods are on other objects, all the new exported methods are on PackageUpload rather than exporting anything else.
 * There are source packages with lots of binaries that sometimes need to be overridden individually (e.g. linux) and API requests aren't especially fast. I've therefore arranged for properties (including overrides) of all binaries in an upload to come back as a list of dicts in a single JSON response, and I've amended Archive.overrideBinaries to take a similar list of dicts as a "changes" parameter, allowing many override changes to be made in a single request.

== LOC Rationale ==

+615, on top of a previous branch that was +91. I think this is valid because this is part of an arc of work (resourced by Ubuntu Engineering) that will culminate in removing lib/lp/soyuz/scripts/queue.py and scripts/ftpmaster-tools/queue for at least -862. While it's possible there'll be one or two more bits and pieces, they shouldn't amount to any more than +156, so this whole arc will be LoC-negative.

== Tests ==

bin/test -vvct nascentupload-ddebs.txt -t distroseriesqueue.txt -t xx-packageupload.txt -t test_distroseriesqueue -t test_packageupload

== Demo and Q/A ==

http://paste.ubuntu.com/1026996/ is my prototype client; I plan to walk through all its functionality against qastaging (or dogfood if I need to make new uploads).

To post a comment you must log in.
Revision history for this message
Benji York (benji) wrote :

It is tough to give an appropriate amount of attention to a branch this big. I suggest breaking it into two or more branches that can get closer to our target of an 800 line diff (https://dev.launchpad.net/PreMergeReviews#line-38).

Perhaps one branch that does the refactoring needed for the interface and another that does the exposing, or maybe separate branches for exposing different subsets of the API.

Thanks.

Revision history for this message
Colin Watson (cjwatson) wrote :

You're probably right. I shall get splitting ...

Revision history for this message
Colin Watson (cjwatson) wrote :

This has now all been merged in smaller pieces.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/archiveuploader/tests/nascentupload-ddebs.txt'
2--- lib/lp/archiveuploader/tests/nascentupload-ddebs.txt 2012-01-20 16:11:11 +0000
3+++ lib/lp/archiveuploader/tests/nascentupload-ddebs.txt 2012-06-08 14:30:37 +0000
4@@ -89,7 +89,8 @@
5
6 >>> switch_dbuser('launchpad')
7
8- >>> bin.queue_root.overrideBinaries(main, devel, None, [main, universe])
9+ >>> bin.queue_root.overrideBinaries(
10+ ... [{"component": main, "section": devel}], [main, universe])
11 True
12 >>> bin.queue_root.acceptFromQueue()
13
14
15=== modified file 'lib/lp/registry/interfaces/distroseries.py'
16--- lib/lp/registry/interfaces/distroseries.py 2012-01-10 09:55:24 +0000
17+++ lib/lp/registry/interfaces/distroseries.py 2012-06-08 14:30:37 +0000
18@@ -547,6 +547,13 @@
19 description=_("Return only items with custom files of this "
20 "type."),
21 required=False),
22+ name=TextLine(title=_("Package or file name"), required=False),
23+ version=TextLine(title=_("Package version"), required=False),
24+ exact_match=Bool(
25+ title=_("Exact match"),
26+ description=_("Whether to filter name and version by exact "
27+ "matching."),
28+ required=False),
29 )
30 # Really IPackageUpload, patched in _schema_circular_imports.py
31 @operation_returns_collection_of(Interface)
32
33=== modified file 'lib/lp/soyuz/browser/queue.py'
34--- lib/lp/soyuz/browser/queue.py 2012-01-01 02:58:52 +0000
35+++ lib/lp/soyuz/browser/queue.py 2012-06-08 14:30:37 +0000
36@@ -1,4 +1,4 @@
37-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
38+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
39 # GNU Affero General Public License version 3 (see the file LICENSE).
40
41 """Browser views for package queue."""
42@@ -385,9 +385,13 @@
43 try:
44 source_overridden = queue_item.overrideSource(
45 new_component, new_section, allowed_components)
46+ binary_changes = [{
47+ "component": new_component,
48+ "section": new_section,
49+ "priority": new_priority,
50+ }]
51 binary_overridden = queue_item.overrideBinaries(
52- new_component, new_section, new_priority,
53- allowed_components)
54+ binary_changes, allowed_components)
55 except QueueInconsistentStateError, info:
56 failure.append("FAILED: %s (%s)" %
57 (queue_item.displayname, info))
58
59=== modified file 'lib/lp/soyuz/configure.zcml'
60--- lib/lp/soyuz/configure.zcml 2012-06-07 20:35:53 +0000
61+++ lib/lp/soyuz/configure.zcml 2012-06-08 14:30:37 +0000
62@@ -157,18 +157,24 @@
63 distroseries
64 pocket
65 changesfile
66+ changes_file_url
67 signing_key
68 archive
69 sources
70+ sourceFileUrls
71 builds
72+ binaryFileUrls
73 customfiles
74 custom_file_urls
75+ customFileUrls
76+ getBinaryProperties
77 date_created
78 sourcepackagerelease
79 component_name
80 concrete_package_copy_job
81 contains_source
82 contains_build
83+ contains_copy
84 contains_translation
85 contains_installer
86 contains_upgrader
87
88=== modified file 'lib/lp/soyuz/doc/distroseriesqueue.txt'
89--- lib/lp/soyuz/doc/distroseriesqueue.txt 2012-01-06 11:08:30 +0000
90+++ lib/lp/soyuz/doc/distroseriesqueue.txt 2012-06-08 14:30:37 +0000
91@@ -648,7 +648,7 @@
92 In addition to these parameters, you must also supply
93 "allowed_components", which is a sequence of IComponent. Any overrides
94 must have the existing and new component in this sequence otherwise
95-QueueInconsistentStateError is raised.
96+QueueAdminUnauthorizedError is raised.
97
98 The alsa-utils source is already in the queue with component "main"
99 and section "base".
100@@ -673,7 +673,7 @@
101 ... allowed_components=(universe,))
102 Traceback (most recent call last):
103 ...
104- QueueInconsistentStateError: No rights to override to restricted
105+ QueueAdminUnauthorizedError: No rights to override to restricted
106
107 Allowing "restricted" still won't work because the original component
108 is "main":
109@@ -683,7 +683,7 @@
110 ... allowed_components=(restricted,))
111 Traceback (most recent call last):
112 ...
113- QueueInconsistentStateError: No rights to override from main
114+ QueueAdminUnauthorizedError: No rights to override from main
115
116 Specifying both main and restricted allows the override to restricted/web.
117 overrideSource() returns True if it completed the task.
118@@ -710,29 +710,25 @@
119 main/base/Important
120
121 >>> from lp.soyuz.enums import PackagePublishingPriority
122- >>> print item.overrideBinaries(
123- ... new_component=restricted,
124- ... new_section=web,
125- ... new_priority=PackagePublishingPriority.EXTRA,
126- ... allowed_components=(universe,))
127- Traceback (most recent call last):
128- ...
129- QueueInconsistentStateError: No rights to override to restricted
130-
131- >>> print item.overrideBinaries(
132- ... new_component=restricted,
133- ... new_section=web,
134- ... new_priority=PackagePublishingPriority.EXTRA,
135- ... allowed_components=(restricted,))
136- Traceback (most recent call last):
137- ...
138- QueueInconsistentStateError: No rights to override from main
139-
140- >>> print item.overrideBinaries(
141- ... new_component=restricted,
142- ... new_section=web,
143- ... new_priority=PackagePublishingPriority.EXTRA,
144- ... allowed_components=(main,restricted))
145+ >>> binary_changes = [{
146+ ... "component": restricted,
147+ ... "section": web,
148+ ... "priority": PackagePublishingPriority.EXTRA,
149+ ... }]
150+ >>> print item.overrideBinaries(
151+ ... binary_changes, allowed_components=(universe,))
152+ Traceback (most recent call last):
153+ ...
154+ QueueAdminUnauthorizedError: No rights to override to restricted
155+
156+ >>> print item.overrideBinaries(
157+ ... binary_changes, allowed_components=(restricted,))
158+ Traceback (most recent call last):
159+ ...
160+ QueueAdminUnauthorizedError: No rights to override from main
161+
162+ >>> print item.overrideBinaries(
163+ ... binary_changes, allowed_components=(main,restricted))
164 True
165 >>> print "%s/%s/%s" % (
166 ... binary_package.component.name,
167
168=== modified file 'lib/lp/soyuz/interfaces/archive.py'
169--- lib/lp/soyuz/interfaces/archive.py 2012-06-06 21:24:57 +0000
170+++ lib/lp/soyuz/interfaces/archive.py 2012-06-08 14:30:37 +0000
171@@ -43,6 +43,8 @@
172 'NoSuchPPA',
173 'NoTokensForTeams',
174 'PocketNotFound',
175+ 'PriorityNotFound',
176+ 'SectionNotFound',
177 'VersionRequiresName',
178 'default_name_by_purpose',
179 'validate_external_dependencies',
180@@ -156,7 +158,7 @@
181
182
183 class ComponentNotFound(NameLookupFailed):
184- """Invalid source name."""
185+ """Invalid component name."""
186 _message_prefix = 'No such component'
187
188
189@@ -165,6 +167,16 @@
190 """Invalid component name."""
191
192
193+class SectionNotFound(NameLookupFailed):
194+ """Invalid section name."""
195+ _message_prefix = "No such section"
196+
197+
198+class PriorityNotFound(NameLookupFailed):
199+ """Invalid priority name."""
200+ _message_prefix = "No such priority"
201+
202+
203 class NoSuchPPA(NameLookupFailed):
204 """Raised when we try to look up an PPA that doesn't exist."""
205 _message_prefix = "No such ppa"
206
207=== modified file 'lib/lp/soyuz/interfaces/binarypackagerelease.py'
208--- lib/lp/soyuz/interfaces/binarypackagerelease.py 2011-12-24 16:54:44 +0000
209+++ lib/lp/soyuz/interfaces/binarypackagerelease.py 2012-06-08 14:30:37 +0000
210@@ -1,4 +1,4 @@
211-# Copyright 2009 Canonical Ltd. This software is licensed under the
212+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
213 # GNU Affero General Public License version 3 (see the file LICENSE).
214
215 # pylint: disable-msg=E0211,E0213
216@@ -98,6 +98,13 @@
217 description=_("True if there binary version was never published for "
218 "the architeture it was built for. False otherwise."))
219
220+ def properties():
221+ """Returns the properties of this binary.
222+
223+ For fast retrieval over the webservice, this is returned as a
224+ dictionary.
225+ """
226+
227 def addFile(file):
228 """Create a BinaryPackageFile record referencing this build
229 and attach the provided library file alias (file).
230
231=== modified file 'lib/lp/soyuz/interfaces/queue.py'
232--- lib/lp/soyuz/interfaces/queue.py 2012-05-30 08:50:50 +0000
233+++ lib/lp/soyuz/interfaces/queue.py 2012-06-08 14:30:37 +0000
234@@ -16,6 +16,7 @@
235 'IPackageUploadCustom',
236 'IPackageUploadSet',
237 'NonBuildableSourceUploadError',
238+ 'QueueAdminUnauthorizedError',
239 'QueueBuildAcceptError',
240 'QueueInconsistentStateError',
241 'QueueSourceAcceptError',
242@@ -26,11 +27,15 @@
243
244 from lazr.enum import DBEnumeratedType
245 from lazr.restful.declarations import (
246+ call_with,
247 error_status,
248 export_as_webservice_entry,
249+ export_read_operation,
250 export_write_operation,
251 exported,
252 operation_for_version,
253+ operation_parameters,
254+ REQUEST_USER,
255 )
256 from lazr.restful.fields import Reference
257 from zope.interface import (
258@@ -38,12 +43,15 @@
259 Interface,
260 )
261 from zope.schema import (
262+ Bool,
263 Choice,
264 Datetime,
265+ Dict,
266 Int,
267 List,
268 TextLine,
269 )
270+from zope.security.interfaces import Unauthorized
271
272 from lp import _
273 from lp.soyuz.enums import PackageUploadStatus
274@@ -67,6 +75,10 @@
275 """
276
277
278+class QueueAdminUnauthorizedError(Unauthorized):
279+ """User not permitted to perform a queue administration operation."""
280+
281+
282 class NonBuildableSourceUploadError(QueueInconsistentStateError):
283 """Source upload will not result in any build record.
284
285@@ -141,6 +153,14 @@
286
287 changesfile = Attribute("The librarian alias for the changes file "
288 "associated with this upload")
289+ changes_file_url = exported(
290+ TextLine(
291+ title=_("Changes file URL"),
292+ description=_("Librarian URL for the changes file associated with "
293+ "this upload. Will be None if the upload was copied "
294+ "from another series."),
295+ required=False, readonly=True),
296+ as_of="devel")
297
298 signing_key = Attribute("Changesfile Signing Key.")
299
300@@ -162,17 +182,18 @@
301 title=_("Archive"), required=True, readonly=True))
302 sources = Attribute("The queue sources associated with this queue item")
303 builds = Attribute("The queue builds associated with the queue item")
304+
305 customfiles = Attribute("Custom upload files associated with this "
306 "queue item")
307-
308 custom_file_urls = exported(
309 List(
310- title=_("Custom File URLs"),
311+ title=_("Custom file URLs"),
312 description=_("Librarian URLs for all the custom files attached "
313 "to this upload."),
314 value_type=TextLine(),
315 required=False,
316- readonly=True))
317+ readonly=True),
318+ ("devel", dict(exported=False)), exported=True)
319
320 displayname = exported(
321 TextLine(
322@@ -191,17 +212,39 @@
323 sourcepackagerelease = Attribute(
324 "The source package release for this item")
325
326- package_name = TextLine(
327- title=_("Name of the uploaded source package"), readonly=True)
328-
329- package_version = TextLine(
330- title=_("Source package version"), readonly=True)
331-
332- component_name = TextLine(
333- title=_("Source package component name"), readonly=True)
334-
335- contains_source = Attribute("whether or not this upload contains sources")
336- contains_build = Attribute("whether or not this upload contains binaries")
337+ package_name = exported(
338+ TextLine(
339+ title=_("Name of the uploaded source package"), readonly=True),
340+ as_of="devel")
341+
342+ package_version = exported(
343+ TextLine(title=_("Source package version"), readonly=True),
344+ as_of="devel")
345+
346+ component_name = exported(
347+ TextLine(title=_("Source package component name"), readonly=True),
348+ as_of="devel")
349+
350+ section_name = exported(
351+ TextLine(title=_("Source package section name"), readonly=True),
352+ as_of="devel")
353+
354+ contains_source = exported(
355+ Bool(
356+ title=_("Whether or not this upload contains sources"),
357+ readonly=True),
358+ as_of="devel")
359+ contains_build = exported(
360+ Bool(
361+ title=_("Whether or not this upload contains binaries"),
362+ readonly=True),
363+ as_of="devel")
364+ contains_copy = exported(
365+ Bool(
366+ title=_("Whether or not this upload contains a copy from another "
367+ "series."),
368+ readonly=True),
369+ as_of="devel")
370 contains_installer = Attribute(
371 "whether or not this upload contains installers images")
372 contains_translation = Attribute(
373@@ -223,8 +266,38 @@
374 on all the binarypackagerelease records arising from the build.
375 """)
376
377- section_name = TextLine(
378- title=_("Source package sectio name"), readonly=True)
379+ @export_read_operation()
380+ @operation_for_version("devel")
381+ def sourceFileUrls():
382+ """URLs for all the source files attached to this upload.
383+
384+ :return: A collection of URLs for this upload.
385+ """
386+
387+ @export_read_operation()
388+ @operation_for_version("devel")
389+ def binaryFileUrls():
390+ """URLs for all the binary files attached to this upload.
391+
392+ :return: A collection of URLs for this upload.
393+ """
394+
395+ @export_read_operation()
396+ @operation_for_version("devel")
397+ def customFileUrls():
398+ """URLs for all the custom files attached to this upload.
399+
400+ :return: A collection of URLs for this upload.
401+ """
402+
403+ @export_read_operation()
404+ @operation_for_version("devel")
405+ def getBinaryProperties():
406+ """The properties of the binaries associated with this queue item.
407+
408+ :return: A list of dictionaries, each containing the properties of a
409+ single binary.
410+ """
411
412 def setNew():
413 """Set queue state to NEW."""
414@@ -329,7 +402,14 @@
415 :param logger: Specify a logger object if required. Mainly for tests.
416 """
417
418- def overrideSource(new_component, new_section, allowed_components):
419+ @operation_parameters(
420+ new_component=TextLine(title=u"The new component name."),
421+ new_section=TextLine(title=u"The new section name."))
422+ @call_with(allowed_components=None, user=REQUEST_USER)
423+ @export_write_operation()
424+ @operation_for_version('devel')
425+ def overrideSource(new_component=None, new_section=None,
426+ allowed_components=None, user=None):
427 """Override the source package contained in this queue item.
428
429 :param new_component: An IComponent to replace the existing one
430@@ -338,6 +418,8 @@
431 in the upload's source.
432 :param allowed_components: A sequence of components that the
433 callsite is allowed to override from and to.
434+ :param user: The user requesting the override change, used if
435+ allowed_components is None.
436
437 :raises QueueInconsistentStateError: if either the existing
438 or the new_component are not in the allowed_components
439@@ -349,27 +431,40 @@
440 :return: True if the source was overridden.
441 """
442
443- def overrideBinaries(new_component, new_section, new_priority,
444- allowed_components):
445- """Override all the binaries in a binary queue item.
446+ @operation_parameters(
447+ changes=List(
448+ title=u"A sequence of changes to apply.",
449+ description=(
450+ u"Each item may have a 'name' item which specifies the binary "
451+ "package name to override; otherwise, the change applies to "
452+ "all binaries in the upload. It may also have 'component', "
453+ "'section', and 'priority' items which replace the "
454+ "corresponding existing one in the upload's overridden "
455+ "binaries."),
456+ value_type=Dict(key_type=TextLine())))
457+ @call_with(allowed_components=None, user=REQUEST_USER)
458+ @export_write_operation()
459+ @operation_for_version('devel')
460+ def overrideBinaries(changes, allowed_components=None, user=None):
461+ """Override binary packages in a binary queue item.
462
463- :param new_component: An IComponent to replace the existing one
464- in the upload's source.
465- :param new_section: An ISection to replace the existing one
466- in the upload's source.
467- :param new_priority: A valid PackagePublishingPriority to replace
468- the existing one in the upload's binaries.
469+ :param changes: A sequence of mappings of changes to apply. Each
470+ change mapping may have a "name" item which specifies the binary
471+ package name to override; otherwise, the change applies to all
472+ binaries in the upload. It may also have "component", "section",
473+ and "priority" items which replace the corresponding existing
474+ one in the upload's overridden binaries. Any missing items are
475+ left unchanged.
476 :param allowed_components: A sequence of components that the
477 callsite is allowed to override from and to.
478+ :param user: The user requesting the override change, used if
479+ allowed_components is None.
480
481 :raises QueueInconsistentStateError: if either the existing
482 or the new_component are not in the allowed_components
483 sequence.
484
485- The override values may be None, in which case they are not
486- changed.
487-
488- :return: True if the binaries were overridden.
489+ :return: True if any binaries were overridden.
490 """
491
492
493@@ -382,13 +477,20 @@
494
495 packageupload = Int(
496 title=_("PackageUpload"), required=True,
497- readonly=False,
498+ readonly=True,
499 )
500
501 build = Int(
502 title=_("The related build"), required=True, readonly=False,
503 )
504
505+ def binaries():
506+ """Returns the properties of the binaries in this build.
507+
508+ For fast retrieval over the webservice, these are returned as a list
509+ of dictionaries, one per binary.
510+ """
511+
512 def publish(logger=None):
513 """Publish this queued source in the distroseries referred to by
514 the parent queue item.
515
516=== modified file 'lib/lp/soyuz/model/binarypackagerelease.py'
517--- lib/lp/soyuz/model/binarypackagerelease.py 2012-04-16 23:02:44 +0000
518+++ lib/lp/soyuz/model/binarypackagerelease.py 2012-06-08 14:30:37 +0000
519@@ -1,4 +1,4 @@
520-# Copyright 2009 Canonical Ltd. This software is licensed under the
521+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
522 # GNU Affero General Public License version 3 (see the file LICENSE).
523
524 # pylint: disable-msg=E0611,W0212
525@@ -131,6 +131,18 @@
526 self.binarypackagename)
527 return distroarchseries_binary_package.currentrelease is None
528
529+ @property
530+ def properties(self):
531+ return {
532+ "name": self.name,
533+ "version": self.version,
534+ "is_new": self.is_new,
535+ "architecture": self.build.arch_tag,
536+ "component": self.component.name,
537+ "section": self.section.name,
538+ "priority": self.priority.name,
539+ }
540+
541 @cachedproperty
542 def files(self):
543 return list(
544@@ -201,4 +213,3 @@
545 def binary_package_version(self):
546 """See `IBinaryPackageReleaseDownloadCount`."""
547 return self.binary_package_release.version
548-
549
550=== modified file 'lib/lp/soyuz/model/queue.py'
551--- lib/lp/soyuz/model/queue.py 2012-05-25 15:31:50 +0000
552+++ lib/lp/soyuz/model/queue.py 2012-06-08 14:30:37 +0000
553@@ -13,6 +13,7 @@
554 'PackageUploadSet',
555 ]
556
557+from itertools import chain
558 import os
559 import shutil
560 import StringIO
561@@ -50,8 +51,10 @@
562 from lp.registry.interfaces.pocket import PackagePublishingPocket
563 from lp.registry.model.sourcepackagename import SourcePackageName
564 from lp.services.config import config
565+from lp.services.database.bulk import load_referencing
566 from lp.services.database.constants import UTC_NOW
567 from lp.services.database.datetimecol import UtcDateTimeCol
568+from lp.services.database.decoratedresultset import DecoratedResultSet
569 from lp.services.database.enumcol import EnumCol
570 from lp.services.database.lpstorm import (
571 IMasterStore,
572@@ -61,22 +64,34 @@
573 SQLBase,
574 sqlvalues,
575 )
576+from lp.services.librarian.browser import ProxiedLibraryFileAlias
577 from lp.services.librarian.interfaces.client import DownloadFailed
578 from lp.services.librarian.model import LibraryFileAlias
579 from lp.services.librarian.utils import copy_and_close
580 from lp.services.mail.signedmessage import strip_pgp_signature
581-from lp.services.propertycache import cachedproperty
582+from lp.services.propertycache import (
583+ cachedproperty,
584+ get_property_cache,
585+ )
586 from lp.soyuz.adapters.notification import notify
587 from lp.soyuz.adapters.overrides import SourceOverride
588 from lp.soyuz.enums import (
589 PackageUploadCustomFormat,
590 PackageUploadStatus,
591 )
592-from lp.soyuz.interfaces.archive import MAIN_ARCHIVE_PURPOSES
593+from lp.soyuz.interfaces.archive import (
594+ ComponentNotFound,
595+ MAIN_ARCHIVE_PURPOSES,
596+ PriorityNotFound,
597+ SectionNotFound,
598+ )
599+from lp.soyuz.interfaces.archivepermission import IArchivePermissionSet
600+from lp.soyuz.interfaces.component import IComponentSet
601 from lp.soyuz.interfaces.packagecopyjob import IPackageCopyJobSource
602 from lp.soyuz.interfaces.publishing import (
603 IPublishingSet,
604 ISourcePackagePublishingHistory,
605+ name_priority_map,
606 )
607 from lp.soyuz.interfaces.queue import (
608 IPackageUpload,
609@@ -86,11 +101,13 @@
610 IPackageUploadSet,
611 IPackageUploadSource,
612 NonBuildableSourceUploadError,
613+ QueueAdminUnauthorizedError,
614 QueueBuildAcceptError,
615 QueueInconsistentStateError,
616 QueueSourceAcceptError,
617 QueueStateWriteProtectedError,
618 )
619+from lp.soyuz.interfaces.section import ISectionSet
620 from lp.soyuz.model.binarypackagename import BinaryPackageName
621 from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease
622 from lp.soyuz.pas import BuildDaemonPackagesArchSpecific
623@@ -230,11 +247,44 @@
624
625 # Join this table to the PackageUploadBuild and the
626 # PackageUploadSource objects which are related.
627- sources = SQLMultipleJoin('PackageUploadSource',
628+ _sources = SQLMultipleJoin('PackageUploadSource',
629+ joinColumn='packageupload')
630+ # Does not include source builds.
631+ _builds = SQLMultipleJoin('PackageUploadBuild',
632 joinColumn='packageupload')
633- # Does not include source builds.
634- builds = SQLMultipleJoin('PackageUploadBuild',
635- joinColumn='packageupload')
636+
637+ @cachedproperty
638+ def sources(self):
639+ return list(self._sources)
640+
641+ def sourceFileUrls(self):
642+ """See `IPackageUpload`."""
643+ if self.contains_source:
644+ return [
645+ ProxiedLibraryFileAlias(
646+ file.libraryfile, self.archive).http_url
647+ for file in self.sourcepackagerelease.files]
648+ else:
649+ return []
650+
651+ @cachedproperty
652+ def builds(self):
653+ return list(self._builds)
654+
655+ def binaryFileUrls(self):
656+ """See `IPackageUpload`."""
657+ return [
658+ ProxiedLibraryFileAlias(file.libraryfile, self.archive).http_url
659+ for build in self.builds
660+ for bpr in build.build.binarypackages
661+ for file in bpr.files]
662+
663+ @property
664+ def changes_file_url(self):
665+ if self.changesfile is not None:
666+ return self.changesfile.getURL()
667+ else:
668+ return None
669
670 def getSourceBuild(self):
671 #avoid circular import
672@@ -250,8 +300,12 @@
673 PackageUploadSource.packageupload == self.id).one()
674
675 # Also the custom files associated with the build.
676- customfiles = SQLMultipleJoin('PackageUploadCustom',
677- joinColumn='packageupload')
678+ _customfiles = SQLMultipleJoin('PackageUploadCustom',
679+ joinColumn='packageupload')
680+
681+ @cachedproperty
682+ def customfiles(self):
683+ return list(self._customfiles)
684
685 @property
686 def custom_file_urls(self):
687@@ -259,6 +313,18 @@
688 return tuple(
689 file.libraryfilealias.getURL() for file in self.customfiles)
690
691+ def customFileUrls(self):
692+ """See `IPackageUpload`."""
693+ return [
694+ ProxiedLibraryFileAlias(
695+ file.libraryfilealias, self.archive).http_url
696+ for file in self.customfiles]
697+
698+ def getBinaryProperties(self):
699+ """See `IPackageUpload`."""
700+ return list(chain.from_iterable(
701+ build.binaries for build in self.builds))
702+
703 def setNew(self):
704 """See `IPackageUpload`."""
705 if self.status == PackageUploadStatus.NEW:
706@@ -544,7 +610,7 @@
707 def acceptFromCopy(self):
708 """See `IPackageUpload`."""
709 assert self.is_delayed_copy, 'Can only process delayed-copies.'
710- assert self.sources.count() == 1, (
711+ assert self._sources.count() == 1, (
712 'Source is mandatory for delayed copies.')
713 self.setAccepted()
714
715@@ -581,7 +647,7 @@
716
717 def _isSingleSourceUpload(self):
718 """Return True if this upload contains only a single source."""
719- return ((self.sources.count() == 1) and
720+ return ((self._sources.count() == 1) and
721 (not bool(self.builds)) and
722 (not bool(self.customfiles)))
723
724@@ -590,12 +656,17 @@
725 @cachedproperty
726 def contains_source(self):
727 """See `IPackageUpload`."""
728- return self.sources
729+ return bool(self.sources)
730
731 @cachedproperty
732 def contains_build(self):
733 """See `IPackageUpload`."""
734- return self.builds
735+ return bool(self.builds)
736+
737+ @cachedproperty
738+ def contains_copy(self):
739+ """See `IPackageUpload`."""
740+ return self.package_copy_job_id is not None
741
742 @cachedproperty
743 def from_build(self):
744@@ -804,18 +875,21 @@
745
746 def addSource(self, spr):
747 """See `IPackageUpload`."""
748+ del get_property_cache(self).sources
749 return PackageUploadSource(
750 packageupload=self,
751 sourcepackagerelease=spr.id)
752
753 def addBuild(self, build):
754 """See `IPackageUpload`."""
755+ del get_property_cache(self).builds
756 return PackageUploadBuild(
757 packageupload=self,
758 build=build.id)
759
760 def addCustom(self, library_file, custom_type):
761 """See `IPackageUpload`."""
762+ del get_property_cache(self).customfiles
763 return PackageUploadCustom(
764 packageupload=self,
765 libraryfilealias=library_file.id,
766@@ -915,6 +989,33 @@
767 """See `IPackageUpload`."""
768 return getUtility(IPackageCopyJobSource).wrap(self.package_copy_job)
769
770+ def _nameToComponent(self, component):
771+ """Helper to convert a possible string component to IComponent."""
772+ try:
773+ if isinstance(component, basestring):
774+ component = getUtility(IComponentSet)[component]
775+ return component
776+ except NotFoundError:
777+ raise ComponentNotFound(component)
778+
779+ def _nameToSection(self, section):
780+ """Helper to convert a possible string section to ISection."""
781+ try:
782+ if isinstance(section, basestring):
783+ section = getUtility(ISectionSet)[section]
784+ return section
785+ except NotFoundError:
786+ raise SectionNotFound(section)
787+
788+ def _nameToPriority(self, priority):
789+ """Helper to convert a possible string priority to its enum."""
790+ try:
791+ if isinstance(priority, basestring):
792+ priority = name_priority_map[priority]
793+ return priority
794+ except KeyError:
795+ raise PriorityNotFound(priority)
796+
797 def _overrideSyncSource(self, new_component, new_section,
798 allowed_components):
799 """Override source on the upload's `PackageCopyJob`, if any."""
800@@ -925,7 +1026,7 @@
801 allowed_component_names = [
802 component.name for component in allowed_components]
803 if copy_job.component_name not in allowed_component_names:
804- raise QueueInconsistentStateError(
805+ raise QueueAdminUnauthorizedError(
806 "No rights to override from %s" % copy_job.component_name)
807 copy_job.addSourceOverride(SourceOverride(
808 copy_job.package_name, new_component, new_section))
809@@ -942,7 +1043,7 @@
810 if old_component not in allowed_components:
811 # The old component is not in the list of allowed components
812 # to override.
813- raise QueueInconsistentStateError(
814+ raise QueueAdminUnauthorizedError(
815 "No rights to override from %s" % old_component.name)
816 source.sourcepackagerelease.override(
817 component=new_component, section=new_section)
818@@ -955,14 +1056,29 @@
819
820 return made_changes
821
822- def overrideSource(self, new_component, new_section, allowed_components):
823+ def overrideSource(self, new_component=None, new_section=None,
824+ allowed_components=None, user=None):
825 """See `IPackageUpload`."""
826 if new_component is None and new_section is None:
827 # Nothing needs overriding, bail out.
828 return False
829
830+ new_component = self._nameToComponent(new_component)
831+ new_section = self._nameToSection(new_section)
832+
833+ if allowed_components is None and user is not None:
834+ # Get a list of components for which the user has rights to
835+ # override to or from.
836+ permission_set = getUtility(IArchivePermissionSet)
837+ permissions = permission_set.componentsForQueueAdmin(
838+ self.distroseries.main_archive, user)
839+ allowed_components = set(
840+ permission.component for permission in permissions)
841+ assert allowed_components is not None, (
842+ "Must provide allowed_components for non-webservice calls.")
843+
844 if new_component not in list(allowed_components) + [None]:
845- raise QueueInconsistentStateError(
846+ raise QueueAdminUnauthorizedError(
847 "No rights to override to %s" % new_component.name)
848
849 return (
850@@ -971,35 +1087,95 @@
851 self._overrideNonSyncSource(
852 new_component, new_section, allowed_components))
853
854- def overrideBinaries(self, new_component, new_section, new_priority,
855- allowed_components):
856+ def _filterBinaryChanges(self, changes):
857+ """Process a binary changes mapping into a more convenient form."""
858+ changes_by_name = {}
859+ changes_for_all = None
860+
861+ for change in changes:
862+ filtered_change = {}
863+ if "component" in change:
864+ filtered_change["component"] = self._nameToComponent(
865+ change["component"])
866+ if "section" in change:
867+ filtered_change["section"] = self._nameToSection(
868+ change["section"])
869+ if "priority" in change:
870+ filtered_change["priority"] = self._nameToPriority(
871+ change["priority"])
872+
873+ if "name" in change:
874+ changes_by_name[change["name"]] = filtered_change
875+ else:
876+ # Changes with no "name" item provide a default for all
877+ # binaries.
878+ changes_for_all = filtered_change
879+
880+ return changes_by_name, changes_for_all
881+
882+ def overrideBinaries(self, changes, allowed_components=None, user=None):
883 """See `IPackageUpload`."""
884 if not self.contains_build:
885 return False
886
887- if (new_component is None and new_section is None and
888- new_priority is None):
889+ if not changes:
890 # Nothing needs overriding, bail out.
891 return False
892
893- if new_component not in allowed_components:
894- raise QueueInconsistentStateError(
895- "No rights to override to %s" % new_component.name)
896-
897+ if allowed_components is None and user is not None:
898+ # Get a list of components for which the user has rights to
899+ # override to or from.
900+ permission_set = getUtility(IArchivePermissionSet)
901+ permissions = permission_set.componentsForQueueAdmin(
902+ self.distroseries.main_archive, user)
903+ allowed_components = set(
904+ permission.component for permission in permissions)
905+ assert allowed_components is not None, (
906+ "Must provide allowed_components for non-webservice calls.")
907+
908+ changes_by_name, changes_for_all = self._filterBinaryChanges(changes)
909+
910+ new_components = set()
911+ for change in changes_by_name.values():
912+ if "component" in change:
913+ new_components.add(change["component"])
914+ if changes_for_all is not None and "component" in changes_for_all:
915+ new_components.add(changes_for_all["component"])
916+ disallowed_components = sorted(
917+ component.name
918+ for component in new_components.difference(allowed_components))
919+ if disallowed_components:
920+ raise QueueAdminUnauthorizedError(
921+ "No rights to override to %s" %
922+ ", ".join(disallowed_components))
923+
924+ made_changes = False
925 for build in self.builds:
926+ # See if the new component requires a new archive on the build.
927+ for component in new_components:
928+ distroarchseries = build.build.distro_arch_series
929+ distribution = distroarchseries.distroseries.distribution
930+ new_archive = distribution.getArchiveByComponent(
931+ component.name)
932+ if new_archive != build.build.archive:
933+ raise QueueInconsistentStateError(
934+ "Overriding component to '%s' failed because it "
935+ "would require a new archive." % component.name)
936+
937 for binarypackage in build.build.binarypackages:
938- if binarypackage.component not in allowed_components:
939- # The old or the new component is not in the list of
940- # allowed components to override.
941- raise QueueInconsistentStateError(
942- "No rights to override from %s" % (
943- binarypackage.component.name))
944- binarypackage.override(
945- component=new_component,
946- section=new_section,
947- priority=new_priority)
948+ change = changes_by_name.get(
949+ binarypackage.name, changes_for_all)
950+ if change is not None:
951+ if binarypackage.component not in allowed_components:
952+ # The old component is not in the list of allowed
953+ # components to override.
954+ raise QueueAdminUnauthorizedError(
955+ "No rights to override from %s" % (
956+ binarypackage.component.name))
957+ binarypackage.override(**change)
958+ made_changes = True
959
960- return bool(self.builds)
961+ return made_changes
962
963
964 class PackageUploadBuild(SQLBase):
965@@ -1014,6 +1190,12 @@
966
967 build = ForeignKey(dbName='build', foreignKey='BinaryPackageBuild')
968
969+ @property
970+ def binaries(self):
971+ """See `IPackageUploadBuild`."""
972+ for binary in self.build.binarypackages:
973+ yield binary.properties
974+
975 def checkComponentAndSection(self):
976 """See `IPackageUploadBuild`."""
977 distroseries = self.packageupload.distroseries
978@@ -1622,7 +1804,30 @@
979 PackageUpload.distroseries == distroseries,
980 *conditions)
981 query = query.order_by(Desc(PackageUpload.id))
982- return query.config(distinct=True)
983+ query = query.config(distinct=True)
984+
985+ def preload_hook(rows):
986+ puses = load_referencing(
987+ PackageUploadSource, rows, ["packageuploadID"])
988+ pubs = load_referencing(
989+ PackageUploadBuild, rows, ["packageuploadID"])
990+ pucs = load_referencing(
991+ PackageUploadCustom, rows, ["packageuploadID"])
992+
993+ for pu in rows:
994+ cache = get_property_cache(pu)
995+ cache.sources = []
996+ cache.builds = []
997+ cache.customfiles = []
998+
999+ for pus in puses:
1000+ get_property_cache(pus.packageupload).sources.append(pus)
1001+ for pub in pubs:
1002+ get_property_cache(pub.packageupload).builds.append(pub)
1003+ for puc in pucs:
1004+ get_property_cache(puc.packageupload).customfiles.append(puc)
1005+
1006+ return DecoratedResultSet(query, pre_iter_hook=preload_hook)
1007
1008 def getBuildByBuildIDs(self, build_ids):
1009 """See `IPackageUploadSet`."""
1010
1011=== modified file 'lib/lp/soyuz/scripts/queue.py'
1012--- lib/lp/soyuz/scripts/queue.py 2012-02-10 10:50:03 +0000
1013+++ lib/lp/soyuz/scripts/queue.py 2012-06-08 14:30:37 +0000
1014@@ -1,4 +1,4 @@
1015-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
1016+# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
1017 # GNU Affero General Public License version 3 (see the file LICENSE).
1018
1019 # pylint: disable-msg=W0231
1020@@ -260,8 +260,8 @@
1021 False: '-',
1022 }
1023 return (
1024- source_tag[bool(queue_item.contains_source)] +
1025- binary_tag[bool(queue_item.contains_build)])
1026+ source_tag[queue_item.contains_source] +
1027+ binary_tag[queue_item.contains_build])
1028
1029 def displayItem(self, queue_item):
1030 """Display one line summary of the queue item provided."""
1031
1032=== modified file 'lib/lp/soyuz/stories/webservice/xx-packageupload.txt'
1033--- lib/lp/soyuz/stories/webservice/xx-packageupload.txt 2012-05-30 14:12:44 +0000
1034+++ lib/lp/soyuz/stories/webservice/xx-packageupload.txt 2012-06-08 14:30:37 +0000
1035@@ -31,6 +31,19 @@
1036 self_link: u'http://.../ubuntu/warty/+upload/11'
1037 status: u'Done'
1038
1039+getPackageUploads can filter on package names.
1040+
1041+ >>> uploads = webservice.named_get(
1042+ ... warty['self_link'], 'getPackageUploads',
1043+ ... name='mozilla').jsonBody()
1044+ >>> len(uploads['entries'])
1045+ 1
1046+ >>> uploads = webservice.named_get(
1047+ ... warty['self_link'], 'getPackageUploads',
1048+ ... name='missing').jsonBody()
1049+ >>> len(uploads['entries'])
1050+ 0
1051+
1052
1053 Retrieving Static Translation Files
1054 ===================================
1055
1056=== modified file 'lib/lp/soyuz/tests/test_distroseriesqueue_ddtp_tarball.py'
1057--- lib/lp/soyuz/tests/test_distroseriesqueue_ddtp_tarball.py 2012-05-25 13:28:31 +0000
1058+++ lib/lp/soyuz/tests/test_distroseriesqueue_ddtp_tarball.py 2012-06-08 14:30:37 +0000
1059@@ -27,10 +27,6 @@
1060 getPolicy,
1061 )
1062 from lp.services.log.logger import DevNullLogger
1063-from lp.soyuz.scripts.queue import (
1064- CommandRunner,
1065- name_queue_map,
1066- )
1067 from lp.soyuz.tests.test_publishing import TestNativePublishingBase
1068 from lp.testing.gpgkeys import import_public_test_keys
1069
1070@@ -68,30 +64,6 @@
1071 def test_accepts_correct_upload(self):
1072 self.uploadTestData("20060728")
1073
1074- def runQueueCommand(self, queue_name, args):
1075- def null_display(text):
1076- pass
1077-
1078- queue = name_queue_map[queue_name]
1079- runner = CommandRunner(
1080- queue, "ubuntutest", "breezy-autotest", True, None, None, None,
1081- display=null_display)
1082- runner.execute(args)
1083-
1084- def test_queue_tool_behaviour(self):
1085- # The queue tool can fetch ddtp-tarball uploads.
1086- self.uploadTestData("20060728")
1087- # Make sure that we can use the librarian files.
1088- transaction.commit()
1089- # Fetch upload into a temporary directory.
1090- self.useTempDir()
1091- self.runQueueCommand("accepted", ["fetch", "trans"])
1092- expected_entries = [
1093- "translations-main_20060728_all.changes",
1094- "translations_main_20060728.tar.gz",
1095- ]
1096- self.assertContentEqual(expected_entries, os.listdir("."))
1097-
1098 def test_publish(self):
1099 upload = self.uploadTestData("20060728")
1100 transaction.commit()
1101
1102=== modified file 'lib/lp/soyuz/tests/test_distroseriesqueue_dist_upgrader.py'
1103--- lib/lp/soyuz/tests/test_distroseriesqueue_dist_upgrader.py 2012-05-25 13:27:41 +0000
1104+++ lib/lp/soyuz/tests/test_distroseriesqueue_dist_upgrader.py 2012-06-08 14:30:37 +0000
1105@@ -23,10 +23,6 @@
1106 )
1107 from lp.services.config import config
1108 from lp.services.log.logger import DevNullLogger
1109-from lp.soyuz.scripts.queue import (
1110- CommandRunner,
1111- name_queue_map,
1112- )
1113 from lp.soyuz.tests.test_publishing import TestNativePublishingBase
1114 from lp.testing.gpgkeys import import_public_test_keys
1115
1116@@ -69,37 +65,18 @@
1117 def test_accepts_correct_upload(self):
1118 self.uploadTestData("20060302.0120")
1119
1120- def runQueueCommand(self, queue_name, args):
1121- def null_display(text):
1122- pass
1123-
1124- queue = name_queue_map[queue_name]
1125- runner = CommandRunner(
1126- queue, "ubuntutest", "breezy-autotest", True, None, None, None,
1127- display=null_display)
1128- runner.execute(args)
1129-
1130- def test_queue_tool_behaviour(self):
1131- # The queue tool can accept, reject, and fetch dist-upgrader
1132- # uploads. See bug #54649.
1133+ def test_accept_reject(self):
1134+ # We can accept and reject dist-upgrader uploads.
1135 upload = self.uploadTestData("20060302.0120")
1136 # Make sure that we can use the librarian files.
1137 transaction.commit()
1138 # Reject from accepted queue (unlikely, would normally be from
1139 # unapproved or new).
1140- self.runQueueCommand("accepted", ["reject", "dist"])
1141+ upload.queue_root.rejectFromQueue(logger=self.logger)
1142 self.assertEqual("REJECTED", upload.queue_root.status.name)
1143 # Accept from rejected queue (also unlikely, but only for testing).
1144- self.runQueueCommand("rejected", ["accept", "dist"])
1145+ upload.queue_root.acceptFromQueue(logger=self.logger)
1146 self.assertEqual("ACCEPTED", upload.queue_root.status.name)
1147- # Fetch upload into a temporary directory.
1148- self.useTempDir()
1149- self.runQueueCommand("accepted", ["fetch", "dist"])
1150- expected_entries = [
1151- "dist-upgrader_20060302.0120_all.changes",
1152- "dist-upgrader_20060302.0120_all.tar.gz",
1153- ]
1154- self.assertContentEqual(expected_entries, os.listdir("."))
1155
1156 def test_bad_upload_remains_in_accepted(self):
1157 # Bad dist-upgrader uploads remain in ACCEPTED.
1158
1159=== modified file 'lib/lp/soyuz/tests/test_packageupload.py'
1160--- lib/lp/soyuz/tests/test_packageupload.py 2012-05-30 08:50:50 +0000
1161+++ lib/lp/soyuz/tests/test_packageupload.py 2012-06-08 14:30:37 +0000
1162@@ -12,9 +12,11 @@
1163 BadRequest,
1164 Unauthorized,
1165 )
1166+from testtools.matchers import Equals
1167 import transaction
1168 from zope.component import getUtility
1169 from zope.security.proxy import removeSecurityProxy
1170+from zope.schema import getFields
1171
1172 from lp.archivepublisher.interfaces.publisherconfig import IPublisherConfigSet
1173 from lp.archiveuploader.tests import datadir
1174@@ -25,6 +27,7 @@
1175 from lp.services.config import config
1176 from lp.services.database.lpstorm import IStore
1177 from lp.services.job.interfaces.job import JobStatus
1178+from lp.services.librarian.browser import ProxiedLibraryFileAlias
1179 from lp.services.log.logger import BufferLogger
1180 from lp.services.mail import stub
1181 from lp.soyuz.adapters.overrides import SourceOverride
1182@@ -37,7 +40,9 @@
1183 from lp.soyuz.interfaces.archivepermission import IArchivePermissionSet
1184 from lp.soyuz.interfaces.component import IComponentSet
1185 from lp.soyuz.interfaces.queue import (
1186+ IPackageUpload,
1187 IPackageUploadSet,
1188+ QueueAdminUnauthorizedError,
1189 QueueInconsistentStateError,
1190 )
1191 from lp.soyuz.interfaces.section import ISectionSet
1192@@ -48,6 +53,7 @@
1193 api_url,
1194 launchpadlib_for,
1195 person_logged_in,
1196+ StormStatementRecorder,
1197 TestCaseWithFactory,
1198 )
1199 from lp.testing.dbuser import switch_dbuser
1200@@ -55,7 +61,10 @@
1201 LaunchpadFunctionalLayer,
1202 LaunchpadZopelessLayer,
1203 )
1204-from lp.testing.matchers import Provides
1205+from lp.testing.matchers import (
1206+ HasQueryCount,
1207+ Provides,
1208+ )
1209
1210
1211 class PackageUploadTestCase(TestCaseWithFactory):
1212@@ -440,8 +449,7 @@
1213 only_allowed_component = self.factory.makeComponent()
1214 section = self.factory.makeSection()
1215 self.assertRaises(
1216- QueueInconsistentStateError,
1217- pu.overrideSource,
1218+ QueueAdminUnauthorizedError, pu.overrideSource,
1219 only_allowed_component, section, [only_allowed_component])
1220
1221 def test_overrideSource_checks_permission_for_new_component(self):
1222@@ -450,8 +458,7 @@
1223 disallowed_component = self.factory.makeComponent()
1224 section = self.factory.makeSection()
1225 self.assertRaises(
1226- QueueInconsistentStateError,
1227- pu.overrideSource,
1228+ QueueAdminUnauthorizedError, pu.overrideSource,
1229 disallowed_component, section, [current_component])
1230
1231 def test_overrideSource_ignores_None_component_change(self):
1232@@ -864,6 +871,20 @@
1233 list(reversed(ordered_uploads)),
1234 list(getUtility(IPackageUploadSet).getAll(series)))
1235
1236+ def test_getAll_can_preload_exported_properties(self):
1237+ # getAll preloads everything exported on the webservice.
1238+ distroseries = self.factory.makeDistroSeries()
1239+ self.factory.makeSourcePackageUpload(distroseries=distroseries)
1240+ self.factory.makeBuildPackageUpload(distroseries=distroseries)
1241+ self.factory.makeCustomPackageUpload(distroseries=distroseries)
1242+ uploads = list(getUtility(IPackageUploadSet).getAll(distroseries))
1243+ with StormStatementRecorder() as recorder:
1244+ for name, field in getFields(IPackageUpload).items():
1245+ if field.queryTaggedValue("lazr.restful.exported") is not None:
1246+ for upload in uploads:
1247+ getattr(upload, name)
1248+ self.assertThat(recorder, HasQueryCount(Equals(0)))
1249+
1250 def test_rejectFromQueue_no_changes_file(self):
1251 # If the PackageUpload has no changesfile, we can still reject it.
1252 pu = self.factory.makePackageUpload()
1253@@ -880,12 +901,13 @@
1254 def setUp(self):
1255 super(TestPackageUploadWebservice, self).setUp()
1256 self.webservice = None
1257-
1258- def makeDistroSeries(self):
1259 self.distroseries = self.factory.makeDistroSeries()
1260 self.main = self.factory.makeComponent("main")
1261 self.factory.makeComponentSelection(
1262 distroseries=self.distroseries, component=self.main)
1263+ self.universe = self.factory.makeComponent("universe")
1264+ self.factory.makeComponentSelection(
1265+ distroseries=self.distroseries, component=self.universe)
1266
1267 def makeQueueAdmin(self, components):
1268 person = self.factory.makePerson()
1269@@ -902,57 +924,336 @@
1270 self.webservice = launchpadlib_for("testing", person)
1271 return self.webservice.load(api_url(obj))
1272
1273+ def makeSourcePackageUpload(self, person, **kwargs):
1274+ with person_logged_in(person):
1275+ upload = self.factory.makeSourcePackageUpload(
1276+ distroseries=self.distroseries, **kwargs)
1277+ transaction.commit()
1278+ spr = upload.sourcepackagerelease
1279+ for extension in ("dsc", "tar.gz"):
1280+ filename = "%s_%s.%s" % (spr.name, spr.version, extension)
1281+ lfa = self.factory.makeLibraryFileAlias(filename=filename)
1282+ spr.addFile(lfa)
1283+ transaction.commit()
1284+ return upload, self.load(upload, person)
1285+
1286+ def makeBinaryPackageUpload(self, person, binarypackagename=None,
1287+ component=None):
1288+ with person_logged_in(person):
1289+ upload = self.factory.makeBuildPackageUpload(
1290+ distroseries=self.distroseries,
1291+ binarypackagename=binarypackagename, component=component)
1292+ self.factory.makeBinaryPackageRelease(
1293+ build=upload.builds[0].build, component=component)
1294+ transaction.commit()
1295+ for build in upload.builds:
1296+ for bpr in build.build.binarypackages:
1297+ filename = "%s_%s_%s.deb" % (
1298+ bpr.name, bpr.version, bpr.build.arch_tag)
1299+ lfa = self.factory.makeLibraryFileAlias(filename=filename)
1300+ bpr.addFile(lfa)
1301+ transaction.commit()
1302+ return upload, self.load(upload, person)
1303+
1304+ def makeCustomPackageUpload(self, person, **kwargs):
1305+ with person_logged_in(person):
1306+ upload = self.factory.makeCustomPackageUpload(
1307+ distroseries=self.distroseries, **kwargs)
1308+ transaction.commit()
1309+ return upload, self.load(upload, person)
1310+
1311 def assertRequiresEdit(self, method_name, **kwargs):
1312 """Test that a web service queue method requires launchpad.Edit."""
1313 with admin_logged_in():
1314 upload = self.factory.makeSourcePackageUpload()
1315 transaction.commit()
1316 ws_upload = self.load(upload)
1317- self.assertRaises(Unauthorized, getattr(ws_upload, method_name),
1318- **kwargs)
1319+ self.assertRaises(
1320+ Unauthorized, getattr(ws_upload, method_name), **kwargs)
1321
1322 def test_edit_permissions(self):
1323 self.assertRequiresEdit("acceptFromQueue")
1324 self.assertRequiresEdit("rejectFromQueue")
1325+ self.assertRequiresEdit("overrideSource", new_component="main")
1326+ self.assertRequiresEdit(
1327+ "overrideBinaries", changes=[{"component": "main"}])
1328
1329 def test_acceptFromQueue_archive_admin(self):
1330 # acceptFromQueue as an archive admin accepts the upload.
1331- self.makeDistroSeries()
1332 person = self.makeQueueAdmin([self.main])
1333- with person_logged_in(person):
1334- upload = self.factory.makeSourcePackageUpload(
1335- distroseries=self.distroseries, component=self.main)
1336- transaction.commit()
1337+ upload, ws_upload = self.makeSourcePackageUpload(
1338+ person, component=self.main)
1339
1340- ws_upload = self.load(upload, person)
1341 self.assertEqual("New", ws_upload.status)
1342 ws_upload.acceptFromQueue()
1343 self.assertEqual("Done", ws_upload.status)
1344
1345 def test_double_accept_raises_BadRequest(self):
1346 # Trying to accept an upload twice returns 400 instead of OOPSing.
1347- self.makeDistroSeries()
1348 person = self.makeQueueAdmin([self.main])
1349+ upload, _ = self.makeSourcePackageUpload(person, component=self.main)
1350+
1351 with person_logged_in(person):
1352- upload = self.factory.makeSourcePackageUpload(
1353- distroseries=self.distroseries, component=self.main)
1354 upload.setAccepted()
1355- transaction.commit()
1356-
1357 ws_upload = self.load(upload, person)
1358 self.assertEqual("Accepted", ws_upload.status)
1359 self.assertRaises(BadRequest, ws_upload.acceptFromQueue)
1360
1361 def test_rejectFromQueue_archive_admin(self):
1362 # rejectFromQueue as an archive admin rejects the upload.
1363- self.makeDistroSeries()
1364 person = self.makeQueueAdmin([self.main])
1365- with person_logged_in(person):
1366- upload = self.factory.makeSourcePackageUpload(
1367- distroseries=self.distroseries, component=self.main)
1368- transaction.commit()
1369+ upload, ws_upload = self.makeSourcePackageUpload(
1370+ person, component=self.main)
1371
1372- ws_upload = self.load(upload, person)
1373 self.assertEqual("New", ws_upload.status)
1374 ws_upload.rejectFromQueue()
1375 self.assertEqual("Rejected", ws_upload.status)
1376+
1377+ def test_source_info(self):
1378+ # API clients can inspect properties of source uploads.
1379+ person = self.makeQueueAdmin([self.universe])
1380+ upload, ws_upload = self.makeSourcePackageUpload(
1381+ person, sourcepackagename="hello", component=self.universe)
1382+
1383+ self.assertTrue(ws_upload.contains_source)
1384+ self.assertFalse(ws_upload.contains_build)
1385+ self.assertFalse(ws_upload.contains_copy)
1386+ self.assertEqual("hello", ws_upload.display_name)
1387+ self.assertEqual(upload.package_version, ws_upload.display_version)
1388+ self.assertEqual("source", ws_upload.display_arches)
1389+ self.assertEqual("hello", ws_upload.package_name)
1390+ self.assertEqual(upload.package_version, ws_upload.package_version)
1391+ self.assertEqual("universe", ws_upload.component_name)
1392+ self.assertEqual(upload.section_name, ws_upload.section_name)
1393+
1394+ def test_source_fetch(self):
1395+ # API clients can fetch files attached to source uploads.
1396+ person = self.makeQueueAdmin([self.universe])
1397+ upload, ws_upload = self.makeSourcePackageUpload(
1398+ person, component=self.universe)
1399+ ws_source_file_urls = ws_upload.sourceFileUrls()
1400+ self.assertNotEqual(0, len(ws_source_file_urls))
1401+ with person_logged_in(person):
1402+ source_file_urls = [
1403+ ProxiedLibraryFileAlias(
1404+ file.libraryfile, upload.archive).http_url
1405+ for file in upload.sourcepackagerelease.files]
1406+ self.assertContentEqual(source_file_urls, ws_source_file_urls)
1407+
1408+ def test_overrideSource_limited_component_permissions(self):
1409+ # Overriding between two components requires queue admin of both.
1410+ person = self.makeQueueAdmin([self.universe])
1411+ upload, ws_upload = self.makeSourcePackageUpload(
1412+ person, component=self.universe)
1413+
1414+ self.assertEqual("New", ws_upload.status)
1415+ self.assertEqual("universe", ws_upload.component_name)
1416+ self.assertRaises(Unauthorized, ws_upload.overrideSource,
1417+ new_component="main")
1418+
1419+ with admin_logged_in():
1420+ upload.overrideSource(
1421+ new_component=self.main,
1422+ allowed_components=[self.main, self.universe])
1423+ transaction.commit()
1424+ self.assertEqual("main", upload.component_name)
1425+ self.assertRaises(Unauthorized, ws_upload.overrideSource,
1426+ new_component="universe")
1427+
1428+ def test_overrideSource_changes_properties(self):
1429+ # Running overrideSource changes the corresponding properties.
1430+ person = self.makeQueueAdmin([self.main, self.universe])
1431+ upload, ws_upload = self.makeSourcePackageUpload(
1432+ person, component=self.universe)
1433+ with person_logged_in(person):
1434+ new_section = self.factory.makeSection()
1435+ transaction.commit()
1436+
1437+ self.assertEqual("New", ws_upload.status)
1438+ self.assertEqual("universe", ws_upload.component_name)
1439+ self.assertNotEqual(new_section.name, ws_upload.section_name)
1440+ ws_upload.overrideSource(
1441+ new_component="main", new_section=new_section.name)
1442+ self.assertEqual("main", ws_upload.component_name)
1443+ self.assertEqual(new_section.name, ws_upload.section_name)
1444+ ws_upload.overrideSource(new_component="universe")
1445+ self.assertEqual("universe", ws_upload.component_name)
1446+
1447+ def test_binary_info(self):
1448+ # API clients can inspect properties of binary uploads.
1449+ person = self.makeQueueAdmin([self.universe])
1450+ upload, ws_upload = self.makeBinaryPackageUpload(
1451+ person, component=self.universe)
1452+ with person_logged_in(person):
1453+ arch = upload.builds[0].build.arch_tag
1454+ bprs = upload.builds[0].build.binarypackages
1455+
1456+ self.assertFalse(ws_upload.contains_source)
1457+ self.assertTrue(ws_upload.contains_build)
1458+ ws_binaries = ws_upload.getBinaryProperties()
1459+ self.assertEqual(len(list(bprs)), len(ws_binaries))
1460+ for bpr, binary in zip(bprs, ws_binaries):
1461+ expected_binary = {
1462+ "is_new": True,
1463+ "name": bpr.name,
1464+ "version": bpr.version,
1465+ "architecture": arch,
1466+ "component": "universe",
1467+ "section": bpr.section.name,
1468+ "priority": bpr.priority.name,
1469+ }
1470+ self.assertContentEqual(expected_binary.keys(), binary.keys())
1471+ for key, value in expected_binary.items():
1472+ self.assertEqual(value, binary[key])
1473+
1474+ def test_binary_fetch(self):
1475+ # API clients can fetch files attached to binary uploads.
1476+ person = self.makeQueueAdmin([self.universe])
1477+ upload, ws_upload = self.makeBinaryPackageUpload(
1478+ person, component=self.universe)
1479+
1480+ ws_binary_file_urls = ws_upload.binaryFileUrls()
1481+ self.assertNotEqual(0, len(ws_binary_file_urls))
1482+ with person_logged_in(person):
1483+ binary_file_urls = [
1484+ ProxiedLibraryFileAlias(
1485+ file.libraryfile, upload.archive).http_url
1486+ for bpr in upload.builds[0].build.binarypackages
1487+ for file in bpr.files]
1488+ self.assertContentEqual(binary_file_urls, ws_binary_file_urls)
1489+
1490+ def test_overrideBinaries_limited_component_permissions(self):
1491+ # Overriding between two components requires queue admin of both.
1492+ person = self.makeQueueAdmin([self.universe])
1493+ upload, ws_upload = self.makeBinaryPackageUpload(
1494+ person, binarypackagename="hello", component=self.universe)
1495+
1496+ self.assertEqual("New", ws_upload.status)
1497+ self.assertEqual(
1498+ set(["universe"]),
1499+ set(binary["component"]
1500+ for binary in ws_upload.getBinaryProperties()))
1501+ self.assertRaises(
1502+ Unauthorized, ws_upload.overrideBinaries,
1503+ changes=[{"component": "main"}])
1504+
1505+ with admin_logged_in():
1506+ upload.overrideBinaries(
1507+ [{"component": self.main}],
1508+ allowed_components=[self.main, self.universe])
1509+ transaction.commit()
1510+
1511+ self.assertEqual(
1512+ set(["main"]),
1513+ set(binary["component"]
1514+ for binary in ws_upload.getBinaryProperties()))
1515+ self.assertRaises(
1516+ Unauthorized, ws_upload.overrideBinaries,
1517+ changes=[{"component": "universe"}])
1518+
1519+ def test_overrideBinaries_disallows_new_archive(self):
1520+ # overrideBinaries refuses to override the component to something
1521+ # that requires a different archive.
1522+ partner = self.factory.makeComponent("partner")
1523+ self.factory.makeComponentSelection(
1524+ distroseries=self.distroseries, component=partner)
1525+ person = self.makeQueueAdmin([self.universe, partner])
1526+ upload, ws_upload = self.makeBinaryPackageUpload(
1527+ person, component=self.universe)
1528+
1529+ self.assertEqual(
1530+ "universe", ws_upload.getBinaryProperties()[0]["component"])
1531+ self.assertRaises(
1532+ BadRequest, ws_upload.overrideBinaries,
1533+ changes=[{"component": "partner"}])
1534+
1535+ def test_overrideBinaries_without_name_changes_all_properties(self):
1536+ # Running overrideBinaries with a change entry containing no "name"
1537+ # field changes the corresponding properties of all binaries.
1538+ person = self.makeQueueAdmin([self.main, self.universe])
1539+ upload, ws_upload = self.makeBinaryPackageUpload(
1540+ person, component=self.universe)
1541+ with person_logged_in(person):
1542+ new_section = self.factory.makeSection()
1543+ transaction.commit()
1544+
1545+ self.assertEqual("New", ws_upload.status)
1546+ for binary in ws_upload.getBinaryProperties():
1547+ self.assertEqual("universe", binary["component"])
1548+ self.assertNotEqual(new_section.name, binary["section"])
1549+ self.assertEqual("OPTIONAL", binary["priority"])
1550+ changes = [{
1551+ "component": "main",
1552+ "section": new_section.name,
1553+ "priority": "extra",
1554+ }]
1555+ ws_upload.overrideBinaries(changes=changes)
1556+ for binary in ws_upload.getBinaryProperties():
1557+ self.assertEqual("main", binary["component"])
1558+ self.assertEqual(new_section.name, binary["section"])
1559+ self.assertEqual("EXTRA", binary["priority"])
1560+
1561+ def test_overrideBinaries_with_name_changes_selected_properties(self):
1562+ # Running overrideBinaries with change entries containing "name"
1563+ # fields changes the corresponding properties of only the selected
1564+ # binaries.
1565+ person = self.makeQueueAdmin([self.main, self.universe])
1566+ upload, ws_upload = self.makeBinaryPackageUpload(
1567+ person, component=self.universe)
1568+ with person_logged_in(person):
1569+ new_section = self.factory.makeSection()
1570+ transaction.commit()
1571+
1572+ self.assertEqual("New", ws_upload.status)
1573+ ws_binaries = ws_upload.getBinaryProperties()
1574+ for binary in ws_binaries:
1575+ self.assertEqual("universe", binary["component"])
1576+ self.assertNotEqual(new_section.name, binary["section"])
1577+ self.assertEqual("OPTIONAL", binary["priority"])
1578+ change_one = {
1579+ "name": ws_binaries[0]["name"],
1580+ "component": "main",
1581+ "priority": "standard",
1582+ }
1583+ change_two = {
1584+ "name": ws_binaries[1]["name"],
1585+ "section": new_section.name,
1586+ }
1587+ ws_upload.overrideBinaries(changes=[change_one, change_two])
1588+ ws_binaries = ws_upload.getBinaryProperties()
1589+ self.assertEqual("main", ws_binaries[0]["component"])
1590+ self.assertNotEqual(new_section.name, ws_binaries[0]["section"])
1591+ self.assertEqual("STANDARD", ws_binaries[0]["priority"])
1592+ self.assertEqual("universe", ws_binaries[1]["component"])
1593+ self.assertEqual(new_section.name, ws_binaries[1]["section"])
1594+ self.assertEqual("OPTIONAL", ws_binaries[1]["priority"])
1595+
1596+ def test_custom_info(self):
1597+ # API clients can inspect properties of custom uploads.
1598+ person = self.makeQueueAdmin([self.universe])
1599+ upload, ws_upload = self.makeCustomPackageUpload(
1600+ person, custom_type=PackageUploadCustomFormat.DEBIAN_INSTALLER,
1601+ filename="debian-installer-images_1.tar.gz")
1602+
1603+ self.assertFalse(ws_upload.contains_source)
1604+ self.assertFalse(ws_upload.contains_build)
1605+ self.assertFalse(ws_upload.contains_copy)
1606+ self.assertEqual(
1607+ "debian-installer-images_1.tar.gz", ws_upload.display_name)
1608+ self.assertEqual("-", ws_upload.display_version)
1609+ self.assertEqual("raw-installer", ws_upload.display_arches)
1610+
1611+ def test_custom_fetch(self):
1612+ # API clients can fetch files attached to custom uploads.
1613+ person = self.makeQueueAdmin([self.universe])
1614+ upload, ws_upload = self.makeCustomPackageUpload(
1615+ person, custom_type=PackageUploadCustomFormat.DEBIAN_INSTALLER,
1616+ filename="debian-installer-images_1.tar.gz")
1617+ ws_custom_file_urls = ws_upload.customFileUrls()
1618+ self.assertNotEqual(0, len(ws_custom_file_urls))
1619+ with person_logged_in(person):
1620+ custom_file_urls = [
1621+ ProxiedLibraryFileAlias(
1622+ file.libraryfilealias, upload.archive).http_url
1623+ for file in upload.customfiles]
1624+ self.assertContentEqual(custom_file_urls, ws_custom_file_urls)
1625
1626=== modified file 'lib/lp/testing/factory.py'
1627--- lib/lp/testing/factory.py 2012-06-08 06:01:50 +0000
1628+++ lib/lp/testing/factory.py 2012-06-08 14:30:37 +0000
1629@@ -3494,7 +3494,7 @@
1630 return upload
1631
1632 def makeBuildPackageUpload(self, distroseries=None,
1633- binarypackagename=None):
1634+ binarypackagename=None, component=None):
1635 """Make a `PackageUpload` with a `PackageUploadBuild` attached."""
1636 if distroseries is None:
1637 distroseries = self.makeDistroSeries()
1638@@ -3503,7 +3503,8 @@
1639 build = self.makeBinaryPackageBuild()
1640 upload.addBuild(build)
1641 self.makeBinaryPackageRelease(
1642- binarypackagename=binarypackagename, build=build)
1643+ binarypackagename=binarypackagename, build=build,
1644+ component=component)
1645 return upload
1646
1647 def makeCustomPackageUpload(self, distroseries=None, custom_type=None,