Merge lp:~salgado/launchpad/vostok-upload-policy into lp:launchpad

Proposed by Guilherme Salgado
Status: Merged
Approved by: Michael Nelson
Approved revision: no longer in the source branch.
Merged at revision: 11450
Proposed branch: lp:~salgado/launchpad/vostok-upload-policy
Merge into: lp:launchpad
Prerequisite: lp:~salgado/launchpad/remove-security-upload-policy
Diff against target: 719 lines (+340/-84)
13 files modified
lib/lp/archiveuploader/nascentupload.py (+6/-31)
lib/lp/archiveuploader/tests/__init__.py (+4/-0)
lib/lp/archiveuploader/tests/nascentupload-announcements.txt (+2/-1)
lib/lp/archiveuploader/tests/nascentupload.txt (+2/-1)
lib/lp/archiveuploader/tests/nascentuploadfile.txt (+3/-2)
lib/lp/archiveuploader/tests/test_buildduploads.py (+130/-4)
lib/lp/archiveuploader/tests/test_nascentupload_documentation.py (+2/-1)
lib/lp/archiveuploader/tests/test_ppauploadprocessor.py (+3/-6)
lib/lp/archiveuploader/tests/test_uploadpolicy.py (+127/-0)
lib/lp/archiveuploader/uploadpolicy.py (+52/-14)
lib/lp/code/model/sourcepackagerecipebuild.py (+2/-5)
lib/lp/soyuz/browser/tests/distroseriesqueue-views.txt (+2/-1)
lib/lp/soyuz/doc/soyuz-set-of-uploads.txt (+5/-18)
To merge this branch: bzr merge lp:~salgado/launchpad/vostok-upload-policy
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) Approve
Review via email: mp+33584@code.launchpad.net

Description of the change

This branch moves the code related to validating the upload type from nascentupload to the upload policy, as a separate method. This makes it easy to unit test that code as well as adding more flexibility for subclasses that might want to validate differently.

It also replaces the three can_upload_* variables with a single enum variable.

To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Beautiful. A few notes, of course:

Lines 199 and 319 of the diff have multiple imports that don't conform to the new import formatting rules:

from foobar import (
    Bar,
    Foo,
    )

In line 379 of the diff, the "else" and the assertion throw the symmetry of the code slightly out of whack. Just an idea, but maybe you can move it out into a separate "else" clause?

    elif upload.binaryful:
        # [...]
    else:
        raise AssertionError("...")

The list of hand-wrapped text lines at line 382 of the diff puzzles me. Why not a single dedent("""triple-quoted multi-line string""")?

Finally, I think I see either an indentation problem or something involving tabs around line 390. Did you run "make lint" over the current form of this branch?

But that's all small fry. The branch is definitely an improvement.

review: Approve
Revision history for this message
Guilherme Salgado (salgado) wrote :

On Wed, 2010-08-25 at 05:27 +0000, Jeroen T. Vermeulen wrote:
> Review: Approve
> Beautiful. A few notes, of course:
>
> Lines 199 and 319 of the diff have multiple imports that don't conform
> to the new import formatting rules:
>
> from foobar import (
> Bar,
> Foo,
> )

Oops, fixed.

>
> In line 379 of the diff, the "else" and the assertion throw the
> symmetry of the code slightly out of whack. Just an idea, but maybe
> you can move it out into a separate "else" clause?
>
> elif upload.binaryful:
> # [...]
> else:
> raise AssertionError("...")

That wfm.

>
> The list of hand-wrapped text lines at line 382 of the diff puzzles
> me. Why not a single dedent("""triple-quoted multi-line string""")?

That's code I just moved around, but I've changed it as you suggest.

>
> Finally, I think I see either an indentation problem or something

Broken indentation indeed; didn't notice when moving the code around.

> involving tabs around line 390. Did you run "make lint" over the
> current form of this branch?

I never did because 'bzr send' would do that for me, but now that I'm
using bzr lp-submit...

>
> But that's all small fry. The branch is definitely an improvement.

Thanks for the review!

If I can make a suggestion as well, it'd be much better if you could
include chunks of the diff in your review[1], for context, instead of
using the line numbers of the diff. That way we don't have to go back
and forth between your comments and the diff (in my case between my mail
client and the merge-proposal's page), but more importantly because the
diff is a moving target, so there's a big chance that the line numbers
have changed when I get to reply to your review. :)

[1] I do that by replying to the m-p email on my mail client, so the
diff is quoted and I just insert my comments at the appropriate places.

Revision history for this message
Michael Nelson (michael.nelson) wrote :

r=me for -r11430..11432.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archiveuploader/nascentupload.py'
--- lib/lp/archiveuploader/nascentupload.py 2010-08-24 08:43:51 +0000
+++ lib/lp/archiveuploader/nascentupload.py 2010-08-25 17:14:48 +0000
@@ -146,10 +146,11 @@
146 UploadError will be raised and sent up to the caller. If this happens146 UploadError will be raised and sent up to the caller. If this happens
147 the caller should call the reject method and process a rejection.147 the caller should call the reject method and process a rejection.
148 """148 """
149 policy = self.policy
149 self.logger.debug("Beginning processing.")150 self.logger.debug("Beginning processing.")
150151
151 try:152 try:
152 self.policy.setDistroSeriesAndPocket(self.changes.suite_name)153 policy.setDistroSeriesAndPocket(self.changes.suite_name)
153 except NotFoundError:154 except NotFoundError:
154 self.reject(155 self.reject(
155 "Unable to find distroseries: %s" % self.changes.suite_name)156 "Unable to find distroseries: %s" % self.changes.suite_name)
@@ -185,36 +186,12 @@
185 isinstance(self.changes.files[0], CustomUploadFile)):186 isinstance(self.changes.files[0], CustomUploadFile)):
186 self.logger.debug("Single Custom Upload detected.")187 self.logger.debug("Single Custom Upload detected.")
187 else:188 else:
188 if self.sourceful and not self.policy.can_upload_source:189 policy.validateUploadType(self)
189 self.reject("Upload is sourceful, but policy refuses "190
190 "sourceful uploads.")191 if self.sourceful and not self.changes.dsc:
191
192 elif self.binaryful and not self.policy.can_upload_binaries:
193 messages = [
194 "Upload rejected because it contains binary packages.",
195 "Ensure you are using `debuild -S`, or an equivalent",
196 "command, to generate only the source package before",
197 "re-uploading.",
198 ]
199 if self.is_ppa:
200 messages.append(
201 "See https://help.launchpad.net/Packaging/PPA for more "
202 "information.")
203 self.reject(" ".join(messages))
204
205 elif (self.sourceful and self.binaryful and
206 not self.policy.can_upload_mixed):
207 self.reject("Upload is source/binary but policy refuses "
208 "mixed uploads.")
209
210 elif self.sourceful and not self.changes.dsc:
211 self.reject(192 self.reject(
212 "Unable to find the DSC file in the source upload.")193 "Unable to find the DSC file in the source upload.")
213194
214 else:
215 # Upload content are consistent with the current policy.
216 pass
217
218 # Apply the overrides from the database. This needs to be done195 # Apply the overrides from the database. This needs to be done
219 # before doing component verifications because the component196 # before doing component verifications because the component
220 # actually comes from overrides for packages that are not NEW.197 # actually comes from overrides for packages that are not NEW.
@@ -227,7 +204,7 @@
227 self.verify_acl()204 self.verify_acl()
228205
229 # Perform policy checks.206 # Perform policy checks.
230 self.policy.checkUpload(self)207 policy.checkUpload(self)
231208
232 # That's all folks.209 # That's all folks.
233 self.logger.debug("Finished checking upload.")210 self.logger.debug("Finished checking upload.")
@@ -996,8 +973,6 @@
996 # so late in the game is that in the973 # so late in the game is that in the
997 # mixed-upload case we only have a974 # mixed-upload case we only have a
998 # sourcepackagerelease to verify here!975 # sourcepackagerelease to verify here!
999 assert self.policy.can_upload_mixed, (
1000 "Current policy does not allow mixed uploads.")
1001 assert sourcepackagerelease, (976 assert sourcepackagerelease, (
1002 "No sourcepackagerelease was found.")977 "No sourcepackagerelease was found.")
1003 binary_package_file.verifySourcePackageRelease(978 binary_package_file.verifySourcePackageRelease(
1004979
=== modified file 'lib/lp/archiveuploader/tests/__init__.py'
--- lib/lp/archiveuploader/tests/__init__.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/tests/__init__.py 2010-08-25 17:14:48 +0000
@@ -117,6 +117,10 @@
117 # We require the changes to be signed but not the dsc117 # We require the changes to be signed but not the dsc
118 self.unsigned_dsc_ok = True118 self.unsigned_dsc_ok = True
119119
120 def validateUploadType(self, upload):
121 """We accept uploads of any type."""
122 pass
123
120 def policySpecificChecks(self, upload):124 def policySpecificChecks(self, upload):
121 """Nothing, let it go."""125 """Nothing, let it go."""
122 pass126 pass
123127
=== modified file 'lib/lp/archiveuploader/tests/nascentupload-announcements.txt'
--- lib/lp/archiveuploader/tests/nascentupload-announcements.txt 2010-08-25 17:14:46 +0000
+++ lib/lp/archiveuploader/tests/nascentupload-announcements.txt 2010-08-25 17:14:48 +0000
@@ -297,9 +297,10 @@
297NEW binary upload to RELEASE pocket via 'sync' policy (we need to297NEW binary upload to RELEASE pocket via 'sync' policy (we need to
298override sync policy to allow binary uploads):298override sync policy to allow binary uploads):
299299
300 >>> from lp.archiveuploader.uploadpolicy import ArchiveUploadType
300 >>> modified_sync_policy = getPolicy(301 >>> modified_sync_policy = getPolicy(
301 ... name='sync', distro='ubuntu', distroseries='hoary')302 ... name='sync', distro='ubuntu', distroseries='hoary')
302 >>> modified_sync_policy.can_upload_binaries = True303 >>> modified_sync_policy.accepted_type = ArchiveUploadType.BINARY_ONLY
303304
304 >>> bar_src = NascentUpload.from_changesfile_path(305 >>> bar_src = NascentUpload.from_changesfile_path(
305 ... datadir('suite/bar_1.0-1_binary/bar_1.0-1_i386.changes'),306 ... datadir('suite/bar_1.0-1_binary/bar_1.0-1_i386.changes'),
306307
=== modified file 'lib/lp/archiveuploader/tests/nascentupload.txt'
--- lib/lp/archiveuploader/tests/nascentupload.txt 2010-08-20 12:11:30 +0000
+++ lib/lp/archiveuploader/tests/nascentupload.txt 2010-08-25 17:14:48 +0000
@@ -499,9 +499,10 @@
499need unsigned changes and binaries uploads (same as security, but also499need unsigned changes and binaries uploads (same as security, but also
500accepts non-security uploads)500accepts non-security uploads)
501501
502 >>> from lp.archiveuploader.uploadpolicy import ArchiveUploadType
502 >>> modified_sync_policy = getPolicy(503 >>> modified_sync_policy = getPolicy(
503 ... name='sync', distro='ubuntu', distroseries='hoary')504 ... name='sync', distro='ubuntu', distroseries='hoary')
504 >>> modified_sync_policy.can_upload_binaries = True505 >>> modified_sync_policy.accepted_type = ArchiveUploadType.BINARY_ONLY
505506
506 >>> ed_bin = NascentUpload.from_changesfile_path(507 >>> ed_bin = NascentUpload.from_changesfile_path(
507 ... datadir('split-upload-test/ed_0.2-20_i386.changes'),508 ... datadir('split-upload-test/ed_0.2-20_i386.changes'),
508509
=== modified file 'lib/lp/archiveuploader/tests/nascentuploadfile.txt'
--- lib/lp/archiveuploader/tests/nascentuploadfile.txt 2010-08-20 12:11:30 +0000
+++ lib/lp/archiveuploader/tests/nascentuploadfile.txt 2010-08-25 17:14:48 +0000
@@ -531,9 +531,10 @@
531The timestamp of the files in the .deb are tested against the policy for being 531The timestamp of the files in the .deb are tested against the policy for being
532too new:532too new:
533533
534 >>> from lp.archiveuploader.uploadpolicy import ArchiveUploadType
534 >>> old_only_policy = getPolicy(535 >>> old_only_policy = getPolicy(
535 ... name='insecure', distro='ubuntu', distroseries='hoary')536 ... name='insecure', distro='ubuntu', distroseries='hoary')
536 >>> old_only_policy.can_upload_binaries = True537 >>> old_only_policy.accepted_type = ArchiveUploadType.BINARY_ONLY
537 >>> old_only_policy.future_time_grace = -5 * 365 * 24 * 60 * 60538 >>> old_only_policy.future_time_grace = -5 * 365 * 24 * 60 * 60
538539
539 >>> ed_binary_deb = DebBinaryUploadFile(ed_deb_path,540 >>> ed_binary_deb = DebBinaryUploadFile(ed_deb_path,
@@ -549,7 +550,7 @@
549550
550 >>> new_only_policy = getPolicy(551 >>> new_only_policy = getPolicy(
551 ... name='insecure', distro='ubuntu', distroseries='hoary')552 ... name='insecure', distro='ubuntu', distroseries='hoary')
552 >>> new_only_policy.can_upload_binaries = True553 >>> new_only_policy.accepted_type = ArchiveUploadType.BINARY_ONLY
553 >>> new_only_policy.earliest_year = 2010554 >>> new_only_policy.earliest_year = 2010
554 >>> ed_binary_deb = DebBinaryUploadFile(ed_deb_path,555 >>> ed_binary_deb = DebBinaryUploadFile(ed_deb_path,
555 ... 'e31eeb0b6b3b87e1ea79378df864ffff',556 ... 'e31eeb0b6b3b87e1ea79378df864ffff',
556557
=== modified file 'lib/lp/archiveuploader/tests/test_buildduploads.py'
--- lib/lp/archiveuploader/tests/test_buildduploads.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/tests/test_buildduploads.py 2010-08-25 17:14:48 +0000
@@ -5,13 +5,139 @@
55
6__metaclass__ = type6__metaclass__ = type
77
8import os
9
10from zope.component import getUtility
11
8from canonical.database.constants import UTC_NOW12from canonical.database.constants import UTC_NOW
9from canonical.launchpad.ftests import import_public_test_keys13from canonical.launchpad.ftests import import_public_test_keys
10from canonical.launchpad.interfaces import PackagePublishingStatus14from canonical.launchpad.interfaces import (
11from lp.archiveuploader.tests.test_securityuploads import (15 PackagePublishingStatus,
12 TestStagedBinaryUploadBase,16 PackageUploadStatus,
13 )17 )
18
19from lp.archiveuploader.tests.test_uploadprocessor import (
20 TestUploadProcessorBase,
21 )
22from lp.registry.interfaces.distribution import IDistributionSet
14from lp.registry.interfaces.pocket import PackagePublishingPocket23from lp.registry.interfaces.pocket import PackagePublishingPocket
24from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
25
26
27class TestStagedBinaryUploadBase(TestUploadProcessorBase):
28 name = 'baz'
29 version = '1.0-1'
30 distribution_name = None
31 distroseries_name = None
32 pocket = None
33 policy = 'buildd'
34 no_mails = True
35
36 @property
37 def distribution(self):
38 return getUtility(IDistributionSet)[self.distribution_name]
39
40 @property
41 def distroseries(self):
42 return self.distribution[self.distroseries_name]
43
44 @property
45 def package_name(self):
46 return "%s_%s" % (self.name, self.version)
47
48 @property
49 def source_dir(self):
50 return self.package_name
51
52 @property
53 def source_changesfile(self):
54 return "%s_source.changes" % self.package_name
55
56 @property
57 def binary_dir(self):
58 return "%s_binary" % self.package_name
59
60 def getBinaryChangesfileFor(self, archtag):
61 return "%s_%s.changes" % (self.package_name, archtag)
62
63 def setUp(self):
64 """Setup environment for staged binaries upload via security policy.
65
66 1. Setup queue directory and other basic attributes
67 2. Override policy options to get security policy and not send emails
68 3. Setup a common UploadProcessor with the overridden options
69 4. Store number of build present before issuing any upload
70 5. Upload the source package via security policy
71 6. Clean log messages.
72 7. Commit transaction, so the upload source can be seen.
73 """
74 super(TestStagedBinaryUploadBase, self).setUp()
75 self.options.context = self.policy
76 self.options.nomails = self.no_mails
77 # Set up the uploadprocessor with appropriate options and logger
78 self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
79 self.builds_before_upload = BinaryPackageBuild.select().count()
80 self.source_queue = None
81 self._uploadSource()
82 self.log.lines = []
83 self.layer.txn.commit()
84
85 def assertBuildsCreated(self, amount):
86 """Assert that a given 'amount' of build records was created."""
87 builds_count = BinaryPackageBuild.select().count()
88 self.assertEqual(
89 self.builds_before_upload + amount, builds_count)
90
91 def _prepareUpload(self, upload_dir):
92 """Place a copy of the upload directory into incoming queue."""
93 os.system("cp -a %s %s" %
94 (os.path.join(self.test_files_dir, upload_dir),
95 os.path.join(self.queue_folder, "incoming")))
96
97 def _uploadSource(self):
98 """Upload and Accept (if necessary) the base source."""
99 self._prepareUpload(self.source_dir)
100 self.uploadprocessor.processChangesFile(
101 os.path.join(self.queue_folder, "incoming", self.source_dir),
102 self.source_changesfile)
103 queue_item = self.uploadprocessor.last_processed_upload.queue_root
104 self.assertTrue(
105 queue_item is not None,
106 "Source Upload Failed\nGot: %s" % "\n".join(self.log.lines))
107 acceptable_statuses = [
108 PackageUploadStatus.NEW,
109 PackageUploadStatus.UNAPPROVED,
110 ]
111 if queue_item.status in acceptable_statuses:
112 queue_item.setAccepted()
113 # Store source queue item for future use.
114 self.source_queue = queue_item
115
116 def _uploadBinary(self, archtag):
117 """Upload the base binary.
118
119 Ensure it got processed and has a respective queue record.
120 Return the IBuild attached to upload.
121 """
122 self._prepareUpload(self.binary_dir)
123 self.uploadprocessor.processChangesFile(
124 os.path.join(self.queue_folder, "incoming", self.binary_dir),
125 self.getBinaryChangesfileFor(archtag))
126 queue_item = self.uploadprocessor.last_processed_upload.queue_root
127 self.assertTrue(
128 queue_item is not None,
129 "Binary Upload Failed\nGot: %s" % "\n".join(self.log.lines))
130 self.assertEqual(queue_item.builds.count(), 1)
131 return queue_item.builds[0].build
132
133 def _createBuild(self, archtag):
134 """Create a build record attached to the base source."""
135 spr = self.source_queue.sources[0].sourcepackagerelease
136 build = spr.createBuild(
137 distro_arch_series=self.distroseries[archtag],
138 pocket=self.pocket, archive=self.distroseries.main_archive)
139 self.layer.txn.commit()
140 return build
15141
16142
17class TestBuilddUploads(TestStagedBinaryUploadBase):143class TestBuilddUploads(TestStagedBinaryUploadBase):
18144
=== modified file 'lib/lp/archiveuploader/tests/test_nascentupload_documentation.py'
--- lib/lp/archiveuploader/tests/test_nascentupload_documentation.py 2010-08-23 13:39:29 +0000
+++ lib/lp/archiveuploader/tests/test_nascentupload_documentation.py 2010-08-25 17:14:48 +0000
@@ -30,6 +30,7 @@
30 getPolicy,30 getPolicy,
31 mock_logger_quiet,31 mock_logger_quiet,
32 )32 )
33from lp.archiveuploader.uploadpolicy import ArchiveUploadType
33from lp.registry.interfaces.distribution import IDistributionSet34from lp.registry.interfaces.distribution import IDistributionSet
34from lp.soyuz.interfaces.component import IComponentSet35from lp.soyuz.interfaces.component import IComponentSet
3536
@@ -52,7 +53,7 @@
52def getUploadForBinary(upload_path):53def getUploadForBinary(upload_path):
53 """Return a NascentUpload object for binaries."""54 """Return a NascentUpload object for binaries."""
54 policy = getPolicy(name='sync', distro='ubuntu', distroseries='hoary')55 policy = getPolicy(name='sync', distro='ubuntu', distroseries='hoary')
55 policy.can_upload_binaries = True56 policy.accepted_type = ArchiveUploadType.BINARY_ONLY
56 return NascentUpload.from_changesfile_path(57 return NascentUpload.from_changesfile_path(
57 datadir(upload_path), policy, mock_logger_quiet)58 datadir(upload_path), policy, mock_logger_quiet)
5859
5960
=== modified file 'lib/lp/archiveuploader/tests/test_ppauploadprocessor.py'
--- lib/lp/archiveuploader/tests/test_ppauploadprocessor.py 2010-08-20 20:31:18 +0000
+++ lib/lp/archiveuploader/tests/test_ppauploadprocessor.py 2010-08-25 17:14:48 +0000
@@ -681,12 +681,9 @@
681 "bar_1.0-1-mixed", "~name16/ubuntu")681 "bar_1.0-1-mixed", "~name16/ubuntu")
682 self.processUpload(self.uploadprocessor, upload_dir)682 self.processUpload(self.uploadprocessor, upload_dir)
683683
684 self.assertEqual(684 self.assertIn(
685 self.uploadprocessor.last_processed_upload.rejection_message,685 'Source/binary (i.e. mixed) uploads are not allowed.',
686 'Upload rejected because it contains binary packages. Ensure '686 self.uploadprocessor.last_processed_upload.rejection_message)
687 'you are using `debuild -S`, or an equivalent command, to '
688 'generate only the source package before re-uploading. See '
689 'https://help.launchpad.net/Packaging/PPA for more information.')
690687
691 def testPGPSignatureNotPreserved(self):688 def testPGPSignatureNotPreserved(self):
692 """PGP signatures should be removed from PPA .changes files.689 """PGP signatures should be removed from PPA .changes files.
693690
=== added file 'lib/lp/archiveuploader/tests/test_uploadpolicy.py'
--- lib/lp/archiveuploader/tests/test_uploadpolicy.py 1970-01-01 00:00:00 +0000
+++ lib/lp/archiveuploader/tests/test_uploadpolicy.py 2010-08-25 17:14:48 +0000
@@ -0,0 +1,127 @@
1#!/usr/bin/python
2#
3# Copyright 2010 Canonical Ltd. This software is licensed under the
4# GNU Affero General Public License version 3 (see the file LICENSE).
5
6from canonical.testing.layers import DatabaseFunctionalLayer
7
8from lp.app.errors import NotFoundError
9from lp.archiveuploader.uploadpolicy import (
10 AbstractUploadPolicy,
11 ArchiveUploadType,
12 )
13from lp.testing import TestCase, TestCaseWithFactory
14
15
16class TestUploadPolicy_validateUploadType(TestCase):
17 """Test what kind (sourceful/binaryful/mixed) of uploads are accepted."""
18
19 def test_sourceful_accepted(self):
20 policy = make_policy(accepted_type=ArchiveUploadType.SOURCE_ONLY)
21 upload = make_fake_upload(sourceful=True)
22
23 policy.validateUploadType(upload)
24
25 self.assertEquals([], upload.rejections)
26
27 def test_binaryful_accepted(self):
28 policy = make_policy(accepted_type=ArchiveUploadType.BINARY_ONLY)
29 upload = make_fake_upload(binaryful=True)
30
31 policy.validateUploadType(upload)
32
33 self.assertEquals([], upload.rejections)
34
35 def test_mixed_accepted(self):
36 policy = make_policy(accepted_type=ArchiveUploadType.MIXED_ONLY)
37 upload = make_fake_upload(sourceful=True, binaryful=True)
38
39 policy.validateUploadType(upload)
40
41 self.assertEquals([], upload.rejections)
42
43 def test_sourceful_not_accepted(self):
44 policy = make_policy(accepted_type=ArchiveUploadType.BINARY_ONLY)
45 upload = make_fake_upload(sourceful=True)
46
47 policy.validateUploadType(upload)
48
49 self.assertIn(
50 'Sourceful uploads are not accepted by this policy.',
51 upload.rejections)
52
53 def test_binaryful_not_accepted(self):
54 policy = make_policy(accepted_type=ArchiveUploadType.SOURCE_ONLY)
55 upload = make_fake_upload(binaryful=True)
56
57 policy.validateUploadType(upload)
58
59 self.assertTrue(len(upload.rejections) > 0)
60 self.assertIn(
61 'Upload rejected because it contains binary packages.',
62 upload.rejections[0])
63
64 def test_mixed_not_accepted(self):
65 policy = make_policy(accepted_type=ArchiveUploadType.SOURCE_ONLY)
66 upload = make_fake_upload(sourceful=True, binaryful=True)
67
68 policy.validateUploadType(upload)
69
70 self.assertIn(
71 'Source/binary (i.e. mixed) uploads are not allowed.',
72 upload.rejections)
73
74 def test_sourceful_when_only_mixed_accepted(self):
75 policy = make_policy(accepted_type=ArchiveUploadType.MIXED_ONLY)
76 upload = make_fake_upload(sourceful=True, binaryful=False)
77
78 policy.validateUploadType(upload)
79
80 self.assertIn(
81 'Sourceful uploads are not accepted by this policy.',
82 upload.rejections)
83
84 def test_binaryful_when_only_mixed_accepted(self):
85 policy = make_policy(accepted_type=ArchiveUploadType.MIXED_ONLY)
86 upload = make_fake_upload(sourceful=False, binaryful=True)
87
88 policy.validateUploadType(upload)
89
90 self.assertTrue(len(upload.rejections) > 0)
91 self.assertIn(
92 'Upload rejected because it contains binary packages.',
93 upload.rejections[0])
94
95
96class TestUploadPolicy(TestCaseWithFactory):
97
98 layer = DatabaseFunctionalLayer
99
100 def test_setDistroSeriesAndPocket_distro_not_found(self):
101 policy = AbstractUploadPolicy()
102 policy.distro = self.factory.makeDistribution()
103 self.assertRaises(
104 NotFoundError, policy.setDistroSeriesAndPocket,
105 'nonexistent_security')
106
107
108class FakeNascentUpload:
109
110 def __init__(self, sourceful, binaryful):
111 self.sourceful = sourceful
112 self.binaryful = binaryful
113 self.is_ppa = False
114 self.rejections = []
115
116 def reject(self, msg):
117 self.rejections.append(msg)
118
119
120def make_fake_upload(sourceful=False, binaryful=False):
121 return FakeNascentUpload(sourceful, binaryful)
122
123
124def make_policy(accepted_type):
125 policy = AbstractUploadPolicy()
126 policy.accepted_type = accepted_type
127 return policy
0128
=== modified file 'lib/lp/archiveuploader/uploadpolicy.py'
--- lib/lp/archiveuploader/uploadpolicy.py 2010-08-25 17:14:46 +0000
+++ lib/lp/archiveuploader/uploadpolicy.py 2010-08-25 17:14:48 +0000
@@ -7,6 +7,7 @@
77
8__all__ = [8__all__ = [
9 "AbstractUploadPolicy",9 "AbstractUploadPolicy",
10 "ArchiveUploadType",
10 "BuildDaemonUploadPolicy",11 "BuildDaemonUploadPolicy",
11 "findPolicyByName",12 "findPolicyByName",
12 "IArchiveUploadPolicy",13 "IArchiveUploadPolicy",
@@ -14,6 +15,8 @@
14 "UploadPolicyError",15 "UploadPolicyError",
15 ]16 ]
1617
18from textwrap import dedent
19
17from zope.component import (20from zope.component import (
18 getGlobalSiteManager,21 getGlobalSiteManager,
19 getUtility,22 getUtility,
@@ -28,6 +31,9 @@
28from lp.registry.interfaces.pocket import PackagePublishingPocket31from lp.registry.interfaces.pocket import PackagePublishingPocket
29from lp.registry.interfaces.series import SeriesStatus32from lp.registry.interfaces.series import SeriesStatus
3033
34from lazr.enum import EnumeratedType, Item
35
36
31# Defined here so that uploadpolicy.py doesn't depend on lp.code.37# Defined here so that uploadpolicy.py doesn't depend on lp.code.
32SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME = 'recipe'38SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME = 'recipe'
33# Number of seconds in an hour (used later)39# Number of seconds in an hour (used later)
@@ -52,6 +58,13 @@
52 """58 """
5359
5460
61class ArchiveUploadType(EnumeratedType):
62
63 SOURCE_ONLY = Item("Source only")
64 BINARY_ONLY = Item("Binary only")
65 MIXED_ONLY = Item("Mixed only")
66
67
55class AbstractUploadPolicy:68class AbstractUploadPolicy:
56 """Encapsulate the policy of an upload to a launchpad archive.69 """Encapsulate the policy of an upload to a launchpad archive.
5770
@@ -63,8 +76,9 @@
63 """76 """
64 implements(IArchiveUploadPolicy)77 implements(IArchiveUploadPolicy)
6578
79 name = 'abstract'
66 options = None80 options = None
67 name = 'abstract'81 accepted_type = None # Must be defined in subclasses.
6882
69 def __init__(self):83 def __init__(self):
70 """Prepare a policy..."""84 """Prepare a policy..."""
@@ -75,14 +89,45 @@
75 self.unsigned_changes_ok = False89 self.unsigned_changes_ok = False
76 self.unsigned_dsc_ok = False90 self.unsigned_dsc_ok = False
77 self.create_people = True91 self.create_people = True
78 self.can_upload_source = True
79 self.can_upload_binaries = True
80 self.can_upload_mixed = True
81 # future_time_grace is in seconds. 28800 is 8 hours92 # future_time_grace is in seconds. 28800 is 8 hours
82 self.future_time_grace = 8 * HOURS93 self.future_time_grace = 8 * HOURS
83 # The earliest year we accept in a deb's file's mtime94 # The earliest year we accept in a deb's file's mtime
84 self.earliest_year = 198495 self.earliest_year = 1984
8596
97 def validateUploadType(self, upload):
98 """Check that the type of the given upload is accepted by this policy.
99
100 When the type (e.g. sourceful, binaryful or mixed) is not accepted,
101 the upload is rejected.
102 """
103 if upload.sourceful and upload.binaryful:
104 if self.accepted_type != ArchiveUploadType.MIXED_ONLY:
105 upload.reject(
106 "Source/binary (i.e. mixed) uploads are not allowed.")
107
108 elif upload.sourceful:
109 if self.accepted_type != ArchiveUploadType.SOURCE_ONLY:
110 upload.reject(
111 "Sourceful uploads are not accepted by this policy.")
112
113 elif upload.binaryful:
114 if self.accepted_type != ArchiveUploadType.BINARY_ONLY:
115 message = dedent("""
116 Upload rejected because it contains binary packages.
117 Ensure you are using `debuild -S`, or an equivalent
118 command, to generate only the source package before
119 re-uploading.""")
120
121 if upload.is_ppa:
122 message += dedent("""
123 See https://help.launchpad.net/Packaging/PPA for
124 more information.""")
125 upload.reject(message)
126
127 else:
128 raise AssertionError(
129 "Upload is not sourceful, binaryful or mixed.")
130
86 def getUploader(self, changes):131 def getUploader(self, changes):
87 """Get the person who is doing the uploading."""132 """Get the person who is doing the uploading."""
88 return changes.signer133 return changes.signer
@@ -171,11 +216,7 @@
171 """The insecure upload policy is used by the poppy interface."""216 """The insecure upload policy is used by the poppy interface."""
172217
173 name = 'insecure'218 name = 'insecure'
174219 accepted_type = ArchiveUploadType.SOURCE_ONLY
175 def __init__(self):
176 AbstractUploadPolicy.__init__(self)
177 self.can_upload_binaries = False
178 self.can_upload_mixed = False
179220
180 def rejectPPAUploads(self, upload):221 def rejectPPAUploads(self, upload):
181 """Insecure policy allows PPA upload."""222 """Insecure policy allows PPA upload."""
@@ -283,14 +324,13 @@
283 """The build daemon upload policy is invoked by the slave scanner."""324 """The build daemon upload policy is invoked by the slave scanner."""
284325
285 name = 'buildd'326 name = 'buildd'
327 accepted_type = ArchiveUploadType.BINARY_ONLY
286328
287 def __init__(self):329 def __init__(self):
288 super(BuildDaemonUploadPolicy, self).__init__()330 super(BuildDaemonUploadPolicy, self).__init__()
289 # We permit unsigned uploads because we trust our build daemons331 # We permit unsigned uploads because we trust our build daemons
290 self.unsigned_changes_ok = True332 self.unsigned_changes_ok = True
291 self.unsigned_dsc_ok = True333 self.unsigned_dsc_ok = True
292 self.can_upload_source = False
293 self.can_upload_mixed = False
294334
295 def setOptions(self, options):335 def setOptions(self, options):
296 AbstractUploadPolicy.setOptions(self, options)336 AbstractUploadPolicy.setOptions(self, options)
@@ -314,15 +354,13 @@
314 """This policy is invoked when processing sync uploads."""354 """This policy is invoked when processing sync uploads."""
315355
316 name = 'sync'356 name = 'sync'
357 accepted_type = ArchiveUploadType.SOURCE_ONLY
317358
318 def __init__(self):359 def __init__(self):
319 AbstractUploadPolicy.__init__(self)360 AbstractUploadPolicy.__init__(self)
320 # We don't require changes or dsc to be signed for syncs361 # We don't require changes or dsc to be signed for syncs
321 self.unsigned_changes_ok = True362 self.unsigned_changes_ok = True
322 self.unsigned_dsc_ok = True363 self.unsigned_dsc_ok = True
323 # We don't want binaries in a sync
324 self.can_upload_mixed = False
325 self.can_upload_binaries = False
326364
327 def policySpecificChecks(self, upload):365 def policySpecificChecks(self, upload):
328 """Perform sync specific checks."""366 """Perform sync specific checks."""
329367
=== modified file 'lib/lp/code/model/sourcepackagerecipebuild.py'
--- lib/lp/code/model/sourcepackagerecipebuild.py 2010-08-24 09:51:26 +0000
+++ lib/lp/code/model/sourcepackagerecipebuild.py 2010-08-25 17:14:48 +0000
@@ -40,6 +40,7 @@
40from canonical.launchpad.webapp import errorlog40from canonical.launchpad.webapp import errorlog
41from lp.app.errors import NotFoundError41from lp.app.errors import NotFoundError
42from lp.archiveuploader.uploadpolicy import (42from lp.archiveuploader.uploadpolicy import (
43 ArchiveUploadType,
43 BuildDaemonUploadPolicy,44 BuildDaemonUploadPolicy,
44 IArchiveUploadPolicy,45 IArchiveUploadPolicy,
45 SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME,46 SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME,
@@ -78,11 +79,7 @@
78 """Policy for uploading the results of a source package recipe build."""79 """Policy for uploading the results of a source package recipe build."""
7980
80 name = SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME81 name = SOURCE_PACKAGE_RECIPE_UPLOAD_POLICY_NAME
8182 accepted_type = ArchiveUploadType.SOURCE_ONLY
82 def __init__(self):
83 super(SourcePackageRecipeUploadPolicy, self).__init__()
84 self.can_upload_source = True
85 self.can_upload_binaries = False
8683
87 def getUploader(self, changes):84 def getUploader(self, changes):
88 """Return the person doing the upload."""85 """Return the person doing the upload."""
8986
=== modified file 'lib/lp/soyuz/browser/tests/distroseriesqueue-views.txt'
--- lib/lp/soyuz/browser/tests/distroseriesqueue-views.txt 2010-08-20 12:11:30 +0000
+++ lib/lp/soyuz/browser/tests/distroseriesqueue-views.txt 2010-08-25 17:14:48 +0000
@@ -253,6 +253,7 @@
253 ... sourcename="foo", distroseries=hoary, version="1.0-2",253 ... sourcename="foo", distroseries=hoary, version="1.0-2",
254 ... status=PackagePublishingStatus.PUBLISHED)254 ... status=PackagePublishingStatus.PUBLISHED)
255255
256 >>> from lp.archiveuploader.uploadpolicy import ArchiveUploadType
256 >>> from canonical.launchpad.ftests import import_public_test_keys257 >>> from canonical.launchpad.ftests import import_public_test_keys
257 >>> from canonical.launchpad.interfaces import IComponentSet258 >>> from canonical.launchpad.interfaces import IComponentSet
258 >>> from lp.soyuz.model.component import ComponentSelection259 >>> from lp.soyuz.model.component import ComponentSelection
@@ -264,7 +265,7 @@
264 >>> trash = ComponentSelection(distroseries=hoary, component=universe)265 >>> trash = ComponentSelection(distroseries=hoary, component=universe)
265 >>> sync_policy = getPolicy(name='sync', distro='ubuntu',266 >>> sync_policy = getPolicy(name='sync', distro='ubuntu',
266 ... distroseries='hoary')267 ... distroseries='hoary')
267 >>> sync_policy.can_upload_binaries = True268 >>> sync_policy.accepted_type = ArchiveUploadType.BINARY_ONLY
268 >>> foo_upload = NascentUpload.from_changesfile_path(269 >>> foo_upload = NascentUpload.from_changesfile_path(
269 ... datadir('suite/foo_1.0-2_multi_binary/foo_1.0-2_i386.changes'),270 ... datadir('suite/foo_1.0-2_multi_binary/foo_1.0-2_i386.changes'),
270 ... sync_policy, mock_logger_quiet)271 ... sync_policy, mock_logger_quiet)
271272
=== modified file 'lib/lp/soyuz/doc/soyuz-set-of-uploads.txt'
--- lib/lp/soyuz/doc/soyuz-set-of-uploads.txt 2010-08-06 14:29:36 +0000
+++ lib/lp/soyuz/doc/soyuz-set-of-uploads.txt 2010-08-25 17:14:48 +0000
@@ -319,22 +319,6 @@
319 Subject: bar_1.0-3_source.changes rejected319 Subject: bar_1.0-3_source.changes rejected
320 ...320 ...
321321
322Check the rejection (instead of failure) of an upload to a series or
323pocket which does not exist. We use the security upload policy for easier
324testing, as unsigned uploads are allowed.
325
326 >>> simulate_upload(
327 ... 'unstable_1.0-1', upload_policy="security", loglevel=logging.ERROR)
328 Rejected uploads: ['unstable_1.0-1']
329
330 >>> read_email()
331 To: Celso Providelo <celso.providelo@canonical.com>
332 Subject: unstable_1.0-1_source.changes rejected
333 ...
334 Rejected:
335 Unable to find distroseries: unstable
336 ...
337
338Force weird behavior with rfc2047 sentences containing '.' on322Force weird behavior with rfc2047 sentences containing '.' on
339bar_1.0-4, which caused bug # 41102.323bar_1.0-4, which caused bug # 41102.
340324
@@ -605,6 +589,9 @@
605589
606== Staged Security Uploads ==590== Staged Security Uploads ==
607591
592XXX: Salgado, 2010-08-25, bug=624078: This entire section should be removed
593but if you do so publish-distro.py (called further down) will hang.
594
608Perform a staged (source-first) security upload, simulating a run of595Perform a staged (source-first) security upload, simulating a run of
609the buildd-queuebuilder inbetween, to verify fix for bug 53437. To be596the buildd-queuebuilder inbetween, to verify fix for bug 53437. To be
610allowed a security upload, we need to use a released distroseries,597allowed a security upload, we need to use a released distroseries,
@@ -617,7 +604,7 @@
617 >>> ss = SectionSelection(distroseries=warty, section=devel)604 >>> ss = SectionSelection(distroseries=warty, section=devel)
618605
619 >>> simulate_upload(606 >>> simulate_upload(
620 ... 'baz_1.0-1', is_new=True, upload_policy="security",607 ... 'baz_1.0-1', is_new=True, upload_policy="anything",
621 ... series="warty-security", distro="ubuntu")608 ... series="warty-security", distro="ubuntu")
622609
623Check there's a SourcePackageRelease with no build.610Check there's a SourcePackageRelease with no build.
@@ -650,7 +637,7 @@
650Upload the i386 binary:637Upload the i386 binary:
651638
652 >>> simulate_upload(639 >>> simulate_upload(
653 ... 'baz_1.0-1_single_binary', is_new=True, upload_policy="security",640 ... 'baz_1.0-1_single_binary', is_new=True, upload_policy="anything",
654 ... distro="ubuntu", series="warty-security")641 ... distro="ubuntu", series="warty-security")
655642
656Should still just have one build, and it should now be FULLYBUILT.643Should still just have one build, and it should now be FULLYBUILT.