Merge lp:~cjwatson/launchpad/copy-custom-uploads into lp:launchpad

[Colin Watson]

Handle custom files in direct copies (and hence PCJs) by synthesising an upload.

== Summary ==

One of the last bugs that cause Ubuntu archive administrators to still require access to lp_publish@cocoplum (a horrendously privileged account) is bug 231371: if we upload installer or upgrader images to -proposed, get them past verification, and copy them to -updates, we have to copy the custom files around by hand. At the moment we have a pile of manual instructions in https://wiki.ubuntu.com/ArchiveAdministration#Moving_Packages_to_Updates for this, but obviously this should be automated.

== Proposed fix ==

A while back, Jeroen wrote CustomUploadsCopier, which deals with copying custom files across when we open a new distroseries. I always rather suspected that it wouldn't be too difficult to hook it up to solve this problem too, and it looks as though I was right. :-)

This does make one aspect of direct copies behave a little bit like delayed copies in that they synthesise an upload, and I considered making this case trigger a delayed copy, but consensus seems to be that delayed copies should be removed at the earliest opportunity, and I can't say I disagree. Besides, CustomUploadsCopier.copyUpload does enough of the work for us that I think it would actually be more code to reuse delayed copies for this.

== LOC Rationale ==

I got it down to +68 with a bit of judicious tidying up of unit tests. Aside from that, this is part of the general arc of tidying up package copying described in https://code.launchpad.net/~cjwatson/launchpad/pcj-reupload/+merge/111124; as indicated there, even with the addition of this branch that's going to come out substantially LoC-negative.

== Tests ==

bin/test -vvct test_copypackage -t test_custom_uploads_copier -t test_factory -t test_publish_ftpmaster

== Demo and Q/A ==

This will involve two publisher runs on dogfood, so it won't be exactly fast, but it shouldn't be difficult. Upload debian-installer to precise-proposed on dogfood, let it build, publish it, then use Archive.copyPackage (or 'sru-release -l dogfood' from lp:ubuntu-archive-tools) to copy it to precise-updates. This should create the appropriate versioned subdirectories and "current" symlinks in dists/precise-updates/main/installer-*/.

== Lint ==

Just one pre-existing and not very interesting entry:

./lib/lp/soyuz/scripts/tests/test_copypackage.py
    1450: E501 line too long (82 characters)

[Brad Crittenden (bac)]

review: Abstain (code)

[Benji York (benji)]

This branch looks good. The replacing of assertEquals with assertEqual is appreciated.

review: Approve (code)

=== modified file 'database/schema/security.cfg'
--- database/schema/security.cfg	2012-06-22 05:36:22 +0000
+++ database/schema/security.cfg	2012-06-25 16:57:21 +0000
@@ -1034,6 +1034,7 @@
 public.packageupload                            = SELECT
 public.packageuploadsource                      = SELECT
 public.packageuploadbuild                       = SELECT
+public.packageuploadcustom                      = SELECT
 public.packaging                                = SELECT, INSERT
 public.person                                   = SELECT
 public.pocketchroot                             = SELECT

=== modified file 'lib/lp/archivepublisher/scripts/publish_ftpmaster.py'
--- lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2012-05-23 00:20:23 +0000
+++ lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2012-06-25 16:57:21 +0000
@@ -18,7 +18,10 @@
 from lp.archivepublisher.interfaces.publisherconfig import IPublisherConfigSet
 from lp.archivepublisher.publishing import GLOBAL_PUBLISHER_LOCK
 from lp.registry.interfaces.distribution import IDistributionSet
-from lp.registry.interfaces.pocket import pocketsuffix
+from lp.registry.interfaces.pocket import (
+    PackagePublishingPocket,
+    pocketsuffix,
+    )
 from lp.registry.interfaces.series import SeriesStatus
 from lp.services.config import config
 from lp.services.database.bulk import load_related
@@ -588,7 +591,11 @@
                 # This is a fresh series.
                 have_fresh_series = True
                 if series.previous_series is not None:
-                    CustomUploadsCopier(series).copy(series.previous_series)
+                    copier = CustomUploadsCopier(
+                        series, PackagePublishingPocket.RELEASE)
+                    copier.copy(
+                        series.previous_series,
+                        PackagePublishingPocket.RELEASE)
                 self.createIndexes(distribution, suites_needing_indexes)
 
         return have_fresh_series

=== modified file 'lib/lp/soyuz/scripts/custom_uploads_copier.py'
--- lib/lp/soyuz/scripts/custom_uploads_copier.py	2012-05-30 10:25:43 +0000
+++ lib/lp/soyuz/scripts/custom_uploads_copier.py	2012-06-25 16:57:21 +0000
@@ -42,17 +42,20 @@
         PackageUploadCustomFormat.DIST_UPGRADER: DistUpgraderUpload,
         }
 
-    def __init__(self, target_series):
+    def __init__(self, target_series,
+                 target_pocket=PackagePublishingPocket.RELEASE):
         self.target_series = target_series
+        self.target_pocket = target_pocket
 
     def isCopyable(self, upload):
         """Is `upload` the kind of `PackageUploadCustom` that we can copy?"""
         return upload.customformat in self.copyable_types
 
-    def getCandidateUploads(self, source_series):
+    def getCandidateUploads(self, source_series,
+                            source_pocket=PackagePublishingPocket.RELEASE):
         """Find custom uploads that may need copying."""
         uploads = source_series.getPackageUploads(
-            custom_type=self.copyable_types.keys())
+            pocket=source_pocket, custom_type=self.copyable_types.keys())
         load_referencing(PackageUploadCustom, uploads, ['packageuploadID'])
         customs = sum([list(upload.customfiles) for upload in uploads], [])
         customs = filter(self.isCopyable, customs)
@@ -77,15 +80,19 @@
         else:
             return (custom_format, series_key)
 
-    def getLatestUploads(self, source_series):
+    def getLatestUploads(self, source_series,
+                         source_pocket=PackagePublishingPocket.RELEASE):
         """Find the latest uploads.
 
         :param source_series: The `DistroSeries` whose uploads to get.
+        :param source_pocket: The `PackagePublishingPocket` to inspect.
         :return: A dict containing the latest uploads, indexed by keys as
             returned by `getKey`.
         """
+        candidate_uploads = self.getCandidateUploads(
+            source_series, source_pocket=source_pocket)
         latest_uploads = {}
-        for upload in self.getCandidateUploads(source_series):
+        for upload in candidate_uploads:
             key = self.getKey(upload)
             if key is not None:
                 latest_uploads.setdefault(key, upload)
@@ -120,16 +127,20 @@
         if target_archive is None:
             return None
         package_upload = self.target_series.createQueueEntry(
-            PackagePublishingPocket.RELEASE, target_archive,
+            self.target_pocket, target_archive,
             changes_file_alias=original_upload.packageupload.changesfile)
         custom = package_upload.addCustom(
             original_upload.libraryfilealias, original_upload.customformat)
         package_upload.setAccepted()
         return custom
 
-    def copy(self, source_series):
-        """Copy uploads from `source_series`."""
-        target_uploads = self.getLatestUploads(self.target_series)
-        for upload in self.getLatestUploads(source_series).itervalues():
+    def copy(self, source_series,
+             source_pocket=PackagePublishingPocket.RELEASE):
+        """Copy uploads from `source_series`-`source_pocket`."""
+        target_uploads = self.getLatestUploads(
+            self.target_series, source_pocket=self.target_pocket)
+        source_uploads = self.getLatestUploads(
+            source_series, source_pocket=source_pocket)
+        for upload in source_uploads.itervalues():
             if not self.isObsolete(upload, target_uploads):
                 self.copyUpload(upload)

=== modified file 'lib/lp/soyuz/scripts/packagecopier.py'
--- lib/lp/soyuz/scripts/packagecopier.py	2012-06-22 23:55:02 +0000
+++ lib/lp/soyuz/scripts/packagecopier.py	2012-06-25 16:57:21 +0000
@@ -49,6 +49,7 @@
     IPackageUploadCustom,
     IPackageUploadSet,
     )
+from lp.soyuz.scripts.custom_uploads_copier import CustomUploadsCopier
 from lp.soyuz.scripts.ftpmasterbase import (
     SoyuzScript,
     SoyuzScriptError,
@@ -731,6 +732,7 @@
         publications.
     """
     copies = []
+    custom_files = []
 
     # Copy source if it's not yet copied.
     source_in_destination = archive.getPublishedSources(
@@ -762,6 +764,8 @@
         copies.append(source_copy)
     else:
         source_copy = source_in_destination.first()
+    if source_copy.packageupload is not None:
+        custom_files.extend(source_copy.packageupload.customfiles)
 
     if include_binaries:
         # Copy missing binaries for the matching architectures in the
@@ -771,10 +775,22 @@
         # arch-indep publications.
         binary_copies = getUtility(IPublishingSet).copyBinariesTo(
             source.getBuiltBinaries(), series, pocket, archive, policy=policy)
-        # XXX cjwatson 2012-06-22 bug=231371: Copy custom uploads.
 
         if binary_copies is not None:
             copies.extend(binary_copies)
+            binary_uploads = set(
+                bpph.binarypackagerelease.build.package_upload
+                for bpph in binary_copies)
+            for binary_upload in binary_uploads:
+                if binary_upload is not None:
+                    custom_files.extend(binary_upload.customfiles)
+
+    if custom_files:
+        # Custom uploads aren't modelled as publication history records, so
+        # we have to send these through the upload queue.
+        custom_copier = CustomUploadsCopier(series, target_pocket=pocket)
+        for custom in custom_files:
+            custom_copier.copyUpload(custom)
 
     # Always ensure the needed builds exist in the copy destination
     # after copying the binaries.

=== modified file 'lib/lp/soyuz/scripts/tests/test_copypackage.py'
--- lib/lp/soyuz/scripts/tests/test_copypackage.py	2012-06-20 09:19:37 +0000
+++ lib/lp/soyuz/scripts/tests/test_copypackage.py	2012-06-25 16:57:21 +0000
@@ -111,11 +111,9 @@
 
         Their filename, mimetype and file contents should be the same.
         """
-        self.assertEquals(
-            old.filename, new.filename, 'Filename mismatch.')
-        self.assertEquals(
-            old.mimetype, new.mimetype, 'MIME type mismatch.')
-        self.assertEquals(old.read(), new.read(), 'Content mismatch.')
+        self.assertEqual(old.filename, new.filename, 'Filename mismatch.')
+        self.assertEqual(old.mimetype, new.mimetype, 'MIME type mismatch.')
+        self.assertEqual(old.read(), new.read(), 'Content mismatch.')
 
     def assertFileIsReset(self, reuploaded_file):
         """Assert the given `ILibraryFileAlias` attributes were reset.
@@ -123,10 +121,10 @@
         The expiration date and the hits counter are reset and the
         last access records was on file creation.
         """
-        self.assertIs(reuploaded_file.expires, None)
-        self.assertEquals(
+        self.assertIsNone(reuploaded_file.expires)
+        self.assertEqual(
             reuploaded_file.last_accessed, reuploaded_file.date_created)
-        self.assertEquals(reuploaded_file.hits, 0)
+        self.assertEqual(reuploaded_file.hits, 0)
 
     def testReUploadFileToTheSameContext(self):
         # Re-uploading a librarian file to the same privacy/server
@@ -139,7 +137,7 @@
         transaction.commit()
 
         self.assertIsNot(old, new)
-        self.assertEquals(
+        self.assertEqual(
             old.restricted, new.restricted, 'New file still private.')
         self.assertSameContent(old, new)
         self.assertFileIsReset(new)
@@ -223,9 +221,8 @@
 
     def assertNewFiles(self, new_files, result):
         """Check new files created during update_files_privacy."""
-        self.assertEquals(
-            sorted([new_file.filename for new_file in new_files]),
-            result)
+        self.assertEqual(
+            sorted([new_file.filename for new_file in new_files]), result)
 
     def _checkSourceFilesPrivacy(self, pub_record, restricted,
                                  expected_n_files):
@@ -233,20 +230,20 @@
         n_files = 0
         source = pub_record.sourcepackagerelease
         for source_file in source.files:
-            self.assertEquals(
+            self.assertEqual(
                 source_file.libraryfile.restricted, restricted,
                 'Privacy mismatch on %s' % source_file.libraryfile.filename)
             n_files += 1
-        self.assertEquals(
+        self.assertEqual(
             source.upload_changesfile.restricted, restricted,
             'Privacy mismatch on %s' % source.upload_changesfile.filename)
         n_files += 1
         for diff in source.package_diffs:
-            self.assertEquals(
+            self.assertEqual(
                 diff.diff_content.restricted, restricted,
                 'Privacy mismatch on %s' % diff.diff_content.filename)
             n_files += 1
-        self.assertEquals(
+        self.assertEqual(
             n_files, expected_n_files,
             'Expected %d and got %d files' % (expected_n_files, n_files))
 
@@ -335,20 +332,20 @@
         n_files = 0
         binary = pub_record.binarypackagerelease
         for binary_file in binary.files:
-            self.assertEquals(
+            self.assertEqual(
                 binary_file.libraryfile.restricted, restricted,
                 'Privacy mismatch on %s' % binary_file.libraryfile.filename)
             n_files += 1
         build = binary.build
-        self.assertEquals(
+        self.assertEqual(
             build.upload_changesfile.restricted, restricted,
             'Privacy mismatch on %s' % build.upload_changesfile.filename)
         n_files += 1
-        self.assertEquals(
+        self.assertEqual(
             build.log.restricted, restricted,
             'Privacy mismatch on %s' % build.log.filename)
         n_files += 1
-        self.assertEquals(
+        self.assertEqual(
             n_files, expected_n_files,
             'Expected %d and got %d files' % (expected_n_files, n_files))
 
@@ -464,18 +461,17 @@
         """
         copy_checker = CopyChecker(
             self.archive, include_binaries=False, allow_delayed_copies=delayed)
-        self.assertIs(
-            None,
+        self.assertIsNone(
             copy_checker.checkCopy(
                 self.source, self.series, self.pocket,
                 check_permissions=False))
         checked_copies = list(copy_checker.getCheckedCopies())
-        self.assertEquals(1, len(checked_copies))
+        self.assertEqual(1, len(checked_copies))
         [checked_copy] = checked_copies
-        self.assertEquals(
+        self.assertEqual(
             BuildSetStatus.NEEDSBUILD,
             checked_copy.getStatusSummaryForBuilds()['status'])
-        self.assertEquals(delayed, checked_copy.delayed)
+        self.assertEqual(delayed, checked_copy.delayed)
 
     def assertCanCopyBinaries(self, delayed=False):
         """Source and binary copy is allowed.
@@ -493,18 +489,17 @@
         """
         copy_checker = CopyChecker(
             self.archive, include_binaries=True, allow_delayed_copies=delayed)
-        self.assertIs(
-            None,
+        self.assertIsNone(
             copy_checker.checkCopy(
                 self.source, self.series, self.pocket,
                 check_permissions=False))
         checked_copies = list(copy_checker.getCheckedCopies())
-        self.assertEquals(1, len(checked_copies))
+        self.assertEqual(1, len(checked_copies))
         [checked_copy] = checked_copies
         self.assertTrue(
             checked_copy.getStatusSummaryForBuilds()['status'] >=
             BuildSetStatus.FULLYBUILT_PENDING)
-        self.assertEquals(delayed, checked_copy.delayed)
+        self.assertEqual(delayed, checked_copy.delayed)
 
     def assertCannotCopySourceOnly(self, msg, person=None,
                                    check_permissions=False):
@@ -518,7 +513,7 @@
             copy_checker.checkCopy, self.source, self.series, self.pocket,
             person, check_permissions)
         checked_copies = list(copy_checker.getCheckedCopies())
-        self.assertEquals(0, len(checked_copies))
+        self.assertEqual(0, len(checked_copies))
 
     def assertCannotCopyBinaries(self, msg):
         """`CopyChecker.checkCopy()` including binaries raises CannotCopy.
@@ -531,7 +526,7 @@
             copy_checker.checkCopy, self.source, self.series, self.pocket,
             None, False)
         checked_copies = list(copy_checker.getCheckedCopies())
-        self.assertEquals(0, len(checked_copies))
+        self.assertEqual(0, len(checked_copies))
 
     def test_cannot_copy_binaries_from_building(self):
         [build] = self.source.createMissingBuilds()
@@ -609,13 +604,12 @@
         with StormStatementRecorder() as recorder:
             copy_checker = CopyChecker(self.archive, include_binaries=False)
             for source in sources:
-                self.assertIs(
-                    None,
+                self.assertIsNone(
                     copy_checker.checkCopy(
                         source, self.series, self.pocket, person=person,
                         check_permissions=check_permissions))
             checked_copies = list(copy_checker.getCheckedCopies())
-            self.assertEquals(nb_of_sources, len(checked_copies))
+            self.assertEqual(nb_of_sources, len(checked_copies))
         return recorder
 
     def test_queries_copy_check(self):
@@ -834,13 +828,11 @@
 
         # At this point copy is allowed with or without binaries.
         copy_checker = CopyChecker(archive, include_binaries=False)
-        self.assertIs(
-            None,
+        self.assertIsNone(
             copy_checker.checkCopy(
                 source, series, pocket, check_permissions=False))
         copy_checker = CopyChecker(archive, include_binaries=True)
-        self.assertIs(
-            None,
+        self.assertIsNone(
             copy_checker.checkCopy(
                 source, series, pocket, check_permissions=False))
 
@@ -852,8 +844,8 @@
 
         # Now source-only copies are allowed.
         copy_checker = CopyChecker(archive, include_binaries=False)
-        self.assertIs(
-            None, copy_checker.checkCopy(
+        self.assertIsNone(
+            copy_checker.checkCopy(
                 source, series, pocket, check_permissions=False))
 
         # Copies with binaries are denied.
@@ -990,8 +982,7 @@
 
         # The first source-only copy is allowed, thus stored in the
         # copy checker inventory.
-        self.assertIs(
-            None,
+        self.assertIsNone(
             copy_checker.checkCopy(
                 source, source.distroseries, source.pocket,
                 check_permissions=False))
@@ -1164,7 +1155,7 @@
         self.test_publisher.prepareBreezyAutotest()
 
     def assertCopied(self, copies, series, arch_tags):
-        self.assertEquals(
+        self.assertEqual(
             [u'foo 666 in %s' % series.name] +
             [u'foo-bin 666 in %s %s' % (series.name, arch_tag)
              for arch_tag in arch_tags],
@@ -1268,9 +1259,9 @@
 
     def assertComponentSectionAndPriority(self, component, source,
                                           destination):
-        self.assertEquals(component, destination.component)
-        self.assertEquals(source.section, destination.section)
-        self.assertEquals(source.priority, destination.priority)
+        self.assertEqual(component, destination.component)
+        self.assertEqual(source.section, destination.section)
+        self.assertEqual(source.priority, destination.priority)
 
     def test_new_publication_overrides(self):
         # When we copy publications, if the destination primary archive has
@@ -1297,7 +1288,7 @@
         [copied_source, copied_bin_i386, copied_bin_hppa] = self.doCopy(
             source, target_archive, nobby, source.pocket, True)
         universe = getUtility(IComponentSet)['universe']
-        self.assertEquals(universe, copied_source.component)
+        self.assertEqual(universe, copied_source.component)
         self.assertComponentSectionAndPriority(
             universe, bin_i386, copied_bin_i386)
         self.assertComponentSectionAndPriority(
@@ -1333,7 +1324,7 @@
 
         [copied_source, copied_bin_i386, copied_bin_hppa] = self.doCopy(
             source, target_archive, nobby, source.pocket, True)
-        self.assertEquals(copied_source.component, existing_source.component)
+        self.assertEqual(copied_source.component, existing_source.component)
         self.assertComponentSectionAndPriority(
             ebin_i386.component, ebin_i386, copied_bin_i386)
         self.assertComponentSectionAndPriority(
@@ -1365,7 +1356,7 @@
 
         [copied_source, copied_bin_i386, copied_bin_hppa] = self.doCopy(
             source, archive, nobby, PackagePublishingPocket.UPDATES, True)
-        self.assertEquals(copied_source.component, source.component)
+        self.assertEqual(copied_source.component, source.component)
         self.assertComponentSectionAndPriority(
             bin_i386.component, bin_i386, copied_bin_i386)
         self.assertComponentSectionAndPriority(
@@ -1387,7 +1378,7 @@
         [copied_source, copied_bin_i386, copied_bin_hppa] = self.doCopy(
             source, target_archive, nobby, source.pocket, True)
         main = getUtility(IComponentSet)['main']
-        self.assertEquals(main, copied_source.component)
+        self.assertEqual(main, copied_source.component)
         self.assertComponentSectionAndPriority(
             main, bin_i386, copied_bin_i386)
         self.assertComponentSectionAndPriority(
@@ -1444,9 +1435,8 @@
             person=target_archive.owner, check_permissions=False,
             send_email=True)
         [notification] = pop_notifications()
-        self.assertEquals(
-            get_ppa_reference(target_archive),
-            notification['X-Launchpad-PPA'])
+        self.assertEqual(
+            get_ppa_reference(target_archive), notification['X-Launchpad-PPA'])
         body = notification.get_payload()[0].get_payload()
         expected = dedent("""\
             Accepted:
@@ -1483,11 +1473,10 @@
             person=source.sourcepackagerelease.creator,
             check_permissions=False, send_email=True)
         [notification, announcement] = pop_notifications()
-        self.assertEquals(
-            'Foo Bar <[e-mail]>', notification['To'])
-        self.assertEquals('[e-mail]', announcement['To'])
+        self.assertEqual('Foo Bar <[e-mail]>', notification['To'])
+        self.assertEqual('[e-mail]', announcement['To'])
         for mail in (notification, announcement):
-            self.assertEquals(
+            self.assertEqual(
                 '[ubuntutest/nobby] foo 1.0-2 (Accepted)', mail['Subject'])
         expected_text = dedent("""\
             foo (1.0-2) unstable; urgency=3Dlow
@@ -1523,7 +1512,7 @@
                     check_permissions=False, send_email=True,
                     sponsored=sponsored_person)
         [notification, announcement] = pop_notifications()
-        self.assertEquals(
+        self.assertEqual(
             'Sponsored <[e-mail]>', announcement['From'])
         self.assertEqual(sponsored_person, copied_source.creator)
 
@@ -1602,11 +1591,9 @@
         notifications = pop_notifications()
         self.assertEqual(1, len(notifications))
         [notification] = notifications
-        self.assertEquals(
-            'Foo Bar <[e-mail]>', notification['To'])
-        self.assertEquals(
-            '[ubuntutest/nobby] foo 1.0-2 (Rejected)',
-            notification['Subject'])
+        self.assertEqual('Foo Bar <[e-mail]>', notification['To'])
+        self.assertEqual(
+            '[ubuntutest/nobby] foo 1.0-2 (Rejected)', notification['Subject'])
         expected_text = (
             "Rejected:\n"
             "foo 1.0-2 in breezy-autotest (a different source with the same "
@@ -1623,7 +1610,7 @@
             [source], target_archive, nobby, source.pocket, False,
             person=target_archive.owner, check_permissions=False,
             send_email=False)
-        self.assertEquals([], pop_notifications())
+        self.assertEqual([], pop_notifications())
 
     def test_copying_unsupported_arch_with_override(self):
         # When the copier is passed an unsupported arch with an override
@@ -1659,9 +1646,7 @@
             person=target_archive.owner, check_permissions=False,
             send_email=False)
 
-        self.assertEqual(
-            target_archive.owner,
-            copied_source.creator)
+        self.assertEqual(target_archive.owner, copied_source.creator)
 
     def test_unsponsored_copy_does_not_set_sponsor(self):
         # If the copy is not sponsored, SPPH.sponsor is none
@@ -1673,9 +1658,42 @@
             person=target_archive.owner, check_permissions=False,
             send_email=False)
 
+        self.assertIsNone(copied_source.sponsor)
+
+    def test_copy_custom_upload_files(self):
+        # Custom upload files are queued for republication when they are
+        # copied.
+        self.test_publisher.breezy_autotest.status = SeriesStatus.CURRENT
+        source = self.test_publisher.getPubSource(
+            pocket=PackagePublishingPocket.PROPOSED)
+        self.test_publisher.getPubBinaries(
+            pocket=PackagePublishingPocket.PROPOSED, pub_source=source)
+        [build] = source.getBuilds()
+        custom_file = self.factory.makeLibraryFileAlias()
+        build.package_upload.addCustom(
+            custom_file, PackageUploadCustomFormat.DIST_UPGRADER)
+        # Make the new librarian file available.
+        self.layer.txn.commit()
+
+        self.doCopy(
+            source, source.archive, source.distroseries,
+            PackagePublishingPocket.UPDATES, include_binaries=True)
+        [upload] = source.distroseries.getPackageUploads(
+            status=PackageUploadStatus.ACCEPTED, archive=source.archive,
+            pocket=PackagePublishingPocket.UPDATES,
+            custom_type=PackageUploadCustomFormat.DIST_UPGRADER)
+
+        # The upload is targeted to the right publishing context.
+        self.assertEqual(source.archive, upload.archive)
+        self.assertEqual(source.distroseries, upload.distroseries)
+        self.assertEqual(PackagePublishingPocket.UPDATES, upload.pocket)
+
+        # It contains only the custom files.
+        self.assertEqual([], list(upload.sources))
+        self.assertEqual([], list(upload.builds))
         self.assertEqual(
-            copied_source.sponsor,
-            None)
+            [custom_file],
+            [custom.libraryfilealias for custom in upload.customfiles])
 
 
 class TestDoDelayedCopy(TestCaseWithFactory, BaseDoCopyTests):
@@ -1696,16 +1714,13 @@
 
         # Make ubuntutest/breezy-autotest CURRENT so uploads to SECURITY
         # pocket can be accepted.
-        self.test_publisher.breezy_autotest.status = (
-            SeriesStatus.CURRENT)
+        self.test_publisher.breezy_autotest.status = SeriesStatus.CURRENT
 
     def assertCopied(self, copy, series, arch_tags):
-        self.assertEquals(
-            copy.sources[0].sourcepackagerelease.title,
-            'foo - 666')
-        self.assertEquals(
-            sorted(arch_tags),
-            sorted([pub.build.arch_tag for pub in copy.builds]))
+        self.assertEqual(
+            copy.sources[0].sourcepackagerelease.title, 'foo - 666')
+        self.assertContentEqual(
+            arch_tags, [pub.build.arch_tag for pub in copy.builds])
 
     def doCopy(self, source, archive, series, pocket, include_binaries):
         return _do_delayed_copy(source, archive, series, pocket, True)
@@ -1758,35 +1773,32 @@
 
         # A delayed-copy `IPackageUpload` record is returned.
         self.assertTrue(delayed_copy.is_delayed_copy)
-        self.assertEquals(
-            PackageUploadStatus.ACCEPTED, delayed_copy.status)
+        self.assertEqual(PackageUploadStatus.ACCEPTED, delayed_copy.status)
 
         # The returned object has a more descriptive 'displayname'
         # attribute than plain `IPackageUpload` instances.
-        self.assertEquals(
+        self.assertEqual(
             'Delayed copy of foocomm - '
             '1.0-2 (source, i386, raw-dist-upgrader)',
             delayed_copy.displayname)
 
         # It is targeted to the right publishing context.
-        self.assertEquals(self.copy_archive, delayed_copy.archive)
-        self.assertEquals(self.copy_series, delayed_copy.distroseries)
-        self.assertEquals(self.copy_pocket, delayed_copy.pocket)
+        self.assertEqual(self.copy_archive, delayed_copy.archive)
+        self.assertEqual(self.copy_series, delayed_copy.distroseries)
+        self.assertEqual(self.copy_pocket, delayed_copy.pocket)
 
         # And it contains the source, build and custom files.
-        self.assertEquals(
+        self.assertEqual(
             [source.sourcepackagerelease],
             [pus.sourcepackagerelease for pus in delayed_copy.sources])
 
         [build] = source.getBuilds()
-        self.assertEquals(
-            [build],
-            [pub.build for pub in delayed_copy.builds])
+        self.assertEqual([build], [pub.build for pub in delayed_copy.builds])
 
         [custom_file] = [
             custom.libraryfilealias
             for custom in build.package_upload.customfiles]
-        self.assertEquals(
+        self.assertEqual(
             [custom_file],
             [custom.libraryfilealias for custom in delayed_copy.customfiles])
 
@@ -1843,23 +1855,20 @@
         # Setup and execute the delayed copy procedure. This should
         # now result in an accepted delayed upload.
         delayed_copy = self.do_delayed_copy(source)
-        self.assertEquals(
-            PackageUploadStatus.ACCEPTED, delayed_copy.status)
+        self.assertEqual(PackageUploadStatus.ACCEPTED, delayed_copy.status)
 
         # And it contains the source, build and custom files.
-        self.assertEquals(
+        self.assertEqual(
             [source.sourcepackagerelease],
             [pus.sourcepackagerelease for pus in delayed_copy.sources])
 
         [build] = source.getBuilds()
-        self.assertEquals(
-            [build],
-            [pub.build for pub in delayed_copy.builds])
+        self.assertEqual([build], [pub.build for pub in delayed_copy.builds])
 
         [custom_file] = [
             custom.libraryfilealias
             for custom in build.package_upload.customfiles]
-        self.assertEquals(
+        self.assertEqual(
             [custom_file],
             [custom.libraryfilealias for custom in delayed_copy.customfiles])
 
@@ -1915,9 +1924,8 @@
         # archive context. Also after this point, the same delayed-copy
         # request will be denied by `CopyChecker`.
         [build_hppa, build_i386] = source.getBuilds()
-        self.assertEquals(
-            [build_i386],
-            [pub.build for pub in delayed_copy.builds])
+        self.assertEqual(
+            [build_i386], [pub.build for pub in delayed_copy.builds])
 
 
 class CopyPackageScriptTestCase(unittest.TestCase):
@@ -2068,8 +2076,7 @@
         self.assertEqual(len(copied), size)
 
         for candidate in copied:
-            self.assertEqual(
-                candidate.status, PackagePublishingStatus.PENDING)
+            self.assertEqual(PackagePublishingStatus.PENDING, candidate.status)
 
         def excludeOlds(found, old_pending_ids):
             return [pub.id for pub in found if pub.id not in old_pending_ids]
@@ -2399,16 +2406,14 @@
         active_warty_architectures = [
             arch.architecturetag for arch in warty.architectures
             if arch.getChroot()]
-        self.assertEqual(
-            active_warty_architectures, ['i386'])
+        self.assertEqual(active_warty_architectures, ['i386'])
 
         # Setup ubuntu/hoary supporting i386 and hppa architetures.
         hoary = ubuntu.getSeries('hoary')
         test_publisher.addFakeChroots(hoary)
         active_hoary_architectures = [
             arch.architecturetag for arch in hoary.architectures]
-        self.assertEqual(
-            sorted(active_hoary_architectures), ['hppa', 'i386'])
+        self.assertEqual(sorted(active_hoary_architectures), ['hppa', 'i386'])
 
         # We will create an architecture-specific source and its binaries
         # for i386 in ubuntu/warty. They will be copied over.
@@ -2500,8 +2505,7 @@
         # architecture (hppa).
         [new_build] = copied_source.getBuilds()
         self.assertEqual(
-            new_build.title,
-            'hppa build of boing 1.0 in ubuntu hoary RELEASE')
+            new_build.title, 'hppa build of boing 1.0 in ubuntu hoary RELEASE')
 
     def testVersionConflictInDifferentPockets(self):
         """Copy-package stops copies conflicting in different pocket.
@@ -2626,8 +2630,7 @@
 
         [copied_source, original_source] = sources
 
-        self.assertEqual(
-            copied_source.pocket, PackagePublishingPocket.UPDATES)
+        self.assertEqual(copied_source.pocket, PackagePublishingPocket.UPDATES)
         self.assertEqual(
             original_source.pocket, PackagePublishingPocket.SECURITY)
 
@@ -2673,8 +2676,7 @@
         copied = copy_helper.mainTask()
 
         [source_copy, i386_copy] = copied
-        self.assertEqual(
-            source_copy.displayname, 'lazy-building 1.0 in hoary')
+        self.assertEqual(source_copy.displayname, 'lazy-building 1.0 in hoary')
         self.assertEqual(i386_copy.displayname, 'lazy-bin 1.0 in hoary i386')
 
         target_archive = copy_helper.destination.archive
@@ -2735,12 +2737,8 @@
             from_suite='warty', to_suite='hoary', to_ppa='mark')
         copied = copy_helper.mainTask()
 
-        self.assertEqual(
-            str(copy_helper.location),
-            'cprov: warty-RELEASE')
-        self.assertEqual(
-            str(copy_helper.destination),
-            'mark: hoary-RELEASE')
+        self.assertEqual('cprov: warty-RELEASE', str(copy_helper.location))
+        self.assertEqual('mark: hoary-RELEASE', str(copy_helper.destination))
 
         target_archive = copy_helper.destination.archive
         self.checkCopies(copied, target_archive, 2)
@@ -3267,10 +3265,10 @@
             section='misc')
 
         checker = CopyChecker(warty.main_archive, include_binaries=False)
-        self.assertIs(
-            None,
-            checker.checkCopy(proposed_source, warty,
-            PackagePublishingPocket.UPDATES, check_permissions=False))
+        self.assertIsNone(
+            checker.checkCopy(
+                proposed_source, warty, PackagePublishingPocket.UPDATES,
+                check_permissions=False))
 
     def testCopySourceWithConflictingFilesInPPAs(self):
         """We can copy source if the source files match, both in name and
@@ -3355,10 +3353,10 @@
         self.layer.txn.commit()
 
         checker = CopyChecker(dest_ppa, include_binaries=False)
-        self.assertIs(
-            None,
-            checker.checkCopy(test2_source, warty,
-            PackagePublishingPocket.RELEASE, check_permissions=False))
+        self.assertIsNone(
+            checker.checkCopy(
+                test2_source, warty, PackagePublishingPocket.RELEASE,
+                check_permissions=False))
 
     def testCopySourceWithExpiredSourcesInDestination(self):
         """We can also copy sources if the destination archive has expired
@@ -3401,7 +3399,7 @@
         switch_dbuser(self.dbuser)
 
         checker = CopyChecker(dest_ppa, include_binaries=False)
-        self.assertIs(
-            None,
-            checker.checkCopy(test2_source, warty,
-            PackagePublishingPocket.RELEASE, check_permissions=False))
+        self.assertIsNone(
+            checker.checkCopy(
+                test2_source, warty, PackagePublishingPocket.RELEASE,
+                check_permissions=False))

=== modified file 'lib/lp/soyuz/scripts/tests/test_custom_uploads_copier.py'
--- lib/lp/soyuz/scripts/tests/test_custom_uploads_copier.py	2012-05-30 10:25:43 +0000
+++ lib/lp/soyuz/scripts/tests/test_custom_uploads_copier.py	2012-06-25 16:57:21 +0000
@@ -6,6 +6,7 @@
 __metaclass__ = type
 
 from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.registry.interfaces.series import SeriesStatus
 from lp.soyuz.enums import (
     ArchivePurpose,
     PackageUploadCustomFormat,
@@ -108,7 +109,7 @@
     # Alas, PackageUploadCustom relies on the Librarian.
     layer = LaunchpadZopelessLayer
 
-    def makeUpload(self, distroseries=None,
+    def makeUpload(self, distroseries=None, pocket=None,
                    custom_type=PackageUploadCustomFormat.DEBIAN_INSTALLER,
                    version=None, arch=None):
         """Create a `PackageUploadCustom`."""
@@ -121,7 +122,7 @@
             arch = self.factory.getUniqueString()
         filename = "%s.tar.gz" % '_'.join([package_name, version, arch])
         package_upload = self.factory.makeCustomPackageUpload(
-            distroseries=distroseries, custom_type=custom_type,
+            distroseries=distroseries, pocket=pocket, custom_type=custom_type,
             filename=filename)
         return package_upload.customfiles[0]
 
@@ -213,6 +214,19 @@
             upload.id for upload in copier.getCandidateUploads(source_series)]
         self.assertEqual(sorted(candidate_ids, reverse=True), candidate_ids)
 
+    def test_getCandidateUploads_filters_by_pocket(self):
+        # getCandidateUploads ignores uploads for other pockets.
+        source_series = self.factory.makeDistroSeries()
+        matching_upload = self.makeUpload(
+            source_series, pocket=PackagePublishingPocket.PROPOSED)
+        nonmatching_upload = self.makeUpload(
+            source_series, pocket=PackagePublishingPocket.BACKPORTS)
+        copier = CustomUploadsCopier(FakeDistroSeries())
+        candidate_uploads = copier.getCandidateUploads(
+            source_series, PackagePublishingPocket.PROPOSED)
+        self.assertContentEqual([matching_upload], candidate_uploads)
+        self.assertNotIn(nonmatching_upload, candidate_uploads)
+
     def test_getKey_includes_format_and_architecture(self):
         # The key returned by getKey consists of custom upload type,
         # and architecture.
@@ -221,7 +235,7 @@
         # component name as well.
         source_series = self.factory.makeDistroSeries()
         upload = self.makeUpload(
-            source_series, PackageUploadCustomFormat.DIST_UPGRADER,
+            source_series, custom_type=PackageUploadCustomFormat.DIST_UPGRADER,
             arch='mips')
         copier = CustomUploadsCopier(FakeDistroSeries())
         expected_key = (
@@ -379,8 +393,8 @@
         # copyUpload copies the original upload into the release pocket,
         # even though the original is more likely to be in another
         # pocket.
-        original_upload = self.makeUpload()
-        original_upload.packageupload.pocket = PackagePublishingPocket.UPDATES
+        original_upload = self.makeUpload(
+            pocket=PackagePublishingPocket.UPDATES)
         target_series = self.factory.makeDistroSeries()
         copier = CustomUploadsCopier(target_series)
         copied_upload = copier.copyUpload(original_upload)
@@ -388,6 +402,21 @@
             PackagePublishingPocket.RELEASE,
             copied_upload.packageupload.pocket)
 
+    def test_copyUpload_to_updates_pocket(self):
+        # copyUpload copies an upload between pockets in the same series if
+        # requested.
+        source_series = self.factory.makeDistroSeries(
+            status=SeriesStatus.CURRENT)
+        original_upload = self.makeUpload(
+            distroseries=source_series,
+            pocket=PackagePublishingPocket.PROPOSED)
+        copier = CustomUploadsCopier(
+            source_series, target_pocket=PackagePublishingPocket.UPDATES)
+        copied_upload = copier.copyUpload(original_upload)
+        self.assertEqual(
+            PackagePublishingPocket.UPDATES,
+            copied_upload.packageupload.pocket)
+
     def test_copyUpload_accepts_upload(self):
         # Uploads created by copyUpload are automatically accepted.
         original_upload = self.makeUpload()

=== modified file 'lib/lp/testing/factory.py'
--- lib/lp/testing/factory.py	2012-06-11 09:55:13 +0000
+++ lib/lp/testing/factory.py	2012-06-25 16:57:21 +0000
@@ -3493,28 +3493,32 @@
             sourcepackagename=sourcepackagename, component=component))
         return upload
 
-    def makeBuildPackageUpload(self, distroseries=None,
-                               binarypackagename=None):
+    def makeBuildPackageUpload(self, distroseries=None, pocket=None,
+                               binarypackagename=None,
+                               source_package_release=None):
         """Make a `PackageUpload` with a `PackageUploadBuild` attached."""
         if distroseries is None:
             distroseries = self.makeDistroSeries()
         upload = self.makePackageUpload(
-            distroseries=distroseries, archive=distroseries.main_archive)
-        build = self.makeBinaryPackageBuild()
+            distroseries=distroseries, archive=distroseries.main_archive,
+            pocket=pocket)
+        build = self.makeBinaryPackageBuild(
+            source_package_release=source_package_release, pocket=pocket)
         upload.addBuild(build)
         self.makeBinaryPackageRelease(
             binarypackagename=binarypackagename, build=build)
         return upload
 
-    def makeCustomPackageUpload(self, distroseries=None, custom_type=None,
-                                filename=None):
+    def makeCustomPackageUpload(self, distroseries=None, pocket=None,
+                                custom_type=None, filename=None):
         """Make a `PackageUpload` with a `PackageUploadCustom` attached."""
         if distroseries is None:
             distroseries = self.makeDistroSeries()
         if custom_type is None:
             custom_type = PackageUploadCustomFormat.DEBIAN_INSTALLER
         upload = self.makePackageUpload(
-            distroseries=distroseries, archive=distroseries.main_archive)
+            distroseries=distroseries, archive=distroseries.main_archive,
+            pocket=pocket)
         file_alias = self.makeLibraryFileAlias(filename=filename)
         upload.addCustom(file_alias, custom_type)
         return upload

=== modified file 'lib/lp/testing/tests/test_factory.py'
--- lib/lp/testing/tests/test_factory.py	2012-01-01 02:58:52 +0000
+++ lib/lp/testing/tests/test_factory.py	2012-06-25 16:57:21 +0000
@@ -1,4 +1,4 @@
-# Copyright 2010-2011 Canonical Ltd.  This software is licensed under the
+# Copyright 2010-2012 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Tests for the Launchpad object factory."""
@@ -781,10 +781,12 @@
         distroseries = self.factory.makeDistroSeries()
         bpn = self.factory.makeBinaryPackageName()
         pu = self.factory.makeBuildPackageUpload(
-            distroseries=distroseries, binarypackagename=bpn)
+            distroseries=distroseries, pocket=PackagePublishingPocket.PROPOSED,
+            binarypackagename=bpn)
         build = list(pu.builds)[0].build
         self.assertEqual(distroseries, pu.distroseries)
         self.assertEqual(distroseries.distribution, pu.archive.distribution)
+        self.assertEqual(PackagePublishingPocket.PROPOSED, pu.pocket)
         release = IStore(distroseries).find(
             BinaryPackageRelease, BinaryPackageRelease.build == build).one()
         self.assertEqual(bpn, release.binarypackagename)
@@ -803,11 +805,12 @@
         custom_type = PackageUploadCustomFormat.ROSETTA_TRANSLATIONS
         filename = self.factory.getUniqueString()
         pu = self.factory.makeCustomPackageUpload(
-            distroseries=distroseries, custom_type=custom_type,
-            filename=filename)
+            distroseries=distroseries, pocket=PackagePublishingPocket.PROPOSED,
+            custom_type=custom_type, filename=filename)
         custom = list(pu.customfiles)[0]
         self.assertEqual(distroseries, pu.distroseries)
         self.assertEqual(distroseries.distribution, pu.archive.distribution)
+        self.assertEqual(PackagePublishingPocket.PROPOSED, pu.pocket)
         self.assertEqual(custom_type, custom.customformat)
         self.assertEqual(filename, custom.libraryfilealias.filename)