Merge lp:~jtv/launchpad/db-bug-793382 into lp:launchpad/db-devel

[Jeroen T. Vermeulen]

[r=henninge][bug=793382] Bring requestUpgrades back to constant query count.

= Summary =

Now that we no longer bundle multiple packages into one PackageCopyJob, the DistroSeries:+localpackagediffs view method requestUpgrades issued extra queries in proportion to the number of copies requested. Creating a Job cost one extra query per package; the associated PackageCopyJob cost another. This request needs to scale, so we'd like the query count to remain constant.

== Proposed fix ==

Provide interfaces for creating multiple Jobs and PlainPackageCopyJobs with single INSERT queries. That makes it difficult to get the objects in memory, but we don't need them: all we need (and that only for Job, really) is the ids that they have been assigned.

Actually it should be possible for Storm to bundle the INSERTs in this way, but that would take quite some work to get right, and it might become fragile if one INSERT ends up being flushed to the database before the next object can be added to the store.

In the IPlainPackageCopyJobSource interface you provide two different kinds of data when creating multiple jobs at once: some parameters are specified once and repeated for all jobs in the batch, others need to be specified for each job separately. The latter are represented as a list of tuples.

== Pre-implementation notes ==

Gavin came up with the idea of having a "create multiple jobs" interface in a utility class. No other interesting ideas came up when discussing the idea on IRC.

== Implementation details ==

The Job multi-create method went into the class. There is an IJobSource interface, but that was not suitable here for several reasons. First, it's implemented in many places that may not have any need for a multi-create; requiring this method would cause unneeded hardship. Second, IJobSource is not a utility for Jobs. Instead it's more the generic base interface for classes built around Job, which may or may not live in separate tables — so it would be ambiguous what kind of ids the method would return. Finally, the job-specific creation methods will have their own signatures, which wouldn't work very well with having them all come from the same interface definition.

You'll notice in the performance test that even though the query count no longer increases by 2 per copy, the query count for the base case went up by 2. I haven't bothered to figure out where those 2 queries came from; I wonder if perhaps it might have been an accounting glitch where the two INSERTs got deferred beyond the point where the query count is collected. That deferral would not be possible with the new code, and that could explain the difference.

== Tests ==

From low-level to high-level:
{{{
./bin/test -vvc lp.services.job.tests.test_job -t createMultiple
./bin/test -vvc lp.soyuz.tests.test_packagecopyjob -t createMultiple
./bin/test -vvc lp.registry.browser.tests.test_distroseries -t requestUpgrades_is_efficient
}}}

== Demo and Q/A ==

I'm not sure the "upgrade packages" button is visible on DistroSeries:+localpackagediffs yet. But if it is, pressing it should make upgradable entries on that page (and any other batches for the same distroseries) show up as "synchronizing..."

Upgradeable entries are ones where the checkbox on the left is active, and the parent version is greater than the version in the distroseries you're looking at.

= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/soyuz/model/packagecopyjob.py
  lib/lp/soyuz/interfaces/packagecopyjob.py
  lib/lp/soyuz/tests/test_packagecopyjob.py
  lib/lp/registry/browser/tests/test_distroseries.py
  lib/lp/services/job/tests/test_job.py
  lib/lp/services/job/model/job.py
  lib/lp/registry/browser/distroseries.py

[Henning Eggers (henninge)]

I am sorry but I cannot find anything wrong with this proposal. :-P

review: Approve

=== modified file 'lib/canonical/launchpad/doc/vocabularies.txt'
--- lib/canonical/launchpad/doc/vocabularies.txt	2011-05-27 19:53:20 +0000
+++ lib/canonical/launchpad/doc/vocabularies.txt	2011-06-09 10:59:00 +0000
@@ -300,8 +300,8 @@
     >>> package_name_terms.count()
     2
     >>> [(term.token, term.title) for term in package_name_terms]
-    [('mozilla-firefox', u'iceweasel huh ?'),
-     ('mozilla-firefox-data', u'Mozilla Firefox Data is .....')]
+    [('mozilla-firefox', u'mozilla-firefox'),
+     ('mozilla-firefox-data', u'mozilla-firefox-data')]
 
 Searching for "mozilla" should return the binary package name above, and
 the source package named "mozilla".
@@ -310,9 +310,9 @@
     >>> package_name_terms.count()
     3
     >>> [(term.token, term.title) for term in package_name_terms]
-    [('mozilla', 'Not uploaded'),
-     ('mozilla-firefox', u'iceweasel huh ?'),
-     ('mozilla-firefox-data', u'Mozilla Firefox Data is .....')]
+    [('mozilla', u'mozilla'),
+     ('mozilla-firefox', u'mozilla-firefox'),
+     ('mozilla-firefox-data', u'mozilla-firefox-data')]
 
 The search does a case-insensitive, substring match.
 
@@ -320,8 +320,8 @@
     >>> package_name_terms.count()
     2
     >>> [(term.token, term.title) for term in package_name_terms]
-    [('linux-2.6.12', u'this kernel is like the crystal method: a temple...'),
-     ('linux-source-2.6.15', u'Source of: linux-2.6.12')]
+    [('linux-2.6.12', u'linux-2.6.12'),
+     ('linux-source-2.6.15', u'linux-source-2.6.15')]
 
 
 BinaryPackageNameVocabulary

=== modified file 'lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt'
--- lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt	2011-02-25 04:19:04 +0000
+++ lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt	2011-06-09 10:59:00 +0000
@@ -1,4 +1,5 @@
-= Operational Statistics and Metrics =
+Operational Statistics and Metrics
+==================================
 
 We make Zope 3 give us real time statistics about Launchpad's operation.
 We can access them via XML-RPC:
@@ -25,7 +26,8 @@
     ...     reset()
     ...
 
-== Number of requests and XML-RPC requests ==
+Number of requests and XML-RPC requests
+---------------------------------------
 
 Even though XML-RPC requests are technically HTTP requests, we do not
 count them as such. Note that the call to obtain statistics will increment
@@ -59,7 +61,8 @@
     requests: 1
     xml-rpc requests: 1
 
-== Number of HTTP requests and success codes ==
+Number of HTTP requests and success codes
+-----------------------------------------
 
     >>> output = http("GET / HTTP/1.1\nHost: bugs.launchpad.dev\n")
     >>> output.getStatus()
@@ -69,7 +72,8 @@
     http requests: 1
     requests: 1
 
-== Number of 404s ==
+Number of 404s
+--------------
 
 Note that retries is incremented too. As per the standard Launchpad
 database policy, this request first uses the slave DB. The requested
@@ -86,7 +90,8 @@
     requests: 1
     retries: 1
 
-== Number of 500 Internal Server Errors (unhandled exceptions) ==
+Number of 500 Internal Server Errors (unhandled exceptions)
+-----------------------------------------------------------
 
 This is normally the number of OOPS pages displayed to the user, but
 may also include the odd case where the OOPS system has failed and a
@@ -129,7 +134,8 @@
     http requests: 1
     requests: 1
 
-== Number of XML-RPC Faults ==
+Number of XML-RPC Faults
+------------------------
 
     >>> try:
     ...     opstats = lp_xmlrpc.invalid() # XXX: Need a HTTP test too
@@ -142,7 +148,8 @@
     xml-rpc requests: 1
 
 
-== Number of soft timeouts ==
+Number of soft timeouts
+-----------------------
 
     >>> from canonical.config import config
     >>> test_data = dedent("""
@@ -162,7 +169,8 @@
     requests: 1
     soft timeouts: 1
 
-== Number of Timeouts ==
+Number of Timeouts
+------------------
 
 We can't reliably track this using the 503 response code as other
 Launchpad code may well return this status and an XML-RPC request may
@@ -190,7 +198,8 @@
     timeouts: 1
 
 
-== HTTP access for Cricket ==
+HTTP access for Cricket
+-----------------------
 
 Stats can also be retrieved via HTTP in cricket-graph format:
 
@@ -219,13 +228,13 @@
     xmlrpc_requests:0@...
     <BLANKLINE>
 
-== No DB access required ==
+No DB access required
+---------------------
 
-Accessing the opstats page as an anonymous user will make no database
-queries. This is important to make it as reliable as possible since we
-use this page for monitoring. Because of this property, the load
-balancers also use this page to determine if a Launchpad instance is
-responsive.
+Accessing the opstats page will make no database queries. This is important to
+make it as reliable as possible since we use this page for monitoring. Because
+of this property, the load balancers also use this page to determine if a
+Launchpad instance is responsive.
 
 To confirm this, we first point all our database connection information
 to somewhere that doesn't exist.
@@ -257,6 +266,20 @@
     <BLANKLINE>
     1XXs:0@...
 
+This is also true if we are provide authentication.
+
+    >>> print http(r"""
+    ... GET /+opstats HTTP/1.1
+    ... Host: launchpad.dev
+    ... Authorization: Basic Zm9vLmJhckBjYW5vbmljYWwuY29tOnRlc3Q=
+    ... """)
+    HTTP/1.1 200 Ok
+    ...
+    Content-Type: text/plain; charset=US-ASCII
+    ...
+    <BLANKLINE>
+    1XXs:0@...
+
 But our database connections are broken.
 
     >>> from canonical.launchpad.interfaces.lpstorm import IStore
@@ -271,4 +294,3 @@
 
     >>> IStore(Person).find(Person, name='janitor').one().name
     u'janitor'
-

=== modified file 'lib/canonical/launchpad/webapp/publication.py'
--- lib/canonical/launchpad/webapp/publication.py	2011-05-27 21:12:25 +0000
+++ lib/canonical/launchpad/webapp/publication.py	2011-06-09 10:59:00 +0000
@@ -330,7 +330,17 @@
         personless account, return the unauthenticated principal.
         """
         auth_utility = getUtility(IPlacelessAuthUtility)
-        principal = auth_utility.authenticate(request)
+        principal = None
+        # +opstats and +haproxy are status URLs that must not query the DB at
+        # all.  This is enforced (see
+        # lib/canonical/launchpad/webapp/dbpolicy.py). If the request is for
+        # one of those two pages, don't even try to authenticate, because we
+        # may fail.  We haven't traversed yet, so we have to sniff the request
+        # this way.  Even though PATH_INFO is always present in real requests,
+        # we need to tread carefully (``get``) because of test requests in our
+        # automated tests.
+        if request.get('PATH_INFO') not in [u'/+opstats', u'/+haproxy']:
+            principal = auth_utility.authenticate(request)
         if principal is None or principal.person is None:
             # This is either an unauthenticated user or a user who
             # authenticated on our OpenID server using a personless account.

=== modified file 'lib/canonical/launchpad/webapp/tests/test_haproxy.py'
--- lib/canonical/launchpad/webapp/tests/test_haproxy.py	2011-02-18 18:12:43 +0000
+++ lib/canonical/launchpad/webapp/tests/test_haproxy.py	2011-06-09 10:59:00 +0000
@@ -35,6 +35,15 @@
         result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
         self.assertEquals(200, result.getStatus())
 
+    def test_authenticated_HAProxyStatusView_works(self):
+        # We don't use authenticated requests, but this keeps us from
+        # generating oopses.
+        result = self.http(
+            u'GET /+haproxy HTTP/1.0\n'
+            u'Authorization: Basic Zm9vLmJhckBjYW5vbmljYWwuY29tOnRlc3Q=\n',
+            handle_errors=False)
+        self.assertEquals(200, result.getStatus())
+
     def test_HAProxyStatusView_going_down_returns_500(self):
         haproxy.set_going_down_flag(True)
         result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
@@ -61,4 +70,3 @@
         haproxy.set_going_down_flag(True)
         result = self.http(u'GET /+haproxy HTTP/1.0', handle_errors=False)
         self.assertEquals(499, result.getStatus())
-

=== modified file 'lib/lp/registry/browser/distroseries.py'
--- lib/lp/registry/browser/distroseries.py	2011-06-08 21:03:29 +0000
+++ lib/lp/registry/browser/distroseries.py	2011-06-09 10:59:00 +0000
@@ -1052,6 +1052,7 @@
     def sync_sources(self, action, data):
         self._sync_sources(action, data)
 
+<<<<<<< TREE
     def getUpgrades(self):
         """Find straightforward package upgrades.
 
@@ -1103,6 +1104,61 @@
             queue = PackageUploadQueue(self.context, None)
             return check_permission("launchpad.Edit", queue)
 
+=======
+    def getUpgrades(self):
+        """Find straightforward package upgrades.
+
+        These are updates for packages that this distroseries shares
+        with a parent series, for which there have been updates in the
+        parent, and which do not have any changes in this series that
+        might complicate a sync.
+
+        :return: A result set of `DistroSeriesDifference`s.
+        """
+        return getUtility(IDistroSeriesDifferenceSource).getSimpleUpgrades(
+            self.context)
+
+    @action(_("Upgrade Packages"), name="upgrade", condition='canUpgrade')
+    def upgrade(self, action, data):
+        """Request synchronization of straightforward package upgrades."""
+        self.requestUpgrades()
+
+    def requestUpgrades(self):
+        """Request sync of packages that can be easily upgraded."""
+        target_distroseries = self.context
+        copies = [
+            (
+                dsd.source_package_name.name,
+                dsd.parent_source_version,
+                dsd.parent_series.main_archive,
+                target_distroseries.main_archive,
+                PackagePublishingPocket.RELEASE,
+            )
+            for dsd in self.getUpgrades()]
+        getUtility(IPlainPackageCopyJobSource).createMultiple(
+            target_distroseries, copies,
+            copy_policy=PackageCopyPolicy.MASS_SYNC)
+
+        self.request.response.addInfoNotification(
+            (u"Upgrades of {context.displayname} packages have been "
+             u"requested. Please give Launchpad some time to complete "
+             u"these.").format(context=self.context))
+
+    def canUpgrade(self, action=None):
+        """Should the form offer a packages upgrade?"""
+        if getFeatureFlag("soyuz.derived_series_sync.enabled") is None:
+            return False
+        elif self.context.status not in UPGRADABLE_SERIES_STATUSES:
+            # A feature freeze precludes blanket updates.
+            return False
+        elif self.getUpgrades().is_empty():
+            # There are no simple updates to perform.
+            return False
+        else:
+            queue = PackageUploadQueue(self.context, None)
+            return check_permission("launchpad.Edit", queue)
+
+>>>>>>> MERGE-SOURCE
 
 class DistroSeriesMissingPackagesView(DistroSeriesDifferenceBaseView,
                                       LaunchpadFormView):

=== modified file 'lib/lp/registry/browser/tests/test_distroseries.py'
--- lib/lp/registry/browser/tests/test_distroseries.py	2011-06-08 21:03:29 +0000
+++ lib/lp/registry/browser/tests/test_distroseries.py	2011-06-09 10:59:00 +0000
@@ -996,6 +996,7 @@
         self.assertEqual(versions['derived'], derived_span[0].string.strip())
         self.assertEqual(versions['parent'], parent_span[0].string.strip())
 
+<<<<<<< TREE
     def test_getUpgrades_shows_updates_in_parent(self):
         # The view's getUpgrades methods lists packages that can be
         # trivially upgraded: changed in the parent, not changed in the
@@ -1128,6 +1129,139 @@
 
 class TestDistroSeriesLocalDifferencesFunctional(TestCaseWithFactory,
                                                  DistroSeriesDifferenceMixin):
+=======
+    def test_getUpgrades_shows_updates_in_parent(self):
+        # The view's getUpgrades methods lists packages that can be
+        # trivially upgraded: changed in the parent, not changed in the
+        # derived series, but present in both.
+        dsd = self.makePackageUpgrade()
+        view = self.makeView(dsd.derived_series)
+        self.assertContentEqual([dsd], view.getUpgrades())
+
+    def enableDerivedSeriesSyncFeature(self):
+        self.useFixture(
+            FeatureFixture(
+                {u'soyuz.derived_series_sync.enabled': u'on'}))
+
+    @with_celebrity_logged_in("admin")
+    def test_upgrades_offered_only_with_feature_flag(self):
+        # The "Upgrade Packages" button will only be shown when a specific
+        # feature flag is enabled.
+        view = self.makeView()
+        self.makePackageUpgrade(view.context)
+        self.assertFalse(view.canUpgrade())
+        self.enableDerivedSeriesSyncFeature()
+        self.assertTrue(view.canUpgrade())
+
+    def test_upgrades_are_offered_if_appropriate(self):
+        # The "Upgrade Packages" button will only be shown to privileged
+        # users.
+        self.enableDerivedSeriesSyncFeature()
+        dsd = self.makePackageUpgrade()
+        view = self.makeView(dsd.derived_series)
+        with celebrity_logged_in("admin"):
+            self.assertTrue(view.canUpgrade())
+        with person_logged_in(self.factory.makePerson()):
+            self.assertFalse(view.canUpgrade())
+        with anonymous_logged_in():
+            self.assertFalse(view.canUpgrade())
+
+    @with_celebrity_logged_in("admin")
+    def test_upgrades_offered_only_if_available(self):
+        # If there are no upgrades, the "Upgrade Packages" button won't
+        # be shown.
+        self.enableDerivedSeriesSyncFeature()
+        view = self.makeView()
+        self.assertFalse(view.canUpgrade())
+        self.makePackageUpgrade(view.context)
+        self.assertTrue(view.canUpgrade())
+
+    @with_celebrity_logged_in("admin")
+    def test_upgrades_not_offered_after_feature_freeze(self):
+        # There won't be an "Upgrade Packages" button once feature
+        # freeze has occurred.  Mass updates would not make sense after
+        # that point.
+        self.enableDerivedSeriesSyncFeature()
+        upgradeable = {}
+        for status in SeriesStatus.items:
+            dsd = self.makePackageUpgrade()
+            dsd.derived_series.status = status
+            view = self.makeView(dsd.derived_series)
+            upgradeable[status] = view.canUpgrade()
+        expected = {
+            SeriesStatus.FUTURE: True,
+            SeriesStatus.EXPERIMENTAL: True,
+            SeriesStatus.DEVELOPMENT: True,
+            SeriesStatus.FROZEN: False,
+            SeriesStatus.CURRENT: False,
+            SeriesStatus.SUPPORTED: False,
+            SeriesStatus.OBSOLETE: False,
+        }
+        self.assertEqual(expected, upgradeable)
+
+    def test_upgrade_creates_sync_jobs(self):
+        # requestUpgrades generates PackageCopyJobs for the upgrades
+        # that need doing.
+        dsd = self.makePackageUpgrade()
+        series = dsd.derived_series
+        with celebrity_logged_in('admin'):
+            series.status = SeriesStatus.DEVELOPMENT
+            series.datereleased = UTC_NOW
+        view = self.makeView(series)
+        view.requestUpgrades()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        jobs = list(
+            job_source.getActiveJobs(series.distribution.main_archive))
+        self.assertEquals(1, len(jobs))
+        job = jobs[0]
+        self.assertEquals(series, job.target_distroseries)
+        self.assertEqual(dsd.source_package_name.name, job.package_name)
+        self.assertEqual(dsd.parent_source_version, job.package_version)
+        self.assertEqual(PackagePublishingPocket.RELEASE, job.target_pocket)
+
+    def test_upgrade_gives_feedback(self):
+        # requestUpgrades doesn't instantly perform package upgrades,
+        # but it shows the user a notice that the upgrades have been
+        # requested.
+        dsd = self.makePackageUpgrade()
+        view = self.makeView(dsd.derived_series)
+        view.requestUpgrades()
+        expected = {
+            "level": BrowserNotificationLevel.INFO,
+            "message":
+                ("Upgrades of {0.displayname} packages have been "
+                 "requested. Please give Launchpad some time to "
+                 "complete these.").format(dsd.derived_series),
+            }
+        observed = map(vars, view.request.response.notifications)
+        self.assertEqual([expected], observed)
+
+    def test_requestUpgrades_is_efficient(self):
+        # A single web request may need to schedule large numbers of
+        # package upgrades.  It must do so without issuing large numbers
+        # of database queries.
+        derived_series, parent_series = self._createChildAndParent()
+        # Take a baseline measure of queries.
+        self.makePackageUpgrade(derived_series=derived_series)
+        flush_database_caches()
+        with StormStatementRecorder() as recorder1:
+            self.makeView(derived_series).requestUpgrades()
+        self.assertThat(recorder1, HasQueryCount(LessThan(12)))
+
+        # The query count does not increase with the number of upgrades.
+        for index in xrange(3):
+            self.makePackageUpgrade(derived_series=derived_series)
+        flush_database_caches()
+        with StormStatementRecorder() as recorder2:
+            self.makeView(derived_series).requestUpgrades()
+        self.assertThat(
+            recorder2,
+            HasQueryCount(Equals(recorder1.count)))
+
+
+class TestDistroSeriesLocalDifferencesFunctional(TestCaseWithFactory,
+                                                 DistroSeriesDifferenceMixin):
+>>>>>>> MERGE-SOURCE
 
     layer = LaunchpadFunctionalLayer
 

=== modified file 'lib/lp/registry/stories/distroseries/xx-distroseries-index.txt'
--- lib/lp/registry/stories/distroseries/xx-distroseries-index.txt	2011-05-24 10:08:33 +0000
+++ lib/lp/registry/stories/distroseries/xx-distroseries-index.txt	2011-06-09 10:59:00 +0000
@@ -51,13 +51,56 @@
     Release manager: None
     Status: Current Stable Release
     Derives from: Warty (4.10) is not derived from another series.
+<<<<<<< TREE
     Derived series:
+=======
+    Derived series: No derived series.
+>>>>>>> MERGE-SOURCE
     Source packages: 3
     Binary packages: 4
 
 On series that have no source or binary packages, the portlet will
 change its text slightly to annouce this:
 
+    >>> anon_browser.open('http://launchpad.dev/debian/sarge')
+    >>> print extract_text(
+    ...     find_portlet(anon_browser.contents, 'Series information'))
+    Series information
+    Distribution: Debian
+    Series: Sarge (3.1)
+    Project drivers: Jeff Waugh, Mark Shuttleworth
+    Release manager: Jeff Waugh
+    Status: Pre-release Freeze
+    Derives from: Woody (3.0)
+    Source packages: No sources imported or published.
+    Binary packages: No binaries imported or published.
+
+The series' derivation parents - rather than the previous series - are
+shown when derivation is enabled, as are the series derived from this
+series:
+
+    >>> from lp.registry.interfaces.distribution import IDistributionSet
+    >>> from lp.testing import celebrity_logged_in
+    >>> from zope.component import getUtility
+
+    >>> with celebrity_logged_in("admin"):
+    ...     debian = getUtility(IDistributionSet).getByName(u"debian")
+    ...     sarge = debian.getSeries(u"sarge")
+    ...     parents = [
+    ...         factory.makeDistroSeries(name=u"dobby"),
+    ...         factory.makeDistroSeries(name=u"knobby")]
+    ...     distro_series_parents = [
+    ...         factory.makeDistroSeriesParent(
+    ...             derived_series=sarge, parent_series=parent)
+    ...         for parent in parents]
+    ...     children = [
+    ...         factory.makeDistroSeries(name=u"bobby"),
+    ...         factory.makeDistroSeries(name=u"tables")]
+    ...     distro_series_children = [
+    ...         factory.makeDistroSeriesParent(
+    ...             derived_series=child, parent_series=sarge)
+    ...         for child in children]
+
     >>> with derivation_enabled:
     ...     anon_browser.open('http://launchpad.dev/debian/sarge')
     >>> print extract_text(
@@ -68,8 +111,13 @@
     Project drivers: Jeff Waugh, Mark Shuttleworth
     Release manager: Jeff Waugh
     Status: Pre-release Freeze
+<<<<<<< TREE
     Derives from: Woody (3.0)
     Derived series:
+=======
+    Derives from: Dobby (...), Knobby (...)
+    Derived series: Bobby (...), Tables (...)
+>>>>>>> MERGE-SOURCE
     Source packages: No sources imported or published.
     Binary packages: No binaries imported or published.
 

=== modified file 'lib/lp/registry/templates/distroseries-details.pt'
--- lib/lp/registry/templates/distroseries-details.pt	2011-05-24 10:08:33 +0000
+++ lib/lp/registry/templates/distroseries-details.pt	2011-06-09 10:59:00 +0000
@@ -46,6 +46,9 @@
       </dd>
     </dl>
 
+    <tal:derivation-not-enabled
+        tal:condition="not:request/features/soyuz.derived_series_ui.enabled">
+
     <dl>
       <dt>Derives from:</dt>
       <dd>
@@ -60,8 +63,32 @@
       </dd>
     </dl>
 
+<<<<<<< TREE
     <dl tal:condition="request/features/soyuz.derived_series_ui.enabled"
         tal:define="all_child_series context/getDerivedSeries">
+=======
+    </tal:derivation-not-enabled>
+
+    <tal:derivation-enabled
+        tal:condition="request/features/soyuz.derived_series_ui.enabled">
+
+    <dl tal:define="parents context/getParentSeries">
+      <dt>Derives from:</dt>
+      <dd tal:condition="parents">
+        <tal:parents repeat="parent parents">
+          <a tal:attributes="href parent/fmt:url"
+             tal:content="parent/named_version">
+          </a><tal:comma condition="not:repeat/parent/end">, </tal:comma>
+        </tal:parents>
+      </dd>
+      <dd tal:condition="not:parents">
+        <tal:name replace="context/named_version"/> is not derived from
+        another series.
+      </dd>
+    </dl>
+
+    <dl tal:define="all_child_series context/getDerivedSeries">
+>>>>>>> MERGE-SOURCE
       <dt>Derived series:</dt>
       <dd>
         <tal:per_child_series repeat="child_series all_child_series">
@@ -70,12 +97,18 @@
             tal:content="child_series/named_version" /><tal:comma
             condition="not: repeat/child_series/end">,</tal:comma>
         </tal:per_child_series>
+<<<<<<< TREE
        <tal:none condition="all_child_series">
+=======
+        <tal:none condition="not:all_child_series">
+>>>>>>> MERGE-SOURCE
           No derived series.
         </tal:none>
       </dd>
     </dl>
 
+    </tal:derivation-enabled>
+
     <dl tal:define="sourcecount context/sourcecount">
       <dt>Source packages:</dt>
       <dd

=== modified file 'lib/lp/services/job/model/job.py'
--- lib/lp/services/job/model/job.py	2011-05-26 14:29:34 +0000
+++ lib/lp/services/job/model/job.py	2011-06-09 10:59:00 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """ORM object representing jobs."""
@@ -26,7 +26,10 @@
 from canonical.database.constants import UTC_NOW
 from canonical.database.datetimecol import UtcDateTimeCol
 from canonical.database.enumcol import EnumCol
-from canonical.database.sqlbase import SQLBase
+from canonical.database.sqlbase import (
+    quote,
+    SQLBase,
+    )
 from lp.services.job.interfaces.job import (
     IJob,
     JobStatus,
@@ -99,6 +102,22 @@
 
     status = property(lambda x: x._status)
 
+    @classmethod
+    def createMultiple(self, store, num_jobs):
+        """Create multiple `Job`s at once.
+
+        :param store: `Store` to ceate the jobs in.
+        :param num_jobs: Number of `Job`s to create.
+        :return: An iterable of `Job.id` values for the new jobs.
+        """
+        job_contents = ["(%s)" % quote(JobStatus.WAITING)] * num_jobs
+        result = store.execute("""
+            INSERT INTO Job (status)
+            VALUES %s
+            RETURNING id
+            """ % ", ".join(job_contents))
+        return [job_id for job_id, in result]
+
     def acquireLease(self, duration=300):
         """See `IJob`."""
         if (self.lease_expires is not None

=== modified file 'lib/lp/services/job/tests/test_job.py'
--- lib/lp/services/job/tests/test_job.py	2011-05-27 08:18:07 +0000
+++ lib/lp/services/job/tests/test_job.py	2011-06-09 10:59:00 +0000
@@ -8,14 +8,9 @@
 
 import pytz
 from storm.locals import Store
-from zope.component import getUtility
 
 from canonical.database.constants import UTC_NOW
-from canonical.launchpad.webapp.interfaces import (
-    DEFAULT_FLAVOR,
-    IStoreSelector,
-    MAIN_STORE,
-    )
+from canonical.launchpad.interfaces.lpstorm import IStore
 from canonical.launchpad.webapp.testing import verifyObject
 from canonical.testing.layers import ZopelessDatabaseLayer
 from lp.services.job.interfaces.job import (
@@ -44,6 +39,22 @@
         job = Job()
         self.assertEqual(job.status, JobStatus.WAITING)
 
+    def test_createMultiple_creates_requested_number_of_jobs(self):
+        job_ids = list(Job.createMultiple(IStore(Job), 3))
+        self.assertEqual(3, len(job_ids))
+        self.assertEqual(3, len(set(job_ids)))
+
+    def test_createMultiple_returns_valid_job_ids(self):
+        job_ids = list(Job.createMultiple(IStore(Job), 3))
+        store = IStore(Job)
+        for job_id in job_ids:
+            self.assertIsNot(None, store.get(Job, job_id))
+
+    def test_createMultiple_sets_status_to_WAITING(self):
+        store = IStore(Job)
+        job = store.get(Job, Job.createMultiple(store, 1)[0])
+        self.assertEqual(JobStatus.WAITING, job.status)
+
     def test_start(self):
         """Job.start should update the object appropriately.
 
@@ -214,8 +225,7 @@
     layer = ZopelessDatabaseLayer
 
     def _sampleData(self):
-        store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
-        return list(store.execute(Job.ready_jobs))
+        return list(IStore(Job).execute(Job.ready_jobs))
 
     def test_ready_jobs(self):
         """Job.ready_jobs should include new jobs."""

=== modified file 'lib/lp/soyuz/interfaces/packagecopyjob.py'
--- lib/lp/soyuz/interfaces/packagecopyjob.py	2011-06-03 08:53:14 +0000
+++ lib/lp/soyuz/interfaces/packagecopyjob.py	2011-06-09 10:59:00 +0000
@@ -1,3 +1,4 @@
+<<<<<<< TREE
 # Copyright 2010-2011 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
@@ -148,3 +149,170 @@
     copy_policy = Choice(
         title=_("Applicable copy policy"),
         values=PackageCopyPolicy, required=True, readonly=True)
+=======
+# Copyright 2010-2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+__all__ = [
+    "IPackageCopyJob",
+    "IPlainPackageCopyJob",
+    "IPlainPackageCopyJobSource",
+    "PackageCopyJobType",
+    ]
+
+from lazr.enum import (
+    DBEnumeratedType,
+    DBItem,
+    )
+from lazr.restful.fields import Reference
+from zope.interface import (
+    Attribute,
+    Interface,
+    )
+from zope.schema import (
+    Bool,
+    Choice,
+    Int,
+    TextLine,
+    )
+
+from canonical.launchpad import _
+from lp.registry.interfaces.distroseries import IDistroSeries
+from lp.services.job.interfaces.job import (
+    IJob,
+    IJobSource,
+    IRunnableJob,
+    )
+from lp.soyuz.enums import PackageCopyPolicy
+from lp.soyuz.interfaces.archive import IArchive
+
+
+class IPackageCopyJob(Interface):
+    """A job that copies packages between `IArchive`s."""
+
+    id = Int(
+        title=_('DB ID'), required=True, readonly=True,
+        description=_("The tracking number for this job."))
+
+    source_archive_id = Int(
+        title=_('Source Archive ID'),
+        required=True, readonly=True)
+
+    source_archive = Reference(
+        schema=IArchive, title=_('Source Archive'),
+        required=True, readonly=True)
+
+    target_archive_id = Int(
+        title=_('Target Archive ID'),
+        required=True, readonly=True)
+
+    target_archive = Reference(
+        schema=IArchive, title=_('Target Archive'),
+        required=True, readonly=True)
+
+    target_distroseries = Reference(
+        schema=IDistroSeries, title=_('Target DistroSeries.'),
+        required=True, readonly=True)
+
+    package_name = TextLine(
+        title=_("Package name"), required=True, readonly=True)
+
+    job = Reference(
+        schema=IJob, title=_('The common Job attributes'),
+        required=True, readonly=True)
+
+    metadata = Attribute('A dict of data about the job.')
+
+
+class PackageCopyJobType(DBEnumeratedType):
+
+    PLAIN = DBItem(1, """
+        Copy packages between archives.
+
+        This job copies one or more packages, optionally including binaries.
+        """)
+
+
+class IPlainPackageCopyJobSource(IJobSource):
+    """An interface for acquiring `IPackageCopyJobs`."""
+
+    def create(package_name, source_archive,
+               target_archive, target_distroseries, target_pocket,
+               include_binaries=False, package_version=None,
+               copy_policy=PackageCopyPolicy.INSECURE):
+        """Create a new `IPlainPackageCopyJob`.
+
+        :param package_name: The name of the source package to copy.
+        :param source_archive: The `IArchive` in which `source_packages` are
+            found.
+        :param target_archive: The `IArchive` to which to copy the packages.
+        :param target_distroseries: The `IDistroSeries` to which to copy the
+            packages.
+        :param target_pocket: The pocket into which to copy the packages. Must
+            be a member of `PackagePublishingPocket`.
+        :param include_binaries: See `do_copy`.
+        :param package_version: The version string for the package version
+            that is to be copied.
+        :param copy_policy: Applicable `PackageCopyPolicy`.
+        """
+
+    def createMultiple(target_distroseries, copy_tasks,
+                       copy_policy=PackageCopyPolicy.INSECURE,
+                       include_binaries=False):
+        """Create multiple new `IPlainPackageCopyJob`s at once.
+
+        :param target_distroseries: The `IDistroSeries` to which to copy the
+            packages.
+        :param copy_tasks: A list of tuples describing the copies to be
+            performed: (package name, package version, source archive,
+            target archive, target pocket).
+        :param copy_policy: Applicable `PackageCopyPolicy`.
+        :param include_binaries: As in `do_copy`.
+        :return: An iterable of `PackageCopyJob` ids.
+        """
+
+    def getActiveJobs(target_archive):
+        """Retrieve all active sync jobs for an archive."""
+
+    def getPendingJobsPerPackage(target_series):
+        """Find pending jobs for each package in `target_series`.
+
+        This is meant for finding jobs that will resolve specific
+        `DistroSeriesDifference`s.
+
+        :param target_series: Target `DistroSeries`; this corresponds to
+            `DistroSeriesDifference.derived_series`.
+        :return: A dict containing as keys the (name, version) tuples for
+            each `DistroSeriesDifference` that has a resolving
+            `PlainPackageCopyJob` pending.  Each of these DSDs maps to its
+            oldest pending job.  The `version` corresponds to
+            `DistroSeriesDifference.parent_source_version`.
+        """
+
+
+class IPlainPackageCopyJob(IRunnableJob):
+    """A no-frills job to copy packages between `IArchive`s."""
+
+    target_pocket = Int(
+        title=_("Target package publishing pocket"), required=True,
+        readonly=True)
+
+    package_version = TextLine(
+        title=_("Package version"), required=True, readonly=True)
+
+    include_binaries = Bool(
+        title=_("Copy binaries"),
+        required=False, readonly=True)
+
+    def addSourceOverride(override):
+        """Add an `ISourceOverride` to the metadata."""
+
+    def getSourceOverride():
+        """Get an `ISourceOverride` from the metadata."""
+
+    copy_policy = Choice(
+        title=_("Applicable copy policy"),
+        values=PackageCopyPolicy, required=True, readonly=True)
+>>>>>>> MERGE-SOURCE

=== modified file 'lib/lp/soyuz/model/binaryandsourcepackagename.py'
--- lib/lp/soyuz/model/binaryandsourcepackagename.py	2010-12-22 02:48:42 +0000
+++ lib/lp/soyuz/model/binaryandsourcepackagename.py	2011-06-09 10:59:00 +0000
@@ -18,9 +18,7 @@
     BatchedCountableIterator,
     NamedSQLObjectHugeVocabulary,
     )
-from lp.registry.model.sourcepackagename import getSourcePackageDescriptions
 from lp.soyuz.interfaces.binarypackagename import IBinaryAndSourcePackageName
-from lp.soyuz.model.binarypackagename import getBinaryPackageDescriptions
 
 
 class BinaryAndSourcePackageName(SQLBase):
@@ -45,15 +43,7 @@
     """
 
     def getTermsWithDescriptions(self, results):
-        # Note that we grab first source package descriptions and then
-        # binary package descriptions, giving preference to the latter,
-        # via the update() call.
-        descriptions = getSourcePackageDescriptions(results, use_names=True)
-        binary_descriptions = getBinaryPackageDescriptions(results,
-                                                           use_names=True)
-        descriptions.update(binary_descriptions)
-        return [SimpleTerm(obj, obj.name,
-                    descriptions.get(obj.name, "Not uploaded"))
+        return [SimpleTerm(obj, obj.name, obj.name)
                 for obj in results]
 
 

=== modified file 'lib/lp/soyuz/model/packagecopyjob.py'
--- lib/lp/soyuz/model/packagecopyjob.py	2011-06-03 09:18:34 +0000
+++ lib/lp/soyuz/model/packagecopyjob.py	2011-06-09 10:59:00 +0000
@@ -23,10 +23,18 @@
     implements,
     )
 
-from canonical.database.enumcol import EnumCol
-from canonical.launchpad.components.decoratedresultset import (
-    DecoratedResultSet,
-    )
+<<<<<<< TREE
+from canonical.database.enumcol import EnumCol
+from canonical.launchpad.components.decoratedresultset import (
+    DecoratedResultSet,
+    )
+=======
+from canonical.database.enumcol import EnumCol
+from canonical.database.sqlbase import sqlvalues
+from canonical.launchpad.components.decoratedresultset import (
+    DecoratedResultSet,
+    )
+>>>>>>> MERGE-SOURCE
 from canonical.launchpad.interfaces.lpstorm import (
     IMasterStore,
     IStore,
@@ -67,125 +75,256 @@
 
     implements(IPackageCopyJob)
 
-    __storm_table__ = 'PackageCopyJob'
-
-    id = Int(primary=True)
-
-    job_id = Int(name='job')
-    job = Reference(job_id, Job.id)
-
-    source_archive_id = Int(name='source_archive')
-    source_archive = Reference(source_archive_id, Archive.id)
-
-    target_archive_id = Int(name='target_archive')
-    target_archive = Reference(target_archive_id, Archive.id)
-
-    target_distroseries_id = Int(name='target_distroseries')
-    target_distroseries = Reference(target_distroseries_id, DistroSeries.id)
-
-    package_name = Unicode('package_name')
-    copy_policy = EnumCol(enum=PackageCopyPolicy)
-
-    job_type = EnumCol(enum=PackageCopyJobType, notNull=True)
-
-    _json_data = Unicode('json_data')
-
-    def __init__(self, source_archive, target_archive, target_distroseries,
-                 job_type, metadata, package_name=None, copy_policy=None):
-        super(PackageCopyJob, self).__init__()
-        self.job = Job()
-        self.job_type = job_type
-        self.source_archive = source_archive
-        self.target_archive = target_archive
-        self.target_distroseries = target_distroseries
-        self.package_name = unicode(package_name)
-        self.copy_policy = copy_policy
-        self._json_data = self.serializeMetadata(metadata)
-
-    @classmethod
-    def serializeMetadata(cls, metadata_dict):
-        """Serialize a dict of metadata into a unicode string."""
-        return simplejson.dumps(metadata_dict).decode('utf-8')
-
-    @property
-    def metadata(self):
-        return simplejson.loads(self._json_data)
-
-    def extendMetadata(self, metadata_dict):
-        """Add metadata_dict to the existing metadata."""
-        existing = self.metadata
-        existing.update(metadata_dict)
-        self._json_data = self.serializeMetadata(existing)
-
-
-class PackageCopyJobDerived(BaseRunnableJob):
-    """Abstract class for deriving from PackageCopyJob."""
-
-    delegates(IPackageCopyJob)
-
-    def __init__(self, job):
-        self.context = job
-
-    @classmethod
-    def get(cls, job_id):
-        """Get a job by id.
-
-        :return: the PackageCopyJob with the specified id, as the current
-            PackageCopyJobDerived subclass.
-        :raises: NotFoundError if there is no job with the specified id, or
-            its job_type does not match the desired subclass.
-        """
-        job = IStore(PackageCopyJob).get(PackageCopyJob, job_id)
-        if job.job_type != cls.class_job_type:
-            raise NotFoundError(
-                'No object found with id %d and type %s' % (job_id,
-                cls.class_job_type.title))
-        return cls(job)
-
-    @classmethod
-    def iterReady(cls):
-        """Iterate through all ready PackageCopyJobs."""
-        jobs = IStore(PackageCopyJob).find(
-            PackageCopyJob,
-            And(PackageCopyJob.job_type == cls.class_job_type,
-                PackageCopyJob.job == Job.id,
-                Job.id.is_in(Job.ready_jobs)))
-        return (cls(job) for job in jobs)
-
-    def getOopsVars(self):
-        """See `IRunnableJob`."""
-        vars = super(PackageCopyJobDerived, self).getOopsVars()
-        vars.extend([
-            ('source_archive_id', self.context.source_archive_id),
-            ('target_archive_id', self.context.target_archive_id),
-            ('target_distroseries_id', self.context.target_distroseries_id),
-            ('package_copy_job_id', self.context.id),
-            ('package_copy_job_type', self.context.job_type.title),
-            ])
-        return vars
-
-    @property
-    def copy_policy(self):
-        """See `PlainPackageCopyJob`."""
-        return self.context.copy_policy
-
-
-class PlainPackageCopyJob(PackageCopyJobDerived):
-    """Job that copies a package from one archive to another."""
-    # This job type serves in different places: it supports copying
-    # packages between archives, but also the syncing of packages from
-    # parents into a derived distroseries.  We may split these into
-    # separate types at some point, but for now we (allenap, bigjools,
-    # jtv) chose to keep it as one.
-
-    implements(IPlainPackageCopyJob)
-
-    class_job_type = PackageCopyJobType.PLAIN
-    classProvides(IPlainPackageCopyJobSource)
-
-    @classmethod
-    def create(cls, package_name, source_archive,
+<<<<<<< TREE
+    __storm_table__ = 'PackageCopyJob'
+
+    id = Int(primary=True)
+
+    job_id = Int(name='job')
+    job = Reference(job_id, Job.id)
+
+    source_archive_id = Int(name='source_archive')
+    source_archive = Reference(source_archive_id, Archive.id)
+
+    target_archive_id = Int(name='target_archive')
+    target_archive = Reference(target_archive_id, Archive.id)
+
+    target_distroseries_id = Int(name='target_distroseries')
+    target_distroseries = Reference(target_distroseries_id, DistroSeries.id)
+
+    package_name = Unicode('package_name')
+    copy_policy = EnumCol(enum=PackageCopyPolicy)
+
+    job_type = EnumCol(enum=PackageCopyJobType, notNull=True)
+
+    _json_data = Unicode('json_data')
+
+    def __init__(self, source_archive, target_archive, target_distroseries,
+                 job_type, metadata, package_name=None, copy_policy=None):
+        super(PackageCopyJob, self).__init__()
+        self.job = Job()
+        self.job_type = job_type
+        self.source_archive = source_archive
+        self.target_archive = target_archive
+        self.target_distroseries = target_distroseries
+        self.package_name = unicode(package_name)
+        self.copy_policy = copy_policy
+        self._json_data = self.serializeMetadata(metadata)
+
+    @classmethod
+    def serializeMetadata(cls, metadata_dict):
+        """Serialize a dict of metadata into a unicode string."""
+        return simplejson.dumps(metadata_dict).decode('utf-8')
+
+    @property
+    def metadata(self):
+        return simplejson.loads(self._json_data)
+
+    def extendMetadata(self, metadata_dict):
+        """Add metadata_dict to the existing metadata."""
+        existing = self.metadata
+        existing.update(metadata_dict)
+        self._json_data = self.serializeMetadata(existing)
+
+
+class PackageCopyJobDerived(BaseRunnableJob):
+    """Abstract class for deriving from PackageCopyJob."""
+
+    delegates(IPackageCopyJob)
+
+    def __init__(self, job):
+        self.context = job
+
+    @classmethod
+    def get(cls, job_id):
+        """Get a job by id.
+
+        :return: the PackageCopyJob with the specified id, as the current
+            PackageCopyJobDerived subclass.
+        :raises: NotFoundError if there is no job with the specified id, or
+            its job_type does not match the desired subclass.
+        """
+        job = IStore(PackageCopyJob).get(PackageCopyJob, job_id)
+        if job.job_type != cls.class_job_type:
+            raise NotFoundError(
+                'No object found with id %d and type %s' % (job_id,
+                cls.class_job_type.title))
+        return cls(job)
+
+    @classmethod
+    def iterReady(cls):
+        """Iterate through all ready PackageCopyJobs."""
+        jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            And(PackageCopyJob.job_type == cls.class_job_type,
+                PackageCopyJob.job == Job.id,
+                Job.id.is_in(Job.ready_jobs)))
+        return (cls(job) for job in jobs)
+
+    def getOopsVars(self):
+        """See `IRunnableJob`."""
+        vars = super(PackageCopyJobDerived, self).getOopsVars()
+        vars.extend([
+            ('source_archive_id', self.context.source_archive_id),
+            ('target_archive_id', self.context.target_archive_id),
+            ('target_distroseries_id', self.context.target_distroseries_id),
+            ('package_copy_job_id', self.context.id),
+            ('package_copy_job_type', self.context.job_type.title),
+            ])
+        return vars
+
+    @property
+    def copy_policy(self):
+        """See `PlainPackageCopyJob`."""
+        return self.context.copy_policy
+
+
+class PlainPackageCopyJob(PackageCopyJobDerived):
+    """Job that copies a package from one archive to another."""
+    # This job type serves in different places: it supports copying
+    # packages between archives, but also the syncing of packages from
+    # parents into a derived distroseries.  We may split these into
+    # separate types at some point, but for now we (allenap, bigjools,
+    # jtv) chose to keep it as one.
+
+    implements(IPlainPackageCopyJob)
+
+    class_job_type = PackageCopyJobType.PLAIN
+    classProvides(IPlainPackageCopyJobSource)
+
+    @classmethod
+    def create(cls, package_name, source_archive,
+=======
+    __storm_table__ = 'PackageCopyJob'
+
+    id = Int(primary=True)
+
+    job_id = Int(name='job')
+    job = Reference(job_id, Job.id)
+
+    source_archive_id = Int(name='source_archive')
+    source_archive = Reference(source_archive_id, Archive.id)
+
+    target_archive_id = Int(name='target_archive')
+    target_archive = Reference(target_archive_id, Archive.id)
+
+    target_distroseries_id = Int(name='target_distroseries')
+    target_distroseries = Reference(target_distroseries_id, DistroSeries.id)
+
+    package_name = Unicode('package_name')
+    copy_policy = EnumCol(enum=PackageCopyPolicy)
+
+    job_type = EnumCol(enum=PackageCopyJobType, notNull=True)
+
+    _json_data = Unicode('json_data')
+
+    def __init__(self, source_archive, target_archive, target_distroseries,
+                 job_type, metadata, package_name=None, copy_policy=None):
+        super(PackageCopyJob, self).__init__()
+        self.job = Job()
+        self.job_type = job_type
+        self.source_archive = source_archive
+        self.target_archive = target_archive
+        self.target_distroseries = target_distroseries
+        self.package_name = unicode(package_name)
+        self.copy_policy = copy_policy
+        self._json_data = self.serializeMetadata(metadata)
+
+    @classmethod
+    def serializeMetadata(cls, metadata_dict):
+        """Serialize a dict of metadata into a unicode string."""
+        return simplejson.dumps(metadata_dict).decode('utf-8')
+
+    @property
+    def metadata(self):
+        return simplejson.loads(self._json_data)
+
+    def extendMetadata(self, metadata_dict):
+        """Add metadata_dict to the existing metadata."""
+        existing = self.metadata
+        existing.update(metadata_dict)
+        self._json_data = self.serializeMetadata(existing)
+
+
+class PackageCopyJobDerived(BaseRunnableJob):
+    """Abstract class for deriving from PackageCopyJob."""
+
+    delegates(IPackageCopyJob)
+
+    def __init__(self, job):
+        self.context = job
+
+    @classmethod
+    def get(cls, job_id):
+        """Get a job by id.
+
+        :return: the PackageCopyJob with the specified id, as the current
+            PackageCopyJobDerived subclass.
+        :raises: NotFoundError if there is no job with the specified id, or
+            its job_type does not match the desired subclass.
+        """
+        job = IStore(PackageCopyJob).get(PackageCopyJob, job_id)
+        if job.job_type != cls.class_job_type:
+            raise NotFoundError(
+                'No object found with id %d and type %s' % (job_id,
+                cls.class_job_type.title))
+        return cls(job)
+
+    @classmethod
+    def iterReady(cls):
+        """Iterate through all ready PackageCopyJobs."""
+        jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            And(PackageCopyJob.job_type == cls.class_job_type,
+                PackageCopyJob.job == Job.id,
+                Job.id.is_in(Job.ready_jobs)))
+        return (cls(job) for job in jobs)
+
+    def getOopsVars(self):
+        """See `IRunnableJob`."""
+        vars = super(PackageCopyJobDerived, self).getOopsVars()
+        vars.extend([
+            ('source_archive_id', self.context.source_archive_id),
+            ('target_archive_id', self.context.target_archive_id),
+            ('target_distroseries_id', self.context.target_distroseries_id),
+            ('package_copy_job_id', self.context.id),
+            ('package_copy_job_type', self.context.job_type.title),
+            ])
+        return vars
+
+    @property
+    def copy_policy(self):
+        """See `PlainPackageCopyJob`."""
+        return self.context.copy_policy
+
+
+class PlainPackageCopyJob(PackageCopyJobDerived):
+    """Job that copies a package from one archive to another."""
+    # This job type serves in different places: it supports copying
+    # packages between archives, but also the syncing of packages from
+    # parents into a derived distroseries.  We may split these into
+    # separate types at some point, but for now we (allenap, bigjools,
+    # jtv) chose to keep it as one.
+
+    implements(IPlainPackageCopyJob)
+
+    class_job_type = PackageCopyJobType.PLAIN
+    classProvides(IPlainPackageCopyJobSource)
+
+    @classmethod
+    def _makeMetadata(cls, target_pocket, package_version, include_binaries):
+        """."""
+        return {
+            'target_pocket': target_pocket.value,
+            'package_version': package_version,
+            'include_binaries': bool(include_binaries),
+        }
+
+    @classmethod
+    def create(cls, package_name, source_archive,
+>>>>>>> MERGE-SOURCE
                target_archive, target_distroseries, target_pocket,
+<<<<<<< TREE
                include_binaries=False, package_version=None,
                copy_policy=PackageCopyPolicy.INSECURE):
         """See `IPlainPackageCopyJobSource`."""
@@ -204,43 +343,149 @@
             copy_policy=copy_policy,
             metadata=metadata)
         IMasterStore(PackageCopyJob).add(job)
+=======
+               include_binaries=False, package_version=None,
+               copy_policy=PackageCopyPolicy.INSECURE):
+        """See `IPlainPackageCopyJobSource`."""
+        assert package_version is not None, "No package version specified."
+        metadata = cls._makeMetadata(
+            target_pocket, package_version, include_binaries)
+        job = PackageCopyJob(
+            job_type=cls.class_job_type,
+            source_archive=source_archive,
+            target_archive=target_archive,
+            target_distroseries=target_distroseries,
+            package_name=package_name,
+            copy_policy=copy_policy,
+            metadata=metadata)
+        IMasterStore(PackageCopyJob).add(job)
+>>>>>>> MERGE-SOURCE
         return cls(job)
 
     @classmethod
-    def getActiveJobs(cls, target_archive):
-        """See `IPlainPackageCopyJobSource`."""
-        jobs = IStore(PackageCopyJob).find(
-            PackageCopyJob,
-            PackageCopyJob.job_type == cls.class_job_type,
-            PackageCopyJob.target_archive == target_archive,
-            Job.id == PackageCopyJob.job_id,
-            Job._status == JobStatus.WAITING)
-        jobs = jobs.order_by(PackageCopyJob.id)
-        return DecoratedResultSet(jobs, cls)
-
-    @classmethod
-    def getPendingJobsForTargetSeries(cls, target_series):
-        """Get upcoming jobs for `target_series`, ordered by age."""
-        raw_jobs = IStore(PackageCopyJob).find(
-            PackageCopyJob,
-            Job.id == PackageCopyJob.job_id,
-            PackageCopyJob.job_type == cls.class_job_type,
-            PackageCopyJob.target_distroseries == target_series,
-            Job._status.is_in(Job.PENDING_STATUSES))
-        raw_jobs = raw_jobs.order_by(PackageCopyJob.id)
-        return DecoratedResultSet(raw_jobs, cls)
-
-    @classmethod
-    def getPendingJobsPerPackage(cls, target_series):
-        """See `IPlainPackageCopyJobSource`."""
-        result = {}
-        # Go through jobs in-order, picking the first matching job for
-        # any (package, version) tuple.  Because of how
-        # getPendingJobsForTargetSeries orders its results, the first
-        # will be the oldest and thus presumably the first to finish.
-        for job in cls.getPendingJobsForTargetSeries(target_series):
-            result.setdefault(job.package_name, job)
-        return result
+<<<<<<< TREE
+    def getActiveJobs(cls, target_archive):
+        """See `IPlainPackageCopyJobSource`."""
+        jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            PackageCopyJob.job_type == cls.class_job_type,
+            PackageCopyJob.target_archive == target_archive,
+            Job.id == PackageCopyJob.job_id,
+            Job._status == JobStatus.WAITING)
+        jobs = jobs.order_by(PackageCopyJob.id)
+        return DecoratedResultSet(jobs, cls)
+
+    @classmethod
+    def getPendingJobsForTargetSeries(cls, target_series):
+        """Get upcoming jobs for `target_series`, ordered by age."""
+        raw_jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            Job.id == PackageCopyJob.job_id,
+            PackageCopyJob.job_type == cls.class_job_type,
+            PackageCopyJob.target_distroseries == target_series,
+            Job._status.is_in(Job.PENDING_STATUSES))
+        raw_jobs = raw_jobs.order_by(PackageCopyJob.id)
+        return DecoratedResultSet(raw_jobs, cls)
+
+    @classmethod
+    def getPendingJobsPerPackage(cls, target_series):
+        """See `IPlainPackageCopyJobSource`."""
+        result = {}
+        # Go through jobs in-order, picking the first matching job for
+        # any (package, version) tuple.  Because of how
+        # getPendingJobsForTargetSeries orders its results, the first
+        # will be the oldest and thus presumably the first to finish.
+        for job in cls.getPendingJobsForTargetSeries(target_series):
+            result.setdefault(job.package_name, job)
+        return result
+=======
+    def _composeJobInsertionTuple(cls, target_distroseries, copy_policy,
+                                  include_binaries, job_id, copy_task):
+        """Create an SQL fragment for inserting a job into the database.
+
+        :return: A string representing an SQL tuple containing initializers
+            for a `PackageCopyJob` in the database (minus `id`, which is
+            assigned automatically).  Contents are escaped for use in SQL.
+        """
+        (
+            package_name,
+            package_version,
+            source_archive,
+            target_archive,
+            target_pocket,
+        ) = copy_task
+        metadata = cls._makeMetadata(
+            target_pocket, package_version, include_binaries)
+        data = (
+            cls.class_job_type, target_distroseries, copy_policy,
+            source_archive, target_archive, package_name, job_id,
+            PackageCopyJob.serializeMetadata(metadata))
+        format_string = "(%s)" % ", ".join(["%s"] * len(data))
+        return format_string % sqlvalues(*data)
+
+    @classmethod
+    def createMultiple(cls, target_distroseries, copy_tasks,
+                       copy_policy=PackageCopyPolicy.INSECURE,
+                       include_binaries=False):
+        """See `IPlainPackageCopyJobSource`."""
+        store = IMasterStore(Job)
+        job_ids = Job.createMultiple(store, len(copy_tasks))
+        job_contents = [
+            cls._composeJobInsertionTuple(
+                target_distroseries, copy_policy, include_binaries, job_id,
+                task)
+            for job_id, task in zip(job_ids, copy_tasks)]
+        result = store.execute("""
+            INSERT INTO PackageCopyJob (
+                job_type,
+                target_distroseries,
+                copy_policy,
+                source_archive,
+                target_archive,
+                package_name,
+                job,
+                json_data)
+            VALUES %s
+            RETURNING id
+            """ % ", ".join(job_contents))
+        return [job_id for job_id, in result]
+
+    @classmethod
+    def getActiveJobs(cls, target_archive):
+        """See `IPlainPackageCopyJobSource`."""
+        jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            PackageCopyJob.job_type == cls.class_job_type,
+            PackageCopyJob.target_archive == target_archive,
+            Job.id == PackageCopyJob.job_id,
+            Job._status == JobStatus.WAITING)
+        jobs = jobs.order_by(PackageCopyJob.id)
+        return DecoratedResultSet(jobs, cls)
+
+    @classmethod
+    def getPendingJobsForTargetSeries(cls, target_series):
+        """Get upcoming jobs for `target_series`, ordered by age."""
+        raw_jobs = IStore(PackageCopyJob).find(
+            PackageCopyJob,
+            Job.id == PackageCopyJob.job_id,
+            PackageCopyJob.job_type == cls.class_job_type,
+            PackageCopyJob.target_distroseries == target_series,
+            Job._status.is_in(Job.PENDING_STATUSES))
+        raw_jobs = raw_jobs.order_by(PackageCopyJob.id)
+        return DecoratedResultSet(raw_jobs, cls)
+
+    @classmethod
+    def getPendingJobsPerPackage(cls, target_series):
+        """See `IPlainPackageCopyJobSource`."""
+        result = {}
+        # Go through jobs in-order, picking the first matching job for
+        # any (package, version) tuple.  Because of how
+        # getPendingJobsForTargetSeries orders its results, the first
+        # will be the oldest and thus presumably the first to finish.
+        for job in cls.getPendingJobsForTargetSeries(target_series):
+            result.setdefault(job.package_name, job)
+        return result
+>>>>>>> MERGE-SOURCE
 
     @property
     def target_pocket(self):

=== modified file 'lib/lp/soyuz/tests/test_packagecopyjob.py'
--- lib/lp/soyuz/tests/test_packagecopyjob.py	2011-06-03 08:53:14 +0000
+++ lib/lp/soyuz/tests/test_packagecopyjob.py	2011-06-09 10:59:00 +0000
@@ -116,7 +116,55 @@
         self.assertEqual("foo", job.package_name)
         self.assertEqual("1.0-1", job.package_version)
         self.assertEquals(False, job.include_binaries)
-        self.assertEquals(PackageCopyPolicy.MASS_SYNC, job.copy_policy)
+<<<<<<< TREE
+        self.assertEquals(PackageCopyPolicy.MASS_SYNC, job.copy_policy)
+=======
+        self.assertEquals(PackageCopyPolicy.MASS_SYNC, job.copy_policy)
+
+    def test_createMultiple_creates_one_job_per_copy(self):
+        mother = self.factory.makeDistroSeriesParent()
+        derived_series = mother.derived_series
+        father = self.factory.makeDistroSeriesParent(
+            derived_series=derived_series)
+        mother_package = self.factory.makeSourcePackageName()
+        father_package = self.factory.makeSourcePackageName()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        copy_tasks = [
+            (
+                mother_package.name,
+                "1.5mother1",
+                mother.parent_series.main_archive,
+                derived_series.main_archive,
+                PackagePublishingPocket.RELEASE,
+                ),
+            (
+                father_package.name,
+                "0.9father1",
+                father.parent_series.main_archive,
+                derived_series.main_archive,
+                PackagePublishingPocket.UPDATES,
+                ),
+            ]
+        job_ids = list(
+            job_source.createMultiple(mother.derived_series, copy_tasks))
+        jobs = list(job_source.getActiveJobs(derived_series.main_archive))
+        self.assertContentEqual(job_ids, [job.id for job in jobs])
+        self.assertEqual(len(copy_tasks), len(set([job.job for job in jobs])))
+        # Get jobs into the same order as copy_tasks, for ease of
+        # comparison.
+        if jobs[0].package_name != mother_package.name:
+            jobs = reversed(jobs)
+        requested_copies = [
+            (
+                job.package_name,
+                job.package_version,
+                job.source_archive,
+                job.target_archive,
+                job.target_pocket,
+                )
+            for job in jobs]
+        self.assertEqual(copy_tasks, requested_copies)
+>>>>>>> MERGE-SOURCE
 
     def test_getActiveJobs(self):
         # getActiveJobs() can retrieve all active jobs for an archive.
@@ -320,202 +368,404 @@
         copied_source_package = archive2.getPublishedSources(
             name="libc", version="2.8-1", exact_match=True).first()
         self.assertIsNot(copied_source_package, None)
-
-    def test___repr__(self):
-        distroseries = self.factory.makeDistroSeries()
-        archive1 = self.factory.makeArchive(distroseries.distribution)
-        archive2 = self.factory.makeArchive(distroseries.distribution)
-        source = getUtility(IPlainPackageCopyJobSource)
-        job = source.create(
-            package_name="foo", source_archive=archive1,
-            target_archive=archive2, target_distroseries=distroseries,
-            target_pocket=PackagePublishingPocket.RELEASE,
-            package_version="1.0-1", include_binaries=True)
-        self.assertEqual(
-            ("<PlainPackageCopyJob to copy package foo from "
-             "{distroseries.distribution.name}/{archive1.name} to "
-             "{distroseries.distribution.name}/{archive2.name}, "
-             "RELEASE pocket, in {distroseries.distribution.name} "
-             "{distroseries.name}, including binaries>").format(
-                distroseries=distroseries, archive1=archive1,
-                archive2=archive2),
-            repr(job))
-
-    def test_getPendingJobsPerPackage_finds_jobs(self):
-        # getPendingJobsPerPackage finds jobs, and the packages they
-        # belong to.
-        dsd = self.factory.makeDistroSeriesDifference()
-        job = self.makeJob(dsd)
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        self.assertEqual(
-            {dsd.source_package_name.name: job},
-            job_source.getPendingJobsPerPackage(dsd.derived_series))
-
-    def test_getPendingJobsPerPackage_ignores_other_distroseries(self):
-        # getPendingJobsPerPackage only looks for jobs on the indicated
-        # distroseries.
-        self.makeJob()
-        other_series = self.factory.makeDistroSeries()
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        self.assertEqual(
-            {}, job_source.getPendingJobsPerPackage(other_series))
-
-    def test_getPendingJobsPerPackage_only_returns_pending_jobs(self):
-        # getPendingJobsPerPackage ignores jobs that have already been
-        # run.
-        dsd = self.factory.makeDistroSeriesDifference()
-        job = self.makeJob(dsd)
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        found_by_state = {}
-        for status in JobStatus.items:
-            removeSecurityProxy(job).job._status = status
-            result = job_source.getPendingJobsPerPackage(dsd.derived_series)
-            if len(result) > 0:
-                found_by_state[status] = result[dsd.source_package_name.name]
-        expected = {
-            JobStatus.WAITING: job,
-            JobStatus.RUNNING: job,
-            JobStatus.SUSPENDED: job,
-        }
-        self.assertEqual(expected, found_by_state)
-
-    def test_getPendingJobsPerPackage_distinguishes_jobs(self):
-        # getPendingJobsPerPackage associates the right job with the
-        # right package.
-        derived_series = self.factory.makeDistroSeries()
-        dsds = [
-            self.factory.makeDistroSeriesDifference(
-                derived_series=derived_series)
-            for counter in xrange(2)]
-        jobs = map(self.makeJob, dsds)
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        self.assertEqual(
-            dict(zip([dsd.source_package_name.name for dsd in dsds], jobs)),
-            job_source.getPendingJobsPerPackage(derived_series))
-
-    def test_getPendingJobsPerPackage_picks_oldest_job_for_dsd(self):
-        # If there are multiple jobs for one package,
-        # getPendingJobsPerPackage picks the oldest.
-        dsd = self.factory.makeDistroSeriesDifference()
-        jobs = [self.makeJob(dsd) for counter in xrange(2)]
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        self.assertEqual(
-            {dsd.source_package_name.name: jobs[0]},
-            job_source.getPendingJobsPerPackage(dsd.derived_series))
-
-    def test_getPendingJobsPerPackage_ignores_dsds_without_jobs(self):
-        # getPendingJobsPerPackage produces no dict entry for packages
-        # that have no pending jobs, even if they do have DSDs.
-        dsd = self.factory.makeDistroSeriesDifference()
-        job_source = getUtility(IPlainPackageCopyJobSource)
-        self.assertEqual(
-            {}, job_source.getPendingJobsPerPackage(dsd.derived_series))
-
-    def test_findMatchingDSDs_matches_all_DSDs_for_job(self):
-        # findMatchingDSDs finds matching DSDs for any of the packages
-        # in the job.
-        dsd = self.factory.makeDistroSeriesDifference()
-        naked_job = removeSecurityProxy(self.makeJob(dsd))
-        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
-
-    def test_findMatchingDSDs_ignores_other_source_series(self):
-        # findMatchingDSDs tries to ignore DSDs that are for different
-        # parent series than the job's source series.  (This can't be
-        # done with perfect precision because the job doesn't keep track
-        # of source distroseries, but in practice it should be good
-        # enough).
-        dsd = self.factory.makeDistroSeriesDifference()
-        naked_job = removeSecurityProxy(self.makeJob(dsd))
-
-        # If the dsd differs only in parent series, that's enough to
-        # make it a non-match.
-        removeSecurityProxy(dsd).parent_series = (
-            self.factory.makeDistroSeries())
-
-        self.assertContentEqual([], naked_job.findMatchingDSDs())
-
-    def test_findMatchingDSDs_ignores_other_packages(self):
-        # findMatchingDSDs does not return DSDs that are similar to the
-        # information in the job, but are for different packages.
-        dsd = self.factory.makeDistroSeriesDifference()
-        self.factory.makeDistroSeriesDifference(
-            derived_series=dsd.derived_series,
-            parent_series=dsd.parent_series)
-        naked_job = removeSecurityProxy(self.makeJob(dsd))
-        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
-
-    def test_addSourceOverride(self):
-        # Test the addOverride method which adds an ISourceOverride to the
-        # metadata.
-        name = self.factory.makeSourcePackageName()
-        component = self.factory.makeComponent()
-        section=self.factory.makeSection()
-        pcj = self.factory.makePlainPackageCopyJob()
-        self.layer.txn.commit()
-        self.layer.switchDbUser('sync_packages')
-
-        override = SourceOverride(
-            source_package_name=name,
-            component=component,
-            section=section)
-        pcj.addSourceOverride(override)
-
-        metadata_component = getUtility(
-            IComponentSet)[pcj.metadata["component_override"]]
-        metadata_section = getUtility(
-            ISectionSet)[pcj.metadata["section_override"]]
-        matcher = MatchesStructure(
-            component=Equals(metadata_component),
-            section=Equals(metadata_section))
-        self.assertThat(override, matcher)
-
-    def test_getSourceOverride(self):
-        # Test the getSourceOverride which gets an ISourceOverride from
-        # the metadata.
-        name = self.factory.makeSourcePackageName()
-        component = self.factory.makeComponent()
-        section=self.factory.makeSection()
-        pcj = self.factory.makePlainPackageCopyJob(
-            package_name=name.name, package_version="1.0")
-        self.layer.txn.commit()
-        self.layer.switchDbUser('sync_packages')
-
-        override = SourceOverride(
-            source_package_name=name,
-            component=component,
-            section=section)
-        pcj.addSourceOverride(override)
-
-        self.assertEqual(override, pcj.getSourceOverride())
-
-    def test_getPolicyImplementation_returns_policy(self):
-        # getPolicyImplementation returns the ICopyPolicy that was
-        # chosen for the job.
-        dsd = self.factory.makeDistroSeriesDifference()
-        for policy in PackageCopyPolicy.items:
-            naked_job = removeSecurityProxy(
-                self.makeJob(dsd, copy_policy=policy))
-            self.assertEqual(
-                policy, naked_job.getPolicyImplementation().enum_value)
-
-
-class TestPlainPackageCopyJobPrivileges(TestCaseWithFactory, LocalTestHelper):
-    """Test that `PlainPackageCopyJob` has the privileges it needs.
-
-    This test looks for errors, not failures.  It's here only to see that
-    these operations don't run into any privilege limitations.
-    """
-
-    layer = LaunchpadZopelessLayer
-
-    def test_findMatchingDSDs(self):
-        job = self.makeJob()
-        transaction.commit()
-        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
-        removeSecurityProxy(job).findMatchingDSDs()
-
-    def test_reportFailure(self):
-        job = self.makeJob()
-        transaction.commit()
-        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
-        removeSecurityProxy(job).reportFailure(CannotCopy("Mommy it hurts"))
+<<<<<<< TREE
+
+    def test___repr__(self):
+        distroseries = self.factory.makeDistroSeries()
+        archive1 = self.factory.makeArchive(distroseries.distribution)
+        archive2 = self.factory.makeArchive(distroseries.distribution)
+        source = getUtility(IPlainPackageCopyJobSource)
+        job = source.create(
+            package_name="foo", source_archive=archive1,
+            target_archive=archive2, target_distroseries=distroseries,
+            target_pocket=PackagePublishingPocket.RELEASE,
+            package_version="1.0-1", include_binaries=True)
+        self.assertEqual(
+            ("<PlainPackageCopyJob to copy package foo from "
+             "{distroseries.distribution.name}/{archive1.name} to "
+             "{distroseries.distribution.name}/{archive2.name}, "
+             "RELEASE pocket, in {distroseries.distribution.name} "
+             "{distroseries.name}, including binaries>").format(
+                distroseries=distroseries, archive1=archive1,
+                archive2=archive2),
+            repr(job))
+
+    def test_getPendingJobsPerPackage_finds_jobs(self):
+        # getPendingJobsPerPackage finds jobs, and the packages they
+        # belong to.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job = self.makeJob(dsd)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {dsd.source_package_name.name: job},
+            job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_getPendingJobsPerPackage_ignores_other_distroseries(self):
+        # getPendingJobsPerPackage only looks for jobs on the indicated
+        # distroseries.
+        self.makeJob()
+        other_series = self.factory.makeDistroSeries()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {}, job_source.getPendingJobsPerPackage(other_series))
+
+    def test_getPendingJobsPerPackage_only_returns_pending_jobs(self):
+        # getPendingJobsPerPackage ignores jobs that have already been
+        # run.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job = self.makeJob(dsd)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        found_by_state = {}
+        for status in JobStatus.items:
+            removeSecurityProxy(job).job._status = status
+            result = job_source.getPendingJobsPerPackage(dsd.derived_series)
+            if len(result) > 0:
+                found_by_state[status] = result[dsd.source_package_name.name]
+        expected = {
+            JobStatus.WAITING: job,
+            JobStatus.RUNNING: job,
+            JobStatus.SUSPENDED: job,
+        }
+        self.assertEqual(expected, found_by_state)
+
+    def test_getPendingJobsPerPackage_distinguishes_jobs(self):
+        # getPendingJobsPerPackage associates the right job with the
+        # right package.
+        derived_series = self.factory.makeDistroSeries()
+        dsds = [
+            self.factory.makeDistroSeriesDifference(
+                derived_series=derived_series)
+            for counter in xrange(2)]
+        jobs = map(self.makeJob, dsds)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            dict(zip([dsd.source_package_name.name for dsd in dsds], jobs)),
+            job_source.getPendingJobsPerPackage(derived_series))
+
+    def test_getPendingJobsPerPackage_picks_oldest_job_for_dsd(self):
+        # If there are multiple jobs for one package,
+        # getPendingJobsPerPackage picks the oldest.
+        dsd = self.factory.makeDistroSeriesDifference()
+        jobs = [self.makeJob(dsd) for counter in xrange(2)]
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {dsd.source_package_name.name: jobs[0]},
+            job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_getPendingJobsPerPackage_ignores_dsds_without_jobs(self):
+        # getPendingJobsPerPackage produces no dict entry for packages
+        # that have no pending jobs, even if they do have DSDs.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {}, job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_findMatchingDSDs_matches_all_DSDs_for_job(self):
+        # findMatchingDSDs finds matching DSDs for any of the packages
+        # in the job.
+        dsd = self.factory.makeDistroSeriesDifference()
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
+
+    def test_findMatchingDSDs_ignores_other_source_series(self):
+        # findMatchingDSDs tries to ignore DSDs that are for different
+        # parent series than the job's source series.  (This can't be
+        # done with perfect precision because the job doesn't keep track
+        # of source distroseries, but in practice it should be good
+        # enough).
+        dsd = self.factory.makeDistroSeriesDifference()
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+
+        # If the dsd differs only in parent series, that's enough to
+        # make it a non-match.
+        removeSecurityProxy(dsd).parent_series = (
+            self.factory.makeDistroSeries())
+
+        self.assertContentEqual([], naked_job.findMatchingDSDs())
+
+    def test_findMatchingDSDs_ignores_other_packages(self):
+        # findMatchingDSDs does not return DSDs that are similar to the
+        # information in the job, but are for different packages.
+        dsd = self.factory.makeDistroSeriesDifference()
+        self.factory.makeDistroSeriesDifference(
+            derived_series=dsd.derived_series,
+            parent_series=dsd.parent_series)
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
+
+    def test_addSourceOverride(self):
+        # Test the addOverride method which adds an ISourceOverride to the
+        # metadata.
+        name = self.factory.makeSourcePackageName()
+        component = self.factory.makeComponent()
+        section=self.factory.makeSection()
+        pcj = self.factory.makePlainPackageCopyJob()
+        self.layer.txn.commit()
+        self.layer.switchDbUser('sync_packages')
+
+        override = SourceOverride(
+            source_package_name=name,
+            component=component,
+            section=section)
+        pcj.addSourceOverride(override)
+
+        metadata_component = getUtility(
+            IComponentSet)[pcj.metadata["component_override"]]
+        metadata_section = getUtility(
+            ISectionSet)[pcj.metadata["section_override"]]
+        matcher = MatchesStructure(
+            component=Equals(metadata_component),
+            section=Equals(metadata_section))
+        self.assertThat(override, matcher)
+
+    def test_getSourceOverride(self):
+        # Test the getSourceOverride which gets an ISourceOverride from
+        # the metadata.
+        name = self.factory.makeSourcePackageName()
+        component = self.factory.makeComponent()
+        section=self.factory.makeSection()
+        pcj = self.factory.makePlainPackageCopyJob(
+            package_name=name.name, package_version="1.0")
+        self.layer.txn.commit()
+        self.layer.switchDbUser('sync_packages')
+
+        override = SourceOverride(
+            source_package_name=name,
+            component=component,
+            section=section)
+        pcj.addSourceOverride(override)
+
+        self.assertEqual(override, pcj.getSourceOverride())
+
+    def test_getPolicyImplementation_returns_policy(self):
+        # getPolicyImplementation returns the ICopyPolicy that was
+        # chosen for the job.
+        dsd = self.factory.makeDistroSeriesDifference()
+        for policy in PackageCopyPolicy.items:
+            naked_job = removeSecurityProxy(
+                self.makeJob(dsd, copy_policy=policy))
+            self.assertEqual(
+                policy, naked_job.getPolicyImplementation().enum_value)
+
+
+class TestPlainPackageCopyJobPrivileges(TestCaseWithFactory, LocalTestHelper):
+    """Test that `PlainPackageCopyJob` has the privileges it needs.
+
+    This test looks for errors, not failures.  It's here only to see that
+    these operations don't run into any privilege limitations.
+    """
+
+    layer = LaunchpadZopelessLayer
+
+    def test_findMatchingDSDs(self):
+        job = self.makeJob()
+        transaction.commit()
+        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
+        removeSecurityProxy(job).findMatchingDSDs()
+
+    def test_reportFailure(self):
+        job = self.makeJob()
+        transaction.commit()
+        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
+        removeSecurityProxy(job).reportFailure(CannotCopy("Mommy it hurts"))
+=======
+
+    def test___repr__(self):
+        distroseries = self.factory.makeDistroSeries()
+        archive1 = self.factory.makeArchive(distroseries.distribution)
+        archive2 = self.factory.makeArchive(distroseries.distribution)
+        source = getUtility(IPlainPackageCopyJobSource)
+        job = source.create(
+            package_name="foo", source_archive=archive1,
+            target_archive=archive2, target_distroseries=distroseries,
+            target_pocket=PackagePublishingPocket.RELEASE,
+            package_version="1.0-1", include_binaries=True)
+        self.assertEqual(
+            ("<PlainPackageCopyJob to copy package foo from "
+             "{distroseries.distribution.name}/{archive1.name} to "
+             "{distroseries.distribution.name}/{archive2.name}, "
+             "RELEASE pocket, in {distroseries.distribution.name} "
+             "{distroseries.name}, including binaries>").format(
+                distroseries=distroseries, archive1=archive1,
+                archive2=archive2),
+            repr(job))
+
+    def test_getPendingJobsPerPackage_finds_jobs(self):
+        # getPendingJobsPerPackage finds jobs, and the packages they
+        # belong to.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job = self.makeJob(dsd)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {dsd.source_package_name.name: job},
+            job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_getPendingJobsPerPackage_ignores_other_distroseries(self):
+        # getPendingJobsPerPackage only looks for jobs on the indicated
+        # distroseries.
+        self.makeJob()
+        other_series = self.factory.makeDistroSeries()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {}, job_source.getPendingJobsPerPackage(other_series))
+
+    def test_getPendingJobsPerPackage_only_returns_pending_jobs(self):
+        # getPendingJobsPerPackage ignores jobs that have already been
+        # run.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job = self.makeJob(dsd)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        found_by_state = {}
+        for status in JobStatus.items:
+            removeSecurityProxy(job).job._status = status
+            result = job_source.getPendingJobsPerPackage(dsd.derived_series)
+            if len(result) > 0:
+                found_by_state[status] = result[dsd.source_package_name.name]
+        expected = {
+            JobStatus.WAITING: job,
+            JobStatus.RUNNING: job,
+            JobStatus.SUSPENDED: job,
+        }
+        self.assertEqual(expected, found_by_state)
+
+    def test_getPendingJobsPerPackage_distinguishes_jobs(self):
+        # getPendingJobsPerPackage associates the right job with the
+        # right package.
+        derived_series = self.factory.makeDistroSeries()
+        dsds = [
+            self.factory.makeDistroSeriesDifference(
+                derived_series=derived_series)
+            for counter in xrange(2)]
+        jobs = map(self.makeJob, dsds)
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            dict(zip([dsd.source_package_name.name for dsd in dsds], jobs)),
+            job_source.getPendingJobsPerPackage(derived_series))
+
+    def test_getPendingJobsPerPackage_picks_oldest_job_for_dsd(self):
+        # If there are multiple jobs for one package,
+        # getPendingJobsPerPackage picks the oldest.
+        dsd = self.factory.makeDistroSeriesDifference()
+        jobs = [self.makeJob(dsd) for counter in xrange(2)]
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {dsd.source_package_name.name: jobs[0]},
+            job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_getPendingJobsPerPackage_ignores_dsds_without_jobs(self):
+        # getPendingJobsPerPackage produces no dict entry for packages
+        # that have no pending jobs, even if they do have DSDs.
+        dsd = self.factory.makeDistroSeriesDifference()
+        job_source = getUtility(IPlainPackageCopyJobSource)
+        self.assertEqual(
+            {}, job_source.getPendingJobsPerPackage(dsd.derived_series))
+
+    def test_findMatchingDSDs_matches_all_DSDs_for_job(self):
+        # findMatchingDSDs finds matching DSDs for any of the packages
+        # in the job.
+        dsd = self.factory.makeDistroSeriesDifference()
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
+
+    def test_findMatchingDSDs_ignores_other_source_series(self):
+        # findMatchingDSDs tries to ignore DSDs that are for different
+        # parent series than the job's source series.  (This can't be
+        # done with perfect precision because the job doesn't keep track
+        # of source distroseries, but in practice it should be good
+        # enough).
+        dsd = self.factory.makeDistroSeriesDifference()
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+
+        # If the dsd differs only in parent series, that's enough to
+        # make it a non-match.
+        removeSecurityProxy(dsd).parent_series = (
+            self.factory.makeDistroSeries())
+
+        self.assertContentEqual([], naked_job.findMatchingDSDs())
+
+    def test_findMatchingDSDs_ignores_other_packages(self):
+        # findMatchingDSDs does not return DSDs that are similar to the
+        # information in the job, but are for different packages.
+        dsd = self.factory.makeDistroSeriesDifference()
+        self.factory.makeDistroSeriesDifference(
+            derived_series=dsd.derived_series,
+            parent_series=dsd.parent_series)
+        naked_job = removeSecurityProxy(self.makeJob(dsd))
+        self.assertContentEqual([dsd], naked_job.findMatchingDSDs())
+
+    def test_addSourceOverride(self):
+        # Test the addOverride method which adds an ISourceOverride to the
+        # metadata.
+        name = self.factory.makeSourcePackageName()
+        component = self.factory.makeComponent()
+        section = self.factory.makeSection()
+        pcj = self.factory.makePlainPackageCopyJob()
+        self.layer.txn.commit()
+        self.layer.switchDbUser('sync_packages')
+
+        override = SourceOverride(
+            source_package_name=name,
+            component=component,
+            section=section)
+        pcj.addSourceOverride(override)
+
+        metadata_component = getUtility(
+            IComponentSet)[pcj.metadata["component_override"]]
+        metadata_section = getUtility(
+            ISectionSet)[pcj.metadata["section_override"]]
+        matcher = MatchesStructure(
+            component=Equals(metadata_component),
+            section=Equals(metadata_section))
+        self.assertThat(override, matcher)
+
+    def test_getSourceOverride(self):
+        # Test the getSourceOverride which gets an ISourceOverride from
+        # the metadata.
+        name = self.factory.makeSourcePackageName()
+        component = self.factory.makeComponent()
+        section = self.factory.makeSection()
+        pcj = self.factory.makePlainPackageCopyJob(
+            package_name=name.name, package_version="1.0")
+        self.layer.txn.commit()
+        self.layer.switchDbUser('sync_packages')
+
+        override = SourceOverride(
+            source_package_name=name,
+            component=component,
+            section=section)
+        pcj.addSourceOverride(override)
+
+        self.assertEqual(override, pcj.getSourceOverride())
+
+    def test_getPolicyImplementation_returns_policy(self):
+        # getPolicyImplementation returns the ICopyPolicy that was
+        # chosen for the job.
+        dsd = self.factory.makeDistroSeriesDifference()
+        for policy in PackageCopyPolicy.items:
+            naked_job = removeSecurityProxy(
+                self.makeJob(dsd, copy_policy=policy))
+            self.assertEqual(
+                policy, naked_job.getPolicyImplementation().enum_value)
+
+
+class TestPlainPackageCopyJobPrivileges(TestCaseWithFactory, LocalTestHelper):
+    """Test that `PlainPackageCopyJob` has the privileges it needs.
+
+    This test looks for errors, not failures.  It's here only to see that
+    these operations don't run into any privilege limitations.
+    """
+
+    layer = LaunchpadZopelessLayer
+
+    def test_findMatchingDSDs(self):
+        job = self.makeJob()
+        transaction.commit()
+        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
+        removeSecurityProxy(job).findMatchingDSDs()
+
+    def test_reportFailure(self):
+        job = self.makeJob()
+        transaction.commit()
+        self.layer.switchDbUser(config.IPlainPackageCopyJobSource.dbuser)
+        removeSecurityProxy(job).reportFailure(CannotCopy("Mommy it hurts"))
+>>>>>>> MERGE-SOURCE