Merge lp:~julian-edwards/launchpad/publish-copy-archives-bug-520520 into lp:launchpad

[Julian Edwards]

Change the ordering of the dependencies for builds to something that makes more sense.

[Julian Edwards (julian-edwards)]

= Summary =
Change the ordering of the dependencies for builds to something that makes
more sense.

== Proposed fix ==
When we send builds to the build slaves, we tell the chroot exactly what
sources.list it needs to install before pulling in build dependencies. The
ordering of that file has a bearing on what packages get installed if two
repos have the same version of something.

This branch makes that ordering sensible and determinate.

== Pre-implementation notes ==
This is the first part of the fix for bug 520520.

== Implementation details ==
The entries are returned in the order that is most useful;
  1. the context archive itself
  2. external dependencies
  3. user-selected archive dependencies
  4. the default primary archive

This is a simple fix in the get_sources_list_for_building() function.

== Tests ==
bin/test -cvvt archive-dependencies.txt

== Demo and Q/A ==

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/lp/soyuz/doc/archive-dependencies.txt
  lib/lp/soyuz/adapters/archivedependencies.py

== Pylint notices ==

lib/lp/soyuz/adapters/archivedependencies.py
    43: [F0401] Unable to import 'lazr.uri' (No module named uri)

[Paul Hummer (rockstar)]

This all looks good. Thanks for the branch!

review: Approve (code)

=== modified file 'lib/lp/soyuz/adapters/archivedependencies.py'
--- lib/lp/soyuz/adapters/archivedependencies.py	2010-01-15 04:03:26 +0000
+++ lib/lp/soyuz/adapters/archivedependencies.py	2010-02-18 15:35:24 +0000
@@ -119,23 +119,40 @@
 def get_sources_list_for_building(build, distroarchseries, sourcepackagename):
     """Return the sources_list entries required to build the given item.
 
+    The entries are returned in the order that is most useful;
+     1. the context archive itself
+     2. external dependencies
+     3. user-selected archive dependencies
+     4. the default primary archive
+
     :param build: a context `IBuild`.
     :param distroarchseries: A `IDistroArchSeries`
     :param sourcepackagename: A source package name (as text)
     :return: a deb sources_list entries (lines).
     """
     deps = []
-
-    # Consider primary archive dependency override. Add the default
-    # primary archive dependencies if it's not present.
-    if build.archive.getArchiveDependency(
-        build.archive.distribution.main_archive) is None:
-        primary_dependencies = _get_default_primary_dependencies(build)
-        deps.extend(primary_dependencies)
+    sources_list_lines = []
+
+    # Add implicit self-dependency for non-primary contexts.
+    if build.archive.purpose in ALLOW_RELEASE_BUILDS:
+        self_dep = [(
+            build.archive, PackagePublishingPocket.RELEASE,
+            get_components_for_building(build))]
+        sources_list_lines = _get_sources_list_for_dependencies(
+            self_dep, distroarchseries)
+
+    # Append external sources_list lines for this archive if it's
+    # specified in the configuration.
+    dependencies = build.archive.external_dependencies
+    if dependencies is not None:
+        for archive_dep in dependencies.splitlines():
+            line = archive_dep % (
+                {'series': distroarchseries.distroseries.name})
+            sources_list_lines.append(line)
 
     # Consider user-selected archive dependencies.
-    primary_component = get_primary_current_component(build.archive, 
-        build.distroseries, sourcepackagename)
+    primary_component = get_primary_current_component(
+        build.archive, build.distroseries, sourcepackagename)
     for archive_dependency in build.archive.dependencies:
         # When the dependency component is undefined, we should use
         # the component where the source is published in the primary
@@ -151,25 +168,15 @@
                 (archive_dependency.dependency, pocket, components)
                 )
 
-    # Add implicit self-dependency for non-primary contexts.
-    if build.archive.purpose in ALLOW_RELEASE_BUILDS:
-        deps.append(
-            (build.archive, PackagePublishingPocket.RELEASE,
-             get_components_for_building(build))
-            )
-
-    sources_list_lines = _get_sources_list_for_dependencies(
-        deps, distroarchseries)
-
-    # Append external sources_list lines for this archive if it's
-    # specified in the configuration.
-    dependencies = build.archive.external_dependencies
-    if dependencies is not None:
-        for archive_dep in dependencies.splitlines():
-            line = archive_dep % (
-                {'series': distroarchseries.distroseries.name})
-            sources_list_lines.append(line)
-
+    # Consider primary archive dependency override. Add the default
+    # primary archive dependencies if it's not present.
+    if build.archive.getArchiveDependency(
+        build.archive.distribution.main_archive) is None:
+        primary_dependencies = _get_default_primary_dependencies(build)
+        deps.extend(primary_dependencies)
+
+    sources_list_lines.extend(
+        _get_sources_list_for_dependencies(deps, distroarchseries))
     return sources_list_lines
 
 def _has_published_binaries(archive, distroarchseries, pocket):

=== modified file 'lib/lp/soyuz/doc/archive-dependencies.txt'
--- lib/lp/soyuz/doc/archive-dependencies.txt	2010-01-16 09:41:17 +0000
+++ lib/lp/soyuz/doc/archive-dependencies.txt	2010-02-18 15:35:24 +0000
@@ -207,7 +207,7 @@
     >>> def print_building_sources_list(candidate):
     ...     sources_list = get_sources_list_for_building(candidate,
     ...         candidate.distroarchseries, candidate.sourcepackagerelease.name)
-    ...     for line in sorted(sources_list):
+    ...     for line in sources_list:
     ...         print line
 
 Note that only the default ubuntu dependencies for a public PPA will be
@@ -240,13 +240,13 @@
     ...     status=PackagePublishingStatus.PUBLISHED)
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
     	main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-security
     	main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-updates
     	main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
 Similarly, populated PPA dependencies are listed in the building
 'sources_list'.
@@ -261,14 +261,14 @@
     ...     getUtility(IComponentSet)['main'])
 
     >>> print_building_sources_list(a_build)
-    deb http://ftpmaster.internal/ubuntu hoary
-    	main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-security
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-updates
-    	main restricted universe multiverse
     deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ppa.launchpad.dev/mark/ppa/ubuntu hoary main
+    deb http://ftpmaster.internal/ubuntu hoary
+    	main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-security
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-updates
+    	main restricted universe multiverse
 
 The authentication information gets added for private PPA
 dependencies.
@@ -277,6 +277,7 @@
     >>> mark.archive.private = True
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://buildd:[e-mail]/mark/ppa/ubuntu
     	hoary main
     deb http://ftpmaster.internal/ubuntu hoary
@@ -285,7 +286,6 @@
         main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-updates
     	main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
 Good enough, let's delete the archive dependency on Mark's PPA.
 
@@ -306,13 +306,13 @@
 dependencies.
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
         main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-security
         main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-updates
         main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
 The default build behaviour will remain unchanged when we override the
 default primary archive dependencies with exactly the same values.
@@ -322,13 +322,13 @@
     ...     getUtility(IComponentSet)['multiverse'])
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
         main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-security
         main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-updates
         main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
     >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
 
@@ -349,11 +349,11 @@
     universe
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
         main universe
     deb http://ftpmaster.internal/ubuntu hoary-security
         main universe
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
     >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
 
@@ -366,8 +366,8 @@
     ...     getUtility(IComponentSet)['restricted'])
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary main restricted
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
     >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
 
@@ -379,15 +379,15 @@
     ...     getUtility(IComponentSet)['multiverse'])
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
         main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-security
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-updates
+        main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-proposed
         main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-security
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-updates
-        main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
     >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
 
@@ -398,15 +398,15 @@
     ...     getUtility(IComponentSet)['multiverse'])
 
     >>> print_building_sources_list(a_build)
+    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
     deb http://ftpmaster.internal/ubuntu hoary
         main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-security
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-updates
+        main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu hoary-backports
         main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-security
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-updates
-        main restricted universe multiverse
-    deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
 
     >>> cprov.archive.removeArchiveDependency(ubuntu.main_archive)
 
@@ -432,13 +432,13 @@
     >>> [partner_build] = pub_source.createMissingBuilds()
 
     >>> print_building_sources_list(partner_build)
-    deb http://ftpmaster.internal/ubuntu hoary
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-security
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-updates
-        main restricted universe multiverse
     deb http://ftpmaster.internal/ubuntu-partner hoary partner
+    deb http://ftpmaster.internal/ubuntu hoary
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-security
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-updates
+        main restricted universe multiverse
 
 
 == External build dependencies ==
@@ -466,14 +466,14 @@
 Now builds in Celso's PPA will use the external dependencies.
 
     >>> print_building_sources_list(a_build)
-    deb http://ftpmaster.internal/ubuntu hoary
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-security
-        main restricted universe multiverse
-    deb http://ftpmaster.internal/ubuntu hoary-updates
-        main restricted universe multiverse
     deb http://ppa.launchpad.dev/cprov/ppa/ubuntu hoary main
+    deb http://user:pass@repository zoing everything
     deb http://user:pass@repository hoary public private
     deb http://user:pass@repository hoary-extra public
-    deb http://user:pass@repository zoing everything
+    deb http://ftpmaster.internal/ubuntu hoary
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-security
+        main restricted universe multiverse
+    deb http://ftpmaster.internal/ubuntu hoary-updates
+        main restricted universe multiverse