Merge lp:~julian-edwards/launchpad/ppa-dependency-schema into lp:launchpad/db-devel

[Julian Edwards]

[Julian Edwards (julian-edwards)]

Submitting to db-devel blank email FTW!

Ok, here's what I did:

= Summary =
Move a PPA's external dependencies from config to the database

A PPA can use external repositories for build dependencies. This was only
done for OEM Services as they have some dependencies that are not in PPAs.
The problem was that with the original implementation done via config items,
the LOSAs need to CP any changes and it turns out that OEM needs a few of
these per week.

So, the dependencies now come from a new database column which is only
editable on the PPA admin form so it's restricted to admins only. We don't
want anyone being able to set this as it breaks the promise that all the
source for a PPA binary is available in another PPA or Ubuntu.

To test, run:
bin/test -cvvt archive.txt -t archivedependencies.txt -t xx-ppa-workflow.txt

And you can see an example of admin page at:
http://launchpad.dev/~mark/+archive/ppa/+admin

[Graham Binns (gmb)]

review: Approve (code)

[Stuart Bishop (stub)]

Discussed on irc, all fine.

patch-2207-05-0.sql

review: Approve (schema)

[Jonathan Lange (jml)]

 * This is highly structured data, which makes me wonder why it's not structured in the database. We agreed that it's ok because:
   * The field is temporary and only being used for migration
   * There are no use cases for analyzing the data.

I'm ok with this to land on the conditions that:
 * there are comments indicating that it's only for OEM migration
 * we add validation code to guarantee that it's clean, in case we want to migrate to structured data later. If the validation code is not in the model, the comments in the interface should warn that there's no model-level validation.
 * we schedule a date to actually delete this column after OEM have migrated.

Also,
 * Please update the comment field to refer to the URL scheme, both in comments.sql and the interface.

review: Approve (schema)

=== modified file 'database/sampledata/current-dev.sql'
--- database/sampledata/current-dev.sql	2009-09-07 01:46:23 +0000
+++ database/sampledata/current-dev.sql	2009-10-07 14:52:13 +0000
@@ -969,18 +969,18 @@
 
 ALTER TABLE archive DISABLE TRIGGER ALL;
 
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (1, 17, NULL, true, NULL, 1, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:12.241774', 15, 1, 8, 5, 1, '2008-09-23 17:29:03.442606', NULL, NULL, NULL, 'Primary Archive for Ubuntu Linux', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (2, 1, NULL, true, NULL, 2, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.863812', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.445921', NULL, NULL, NULL, 'Primary Archive for Redhat Advanced Server', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (3, 1, NULL, true, NULL, 3, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.864941', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.446557', NULL, NULL, NULL, 'Primary Archive for Debian GNU/Linux', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (4, 1, NULL, true, NULL, 4, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.865502', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.44689', NULL, NULL, NULL, 'Primary Archive for The Gentoo Linux', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (5, 1, NULL, true, NULL, 5, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.866015', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447202', NULL, NULL, NULL, 'Primary Archive for Kubuntu - Free KDE-based Linux', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (7, 4, NULL, true, NULL, 7, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.866529', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447515', NULL, NULL, NULL, 'Primary Archive for GuadaLinex: Linux for Andalucia', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (8, 17, NULL, true, NULL, 8, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.867154', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447851', NULL, NULL, NULL, 'Primary Archive for Ubuntu Test', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (9, 28, 'packages to help my friends.', true, 1024, 1, 2, false, 3, 3, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.867684', 4, 0, 3, 1, 0, '2008-09-23 17:29:03.448178', NULL, NULL, NULL, 'PPA for Celso Providelo', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (10, 1, 'packages to help the humanity (you know, ubuntu)', true, 1024, 1, 2, false, 1, 1, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.868202', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.448488', NULL, NULL, NULL, 'PPA for Mark Shuttleworth', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (11, 52, 'I am not allowed to say, I have no privs.', true, 1024, 1, 2, false, 0, 0, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.868709', 1, 0, 0, 1, 0, '2008-09-23 17:29:03.448797', NULL, NULL, NULL, 'PPA for No Privileges Person', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (12, 17, 'Partner archive', true, NULL, 1, 4, false, NULL, NULL, NULL, NULL, NULL, false, 'partner', true, '2008-05-27 18:15:15.869209', 1, 0, 1, 0, 0, '2008-09-23 17:29:03.449157', NULL, NULL, NULL, 'Partner Archive for Ubuntu Linux', 0);
-INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score) VALUES (13, 17, 'Partner archive', true, NULL, 8, 4, false, NULL, NULL, NULL, NULL, NULL, false, 'partner', true, '2008-05-27 18:15:15.869732', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.449471', NULL, NULL, NULL, 'Partner Archive for Ubuntu Test', 0);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (1, 17, NULL, true, NULL, 1, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:12.241774', 15, 1, 8, 5, 1, '2008-09-23 17:29:03.442606', NULL, NULL, NULL, 'Primary Archive for Ubuntu Linux', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (2, 1, NULL, true, NULL, 2, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.863812', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.445921', NULL, NULL, NULL, 'Primary Archive for Redhat Advanced Server', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (3, 1, NULL, true, NULL, 3, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.864941', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.446557', NULL, NULL, NULL, 'Primary Archive for Debian GNU/Linux', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (4, 1, NULL, true, NULL, 4, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.865502', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.44689', NULL, NULL, NULL, 'Primary Archive for The Gentoo Linux', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (5, 1, NULL, true, NULL, 5, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.866015', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447202', NULL, NULL, NULL, 'Primary Archive for Kubuntu - Free KDE-based Linux', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (7, 4, NULL, true, NULL, 7, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.866529', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447515', NULL, NULL, NULL, 'Primary Archive for GuadaLinex: Linux for Andalucia', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (8, 17, NULL, true, NULL, 8, 1, false, NULL, NULL, NULL, NULL, NULL, false, 'primary', true, '2008-05-27 18:15:15.867154', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.447851', NULL, NULL, NULL, 'Primary Archive for Ubuntu Test', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (9, 28, 'packages to help my friends.', true, 1024, 1, 2, false, 3, 3, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.867684', 4, 0, 3, 1, 0, '2008-09-23 17:29:03.448178', NULL, NULL, NULL, 'PPA for Celso Providelo', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (10, 1, 'packages to help the humanity (you know, ubuntu)', true, 1024, 1, 2, false, 1, 1, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.868202', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.448488', NULL, NULL, NULL, 'PPA for Mark Shuttleworth', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (11, 52, 'I am not allowed to say, I have no privs.', true, 1024, 1, 2, false, 0, 0, NULL, NULL, NULL, true, 'ppa', true, '2008-05-27 18:15:15.868709', 1, 0, 0, 1, 0, '2008-09-23 17:29:03.448797', NULL, NULL, NULL, 'PPA for No Privileges Person', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (12, 17, 'Partner archive', true, NULL, 1, 4, false, NULL, NULL, NULL, NULL, NULL, false, 'partner', true, '2008-05-27 18:15:15.869209', 1, 0, 1, 0, 0, '2008-09-23 17:29:03.449157', NULL, NULL, NULL, 'Partner Archive for Ubuntu Linux', 0, NULL);
+INSERT INTO archive (id, owner, description, enabled, authorized_size, distribution, purpose, private, sources_cached, binaries_cached, package_description_cache, fti, buildd_secret, require_virtualized, name, publish, date_updated, total_count, pending_count, succeeded_count, failed_count, building_count, date_created, signing_key, removed_binary_retention_days, num_old_versions_published, displayname, relative_build_score, external_dependencies) VALUES (13, 17, 'Partner archive', true, NULL, 8, 4, false, NULL, NULL, NULL, NULL, NULL, false, 'partner', true, '2008-05-27 18:15:15.869732', 0, 0, 0, 0, 0, '2008-09-23 17:29:03.449471', NULL, NULL, NULL, 'Partner Archive for Ubuntu Test', 0, NULL);
 
 
 ALTER TABLE archive ENABLE TRIGGER ALL;

=== modified file 'database/schema/comments.sql'
--- database/schema/comments.sql	2009-09-25 08:46:57 +0000
+++ database/schema/comments.sql	2009-10-07 14:52:13 +0000
@@ -1865,6 +1865,7 @@
 COMMENT ON COLUMN Archive.removed_binary_retention_days IS 'The number of days before superseded or deleted binary files are expired in the librarian, or zero for never.';
 COMMENT ON COLUMN Archive.num_old_versions_published IS 'The number of versions of a package to keep published before older versions are superseded.';
 COMMENT ON COLUMN Archive.relative_build_score IS 'A delta to the build score that is applied to all builds in this archive.';
+COMMENT ON COLUMN Archive.external_dependencies IS 'Newline-separated list of repositories to be used to retrieve any external build dependencies when building packages in this archive, in the format: deb http[s]://[user:pass@]<host>[/path] %(series)s[-pocket] [components]  The series variable is replaced with the series name of the context build.  This column is specifically and only intended for OEM migration to Launchpad and should be re-examined in October 2010 to see if it is still relevant.';
 
 -- ArchiveAuthToken
 

=== added file 'database/schema/patch-2207-05-0.sql'
--- database/schema/patch-2207-05-0.sql	1970-01-01 00:00:00 +0000
+++ database/schema/patch-2207-05-0.sql	2009-10-07 14:52:13 +0000
@@ -0,0 +1,12 @@
+-- Copyright 2009 Canonical Ltd.  This software is licensed under the
+-- GNU Affero General Public License version 3 (see the file LICENSE).
+
+SET client_min_messages=ERROR;
+
+
+-- Add a column for external archive dependencies.
+ALTER TABLE Archive
+  ADD COLUMN external_dependencies text DEFAULT NULL;
+
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 05, 0);

=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf	2009-10-01 12:06:50 +0000
+++ lib/canonical/config/schema-lazr.conf	2009-10-07 14:52:13 +0000
@@ -1418,13 +1418,6 @@
 
 
 [ppa.master]
-# Line-separated repository lines to be used as build dependencies, in
-# the following format:
-#   deb [user:pass@]<host>[/path] %(series)s[-pocket] [components]
-# 'series' variable is replaced with the series name of the context build.
-# datatype: string
-dependencies: none
-
 
 [poimport]
 # The database user which will be used by this process.

=== modified file 'lib/lp/soyuz/adapters/archivedependencies.py'
--- lib/lp/soyuz/adapters/archivedependencies.py	2009-08-16 12:38:12 +0000
+++ lib/lp/soyuz/adapters/archivedependencies.py	2009-10-07 14:52:13 +0000
@@ -36,7 +36,6 @@
     'pocket_dependencies',
     ]
 
-from canonical.config import config
 from lp.registry.interfaces.pocket import (
     PackagePublishingPocket, pocketsuffix)
 from lp.soyuz.interfaces.archive import ArchivePurpose, ALLOW_RELEASE_BUILDS
@@ -161,11 +160,9 @@
 
     # Append external sources_list lines for this archive if it's
     # specified in the configuration.
-    archive_config_key = 'ppa.%s_%s' % (
-        build.archive.owner.name, build.archive.name)
-    if archive_config_key in config:
-        archive_config = config[archive_config_key]
-        for archive_dep in archive_config.dependencies.splitlines():
+    dependencies = build.archive.external_dependencies
+    if dependencies is not None:
+        for archive_dep in dependencies.splitlines():
             line = archive_dep % (
                 {'series': build.distroarchseries.distroseries.name})
             sources_list_lines.append(line)

=== modified file 'lib/lp/soyuz/browser/archive.py'
--- lib/lp/soyuz/browser/archive.py	2009-09-18 21:00:18 +0000
+++ lib/lp/soyuz/browser/archive.py	2009-10-07 14:52:13 +0000
@@ -28,6 +28,7 @@
 
 from datetime import datetime, timedelta
 import pytz
+from urlparse import urlparse
 
 from zope.app.form.browser import TextAreaWidget
 from zope.component import getUtility
@@ -1754,7 +1755,10 @@
 class ArchiveAdminView(BaseArchiveEditView):
 
     field_names = ['enabled', 'private', 'require_virtualized',
-                   'buildd_secret', 'authorized_size', 'relative_build_score']
+                   'buildd_secret', 'authorized_size', 'relative_build_score',
+                   'external_dependencies']
+
+    custom_widget('external_dependencies', TextAreaWidget, height=3)
 
     def validate_save(self, action, data):
         """Validate the save action on ArchiveAdminView.
@@ -1773,12 +1777,44 @@
             self.setFieldError(
                 'private',
                 'Private teams may not have public archives.')
-
         elif data.get('buildd_secret') is not None and not data['private']:
             self.setFieldError(
                 'buildd_secret',
                 'Do not specify for non-private archives')
 
+        # Check the external_dependencies field.
+        ext_deps =  data.get('external_dependencies')
+        if ext_deps is not None:
+            errors = self.validate_external_dependencies(ext_deps)
+            if len(errors) != 0:
+                error_text = "\n".join(errors)
+                self.setFieldError('external_dependencies', error_text)
+
+    def validate_external_dependencies(self, ext_deps):
+        """Validate the external_dependencies field.
+
+        :param ext_deps: The dependencies form field to check.
+        """
+        errors = []
+        # The field can consist of multiple entries separated by
+        # newlines, so process each in turn.
+        for dep in ext_deps.splitlines():
+            try:
+                deb, url, suite, components = dep.split(" ", 3)
+            except ValueError:
+                errors.append(
+                    "'%s' is not a complete and valid sources.list entry"
+                        % dep)
+                continue
+
+            if deb != "deb":
+                errors.append("%s: Must start with 'deb'" % dep)
+            url_components = urlparse(url)
+            if not url_components[0] or not url_components[1]:
+                errors.append("%s: Invalid URL" % dep)
+
+        return errors
+
     @property
     def owner_is_private_team(self):
         """Is the owner a private team?

=== modified file 'lib/lp/soyuz/browser/tests/archive-views.txt'
--- lib/lp/soyuz/browser/tests/archive-views.txt	2009-10-01 07:05:22 +0000
+++ lib/lp/soyuz/browser/tests/archive-views.txt	2009-10-07 14:52:13 +0000
@@ -1349,3 +1349,50 @@
 
     >>> print copy.status.name
     ACCEPTED
+
+
+== External dependencies validation ==
+
+The ArchiveAdminView checks the external_dependencies form data to see if
+it's a valid sources.list entry.
+
+    >>> ppa_archive_view = create_initialized_view(
+    ...     cprov.archive, name="+admin")
+
+The validate_external_dependencies() method is called when validating and will
+return a list of errors if the data dis not validate.  A valid entry is of the
+form:
+    deb scheme://domain/ suite component[s]
+
+    >>> print ppa_archive_view.validate_external_dependencies(
+    ...     "deb http://example.com/ karmic main")
+    []
+
+Multiple entries are valid, separated by newlines:
+
+    >>> print ppa_archive_view.validate_external_dependencies(
+    ...     "deb http://example.com/ karmic main\n"
+    ...     "deb http://example.com/ karmic restricted")
+    []
+
+If the line does not start with the word "deb" it fails:
+
+    >>> print ppa_archive_view.validate_external_dependencies(
+    ...     "deb http://example.com/ karmic universe\n"
+    ...     "dab http://example.com/ karmic main")
+    ["dab http://example.com/ karmic main: Must start with 'deb'"]
+
+If the line has too few parts it fails.  Here we're missing a suite:
+
+    >>> print ppa_archive_view.validate_external_dependencies(
+    ...     "deb http://example.com/ karmic universe\n"
+    ...     "deb http://example.com/ main")
+    ["'deb http://example.com/ main'
+        is not a complete and valid sources.list entry"]
+
+If the URL looks invalid, it fails:
+
+    >>> print ppa_archive_view.validate_external_dependencies(
+    ...     "deb http://example.com/ karmic universe\n"
+    ...     "deb example.com/ karmic main")
+    ['deb example.com/ karmic main: Invalid URL']

=== modified file 'lib/lp/soyuz/configure.zcml'
--- lib/lp/soyuz/configure.zcml	2009-09-29 07:21:40 +0000
+++ lib/lp/soyuz/configure.zcml	2009-10-07 14:52:13 +0000
@@ -399,8 +399,8 @@
         <require
             permission="launchpad.Commercial"
             set_attributes="authorized_size buildd_secret
-                            enabled private require_virtualized
-                            relative_build_score"/>
+                            enabled external_dependencies private
+                            require_virtualized relative_build_score "/>
         <require
             permission="launchpad.Admin"
             set_attributes="distribution name signing_key"/>

=== modified file 'lib/lp/soyuz/doc/archive-dependencies.txt'
--- lib/lp/soyuz/doc/archive-dependencies.txt	2009-08-28 07:34:44 +0000
+++ lib/lp/soyuz/doc/archive-dependencies.txt	2009-10-07 14:52:13 +0000
@@ -441,38 +441,27 @@
 
 == External build dependencies ==
 
-Via a static configuration change, any PPA hosted in launchpad can be
+Via an administrator change, any PPA hosted in launchpad can be
 assigned to one or more 'external' build dependencies additionally to
 the internal ones.
 
-There is a configuration category called 'ppa' which can be extended
-for any hosted PPA. They are named as following:
-
- [ppa.<owner_name>_<ppa_name>]
-
-The 'ppa' categories contain the 'dependencies' field, which is a
-multi-line string listing the external dependencies in the debian
-sources_list format.
-
- deb [user:pass@]<host>[/path] %(series)s[-pocket] [components]
+There is a column on IArchive called 'external_dependencies' which can be set
+for any hosted PPA.  It is a string listing the comma-separated external
+dependencies in the debian sources_list format.
+
+ deb http[s]://[user:pass@]<host>[/path] %(series)s[-pocket] [components]
 
 The '%(series)s' part is optional and will be replaced on-the-fly with
 the series name for the build record being dispatched.
 
-We will create a configuration for Celso's PPA.
-
-    >>> from canonical.config import config
-    >>> cprov_deps = """
-    ... [ppa.cprov_ppa]
-    ... dependencies:
-    ...   deb http://user:pass@repository zoing everything
-    ...   deb http://user:pass@repository %(series)s public private
-    ...   deb http://user:pass@repository %(series)s-extra public
-    ... """
-    >>> config.push('test_deps', cprov_deps)
-
-Configuration in place, now builds in Celso's PPA will use the
-external dependencies.
+We will create some dependencies for Celso's PPA.
+
+    >>> cprov.archive.external_dependencies = (
+    ... "deb http://user:pass@repository zoing everything\n"
+    ... "deb http://user:pass@repository %(series)s public private\n"
+    ... "deb http://user:pass@repository %(series)s-extra public")
+
+Now builds in Celso's PPA will use the external dependencies.
 
     >>> print_building_sources_list(a_build)
     deb http://ftpmaster.internal/ubuntu hoary
@@ -486,6 +475,3 @@
     deb http://user:pass@repository hoary-extra public
     deb http://user:pass@repository zoing everything
 
-Tests done, we can remove the extra configuration content we've added.
-
-    >>> unused = config.pop('test_deps')

=== modified file 'lib/lp/soyuz/doc/archive.txt'
--- lib/lp/soyuz/doc/archive.txt	2009-09-23 07:41:13 +0000
+++ lib/lp/soyuz/doc/archive.txt	2009-10-07 14:52:13 +0000
@@ -58,11 +58,11 @@
     >>> cprov_archive.failed_count
     1
 
-relative_build_score is a property that can be set only by LP admins and read
-by anyone.  It is a signed integer that represents a delta to all the build
-scores for builds done in the archive.
+relative_build_score and external_dependencies are properties that can be set
+only by LP admins and read by anyone.
 
-The default value is zero:
+relative_build_score is a signed integer that represents a delta to all the
+build scores for builds done in the archive.  The default value is zero:
 
     >>> cprov_archive.relative_build_score
     0
@@ -74,10 +74,28 @@
     ...
     Unauthorized: (..., 'relative_build_score', 'launchpad.Commercial')
 
-As a Launchpad admin, it will work.
+external_dependencies is a text field that should contain a comma-separated
+list of sources.list entries in the format:
+deb http[s]://[user:pass@]<host>[/path] %(series)s[-pocket] [components]
+where the series variable is replaced with the series name of the context
+build.  This allows an admin to set external repositories as a source for
+build dependencies on the context PPA.  Its default value is None:
+
+    >>> print cprov_archive.external_dependencies
+    None
+
+Amending it as an unprivileged user results in failure:
+
+    >>> cprov_archive.external_dependencies = "test"
+    Traceback (most recent call last):
+    ...
+    Unauthorized: (..., 'external_dependencies', 'launchpad.Commercial')
+
+As a Launchpad admin, setting these properties will work.
 
     >>> login("[e-mail]")
     >>> cprov_archive.relative_build_score = 100
+    >>> cprov_archive.external_dependencies = "test"
 
 The buildd_secret is used by the slave scanner when generating a
 sources.list entry for the builder to access a private archive.  It is

=== modified file 'lib/lp/soyuz/interfaces/archive.py'
--- lib/lp/soyuz/interfaces/archive.py	2009-09-23 07:41:13 +0000
+++ lib/lp/soyuz/interfaces/archive.py	2009-10-07 14:52:13 +0000
@@ -253,6 +253,18 @@
         description=_(
             "A delta to apply to all build scores for this archive."))
 
+    external_dependencies = Text(
+        title=_("External dependencies"), required=False, readonly=False,
+        description=_(
+            "Newline-separated list of repositories to be used to retrieve "
+            "any external build dependencies when building packages in this "
+            "archive, in the format:\n"
+            "deb http[s]://[user:pass@]<host>[/path] %(series)s[-pocket] "
+                "[components]\n"
+            "The series variable is replaced with the series name of the "
+            "context build.\n"
+            "NOTE: This is for migration of OEM PPAs only!"))
+
     def getSourcesForDeletion(name=None, status=None, distroseries=None):
         """All `ISourcePackagePublishingHistory` available for deletion.
 

=== modified file 'lib/lp/soyuz/model/archive.py'
--- lib/lp/soyuz/model/archive.py	2009-09-30 14:30:46 +0000
+++ lib/lp/soyuz/model/archive.py	2009-10-07 14:52:13 +0000
@@ -174,6 +174,12 @@
     relative_build_score = IntCol(
         dbName='relative_build_score', notNull=True, default=0)
 
+    # This field is specifically and only intended for OEM migration to
+    # Launchpad and should be re-examined in October 2010 to see if it
+    # is still relevant.
+    external_dependencies = StringCol(
+        dbName='external_dependencies', notNull=False, default=None)
+
     def _init(self, *args, **kw):
         """Provide the right interface for URL traversal."""
         SQLBase._init(self, *args, **kw)

=== modified file 'lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt'
--- lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt	2009-09-18 15:24:30 +0000
+++ lib/lp/soyuz/stories/ppa/xx-ppa-workflow.txt	2009-10-07 14:52:13 +0000
@@ -345,6 +345,7 @@
  * Set a maximum disk size: uploads will be rejected if the resulting
    PPA size is exceeding the authorized size.
  * Set a per-archive build score delta.
+ * Set external archive dependencies
 
 In this case, the administrator may wish to amend the PPA so that it is
 set up like the ubuntu security PPA, which is private but does not
@@ -358,6 +359,8 @@
     True
     >>> admin_browser.getControl(name="field.relative_build_score").value
     '0'
+    >>> admin_browser.getControl(name="field.external_dependencies").value
+    ''
 
     >>> admin_browser.getControl(name="field.enabled").value = False
     >>> admin_browser.getControl(name="field.private").value = True
@@ -367,6 +370,9 @@
     >>> admin_browser.getControl(name="field.authorized_size").value = '1'
     >>> admin_browser.getControl(
     ...     name="field.relative_build_score").value = '199'
+    >>> admin_browser.getControl(
+    ...     name="field.external_dependencies"
+    ...     ).value = "deb http://my.spethial.repo.com/ %(series)s main"
     >>> admin_browser.getControl("Save").click()
 
 Once confirmed the administrator is sent to the PPA page where he can
@@ -379,16 +385,33 @@
     ...     print msg
     This archive has been disabled.
 
-We need go back to the "Administer archive" page to see the build score change
-that was made:
+We need go back to the "Administer archive" page to see the build score and
+external dependencies changes that were made:
 
     >>> admin_browser.getLink("Administer archive").click()
     >>> admin_browser.getControl(name="field.relative_build_score").value
     '199'
+    >>> admin_browser.getControl(name="field.external_dependencies").value
+    'deb http://my.spethial.repo.com/ %(series)s main'
+
+The external dependencies field is validated to make sure it looks like
+a sources.list entry.  If the field fails validation an error is displayed.
+
+    >>> admin_browser.getControl(
+    ...     name="field.external_dependencies"
+    ...     ).value = "deb not_a_url"
+    >>> admin_browser.getControl("Save").click()
+    >>> for error in get_feedback_messages(admin_browser.contents):
+    ...     print error
+    There is 1 error.
+    'deb not_a_url' is not a complete and valid sources.list entry
+
 
 When the archive is private, the buildd secret must also be set, or an
 error is issued:
 
+    >>> admin_browser.getControl(
+    ...     name="field.external_dependencies").value = ""
     >>> admin_browser.getControl(name="field.private").value = True
     >>> admin_browser.getControl(name="field.buildd_secret").value = ""
     >>> admin_browser.getControl("Save").click()