Merge lp:~salgado/launchpad/layer-specific-navigation into lp:launchpad/db-devel

[Guilherme Salgado]

This branch makes it possible to register navigation classes for a
specific layer.

This is to allow us to use a different navigation class for, say,
IDistribution on the vostok vhost.

One issue I've encountered is that the navigation directive will
unconditionally register the nav classes for IXMLRPCRequest (the layer
of the xmlrpc vhost) as well, and that will cause a
ConfigurationConflict when you have more than one navigation class used
for a given context. One solution to this is to add yet another
parameter to the navigation directive (used_for_xmlrpc=True) and use
that to decide whether or not to register the nav class for
IXMLRPCRequest.

I'm not particularly happy with that solution as it doesn't make it
possible for vostok to overwrite the navigation that is used on the
xmlrpc vhost, but that may not be a problem in practice as we need
custom navigation classes only because we want to remove traversals from
the existing ones.

[Paul Hummer (rockstar)]

I approved this in the devel proposal

=== modified file '.bzrignore'
--- .bzrignore	2010-07-15 15:57:40 +0000
+++ .bzrignore	2010-08-12 19:18:45 +0000
@@ -56,7 +56,6 @@
 .bazaar
 .cache
 .subversion
-lib/canonical/buildd/launchpad-files
 .testrepository
 .memcache.pid
 ./pipes
@@ -77,3 +76,4 @@
 lib/canonical/launchpad-buildd_*_source.build
 lib/canonical/launchpad-buildd_*_source.changes
 lib/canonical/buildd/debian/*
+lib/canonical/buildd/launchpad-files/*
 
=== modified file 'README'
--- README	2010-04-06 20:07:30 +0000
+++ README	2010-08-12 19:18:45 +0000
@@ -1,4 +1,106 @@
-This is the top level project, that supplies the infrastructure for testing,
-and running launchpad.
-
-Documentation is in the doc directory or on the wiki.
+====================
+README for Launchpad
+====================
+
+Launchpad is an open source suite of tools that help people and teams to work
+together on software projects.  Unlike many open source projects, Launchpad
+isn't something you install and run yourself (although you are welcome to do
+so), instead, contributors help make <https://launchpad.net> better.
+
+Launchpad is a project of Canonical <http://www.canonical.com> and has
+received many contributions from many wonderful people
+<https://dev.launchpad.net/Contributions>.
+
+If you want help using Launchpad, then please visit our help wiki at:
+
+    https://help.launchpad.net
+
+If you'd like to contribute to Launchpad, have a look at:
+
+    https://dev.launchpad.net
+
+Alternatively, have a poke around in the code, which you probably already know
+how to get if you are reading this file.
+
+
+Getting started
+===============
+
+There's a full guide for getting up-and-running with a development Launchpad
+environment at <https://dev.launchpad.net/Getting>.  When you are ready to
+submit a patch, please consult <https://dev.launchpad.net/PatchSubmission>.
+
+Our bug tracker is at <https://bugs.launchpad.net/launchpad/> and you can get
+the source code any time by doing:
+
+  $ bzr branch lp:launchpad
+
+
+Navigating the tree
+-------------------
+
+The Launchpad tree is big, messy and changing.  Sorry about that.  Don't panic
+though, it can sense fear.  Keep a firm grip on `grep` and pay attention to
+these important top-level folders:
+
+  bin/, utilities/
+    Where you will find scripts intended for developers and admins.  There's
+    no rhyme or reason to what goes in bin/ and what goes in utilities/, so
+    take a look in both. bin/ will be empty in a fresh checkout, the actual
+    content lives in 'buildout-templates'.
+
+  configs/
+    Configuration files for various kinds of Launchpad instances.
+    'development' and 'testrunner' are of particular interest to developers.
+
+  cronscripts/
+    Scripts that are run on actual production instances of Launchpad as
+    cronjobs.
+
+  daemons/
+    Entry points for various daemons that form part of Launchpad
+
+  database/
+    Our database schema, our sample data, and some other stuff that causes
+    fear.
+
+  doc/
+    General system-wide documentation. You can also find documentation on
+    <https://dev.launchpad.net>, in docstrings and in doctests.
+
+  lib/
+    Where the vast majority of the code lives, along with our templates, tests
+    and the bits of our documentation that are written as doctests. 'lp' and
+    'canonical' are the two most interesting packages. Note that 'canonical'
+    is deprecated in favour of 'lp'.  To learn more about how the 'lp' package
+    is laid out, take a look at its docstring.
+
+  Makefile
+    Ahh, bliss.  The Makefile has all sorts of goodies.  If you spend any
+    length of time hacking on Launchpad, you'll use it often.  The most
+    important targets are 'make clean', 'make compile', 'make schema', 'make
+    run' and 'make run_all'.
+
+  scripts/
+    Scripts that are run on actual production instances of Launchpad,
+    generally triggered by some automatic process.
+
+
+You can spend years hacking on Launchpad full-time and not know what all of
+the files in the top-level directory are for.  However, here's a guide to some
+of the ones that come up from time to time.
+
+  buildout-templates/
+    Templates that are generated into actual files, normally bin/ scripts,
+    when buildout is run. If you want to change the behaviour of bin/test,
+    look here.
+
+  bzrplugins/, optionalbzrplugins/
+    Bazaar plugins used in running Launchpad.
+
+  sourcecode/
+    A directory into which we symlink branches of some of Launchpad's
+    dependencies.  Don't ask.
+
+You never have to care about 'benchmarks', 'override-includes' or
+'package-includes'.
 
=== renamed file 'daemons/buildd-slave-example.conf' => 'lib/canonical/buildd/buildd-slave-example.conf'
=== modified file 'lib/canonical/buildd/debian/changelog'
--- lib/canonical/buildd/debian/changelog	2010-08-05 21:12:36 +0000
+++ lib/canonical/buildd/debian/changelog	2010-08-12 19:18:45 +0000
@@ -1,9 +1,19 @@
 launchpad-buildd (68) UNRELEASED; urgency=low
 
+  [ William Grant ]
   * Take an 'arch_tag' argument, so the master can override the slave
     architecture.
 
- -- William Grant <[e-mail]>  Sun, 01 Aug 2010 22:00:32 +1000
+  [ Jelmer Vernooij ]
+
+  * Explicitly use source format 1.0.
+  * Add LSB information to init script.
+  * Use debhelper >= 5 (available in dapper, not yet deprecated in
+    maverick).
+  * Fix spelling in description.
+  * Install example buildd configuration.
+
+ -- Jelmer Vernooij <[e-mail]>  Thu, 12 Aug 2010 17:04:14 +0200
 
 launchpad-buildd (67) hardy-cat; urgency=low
 
 
=== added file 'lib/canonical/buildd/debian/compat'
--- lib/canonical/buildd/debian/compat	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/debian/compat	2010-08-12 19:18:45 +0000
@@ -0,0 +1,1 @@
+5
 
=== modified file 'lib/canonical/buildd/debian/control'
--- lib/canonical/buildd/debian/control	2010-05-19 15:50:27 +0000
+++ lib/canonical/buildd/debian/control	2010-08-12 19:18:45 +0000
@@ -3,15 +3,15 @@
 Priority: extra
 Maintainer: Adam Conrad <[e-mail]>
 Standards-Version: 3.5.9
-Build-Depends-Indep: debhelper (>= 4)
+Build-Depends-Indep: debhelper (>= 5)
 
 Package: launchpad-buildd
 Section: misc
 Architecture: all
-Depends: python-twisted-core, python-twisted-web, debootstrap, dpkg-dev, linux32, file, bzip2, sudo, ntpdate, adduser, apt-transport-https, lsb-release, apache2
+Depends: python-twisted-core, python-twisted-web, debootstrap, dpkg-dev, linux32, file, bzip2, sudo, ntpdate, adduser, apt-transport-https, lsb-release, apache2, ${misc:Depends}
 Description: Launchpad buildd slave
  This is the launchpad buildd slave package. It contains everything needed to
  get a launchpad buildd going apart from the database manipulation required to
  tell launchpad about the slave instance. If you are creating more than one
- slave instance on the same computer, be sure to give them independant configs
- and independant filecaches etc.
+ slave instance on the same computer, be sure to give them independent configs
+ and independent filecaches etc.
 
=== added file 'lib/canonical/buildd/debian/launchpad-buildd.examples'
--- lib/canonical/buildd/debian/launchpad-buildd.examples	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/debian/launchpad-buildd.examples	2010-08-12 19:18:45 +0000
@@ -0,0 +1,1 @@
+buildd-slave-example.conf
 
=== modified file 'lib/canonical/buildd/debian/launchpad-buildd.init'
--- lib/canonical/buildd/debian/launchpad-buildd.init	2010-03-31 17:10:21 +0000
+++ lib/canonical/buildd/debian/launchpad-buildd.init	2010-08-12 19:18:45 +0000
@@ -8,6 +8,16 @@
 #
 # Author:       Daniel Silverstone <[e-mail]>
 
+### BEGIN INIT INFO
+# Provides:          launchpad_buildd
+# Required-Start:    $local_fs $network $syslog $time
+# Required-Stop:     $local_fs $network $syslog $time
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# X-Interactive:     false
+# Short-Description: Start/stop launchpad buildds
+### END INIT INFO
+
 set -e
 
 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 
=== modified file 'lib/canonical/buildd/debian/rules'
--- lib/canonical/buildd/debian/rules	2010-07-18 08:49:02 +0000
+++ lib/canonical/buildd/debian/rules	2010-08-12 19:18:45 +0000
@@ -3,7 +3,6 @@
 # Copyright 2009, 2010 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-export DH_COMPAT=4
 export DH_OPTIONS
 
 # This is an incomplete debian rules file for making the launchpad-buildd deb
@@ -41,6 +40,7 @@
 	  etc/launchpad-buildd \
 	  usr/share/launchpad-buildd/canonical/launchpad/daemons \
 	  usr/share/doc/launchpad-buildd
+	dh_installexamples
 
 	# Do installs here
 	touch $(pytarget)/../launchpad/__init__.py
@@ -89,10 +89,11 @@
 clean:
 	dh_clean
 
-package:
-	mkdir -p launchpad-files
+prepare:
 	install -m644 $(daemons)/buildd-slave.tac launchpad-files/buildd-slave.tac
 	cp ../launchpad/daemons/tachandler.py launchpad-files/tachandler.py
+
+package: prepare
 	debuild -uc -us -S
 
 build:
 
=== added directory 'lib/canonical/buildd/debian/source'
=== added file 'lib/canonical/buildd/debian/source/format'
--- lib/canonical/buildd/debian/source/format	1970-01-01 00:00:00 +0000
+++ lib/canonical/buildd/debian/source/format	2010-08-12 19:18:45 +0000
@@ -0,0 +1,1 @@
+1.0
 
=== added directory 'lib/canonical/buildd/launchpad-files'
=== modified file 'lib/canonical/launchpad/doc/navigation.txt'
--- lib/canonical/launchpad/doc/navigation.txt	2010-08-02 02:13:52 +0000
+++ lib/canonical/launchpad/doc/navigation.txt	2010-08-12 19:18:45 +0000
@@ -234,7 +234,7 @@
 
 == ZCML for browser:navigation ==
 
-The zcml processor `browser:navigation` processes navigation classes.
+The zcml processor `browser:navigation` registers navigation classes.
 
     >>> class ThingSetView:
     ...
@@ -266,6 +266,14 @@
     ... </configure>
     ... """)
 
+Once registered, we can look the navigation up using getMultiAdapter().
+
+    >>> from zope.component import getMultiAdapter
+    >>> from canonical.launchpad.webapp.servers import LaunchpadTestRequest
+    >>> from zope.publisher.interfaces.browser import IBrowserPublisher
+    >>> navigation = getMultiAdapter(
+    ...     (thingset, LaunchpadTestRequest()), IBrowserPublisher, name='')
+
 This time, we get the view object for the page that was registered.
 
     >>> navigation.publishTraverse(request, 'thingview')
 
=== modified file 'lib/canonical/launchpad/webapp/metazcml.py'
--- lib/canonical/launchpad/webapp/metazcml.py	2010-08-05 21:53:44 +0000
+++ lib/canonical/launchpad/webapp/metazcml.py	2010-08-12 19:18:45 +0000
@@ -193,6 +193,10 @@
 class INavigationDirective(IGlueDirective):
     """Hook up traversal etc."""
 
+    layer = GlobalInterface(
+        title=u"The layer where this navigation is going to be available.",
+        required=False)
+
 
 class IFeedsDirective(IGlueDirective):
     """Hook up feeds."""
@@ -267,7 +271,7 @@
                           layer=layer, class_=feedclass)
 
 
-def navigation(_context, module, classes):
+def navigation(_context, module, classes, layer=IDefaultBrowserLayer):
     """Handler for the `INavigationDirective`."""
     if not inspect.ismodule(module):
         raise TypeError("module attribute must be a module: %s, %s" %
@@ -281,19 +285,11 @@
         for_ = [navclass.usedfor]
 
         # Register the navigation as the traversal component.
-        layer = IDefaultBrowserLayer
         provides = IBrowserPublisher
         name = ''
         view(_context, factory, layer, name, for_,
                 permission=PublicPermission, provides=provides,
                 allowed_interface=[IBrowserPublisher])
-        #view(_context, factory, layer, name, for_,
-        #     permission=PublicPermission, provides=provides)
-
-        # Also register the navigation as a traversal component for XMLRPC.
-        xmlrpc_layer = IXMLRPCRequest
-        view(_context, factory, xmlrpc_layer, name, for_,
-             permission=PublicPermission, provides=provides)
 
 
 class InterfaceInstanceDispatcher:
 
=== added file 'lib/canonical/launchpad/webapp/tests/test_navigation.py'
--- lib/canonical/launchpad/webapp/tests/test_navigation.py	1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/webapp/tests/test_navigation.py	2010-08-12 19:18:45 +0000
@@ -0,0 +1,107 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+from zope.component import ComponentLookupError, getMultiAdapter
+from zope.configuration import xmlconfig
+from zope.interface import implements, Interface
+from zope.publisher.interfaces.browser import (
+    IBrowserPublisher, IDefaultBrowserLayer)
+from zope.testing.cleanup import cleanUp
+
+from canonical.launchpad.webapp import Navigation
+
+from lp.testing import TestCase
+
+
+class TestNavigationDirective(TestCase):
+
+    def test_default_layer(self):
+        # By default all navigation classes are registered for
+        # IDefaultBrowserLayer.
+        directive = """ 
+            <browser:navigation
+                module="%(this)s" classes="ThingNavigation"/>
+            """ % dict(this=this)
+        xmlconfig.string(zcml_configure % directive)
+        navigation = getMultiAdapter(
+            (Thing(), DefaultBrowserLayer()), IBrowserPublisher, name='')
+        self.assertIsInstance(navigation, ThingNavigation)
+
+    def test_specific_layer(self):
+        # If we specify a layer when registering a navigation class, it will
+        # only be available on that layer.
+        directive = """
+            <browser:navigation
+                module="%(this)s" classes="OtherThingNavigation"
+                layer="%(this)s.IOtherLayer" />
+            """ % dict(this=this)
+        xmlconfig.string(zcml_configure % directive)
+        self.assertRaises(
+            ComponentLookupError,
+            getMultiAdapter,
+            (Thing(), DefaultBrowserLayer()), IBrowserPublisher, name='')
+
+        navigation = getMultiAdapter(
+            (Thing(), OtherLayer()), IBrowserPublisher, name='')
+        self.assertIsInstance(navigation, OtherThingNavigation)
+
+    def test_multiple_navigations_for_single_context(self):
+        # It is possible to have multiple navigation classes for a given
+        # context class as long as they are registered for different layers.
+        directive = """ 
+            <browser:navigation
+                module="%(this)s" classes="ThingNavigation"/>
+            <browser:navigation
+                module="%(this)s" classes="OtherThingNavigation"
+                layer="%(this)s.IOtherLayer" />
+            """ % dict(this=this)
+        xmlconfig.string(zcml_configure % directive)
+
+        navigation = getMultiAdapter(
+            (Thing(), DefaultBrowserLayer()), IBrowserPublisher, name='')
+        other_navigation = getMultiAdapter(
+            (Thing(), OtherLayer()), IBrowserPublisher, name='')
+        self.assertNotEqual(navigation, other_navigation)
+
+    def tearDown(self):
+        TestCase.tearDown(self)
+        cleanUp()
+
+
+class DefaultBrowserLayer:
+    implements(IDefaultBrowserLayer)
+
+
+class IThing(Interface):
+    pass
+
+
+class Thing(object):
+    implements(IThing)
+
+
+class ThingNavigation(Navigation):
+    usedfor = IThing
+
+
+class OtherThingNavigation(Navigation):
+    usedfor = IThing
+
+
+class IOtherLayer(Interface):
+    pass
+
+
+class OtherLayer:
+    implements(IOtherLayer)
+
+
+this = "canonical.launchpad.webapp.tests.test_navigation"
+zcml_configure = """
+    <configure xmlns:browser="http://namespaces.zope.org/browser">
+      <include package="canonical.launchpad.webapp" file="meta.zcml" />
+      %s
+    </configure>
+    """
 
=== added symlink 'lib/deb822.py'
=== target is u'../sourcecode/python-debian/lib/deb822.py'
=== added symlink 'lib/debian'
=== target is u'../sourcecode/python-debian/lib/debian'
=== modified file 'lib/lp/archivepublisher/config.py'
--- lib/lp/archivepublisher/config.py	2010-07-07 06:28:03 +0000
+++ lib/lp/archivepublisher/config.py	2010-08-12 19:18:45 +0000
@@ -120,18 +120,15 @@
                 config_segment["archtags"].append(
                     dar.architecturetag.encode('utf-8'))
 
-            if not dr.lucilleconfig:
-                raise LucilleConfigError(
-                    'No Lucille configuration section for %s' % dr.name)
-
-            strio = StringIO(dr.lucilleconfig.encode('utf-8'))
-            config_segment["config"] = ConfigParser()
-            config_segment["config"].readfp(strio)
-            strio.close()
-            config_segment["components"] = config_segment["config"].get(
-                "publishing", "components").split(" ")
-
-            self._distroseries[distroseries_name] = config_segment
+            if dr.lucilleconfig:
+                strio = StringIO(dr.lucilleconfig.encode('utf-8'))
+                config_segment["config"] = ConfigParser()
+                config_segment["config"].readfp(strio)
+                strio.close()
+                config_segment["components"] = config_segment["config"].get(
+                    "publishing", "components").split(" ")
+
+                self._distroseries[distroseries_name] = config_segment
 
         strio = StringIO(distribution.lucilleconfig.encode('utf-8'))
         self._distroconfig = ConfigParser()
@@ -144,11 +141,19 @@
         # Because dicts iterate for keys only; this works to get dr names
         return self._distroseries.keys()
 
+    def series(self, dr):
+        try:
+            return self._distroseries[dr]
+        except KeyError:
+            raise LucilleConfigError(
+                'No Lucille config section for %s in %s' %
+                    (dr, self.distroName))
+
     def archTagsForSeries(self, dr):
-        return self._distroseries[dr]["archtags"]
+        return self.series(dr)["archtags"]
 
     def componentsForSeries(self, dr):
-        return self._distroseries[dr]["components"]
+        return self.series(dr)["components"]
 
     def _extractConfigInfo(self):
         """Extract configuration information into the attributes we use"""
 
=== modified file 'lib/lp/archivepublisher/ftparchive.py'
--- lib/lp/archivepublisher/ftparchive.py	2009-10-26 18:40:04 +0000
+++ lib/lp/archivepublisher/ftparchive.py	2010-08-12 19:18:45 +0000
@@ -138,6 +138,14 @@
         self._config = config
         self._diskpool = diskpool
         self.distro = distro
+        self.distroseries = []
+        for distroseries in self.distro.series:
+            if not distroseries.name in self._config.distroSeriesNames():
+                self.log.warning("Distroseries %s in %s doesn't have "
+                    "a lucille configuration.", distroseries.name,
+                    self.distro.name)
+            else:
+                self.distroseries.append(distroseries)
         self.publisher = publisher
         self.release_files_needed = {}
 
@@ -185,7 +193,7 @@
         # iterate over the pockets, and do the suffix check inside
         # createEmptyPocketRequest; that would also allow us to replace
         # the == "" check we do there by a RELEASE match
-        for distroseries in self.distro:
+        for distroseries in self.distroseries:
             components = self._config.componentsForSeries(distroseries.name)
             for pocket, suffix in pocketsuffix.items():
                 if not fullpublish:
@@ -366,7 +374,7 @@
 
     def generateOverrides(self, fullpublish=False):
         """Collect packages that need overrides, and generate them."""
-        for distroseries in self.distro.series:
+        for distroseries in self.distroseries:
             for pocket in PackagePublishingPocket.items:
                 if not fullpublish:
                     if not self.publisher.isDirty(distroseries, pocket):
@@ -629,7 +637,7 @@
 
     def generateFileLists(self, fullpublish=False):
         """Collect currently published FilePublishings and write filelists."""
-        for distroseries in self.distro.series:
+        for distroseries in self.distroseries:
             for pocket in pocketsuffix:
                 if not fullpublish:
                     if not self.publisher.isDirty(distroseries, pocket):
 
=== modified file 'lib/lp/archivepublisher/tests/test_config.py'
--- lib/lp/archivepublisher/tests/test_config.py	2010-07-18 00:24:06 +0000
+++ lib/lp/archivepublisher/tests/test_config.py	2010-08-12 19:18:45 +0000
@@ -5,36 +5,48 @@
 
 __metaclass__ = type
 
-import unittest
-
 from zope.component import getUtility
 
 from canonical.config import config
 from canonical.launchpad.interfaces import IDistributionSet
 from canonical.testing import LaunchpadZopelessLayer
 
-
-class TestConfig(unittest.TestCase):
+from lp.archivepublisher.config import Config, LucilleConfigError
+from lp.testing import TestCaseWithFactory
+
+
+class TestConfig(TestCaseWithFactory):
     layer = LaunchpadZopelessLayer
 
     def setUp(self):
+        super(TestConfig, self).setUp()
         self.layer.switchDbUser(config.archivepublisher.dbuser)
         self.ubuntutest = getUtility(IDistributionSet)['ubuntutest']
 
+    def testMissingDistroSeries(self):
+        distroseries = self.factory.makeDistroSeries(
+            distribution=self.ubuntutest, name="somename")
+        d = Config(self.ubuntutest)
+        dsns = d.distroSeriesNames()
+        self.assertEquals(len(dsns), 2)
+        self.assertEquals(dsns[0], "breezy-autotest")
+        self.assertEquals(dsns[1], "hoary-test")
+        self.assertRaises(LucilleConfigError,
+            d.archTagsForSeries, "somename")
+        self.assertRaises(LucilleConfigError,
+            d.archTagsForSeries, "unknown")
+
     def testInstantiate(self):
         """Config should instantiate"""
-        from lp.archivepublisher.config import Config
         d = Config(self.ubuntutest)
 
     def testDistroName(self):
         """Config should be able to return the distroName"""
-        from lp.archivepublisher.config import Config
         d = Config(self.ubuntutest)
         self.assertEqual(d.distroName, "ubuntutest")
 
     def testDistroSeriesNames(self):
         """Config should return two distroseries names"""
-        from lp.archivepublisher.config import Config
         d = Config(self.ubuntutest)
         dsns = d.distroSeriesNames()
         self.assertEquals(len(dsns), 2)
@@ -43,14 +55,12 @@
 
     def testArchTagsForSeries(self):
         """Config should have the arch tags for the drs"""
-        from lp.archivepublisher.config import Config
         d = Config(self.ubuntutest)
         archs = d.archTagsForSeries("hoary-test")
         self.assertEquals(len(archs), 2)
 
     def testDistroConfig(self):
         """Config should have parsed a distro config"""
-        from lp.archivepublisher.config import Config
         d = Config(self.ubuntutest)
         # NOTE: Add checks here when you add stuff in util.py
         self.assertEquals(d.stayofexecution, 5)
 
=== modified file 'lib/lp/archivepublisher/tests/test_ftparchive.py'
--- lib/lp/archivepublisher/tests/test_ftparchive.py	2010-07-18 00:24:06 +0000
+++ lib/lp/archivepublisher/tests/test_ftparchive.py	2010-08-12 19:18:45 +0000
@@ -15,7 +15,7 @@
 from zope.component import getUtility
 
 from canonical.config import config
-from canonical.launchpad.scripts.logger import QuietFakeLogger
+from canonical.launchpad.scripts.logger import BufferLogger, QuietFakeLogger
 from canonical.testing import LaunchpadZopelessLayer
 from lp.archivepublisher.config import Config
 from lp.archivepublisher.diskpool import DiskPool
@@ -23,6 +23,7 @@
 from lp.archivepublisher.publishing import Publisher
 from lp.registry.interfaces.distribution import IDistributionSet
 from lp.registry.interfaces.pocket import PackagePublishingPocket
+from lp.testing import TestCaseWithFactory
 
 
 def sanitize_apt_ftparchive_Sources_output(text):
@@ -55,10 +56,11 @@
         return self._result[i:j]
 
 
-class TestFTPArchive(unittest.TestCase):
+class TestFTPArchive(TestCaseWithFactory):
     layer = LaunchpadZopelessLayer
 
     def setUp(self):
+        super(TestFTPArchive, self).setUp()
         self.layer.switchDbUser(config.archivepublisher.dbuser)
 
         self._distribution = getUtility(IDistributionSet)['ubuntutest']
@@ -79,6 +81,7 @@
         self._publisher = SamplePublisher(self._archive)
 
     def tearDown(self):
+        super(TestFTPArchive, self).tearDown()
         shutil.rmtree(self._config.distroroot)
 
     def _verifyFile(self, filename, directory, output_filter=None):
@@ -116,6 +119,19 @@
             self._publisher)
         return fa
 
+    def test_NoLucilleConfig(self):
+        # Distroseries without a lucille configuration get ignored
+        # and trigger a warning, they don't break the publisher
+        logger = BufferLogger()
+        publisher = Publisher(
+            logger, self._config, self._dp, self._archive)
+        self.factory.makeDistroSeries(self._distribution, name="somename")
+        fa = FTPArchiveHandler(logger, self._config, self._dp,
+                               self._distribution, publisher)
+        fa.createEmptyPocketRequests(fullpublish=True)
+        self.assertEquals("WARNING: Distroseries somename in ubuntutest doesn't "
+            "have a lucille configuration.\n", logger.buffer.getvalue())
+
     def test_getSourcesForOverrides(self):
         # getSourcesForOverrides returns a list of tuples containing:
         # (sourcename, suite, component, section)
 
=== modified file 'lib/lp/archiveuploader/tests/__init__.py'
--- lib/lp/archiveuploader/tests/__init__.py	2010-05-04 15:38:08 +0000
+++ lib/lp/archiveuploader/tests/__init__.py	2010-08-12 19:18:45 +0000
@@ -1,6 +1,10 @@
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
+"""Tests for the archive uploader."""
+
+from __future__ import with_statement
+
 __metaclass__ = type
 
 __all__ = ['datadir', 'getPolicy', 'insertFakeChangesFile',
@@ -25,6 +29,7 @@
         raise ValueError("Path is not relative: %s" % path)
     return os.path.join(here, 'data', path)
 
+
 def insertFakeChangesFile(fileID, path=None):
     """Insert a fake changes file into the librarian.
 
@@ -34,11 +39,11 @@
     """
     if path is None:
         path = datadir("ed-0.2-21/ed_0.2-21_source.changes")
-    changes_file_obj = open(path, 'r')
-    test_changes_file = changes_file_obj.read()
-    changes_file_obj.close()
+    with open(path, 'r') as changes_file_obj:
+        test_changes_file = changes_file_obj.read()
     fillLibrarianFile(fileID, content=test_changes_file)
 
+
 def insertFakeChangesFileForAllPackageUploads():
     """Ensure all the PackageUpload records point to a valid changes file."""
     for id in set(pu.changesfile.id for pu in PackageUploadSet()):
@@ -53,6 +58,7 @@
         self.distroseries = distroseries
         self.buildid = buildid
 
+
 def getPolicy(name='anything', distro='ubuntu', distroseries=None,
               buildid=None):
     """Build and return an Upload Policy for the given context."""
 
=== modified file 'lib/lp/archiveuploader/tests/test_buildduploads.py'
--- lib/lp/archiveuploader/tests/test_buildduploads.py	2010-07-18 00:26:33 +0000
+++ lib/lp/archiveuploader/tests/test_buildduploads.py	2010-08-12 19:18:45 +0000
@@ -7,7 +7,6 @@
 
 from lp.archiveuploader.tests.test_securityuploads import (
     TestStagedBinaryUploadBase)
-from lp.archiveuploader.uploadprocessor import UploadProcessor
 from lp.registry.interfaces.pocket import PackagePublishingPocket
 from canonical.database.constants import UTC_NOW
 from canonical.launchpad.interfaces import PackagePublishingStatus
@@ -84,8 +83,8 @@
         """Setup an UploadProcessor instance for a given buildd context."""
         self.options.context = self.policy
         self.options.buildid = str(build_candidate.id)
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(
+            self.layer.txn)
 
     def testDelayedBinaryUpload(self):
         """Check if Soyuz copes with delayed binary uploads.
 
=== modified file 'lib/lp/archiveuploader/tests/test_ppauploadprocessor.py'
--- lib/lp/archiveuploader/tests/test_ppauploadprocessor.py	2010-08-02 02:13:52 +0000
+++ lib/lp/archiveuploader/tests/test_ppauploadprocessor.py	2010-08-12 19:18:45 +0000
@@ -18,7 +18,6 @@
 from zope.security.proxy import removeSecurityProxy
 
 from lp.app.errors import NotFoundError
-from lp.archiveuploader.uploadprocessor import UploadProcessor
 from lp.archiveuploader.tests.test_uploadprocessor import (
     TestUploadProcessorBase)
 from canonical.config import config
@@ -74,8 +73,7 @@
 
         # Set up the uploadprocessor with appropriate options and logger
         self.options.context = 'insecure'
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
     def assertEmail(self, contents=None, recipients=None,
                     ppa_header='name16'):
@@ -1224,8 +1222,7 @@
 
         # Re-initialize uploadprocessor since it depends on the new
         # transaction reset by switchDbUser.
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
     def testPPASizeQuotaSourceRejection(self):
         """Verify the size quota check for PPA uploads.
 
=== modified file 'lib/lp/archiveuploader/tests/test_recipeuploads.py'
--- lib/lp/archiveuploader/tests/test_recipeuploads.py	2010-07-22 15:24:02 +0000
+++ lib/lp/archiveuploader/tests/test_recipeuploads.py	2010-08-12 19:18:45 +0000
@@ -12,7 +12,6 @@
 
 from lp.archiveuploader.tests.test_uploadprocessor import (
     TestUploadProcessorBase)
-from lp.archiveuploader.uploadprocessor import UploadProcessor
 from lp.buildmaster.interfaces.buildbase import BuildStatus
 from lp.code.interfaces.sourcepackagerecipebuild import (
     ISourcePackageRecipeBuildSource)
@@ -42,8 +41,8 @@
         self.options.context = 'recipe'
         self.options.buildid = self.build.id
 
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(
+            self.layer.txn)
 
     def testSetsBuildAndState(self):
         # Ensure that the upload processor correctly links the SPR to
 
=== modified file 'lib/lp/archiveuploader/tests/test_securityuploads.py'
--- lib/lp/archiveuploader/tests/test_securityuploads.py	2010-07-18 00:26:33 +0000
+++ lib/lp/archiveuploader/tests/test_securityuploads.py	2010-08-12 19:18:45 +0000
@@ -11,7 +11,6 @@
 
 from lp.archiveuploader.tests.test_uploadprocessor import (
     TestUploadProcessorBase)
-from lp.archiveuploader.uploadprocessor import UploadProcessor
 from lp.registry.interfaces.pocket import PackagePublishingPocket
 from lp.soyuz.model.binarypackagebuild import BinaryPackageBuild
 from lp.soyuz.model.processor import ProcessorFamily
@@ -70,8 +69,7 @@
         self.options.context = self.policy
         self.options.nomails = self.no_mails
         # Set up the uploadprocessor with appropriate options and logger
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
         self.builds_before_upload = BinaryPackageBuild.select().count()
         self.source_queue = None
         self._uploadSource()
@@ -232,8 +230,7 @@
         """
         build_candidate = self._createBuild('i386')
         self.options.buildid = str(build_candidate.id)
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         build_used = self._uploadBinary('i386')
 
@@ -254,8 +251,7 @@
         """
         build_candidate = self._createBuild('hppa')
         self.options.buildid = str(build_candidate.id)
-        self.uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        self.uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         self.assertRaises(AssertionError, self._uploadBinary, 'i386')
 
 
=== modified file 'lib/lp/archiveuploader/tests/test_uploadprocessor.py'
--- lib/lp/archiveuploader/tests/test_uploadprocessor.py	2010-08-02 02:13:52 +0000
+++ lib/lp/archiveuploader/tests/test_uploadprocessor.py	2010-08-12 19:18:45 +0000
@@ -23,8 +23,10 @@
 from zope.security.proxy import removeSecurityProxy
 
 from lp.app.errors import NotFoundError
-from lp.archiveuploader.uploadpolicy import AbstractUploadPolicy
+from lp.archiveuploader.uploadpolicy import (AbstractUploadPolicy,
+    findPolicyByOptions)
 from lp.archiveuploader.uploadprocessor import UploadProcessor
+from lp.buildmaster.interfaces.buildbase import BuildStatus
 from canonical.config import config
 from canonical.database.constants import UTC_NOW
 from lp.soyuz.model.archivepermission import ArchivePermission
@@ -59,7 +61,7 @@
     ISourcePackageNameSet)
 from lp.services.mail import stub
 from canonical.launchpad.testing.fakepackager import FakePackager
-from lp.testing import TestCaseWithFactory
+from lp.testing import TestCase, TestCaseWithFactory
 from lp.testing.mail_helpers import pop_notifications
 from canonical.launchpad.webapp.errorlog import ErrorReportingUtility
 from canonical.testing import LaunchpadZopelessLayer
@@ -113,7 +115,8 @@
         super(TestUploadProcessorBase, self).setUp()
 
         self.queue_folder = tempfile.mkdtemp()
-        os.makedirs(os.path.join(self.queue_folder, "incoming"))
+        self.incoming_folder = os.path.join(self.queue_folder, "incoming")
+        os.makedirs(self.incoming_folder)
 
         self.test_files_dir = os.path.join(config.root,
             "lib/lp/archiveuploader/tests/data/suite")
@@ -139,6 +142,30 @@
         shutil.rmtree(self.queue_folder)
         super(TestUploadProcessorBase, self).tearDown()
 
+    def getUploadProcessor(self, txn):
+        def getPolicy(distro):
+            self.options.distro = distro.name
+            return findPolicyByOptions(self.options)
+        return UploadProcessor(
+            self.options.base_fsroot, self.options.dryrun,
+            self.options.nomails,
+            self.options.keep, getPolicy, txn, self.log)
+
+    def publishPackage(self, packagename, version, source=True,
+                        archive=None):
+        """Publish a single package that is currently NEW in the queue."""
+        queue_items = self.breezy.getQueueItems(
+            status=PackageUploadStatus.NEW, name=packagename,
+            version=version, exact_match=True, archive=archive)
+        self.assertEqual(queue_items.count(), 1)
+        queue_item = queue_items[0]
+        queue_item.setAccepted()
+        if source:
+            pubrec = queue_item.sources[0].publish(self.log)
+        else:
+            pubrec = queue_item.builds[0].publish(self.log)
+        return pubrec
+
     def assertLogContains(self, line):
         """Assert if a given line is present in the log messages."""
         self.assertTrue(line in self.log.lines,
@@ -208,25 +235,29 @@
             filename, len(content), StringIO(content),
             'application/x-gtar')
 
-    def queueUpload(self, upload_name, relative_path="", test_files_dir=None):
+    def queueUpload(self, upload_name, relative_path="", test_files_dir=None,
+            queue_entry=None):
         """Queue one of our test uploads.
 
-        upload_name is the name of the test upload directory. It is also
+        upload_name is the name of the test upload directory. If there 
+        is no explicit queue entry name specified, it is also
         the name of the queue entry directory we create.
         relative_path is the path to create inside the upload, eg
         ubuntu/~malcc/default. If not specified, defaults to "".
 
         Return the path to the upload queue entry directory created.
         """
+        if queue_entry is None:
+            queue_entry = upload_name
         target_path = os.path.join(
-            self.queue_folder, "incoming", upload_name, relative_path)
+            self.incoming_folder, queue_entry, relative_path)
         if test_files_dir is None:
             test_files_dir = self.test_files_dir
         upload_dir = os.path.join(test_files_dir, upload_name)
         if relative_path:
             os.makedirs(os.path.dirname(target_path))
         shutil.copytree(upload_dir, target_path)
-        return os.path.join(self.queue_folder, "incoming", upload_name)
+        return os.path.join(self.incoming_folder, queue_entry)
 
     def processUpload(self, processor, upload_dir):
         """Process an upload queue entry directory.
@@ -248,8 +279,7 @@
         self.layer.txn.commit()
         if policy is not None:
             self.options.context = policy
-        return UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        return self.getUploadProcessor(self.layer.txn)
 
     def assertEmail(self, contents=None, recipients=None):
         """Check last email content and recipients.
@@ -341,24 +371,9 @@
             "Expected acceptance email not rejection. Actually Got:\n%s"
                 % raw_msg)
 
-    def _publishPackage(self, packagename, version, source=True,
-                        archive=None):
-        """Publish a single package that is currently NEW in the queue."""
-        queue_items = self.breezy.getQueueItems(
-            status=PackageUploadStatus.NEW, name=packagename,
-            version=version, exact_match=True, archive=archive)
-        self.assertEqual(queue_items.count(), 1)
-        queue_item = queue_items[0]
-        queue_item.setAccepted()
-        if source:
-            pubrec = queue_item.sources[0].publish(self.log)
-        else:
-            pubrec = queue_item.builds[0].publish(self.log)
-        return pubrec
-
     def testInstantiate(self):
         """UploadProcessor should instantiate"""
-        up = UploadProcessor(self.options, None, self.log)
+        up = self.getUploadProcessor(None)
 
     def testLocateDirectories(self):
         """Return a sorted list of subdirs in a directory.
@@ -372,7 +387,7 @@
             os.mkdir("%s/dir1" % testdir)
             os.mkdir("%s/dir2" % testdir)
 
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             located_dirs = up.locateDirectories(testdir)
             self.assertEqual(located_dirs, ['dir1', 'dir2', 'dir3'])
         finally:
@@ -390,7 +405,7 @@
             open("%s/2_source.changes" % testdir, "w").close()
             open("%s/3.not_changes" % testdir, "w").close()
 
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             located_files = up.locateChangesFiles(testdir)
             self.assertEqual(
                 located_files, ["2_source.changes", "1.changes"])
@@ -418,7 +433,7 @@
 
             # Move it
             self.options.base_fsroot = testdir
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             up.moveUpload(upload, target_name)
 
             # Check it moved
@@ -439,7 +454,7 @@
 
             # Remove it
             self.options.base_fsroot = testdir
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             up.moveProcessedUpload(upload, "accepted")
 
             # Check it was removed, not moved
@@ -462,7 +477,7 @@
 
             # Move it
             self.options.base_fsroot = testdir
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             up.moveProcessedUpload(upload, "rejected")
 
             # Check it moved
@@ -485,7 +500,7 @@
 
             # Remove it
             self.options.base_fsroot = testdir
-            up = UploadProcessor(self.options, None, self.log)
+            up = self.getUploadProcessor(None)
             up.removeUpload(upload)
 
             # Check it was removed, not moved
@@ -498,7 +513,7 @@
 
     def testOrderFilenames(self):
         """orderFilenames sorts _source.changes ahead of other files."""
-        up = UploadProcessor(self.options, None, self.log)
+        up = self.getUploadProcessor(None)
 
         self.assertEqual(["d_source.changes", "a", "b", "c"],
             up.orderFilenames(["b", "a", "d_source.changes", "c"]))
@@ -522,8 +537,7 @@
         # Register our broken upload policy
         AbstractUploadPolicy._registerPolicy(BrokenUploadPolicy)
         self.options.context = 'broken'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload a package to Breezy.
         upload_dir = self.queueUpload("baz_1.0-1")
@@ -634,7 +648,7 @@
         # Upload 'bar-1.0-1' source and binary to ubuntu/breezy.
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
-        bar_source_pub = self._publishPackage('bar', '1.0-1')
+        bar_source_pub = self.publishPackage('bar', '1.0-1')
         [bar_original_build] = bar_source_pub.createMissingBuilds()
 
         # Move the source from the accepted queue.
@@ -653,7 +667,7 @@
         self.processUpload(uploadprocessor, upload_dir)
         self.assertEqual(
             uploadprocessor.last_processed_upload.is_rejected, False)
-        bar_bin_pubs = self._publishPackage('bar', '1.0-1', source=False)
+        bar_bin_pubs = self.publishPackage('bar', '1.0-1', source=False)
         # Mangle its publishing component to "restricted" so we can check
         # the copy archive ancestry override later.
         restricted = getUtility(IComponentSet)["restricted"]
@@ -746,14 +760,14 @@
         # Upload 'bar-1.0-1' source and binary to ubuntu/breezy.
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
-        bar_source_pub = self._publishPackage('bar', '1.0-1')
+        bar_source_pub = self.publishPackage('bar', '1.0-1')
         [bar_original_build] = bar_source_pub.createMissingBuilds()
 
         self.options.context = 'buildd'
         self.options.buildid = bar_original_build.id
         upload_dir = self.queueUpload("bar_1.0-1_binary")
         self.processUpload(uploadprocessor, upload_dir)
-        [bar_binary_pub] = self._publishPackage("bar", "1.0-1", source=False)
+        [bar_binary_pub] = self.publishPackage("bar", "1.0-1", source=False)
 
         # Prepare ubuntu/breezy-autotest to build sources in i386.
         breezy_autotest = self.ubuntu['breezy-autotest']
@@ -803,7 +817,7 @@
         # Upload 'bar-1.0-1' source and binary to ubuntu/breezy.
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
-        bar_source_old = self._publishPackage('bar', '1.0-1')
+        bar_source_old = self.publishPackage('bar', '1.0-1')
 
         # Upload 'bar-1.0-1' source and binary to ubuntu/breezy.
         upload_dir = self.queueUpload("bar_1.0-2")
@@ -816,7 +830,7 @@
         self.options.buildid = bar_original_build.id
         upload_dir = self.queueUpload("bar_1.0-2_binary")
         self.processUpload(uploadprocessor, upload_dir)
-        [bar_binary_pub] = self._publishPackage("bar", "1.0-2", source=False)
+        [bar_binary_pub] = self.publishPackage("bar", "1.0-2", source=False)
 
         # Create a COPY archive for building in non-virtual builds.
         uploader = getUtility(IPersonSet).getByName('name16')
@@ -971,7 +985,7 @@
         partner_archive = getUtility(IArchiveSet).getByDistroPurpose(
             self.ubuntu, ArchivePurpose.PARTNER)
         self.assertTrue(partner_archive)
-        self._publishPackage("foocomm", "1.0-1", archive=partner_archive)
+        self.publishPackage("foocomm", "1.0-1", archive=partner_archive)
 
         # Check the publishing record's archive and component.
         foocomm_spph = SourcePackagePublishingHistory.selectOneBy(
@@ -1015,7 +1029,7 @@
         self.assertEqual(foocomm_bpr.component.name, 'partner')
 
         # Publish the upload so we can check the publishing record.
-        self._publishPackage("foocomm", "1.0-1", source=False)
+        self.publishPackage("foocomm", "1.0-1", source=False)
 
         # Check the publishing record's archive and component.
         foocomm_bpph = BinaryPackagePublishingHistory.selectOneBy(
@@ -1054,14 +1068,14 @@
         # Accept and publish the upload.
         partner_archive = getUtility(IArchiveSet).getByDistroPurpose(
             self.ubuntu, ArchivePurpose.PARTNER)
-        self._publishPackage("foocomm", "1.0-1", archive=partner_archive)
+        self.publishPackage("foocomm", "1.0-1", archive=partner_archive)
 
         # Now do the same thing with a binary package.
         upload_dir = self.queueUpload("foocomm_1.0-1_binary")
         self.processUpload(uploadprocessor, upload_dir)
 
         # Accept and publish the upload.
-        self._publishPackage("foocomm", "1.0-1", source=False,
+        self.publishPackage("foocomm", "1.0-1", source=False,
                              archive=partner_archive)
 
         # Upload the next source version of the package.
@@ -1105,8 +1119,7 @@
         self.breezy.status = SeriesStatus.CURRENT
         self.layer.txn.commit()
         self.options.context = 'insecure'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload a package for Breezy.
         upload_dir = self.queueUpload("foocomm_1.0-1_proposed")
@@ -1124,8 +1137,7 @@
         self.breezy.status = SeriesStatus.CURRENT
         self.layer.txn.commit()
         self.options.context = 'insecure'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload a package for Breezy.
         upload_dir = self.queueUpload("foocomm_1.0-1")
@@ -1140,8 +1152,7 @@
         pocket and ensure it fails."""
         # Set up the uploadprocessor with appropriate options and logger.
         self.options.context = 'insecure'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload a package for Breezy.
         upload_dir = self.queueUpload("foocomm_1.0-1_updates")
@@ -1302,8 +1313,7 @@
         used.
         That exception will then initiate the creation of an OOPS report.
         """
-        processor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        processor = self.getUploadProcessor(self.layer.txn)
 
         upload_dir = self.queueUpload("foocomm_1.0-1_proposed")
         bogus_changesfile_data = '''
@@ -1346,8 +1356,7 @@
         self.setupBreezy()
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the source first to enable the binary later:
         upload_dir = self.queueUpload("bar_1.0-1_lzma")
@@ -1357,7 +1366,7 @@
         self.assertTrue(
             "rejected" not in raw_msg,
             "Failed to upload bar source:\n%s" % raw_msg)
-        self._publishPackage("bar", "1.0-1")
+        self.publishPackage("bar", "1.0-1")
         # Clear out emails generated during upload.
         ignore = pop_notifications()
 
@@ -1456,15 +1465,14 @@
             permission=ArchivePermissionType.UPLOAD, person=uploader,
             component=restricted)
 
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the first version and accept it to make it known in
         # Ubuntu.  The uploader has rights to upload NEW packages to
         # components that he does not have direct rights to.
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
-        bar_source_pub = self._publishPackage('bar', '1.0-1')
+        bar_source_pub = self.publishPackage('bar', '1.0-1')
         # Clear out emails generated during upload.
         ignore = pop_notifications()
 
@@ -1509,15 +1517,14 @@
             permission=ArchivePermissionType.UPLOAD, person=uploader,
             component=restricted)
 
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the first version and accept it to make it known in
         # Ubuntu.  The uploader has rights to upload NEW packages to
         # components that he does not have direct rights to.
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
-        bar_source_pub = self._publishPackage('bar', '1.0-1')
+        bar_source_pub = self.publishPackage('bar', '1.0-1')
         # Clear out emails generated during upload.
         ignore = pop_notifications()
 
@@ -1590,8 +1597,7 @@
         # with pointer to the Soyuz questions in Launchpad and the
         # reason why the message was sent to the current recipients.
         self.setupBreezy()
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         upload_dir = self.queueUpload("bar_1.0-1", "boing")
         self.processUpload(uploadprocessor, upload_dir)
@@ -1636,8 +1642,7 @@
         self.setupBreezy()
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the source.
         upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
@@ -1655,8 +1660,7 @@
             permitted_formats=[SourcePackageFormat.FORMAT_3_0_QUILT])
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the source.
         upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
@@ -1666,7 +1670,7 @@
         self.assertTrue(
             "rejected" not in raw_msg,
             "Failed to upload bar source:\n%s" % raw_msg)
-        spph = self._publishPackage("bar", "1.0-1")
+        spph = self.publishPackage("bar", "1.0-1")
 
         self.assertEquals(
             sorted((sprf.libraryfile.filename, sprf.filetype)
@@ -1689,8 +1693,7 @@
             permitted_formats=[SourcePackageFormat.FORMAT_3_0_QUILT])
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the first source.
         upload_dir = self.queueUpload("bar_1.0-1_3.0-quilt")
@@ -1700,7 +1703,7 @@
         self.assertTrue(
             "rejected" not in raw_msg,
             "Failed to upload bar source:\n%s" % raw_msg)
-        spph = self._publishPackage("bar", "1.0-1")
+        spph = self.publishPackage("bar", "1.0-1")
 
         # Upload another source sharing the same (component) orig.
         upload_dir = self.queueUpload("bar_1.0-2_3.0-quilt_without_orig")
@@ -1728,8 +1731,7 @@
             permitted_formats=[SourcePackageFormat.FORMAT_3_0_NATIVE])
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the source.
         upload_dir = self.queueUpload("bar_1.0_3.0-native")
@@ -1739,7 +1741,7 @@
         self.assertTrue(
             "rejected" not in raw_msg,
             "Failed to upload bar source:\n%s" % raw_msg)
-        spph = self._publishPackage("bar", "1.0")
+        spph = self.publishPackage("bar", "1.0")
 
         self.assertEquals(
             sorted((sprf.libraryfile.filename, sprf.filetype)
@@ -1754,8 +1756,7 @@
         self.setupBreezy()
         self.layer.txn.commit()
         self.options.context = 'absolutely-anything'
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         # Upload the source.
         upload_dir = self.queueUpload("bar_1.0-1_1.0-bzip2")
@@ -1772,8 +1773,7 @@
         self.setupBreezy()
         breezy = self.ubuntu['breezy']
         breezy.status = SeriesStatus.CURRENT
-        uploadprocessor = UploadProcessor(
-            self.options, self.layer.txn, self.log)
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
 
         upload_dir = self.queueUpload("bar_1.0-1")
         self.processUpload(uploadprocessor, upload_dir)
 
=== modified file 'lib/lp/archiveuploader/uploadprocessor.py'
--- lib/lp/archiveuploader/uploadprocessor.py	2010-08-04 00:30:56 +0000
+++ lib/lp/archiveuploader/uploadprocessor.py	2010-08-12 19:18:45 +0000
@@ -60,7 +60,7 @@
 from lp.archiveuploader.nascentupload import (
     NascentUpload, FatalUploadError, EarlyReturnUploadError)
 from lp.archiveuploader.uploadpolicy import (
-    findPolicyByOptions, UploadPolicyError)
+    UploadPolicyError)
 from lp.soyuz.interfaces.archive import IArchiveSet, NoSuchPPA
 from lp.registry.interfaces.distribution import IDistributionSet
 from lp.registry.interfaces.person import IPersonSet
@@ -108,16 +108,33 @@
 class UploadProcessor:
     """Responsible for processing uploads. See module docstring."""
 
-    def __init__(self, options, ztm, log):
-        self.options = options
+    def __init__(self, base_fsroot, dry_run, no_mails, keep, policy_for_distro,
+                 ztm, log):
+        """Create a new upload processor.
+
+        :param base_fsroot: Root path for queue to use
+        :param dry_run: Run but don't commit changes to database
+        :param no_mails: Don't send out any emails
+        :param builds: Interpret leaf names as build ids
+        :param keep: Leave the files in place, don't move them away
+        :param policy_for_distro: callback to obtain Policy object for a
+            distribution
+        :param ztm: Database transaction to use
+        :param log: Logger to use for reporting
+        """
+        self.base_fsroot = base_fsroot
+        self.dry_run = dry_run
+        self.keep = keep
+        self.last_processed_upload = None
+        self.log = log
+        self.no_mails = no_mails
+        self._getPolicyForDistro = policy_for_distro
         self.ztm = ztm
-        self.log = log
-        self.last_processed_upload = None
 
-    def processUploadQueue(self):
+    def processUploadQueue(self, leaf_name=None):
         """Search for uploads, and process them.
 
-	Uploads are searched for in the 'incoming' directory inside the
+        Uploads are searched for in the 'incoming' directory inside the
         base_fsroot.
 
         This method also creates the 'incoming', 'accepted', 'rejected', and
@@ -127,19 +144,22 @@
             self.log.debug("Beginning processing")
 
             for subdir in ["incoming", "accepted", "rejected", "failed"]:
-                full_subdir = os.path.join(self.options.base_fsroot, subdir)
+                full_subdir = os.path.join(self.base_fsroot, subdir)
                 if not os.path.exists(full_subdir):
                     self.log.debug("Creating directory %s" % full_subdir)
                     os.mkdir(full_subdir)
 
-            fsroot = os.path.join(self.options.base_fsroot, "incoming")
+            fsroot = os.path.join(self.base_fsroot, "incoming")
             uploads_to_process = self.locateDirectories(fsroot)
             self.log.debug("Checked in %s, found %s"
                            % (fsroot, uploads_to_process))
             for upload in uploads_to_process:
                 self.log.debug("Considering upload %s" % upload)
+                if leaf_name is not None and upload != leaf_name:
+                    self.log.debug("Skipping %s -- does not match %s" % (
+                        upload, leaf_name))
+                    continue
                 self.processUpload(fsroot, upload)
-
         finally:
             self.log.debug("Rolling back any remaining transactions.")
             self.ztm.abort()
@@ -152,16 +172,7 @@
         is 'failed', otherwise it is the worst of the results from the
         individual changes files, in order 'failed', 'rejected', 'accepted'.
 
-        If the leafname option is set but its value is not the same as the
-        name of the upload directory, skip it entirely.
-
         """
-        if (self.options.leafname is not None and
-            upload != self.options.leafname):
-            self.log.debug("Skipping %s -- does not match %s" % (
-                upload, self.options.leafname))
-            return
-
         upload_path = os.path.join(fsroot, upload)
         changes_files = self.locateChangesFiles(upload_path)
 
@@ -242,7 +253,7 @@
             # Skip lockfile deletion, see similar code in lp.poppy.hooks.
             fsroot_lock.release(skip_delete=True)
 
-        sorted_dir_names =  sorted(
+        sorted_dir_names = sorted(
             dir_name
             for dir_name in dir_names
             if os.path.isdir(os.path.join(fsroot, dir_name)))
@@ -321,8 +332,7 @@
                          "https://help.launchpad.net/Packaging/PPA#Uploading "
                          "and update your configuration.")))
         self.log.debug("Finding fresh policy")
-        self.options.distro = distribution.name
-        policy = findPolicyByOptions(self.options)
+        policy = self._getPolicyForDistro(distribution)
         policy.archive = archive
 
         # DistroSeries overriding respect the following precedence:
@@ -396,7 +406,7 @@
             # when transaction is committed) this will cause any emails sent
             # sent by do_reject to be lost.
             notify = True
-            if self.options.dryrun or self.options.nomails:
+            if self.dry_run or self.no_mails:
                 notify = False
             if upload.is_rejected:
                 result = UploadStatusEnum.REJECTED
@@ -415,7 +425,7 @@
                 for msg in upload.rejections:
                     self.log.warn("\t%s" % msg)
 
-            if self.options.dryrun:
+            if self.dry_run:
                 self.log.info("Dry run, aborting transaction.")
                 self.ztm.abort()
             else:
@@ -434,7 +444,7 @@
         This includes moving the given upload directory and moving the
         matching .distro file, if it exists.
         """
-        if self.options.keep or self.options.dryrun:
+        if self.keep or self.dry_run:
             self.log.debug("Keeping contents untouched")
             return
 
@@ -462,21 +472,21 @@
         This includes moving the given upload directory and moving the
         matching .distro file, if it exists.
         """
-        if self.options.keep or self.options.dryrun:
+        if self.keep or self.dry_run:
             self.log.debug("Keeping contents untouched")
             return
 
         pathname = os.path.basename(upload)
 
         target_path = os.path.join(
-            self.options.base_fsroot, subdir_name, pathname)
+            self.base_fsroot, subdir_name, pathname)
         self.log.debug("Moving upload directory %s to %s" %
             (upload, target_path))
         shutil.move(upload, target_path)
 
         distro_filename = upload + ".distro"
         if os.path.isfile(distro_filename):
-            target_path = os.path.join(self.options.base_fsroot, subdir_name,
+            target_path = os.path.join(self.base_fsroot, subdir_name,
                                        os.path.basename(distro_filename))
             self.log.debug("Moving distro file %s to %s" % (distro_filename,
                                                             target_path))
 
=== modified file 'lib/lp/buildmaster/model/buildfarmjob.py'
--- lib/lp/buildmaster/model/buildfarmjob.py	2010-06-16 16:01:08 +0000
+++ lib/lp/buildmaster/model/buildfarmjob.py	2010-08-12 19:18:45 +0000
@@ -16,7 +16,7 @@
 
 import pytz
 
-from storm.expr import And, Coalesce, Desc, Join, LeftJoin, Select
+from storm.expr import Coalesce, Desc, LeftJoin, Or
 from storm.locals import Bool, DateTime, Int, Reference, Storm
 from storm.store import Store
 
@@ -365,49 +365,38 @@
         # Currently only package builds can be private (via their
         # related archive), but not all build farm jobs will have a
         # related package build - hence the left join.
-        left_join_pkg_builds = LeftJoin(
-            BuildFarmJob,
-            Join(
+        origin = [BuildFarmJob]
+        left_join_archive = [
+            LeftJoin(
                 PackageBuild,
-                Archive,
-                And(PackageBuild.archive == Archive.id)),
-            PackageBuild.build_farm_job == BuildFarmJob.id)
-
-        filtered_builds = IStore(BuildFarmJob).using(
-            left_join_pkg_builds).find(BuildFarmJob, *extra_clauses)
+                PackageBuild.build_farm_job == BuildFarmJob.id),
+            LeftJoin(
+                Archive, PackageBuild.archive == Archive.id),
+            ]
 
         if user is None:
             # Anonymous requests don't get to see private builds at all.
-            filtered_builds = filtered_builds.find(
-                Coalesce(Archive.private, False) == False)
+            origin.extend(left_join_archive)
+            extra_clauses.append(Coalesce(Archive.private, False) == False)
 
         elif user.inTeam(getUtility(ILaunchpadCelebrities).admin):
             # Admins get to see everything.
             pass
         else:
-            # Everyone else sees a union of all public builds and the
+            # Everyone else sees all public builds and the
             # specific private builds to which they have access.
-            filtered_builds = filtered_builds.find(
-                Coalesce(Archive.private, False) == False)
-
-            user_teams_subselect = Select(
-                TeamParticipation.teamID,
-                where=And(
-                   TeamParticipation.personID == user.id,
-                   TeamParticipation.teamID == Archive.ownerID))
-            private_builds_for_user = IStore(BuildFarmJob).find(
-                BuildFarmJob,
-                PackageBuild.build_farm_job == BuildFarmJob.id,
-                PackageBuild.archive == Archive.id,
-                Archive.private == True,
-                Archive.ownerID.is_in(user_teams_subselect),
-                *extra_clauses)
-
-            filtered_builds = filtered_builds.union(
-                private_builds_for_user)
+            origin.extend(left_join_archive)
+            origin.append(LeftJoin(
+                TeamParticipation, TeamParticipation.teamID == Archive.ownerID))
+            extra_clauses.append(
+                Or(Coalesce(Archive.private, False) == False,
+                   TeamParticipation.person == user))
+
+        filtered_builds = IStore(BuildFarmJob).using(*origin).find(
+            BuildFarmJob, *extra_clauses)
 
         filtered_builds.order_by(
             Desc(BuildFarmJob.date_finished), BuildFarmJob.id)
+        filtered_builds.config(distinct=True)
 
         return filtered_builds
-
 
=== modified file 'lib/lp/soyuz/scripts/soyuz_process_upload.py'
--- lib/lp/soyuz/scripts/soyuz_process_upload.py	2010-05-04 15:38:08 +0000
+++ lib/lp/soyuz/scripts/soyuz_process_upload.py	2010-08-12 19:18:45 +0000
@@ -8,6 +8,7 @@
 
 import os
 
+from lp.archiveuploader.uploadpolicy import findPolicyByOptions
 from lp.archiveuploader.uploadprocessor import UploadProcessor
 from lp.services.scripts.base import (
     LaunchpadCronScript, LaunchpadScriptFailure)
@@ -74,8 +75,13 @@
                 "%s is not a directory" % self.options.base_fsroot)
 
         self.logger.debug("Initialising connection.")
-        UploadProcessor(
-            self.options, self.txn, self.logger).processUploadQueue()
+        def getPolicy(distro):
+            self.options.distro = distro.name
+            return findPolicyByOptions(self.options)
+        processor = UploadProcessor(self.options.base_fsroot, 
+            self.options.dryrun, self.options.nomails, self.options.keep,
+            getPolicy, self.txn, self.logger)
+        processor.processUploadQueue(self.options.leafname)
 
     @property
     def lockfilename(self):
 
=== modified file 'utilities/sourcedeps.conf'
--- utilities/sourcedeps.conf	2010-08-03 14:59:22 +0000
+++ utilities/sourcedeps.conf	2010-08-12 19:18:45 +0000
@@ -12,5 +12,6 @@
 pygettextpo lp:~launchpad-pqm/pygettextpo/trunk;revno=24
 pygpgme lp:~launchpad-pqm/pygpgme/devel;revno=49
 subvertpy lp:~launchpad-pqm/subvertpy/trunk;revno=2042
+python-debian lp:~launchpad-pqm/python-debian/devel;revno=185
 testresources lp:~launchpad-pqm/testresources/dev;revno=16
 shipit lp:~launchpad-pqm/shipit/trunk;revno=8909 optional