Merge lp:~julian-edwards/launchpad/publisher-config-db-schema into lp:launchpad

[Julian Edwards]

= Summary =
New PublisherConfig table

== Proposed fix ==
This branch adds a new PublisherConfig table which will eventually deprecate
the archivepublisher config section.

The schema allows for separate configurations for each hosted distribution
which is required as part of the Derived Distributions feature. When we start
hosting multiple distributions, the single configuration that currently exists
for the purposes of publishing Ubuntu will not suffice. The configured
options are:

 * The base path for the archive
 * The URL to the archive

It's entirely possible that custom distros will be hosted in an entirely
different disk area to Ubuntu, so we need to retain this configurability for
each distro.

The intention is to add a trivial LaunchpadEditForm page after this lands so
that we can set up the data, fix the rest of the code to use it, and delete
the original config.

== Implementation details ==
Fairly trivial schema change plus new model code.

== Tests ==
bin/test -cvv test_publisherconfig

== Demo and Q/A ==
n/a yet

[Benji York (benji)]

This looks good.

The only thing I thought you might want to know was that the imports in
this section of lib/lp/archivepublisher/model/publisherconfig.py aren't
sorted:

    from storm.locals import (
        Int,
        Reference,
        Storm,
        RawStr,
        )

review: Approve (code)

[Julian Edwards (julian-edwards)]

Benji, thanks to "bzr send" this MP is targeted to devel instead of db-devel.... I'm going to file a new one if you wouldn't mind blessing it!

review: Needs Resubmitting

=== modified file 'database/schema/launchpad_session.sql'
--- database/schema/launchpad_session.sql	2010-09-10 09:45:45 +0000
+++ database/schema/launchpad_session.sql	2011-03-07 15:37:44 +0000
@@ -29,3 +29,28 @@
 GRANT SELECT, INSERT, UPDATE, DELETE ON TimeLimitedToken TO session;
 -- And the garbo needs to run on it too.
 GRANT SELECT, DELETE ON TimeLimitedToken TO session;
+
+
+-- This helper needs to exist in the session database so the BulkPruner
+-- can clean up unwanted sessions.
+CREATE OR REPLACE FUNCTION cursor_fetch(cur refcursor, n integer)
+RETURNS SETOF record LANGUAGE plpgsql AS
+$$
+DECLARE
+    r record;
+    count integer;
+BEGIN
+    FOR count IN 1..n LOOP
+        FETCH FORWARD FROM cur INTO r;
+        IF NOT FOUND THEN
+            RETURN;
+        END IF;
+        RETURN NEXT r;
+    END LOOP;
+END;
+$$;
+
+COMMENT ON FUNCTION cursor_fetch(refcursor, integer) IS
+'Fetch the next n items from a cursor. Work around for not being able to use FETCH inside a SELECT statement.';
+
+GRANT EXECUTE ON FUNCTION cursor_fetch(refcursor, integer) TO session;

=== added file 'database/schema/patch-2208-99-0.sql'
--- database/schema/patch-2208-99-0.sql	1970-01-01 00:00:00 +0000
+++ database/schema/patch-2208-99-0.sql	2011-03-07 15:37:44 +0000
@@ -0,0 +1,14 @@
+SET client_min_messages=ERROR;
+
+CREATE TABLE PublisherConfig (
+    id serial PRIMARY KEY,
+    distribution integer NOT NULL CONSTRAINT publisherconfig__distribution__fk REFERENCES distribution,
+    root_dir text NOT NULL,
+    base_url text NOT NULL,
+    copy_base_url text NOT NULL
+);
+    
+CREATE UNIQUE INDEX publisherconfig__distribution__idx
+    ON PublisherConfig(distribution);
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2208, 99, 0);

=== modified file 'database/schema/security.cfg'
--- database/schema/security.cfg	2011-03-07 07:27:56 +0000
+++ database/schema/security.cfg	2011-03-07 15:37:44 +0000
@@ -260,6 +260,7 @@
 public.productrelease                   = SELECT, INSERT, UPDATE, DELETE
 public.productreleasefile               = SELECT, INSERT, DELETE
 public.productseriescodeimport          = SELECT, INSERT, UPDATE
+public.publisherconfig                  = SELECT, INSERT, UPDATE, DELETE
 public.project                          = SELECT
 public.projectbounty                    = SELECT, INSERT, UPDATE
 public.questionbug                      = SELECT, INSERT, DELETE

=== modified file 'lib/lp/archivepublisher/config.py'
--- lib/lp/archivepublisher/config.py	2010-10-17 13:35:20 +0000
+++ lib/lp/archivepublisher/config.py	2011-03-07 15:37:44 +0000
@@ -83,10 +83,6 @@
     return pubconf
 
 
-class LucilleConfigError(Exception):
-    """Lucille configuration was not present."""
-
-
 class Config(object):
     """Manage a publisher configuration from the database. (Read Only)
     This class provides a useful abstraction so that if we change

=== modified file 'lib/lp/archivepublisher/deathrow.py'
--- lib/lp/archivepublisher/deathrow.py	2010-10-17 13:35:20 +0000
+++ lib/lp/archivepublisher/deathrow.py	2011-03-07 15:37:44 +0000
@@ -17,7 +17,6 @@
 from lp.archivepublisher import ELIGIBLE_DOMINATION_STATES
 from lp.archivepublisher.config import (
     getPubConfig,
-    LucilleConfigError,
     )
 from lp.archivepublisher.diskpool import DiskPool
 from lp.archivepublisher.utils import process_in_batches
@@ -40,12 +39,8 @@
          the one provided by the publishing-configuration, it will be only
          used for PRIMARY archives.
     """
-    log.debug("Grab Lucille config.")
-    try:
-        pubconf = getPubConfig(archive)
-    except LucilleConfigError, info:
-        log.error(info)
-        raise
+    log.debug("Grab publisher config.")
+    pubconf = getPubConfig(archive)
 
     if (pool_root_override is not None and
         archive.purpose == ArchivePurpose.PRIMARY):

=== added file 'lib/lp/archivepublisher/interfaces/publisherconfig.py'
--- lib/lp/archivepublisher/interfaces/publisherconfig.py	1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/interfaces/publisherconfig.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,58 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+# pylint: disable-msg=E0211,E0213
+
+"""PublisherConfig interface."""
+
+__metaclass__ = type
+
+__all__ = [
+    'IPublisherConfig',
+    'IPublisherConfigSet',
+    ]
+
+from lazr.restful.fields import Reference
+from zope.interface import Interface
+from zope.schema import (
+    Int,
+    TextLine,
+    )
+
+from canonical.launchpad import _
+from lp.registry.interfaces.distribution import IDistribution
+
+
+class IPublisherConfig(Interface):
+    """`PublisherConfig` interface."""
+
+    id = Int(title=_('ID'), required=True, readonly=True)
+
+    distribution = Reference(
+        IDistribution, title=_("Distribution"), required=True,
+        description=_("The Distribution for this configuration."))
+
+    root_dir = TextLine(
+        title=_("Root Directory"), required=True,
+        description=_("The root directory for published archives."))
+
+    base_url = TextLine(
+        title=_("Base URL"), required=True,
+        description=_("The base URL for published archives"))
+
+    copy_base_url = TextLine(
+        title=_("Copy Base URL"), required=True,
+        description=_("The base URL for published copy archives"))
+
+
+class IPublisherConfigSet(Interface):
+    """`PublisherConfigSet` interface."""
+
+    def new(distribution, root_dir, base_url, copy_base_url):
+        """Create a new `PublisherConfig`."""
+
+    def getByDistribution(distribution):
+        """Get the config for a a distribution.
+
+        :param distribution: An `IDistribution`
+        """

=== added file 'lib/lp/archivepublisher/model/publisherconfig.py'
--- lib/lp/archivepublisher/model/publisherconfig.py	1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/model/publisherconfig.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,68 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Database class for table PublisherConfig."""
+
+__metaclass__ = type
+
+__all__ = [
+    'PublisherConfig',
+    'PublisherConfigSet',
+    ]
+
+from storm.locals import (
+    Int,
+    RawStr,
+    Reference,
+    Storm,
+    )
+from zope.interface import implements
+
+from canonical.launchpad.interfaces.lpstorm import (
+    IMasterStore,
+    )
+from lp.archivepublisher.interfaces.publisherconfig import (
+    IPublisherConfig,
+    IPublisherConfigSet,
+    )
+
+
+class PublisherConfig(Storm):
+    """See `IArchiveAuthToken`."""
+    implements(IPublisherConfig)
+    __storm_table__ = 'PublisherConfig'
+
+    id = Int(primary=True)
+
+    distribution_id = Int(name='distribution', allow_none=False)
+    distribution = Reference(distribution_id, 'Distribution.id')
+
+    root_dir = RawStr(name='root_dir', allow_none=False)
+
+    base_url = RawStr(name='base_url', allow_none=False)
+
+    copy_base_url = RawStr(name='copy_base_url', allow_none=False)
+
+
+class PublisherConfigSet:
+    """See `IPublisherConfigSet`."""
+    implements(IPublisherConfigSet)
+    title = "Soyuz Publisher Configurations"
+
+    def new(self, distribution, root_dir, base_url, copy_base_url):
+        """Make and return a new `PublisherConfig`."""
+        store = IMasterStore(PublisherConfig)
+        pubconf = PublisherConfig()
+        pubconf.distribution = distribution
+        pubconf.root_dir = root_dir
+        pubconf.base_url = base_url
+        pubconf.copy_base_url = copy_base_url
+        store.add(pubconf)
+        return pubconf
+
+    def getByDistribution(self, distribution):
+        """See `IArchiveAuthTokenSet`."""
+        store = IMasterStore(PublisherConfig)
+        return store.find(
+            PublisherConfig,
+            PublisherConfig.distribution_id == distribution.id).one()

=== modified file 'lib/lp/archivepublisher/publishing.py'
--- lib/lp/archivepublisher/publishing.py	2011-02-04 09:07:36 +0000
+++ lib/lp/archivepublisher/publishing.py	2011-03-07 15:37:44 +0000
@@ -21,7 +21,6 @@
 from lp.archivepublisher import HARDCODED_COMPONENT_ORDER
 from lp.archivepublisher.config import (
     getPubConfig,
-    LucilleConfigError,
     )
 from lp.archivepublisher.diskpool import DiskPool
 from lp.archivepublisher.domination import Dominator
@@ -120,11 +119,7 @@
     else:
         log.debug("Finding configuration for '%s' PPA."
                   % archive.owner.name)
-    try:
-        pubconf = getPubConfig(archive)
-    except LucilleConfigError, info:
-        log.error(info)
-        raise
+    pubconf = getPubConfig(archive)
 
     disk_pool = _getDiskPool(pubconf, log)
 

=== added file 'lib/lp/archivepublisher/tests/test_publisherconfig.py'
--- lib/lp/archivepublisher/tests/test_publisherconfig.py	1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/tests/test_publisherconfig.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,66 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for publisherConfig model class."""
+
+__metaclass__ = type
+
+
+from storm.store import Store
+from storm.exceptions import IntegrityError
+from zope.component import getUtility
+from zope.interface.verify import verifyObject
+
+from canonical.testing.layers import ZopelessDatabaseLayer
+from lp.archivepublisher.interfaces.publisherconfig import (
+    IPublisherConfig,
+    IPublisherConfigSet,
+    )
+from lp.testing import TestCaseWithFactory
+
+
+class TestPublisherConfig(TestCaseWithFactory):
+    """Test the `PublisherConfig` model."""
+    layer = ZopelessDatabaseLayer
+
+    def setUp(self):
+        TestCaseWithFactory.setUp(self)
+        self.distribution = self.factory.makeDistribution(name='conftest')
+
+    def test_verify_interface(self):
+        # Test the interface for the model.
+        pubconf = self.factory.makePublisherConfig()
+        verified = verifyObject(IPublisherConfig, pubconf)
+        self.assertTrue(verified)
+
+    def test_properties(self):
+        # Test the model properties.
+        ROOT_DIR = "rootdir/test"
+        BASE_URL = "http://base.url"
+        COPY_BASE_URL = "http://base.url"
+        pubconf = self.factory.makePublisherConfig(
+            distribution=self.distribution,
+            root_dir=ROOT_DIR,
+            base_url=BASE_URL,
+            copy_base_url=COPY_BASE_URL,
+            )
+
+        self.assertEqual(self.distribution.name, pubconf.distribution.name)
+        self.assertEqual(ROOT_DIR, pubconf.root_dir)
+        self.assertEqual(BASE_URL, pubconf.base_url)
+        self.assertEqual(COPY_BASE_URL, pubconf.copy_base_url)
+
+    def test_one_config_per_distro(self):
+        # Only one config for each distro is allowed.
+        pubconf = self.factory.makePublisherConfig(self.distribution)
+        pubconf2 = self.factory.makePublisherConfig(self.distribution)
+        store = Store.of(pubconf)
+        self.assertRaises(IntegrityError, store.flush)
+
+    def test_getByDistribution(self):
+        # Test that IPublisherConfigSet.getByDistribution works.
+        pubconf = self.factory.makePublisherConfig(
+            distribution=self.distribution)
+        pubconf = getUtility(IPublisherConfigSet).getByDistribution(
+            self.distribution)
+        self.assertEqual(self.distribution.name, pubconf.distribution.name)

=== modified file 'lib/lp/archivepublisher/zcml/configure.zcml'
--- lib/lp/archivepublisher/zcml/configure.zcml	2009-07-13 18:15:02 +0000
+++ lib/lp/archivepublisher/zcml/configure.zcml	2011-03-07 15:37:44 +0000
@@ -2,8 +2,30 @@
      GNU Affero General Public License version 3 (see the file LICENSE).
 -->
 
-<configure xmlns="http://namespaces.zope.org/zope">
+<configure
+    xmlns="http://namespaces.zope.org/zope"
+    xmlns:browser="http://namespaces.zope.org/browser"
+    xmlns:i18n="http://namespaces.zope.org/i18n"
+    xmlns:webservice="http://namespaces.canonical.com/webservice"
+    xmlns:xmlrpc="http://namespaces.zope.org/xmlrpc"
+    i18n_domain="launchpad">
+
     <include package="lp.archivepublisher.zcml"
              file="archivesigningkey.zcml" />
+
+    <securedutility
+        class="lp.archivepublisher.model.publisherconfig.PublisherConfigSet"
+        provides="lp.archivepublisher.interfaces.publisherconfig.IPublisherConfigSet">
+        <allow
+            interface="lp.archivepublisher.interfaces.publisherconfig.IPublisherConfigSet"/>
+    </securedutility>
+
+    <class
+        class="lp.archivepublisher.model.publisherconfig.PublisherConfig">
+        <allow
+            interface="lp.archivepublisher.interfaces.publisherconfig.IPublisherConfig" />
+    </class>
+
+
 </configure>
 

=== modified file 'lib/lp/bugs/configure.zcml'
--- lib/lp/bugs/configure.zcml	2011-03-02 23:08:54 +0000
+++ lib/lp/bugs/configure.zcml	2011-03-07 15:37:44 +0000
@@ -1056,6 +1056,10 @@
         class="lp.bugs.mail.bugnotificationrecipients.BugNotificationRecipients">
         <allow
             interface="canonical.launchpad.interfaces.launchpad.INotificationRecipientSet"/>
+        <!-- BugNotificationRecipients provides the following
+             attributes/methods in addition. -->
+        <allow
+            attributes="subscription_filters addFilter"/>
     </class>
     <securedutility
         provides="lp.bugs.interfaces.bugnotification.IBugNotificationSet"

=== modified file 'lib/lp/bugs/doc/bugnotification-sending.txt'
--- lib/lp/bugs/doc/bugnotification-sending.txt	2011-02-22 10:44:48 +0000
+++ lib/lp/bugs/doc/bugnotification-sending.txt	2011-03-07 15:37:44 +0000
@@ -21,8 +21,10 @@
 
     >>> def print_notification_headers(email_notification):
     ...     for header in ['To', 'From', 'Subject',
-    ...                    'X-Launchpad-Message-Rationale']:
-    ...         print "%s: %s" % (header, email_notification[header])
+    ...                    'X-Launchpad-Message-Rationale',
+    ...                    'X-Launchpad-Subscription-Filter']:
+    ...         if email_notification[header]:
+    ...             print "%s: %s" % (header, email_notification[header])
 
     >>> def print_notification(email_notification):
     ...     print_notification_headers(email_notification)
@@ -676,7 +678,6 @@
 been set properly.
 
     >>> from lp.bugs.model.bugnotification import BugNotification
-    >>> from lp.bugs.enum import BugNotificationStatus
     >>> for notification in BugNotification.select(orderBy='id')[-8:]:
     ...     if notification.is_comment:
     ...         identifier = 'comment'
@@ -1247,6 +1248,7 @@
     >>> with lp_dbuser():
     ...     filter = subscription_no_priv.newBugFilter()
     ...     filter.bug_notification_level = BugNotificationLevel.COMMENTS
+    ...     filter.description = u"Allow-comments filter"
 
     >>> comment = getUtility(IMessageSet).fromText(
     ...     'subject', 'another comment.', sample_person,
@@ -1279,12 +1281,14 @@
     From: Sample Person <[e-mail]>
     Subject: [Bug 1] subject
     X-Launchpad-Message-Rationale: Subscriber (Mozilla Firefox)
+    X-Launchpad-Subscription-Filter: Allow-comments filter
     <BLANKLINE>
     another comment.
     <BLANKLINE>
     --
     You received this bug notification because you are subscribed to Mozilla
     Firefox.
+    Matching filters: Allow-comments filter
     ...
     ----------------------------------------------------------------------
     To: [e-mail]
@@ -1390,6 +1394,7 @@
     From: Sample Person <[e-mail]>
     Subject: [Bug 1] subject
     X-Launchpad-Message-Rationale: Subscriber (Mozilla Firefox)
+    X-Launchpad-Subscription-Filter: Allow-comments filter
     <BLANKLINE>
     no comment for no-priv.
     <BLANKLINE>
@@ -1400,6 +1405,7 @@
     --
     You received this bug notification because you are subscribed to Mozilla
     Firefox.
+    Matching filters: Allow-comments filter
     ...
     ----------------------------------------------------------------------
     To: [e-mail]

=== modified file 'lib/lp/bugs/enum.py'
--- lib/lp/bugs/enum.py	2011-03-05 00:06:26 +0000
+++ lib/lp/bugs/enum.py	2011-03-07 15:37:44 +0000
@@ -57,18 +57,18 @@
 
 class BugNotificationStatus(DBEnumeratedType):
     """The status of a bug notification.
-    
+
     A notification may be pending, sent, or omitted."""
 
     PENDING = DBItem(10, """
         Pending
-        
+
         The notification has not yet been sent.
         """)
 
     OMITTED = DBItem(20, """
         Omitted
-        
+
         The system considered sending the notification, but omitted it.
         This is generally because the action reported by the notification
         was immediately undone.
@@ -76,6 +76,6 @@
 
     SENT = DBItem(30, """
         Sent
-        
+
         The notification has been sent.
         """)

=== modified file 'lib/lp/bugs/mail/bugnotificationrecipients.py'
--- lib/lp/bugs/mail/bugnotificationrecipients.py	2011-03-02 14:07:23 +0000
+++ lib/lp/bugs/mail/bugnotificationrecipients.py	2011-03-07 15:37:44 +0000
@@ -58,6 +58,7 @@
         """
         NotificationRecipientSet.__init__(self)
         self.duplicateof = duplicateof
+        self.subscription_filters = set()
 
     def _addReason(self, person, reason, header):
         """Adds a reason (text and header) for a person.
@@ -127,3 +128,13 @@
         else:
             text = "are the registrant for %s" % upstream.displayname
         self._addReason(person, text, reason)
+
+    def update(self, recipient_set):
+        """See `INotificationRecipientSet`."""
+        super(BugNotificationRecipients, self).update(recipient_set)
+        self.subscription_filters.update(
+            recipient_set.subscription_filters)
+
+    def addFilter(self, subscription_filter):
+        if subscription_filter is not None:
+            self.subscription_filters.add(subscription_filter)

=== modified file 'lib/lp/bugs/model/bugnotification.py'
--- lib/lp/bugs/model/bugnotification.py	2011-02-25 16:19:58 +0000
+++ lib/lp/bugs/model/bugnotification.py	2011-03-07 15:37:44 +0000
@@ -150,12 +150,23 @@
             reason_body, reason_header = recipients.getReason(recipient)
             sql_values.append('(%s, %s, %s, %s)' % sqlvalues(
                 bug_notification, recipient, reason_header, reason_body))
+
         # We add all the recipients in a single SQL statement to make
         # this a bit more efficient for bugs with many subscribers.
         store.execute("""
             INSERT INTO BugNotificationRecipient
               (bug_notification, person, reason_header, reason_body)
             VALUES %s;""" % ', '.join(sql_values))
+
+        if len(recipients.subscription_filters) > 0:
+            filter_link_sql = [
+                "(%s, %s)" % sqlvalues(bug_notification, filter.id)
+                for filter in recipients.subscription_filters]
+            store.execute("""
+                INSERT INTO BugNotificationFilter
+                  (bug_notification, bug_subscription_filter)
+                VALUES %s;""" % ", ".join(filter_link_sql))
+
         return bug_notification
 
 

=== modified file 'lib/lp/bugs/model/structuralsubscription.py'
--- lib/lp/bugs/model/structuralsubscription.py	2011-03-04 02:29:09 +0000
+++ lib/lp/bugs/model/structuralsubscription.py	2011-03-07 15:37:44 +0000
@@ -585,14 +585,15 @@
     else:
         subscribers = []
         query_results = source.find(
-            (Person, StructuralSubscription),
-            *constraints).config(distinct=True)
-        for person, subscription in query_results:
+            (Person, StructuralSubscription, BugSubscriptionFilter),
+            *constraints)
+        for person, subscription, filter in query_results:
             # Set up results.
             if person not in recipients:
                 subscribers.append(person)
                 recipients.addStructuralSubscriber(
                     person, subscription.target)
+            recipients.addFilter(filter)
         return subscribers
 
 

=== modified file 'lib/lp/scripts/garbo.py'
--- lib/lp/scripts/garbo.py	2011-02-23 10:28:53 +0000
+++ lib/lp/scripts/garbo.py	2011-03-07 15:37:44 +0000
@@ -74,6 +74,7 @@
     LaunchpadCronScript,
     SilentLaunchpadScriptFailure,
     )
+from lp.services.session.model import SessionData
 from lp.translations.interfaces.potemplate import IPOTemplateSet
 from lp.translations.model.potranslation import POTranslation
 
@@ -107,10 +108,14 @@
     # from. Must be overridden.
     target_table_class = None
 
-    # The column name in target_table we use as the integer key. May be
-    # overridden.
+    # The column name in target_table we use as the key. The type must
+    # match that returned by the ids_to_prune_query and the
+    # target_table_key_type. May be overridden.
     target_table_key = 'id'
 
+    # SQL type of the target_table_key. May be overridden.
+    target_table_key_type = 'integer'
+
     # An SQL query returning a list of ids to remove from target_table.
     # The query must return a single column named 'id' and should not
     # contain duplicates. Must be overridden.
@@ -119,10 +124,23 @@
     # See `TunableLoop`. May be overridden.
     maximum_chunk_size = 10000
 
+    # Optional extra WHERE clause fragment for the deletion to skip
+    # arbitrary rows flagged for deletion. For example, skip rows
+    # that might have been modified since the set of ids_to_prune
+    # was calculated.
+    extra_prune_clause = None
+
+    def getStore(self):
+        """The master Store for the table we are pruning.
+
+        May be overridden.
+        """
+        return IMasterStore(self.target_table_class)
+
     def __init__(self, log, abort_time=None):
         super(BulkPruner, self).__init__(log, abort_time)
 
-        self.store = IMasterStore(self.target_table_class)
+        self.store = self.getStore()
         self.target_table_name = self.target_table_class.__storm_table__
 
         # Open the cursor.
@@ -138,12 +156,20 @@
 
     def __call__(self, chunk_size):
         """See `ITunableLoop`."""
+        if self.extra_prune_clause:
+            extra = "AND (%s)" % self.extra_prune_clause
+        else:
+            extra = ""
         result = self.store.execute("""
-            DELETE FROM %s WHERE %s IN (
+            DELETE FROM %s
+            WHERE %s IN (
                 SELECT id FROM
-                cursor_fetch('bulkprunerid', %d) AS f(id integer))
+                cursor_fetch('bulkprunerid', %d) AS f(id %s))
+                %s
             """
-            % (self.target_table_name, self.target_table_key, chunk_size))
+            % (
+                self.target_table_name, self.target_table_key,
+                chunk_size, self.target_table_key_type, extra))
         self._num_removed = result.rowcount
         transaction.commit()
 
@@ -157,9 +183,7 @@
 
     XXX bug=723596 StuartBishop: This job only needs to run once per month.
     """
-
     target_table_class = POTranslation
-
     ids_to_prune_query = """
         SELECT POTranslation.id AS id FROM POTranslation
         EXCEPT (
@@ -186,6 +210,45 @@
         """
 
 
+class SessionPruner(BulkPruner):
+    """Base class for session removal."""
+
+    target_table_class = SessionData
+    target_table_key = 'client_id'
+    target_table_key_type = 'text'
+
+
+class AntiqueSessionPruner(SessionPruner):
+    """Remove sessions not accessed for 60 days"""
+
+    ids_to_prune_query = """
+        SELECT client_id AS id FROM SessionData
+        WHERE last_accessed < CURRENT_TIMESTAMP - CAST('60 days' AS interval)
+        """
+
+
+class UnusedSessionPruner(SessionPruner):
+    """Remove sessions older than 1 day with no authentication credentials."""
+
+    ids_to_prune_query = """
+        SELECT client_id AS id FROM SessionData
+        WHERE
+            last_accessed < CURRENT_TIMESTAMP - CAST('1 day' AS interval)
+            AND client_id NOT IN (
+                SELECT client_id
+                FROM SessionPkgData
+                WHERE
+                    product_id = 'launchpad.authenticateduser'
+                    AND key='logintime')
+        """
+
+    # Don't delete a session if it has been used between calculating
+    # the list of sessions to remove and the current iteration.
+    prune_extra_clause = """
+        last_accessed < CURRENT_TIMESTAMP - CAST('1 day' AS interval)
+        """
+
+
 class OAuthNoncePruner(TunableLoop):
     """An ITunableLoop to prune old OAuthNonce records.
 
@@ -977,6 +1040,8 @@
         RevisionCachePruner,
         BugHeatUpdater,
         BugWatchScheduler,
+        AntiqueSessionPruner,
+        UnusedSessionPruner,
         ]
     experimental_tunable_loops = []
 

=== modified file 'lib/lp/scripts/tests/test_garbo.py'
--- lib/lp/scripts/tests/test_garbo.py	2011-02-28 11:50:34 +0000
+++ lib/lp/scripts/tests/test_garbo.py	2011-03-07 15:37:44 +0000
@@ -10,7 +10,6 @@
     datetime,
     timedelta,
     )
-import operator
 import time
 
 from pytz import UTC
@@ -18,7 +17,10 @@
     Min,
     SQL,
     )
-from storm.locals import Storm, Int
+from storm.locals import (
+    Int,
+    Storm,
+    )
 from storm.store import Store
 import transaction
 from zope.component import getUtility
@@ -69,13 +71,19 @@
     PersonCreationRationale,
     )
 from lp.scripts.garbo import (
+    AntiqueSessionPruner,
     BulkPruner,
     DailyDatabaseGarbageCollector,
     HourlyDatabaseGarbageCollector,
     OpenIDConsumerAssociationPruner,
+    UnusedSessionPruner,
     )
 from lp.services.job.model.job import Job
 from lp.services.log.logger import BufferLogger
+from lp.services.session.model import (
+    SessionData,
+    SessionPkgData,
+    )
 from lp.testing import (
     TestCase,
     TestCaseWithFactory,
@@ -181,6 +189,97 @@
             pruner(chunk_size)
 
 
+class TestSessionPruner(TestCase):
+    layer = ZopelessDatabaseLayer
+
+    def setUp(self):
+        super(TestCase, self).setUp()
+
+        # Session database isn't reset between tests. We need to do this
+        # manually.
+        nuke_all_sessions = IMasterStore(SessionData).find(SessionData).remove
+        nuke_all_sessions()
+        self.addCleanup(nuke_all_sessions)
+
+        recent = datetime.now(UTC)
+        yesterday = recent - timedelta(days=1)
+        ancient = recent - timedelta(days=61)
+
+        def make_session(client_id, accessed, authenticated=None):
+            session_data = SessionData()
+            session_data.client_id = client_id
+            session_data.last_accessed = accessed
+            IMasterStore(SessionData).add(session_data)
+
+            if authenticated:
+                session_pkg_data = SessionPkgData()
+                session_pkg_data.client_id = client_id
+                session_pkg_data.product_id = u'launchpad.authenticateduser'
+                session_pkg_data.key = u'logintime'
+                session_pkg_data.pickle = 'value is ignored'
+                IMasterStore(SessionPkgData).add(session_pkg_data)
+
+        make_session(u'recent_auth', recent, True)
+        make_session(u'recent_unauth', recent, False)
+        make_session(u'yesterday_auth', yesterday, True)
+        make_session(u'yesterday_unauth', yesterday, False)
+        make_session(u'ancient_auth', ancient, True)
+        make_session(u'ancient_unauth', ancient, False)
+
+    def sessionExists(self, client_id):
+        store = IMasterStore(SessionData)
+        return not store.find(
+            SessionData, SessionData.client_id == client_id).is_empty()
+
+    def test_antique_session_pruner(self):
+        chunk_size = 2
+        log = BufferLogger()
+        pruner = AntiqueSessionPruner(log)
+        try:
+            while not pruner.isDone():
+                pruner(chunk_size)
+        finally:
+            pruner.cleanUp()
+
+        expected_sessions = set([
+            u'recent_auth',
+            u'recent_unauth',
+            u'yesterday_auth',
+            u'yesterday_unauth',
+            # u'ancient_auth',
+            # u'ancient_unauth',
+            ])
+
+        found_sessions = set(
+            IMasterStore(SessionData).find(SessionData.client_id))
+
+        self.assertEqual(expected_sessions, found_sessions)
+
+    def test_unused_session_pruner(self):
+        chunk_size = 2
+        log = BufferLogger()
+        pruner = UnusedSessionPruner(log)
+        try:
+            while not pruner.isDone():
+                pruner(chunk_size)
+        finally:
+            pruner.cleanUp()
+
+        expected_sessions = set([
+            u'recent_auth',
+            u'recent_unauth',
+            u'yesterday_auth',
+            # u'yesterday_unauth',
+            u'ancient_auth',
+            # u'ancient_unauth',
+            ])
+
+        found_sessions = set(
+            IMasterStore(SessionData).find(SessionData.client_id))
+
+        self.assertEqual(expected_sessions, found_sessions)
+
+
 class TestGarbo(TestCaseWithFactory):
     layer = LaunchpadZopelessLayer
 
@@ -209,7 +308,7 @@
         return collector
 
     def test_OAuthNoncePruner(self):
-        now = datetime.utcnow().replace(tzinfo=UTC)
+        now = datetime.now(UTC)
         timestamps = [
             now - timedelta(days=2), # Garbage
             now - timedelta(days=1) - timedelta(seconds=60), # Garbage
@@ -287,7 +386,7 @@
         self.failUnless(earliest >= now - 24*60*60, 'Still have old nonces')
 
     def test_CodeImportResultPruner(self):
-        now = datetime.utcnow().replace(tzinfo=UTC)
+        now = datetime.now(UTC)
         store = IMasterStore(CodeImportResult)
 
         results_to_keep_count = (
@@ -344,7 +443,7 @@
             >= now - timedelta(days=30))
 
     def test_CodeImportEventPruner(self):
-        now = datetime.utcnow().replace(tzinfo=UTC)
+        now = datetime.now(UTC)
         store = IMasterStore(CodeImportResult)
 
         LaunchpadZopelessLayer.switchDbUser('testadmin')

=== modified file 'lib/lp/services/configure.zcml'
--- lib/lp/services/configure.zcml	2011-03-04 17:05:09 +0000
+++ lib/lp/services/configure.zcml	2011-03-07 15:37:44 +0000
@@ -16,5 +16,6 @@
   <include package=".salesforce" />
   <include package=".scripts" />
   <include package=".search" />
+  <include package=".session" />
   <include package=".worlddata" />
 </configure>

=== added directory 'lib/lp/services/session'
=== added file 'lib/lp/services/session/__init__.py'
=== added file 'lib/lp/services/session/adapters.py'
--- lib/lp/services/session/adapters.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/session/adapters.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,40 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Session adapters."""
+
+__metaclass__ = type
+__all__ = []
+
+
+from zope.component import adapter
+from zope.interface import implementer
+
+from canonical.database.sqlbase import session_store
+from canonical.launchpad.interfaces.lpstorm import (
+    IMasterStore,
+    ISlaveStore,
+    IStore,
+    )
+from lp.services.session.interfaces import IUseSessionStore
+
+
+@adapter(IUseSessionStore)
+@implementer(IMasterStore)
+def session_master_store(cls):
+    """Adapt a Session database object to an `IMasterStore`."""
+    return session_store()
+
+
+@adapter(IUseSessionStore)
+@implementer(ISlaveStore)
+def session_slave_store(cls):
+    """Adapt a Session database object to an `ISlaveStore`."""
+    return session_store()
+
+
+@adapter(IUseSessionStore)
+@implementer(IStore)
+def session_default_store(cls):
+    """Adapt an Session database object to an `IStore`."""
+    return session_store()

=== added file 'lib/lp/services/session/configure.zcml'
--- lib/lp/services/session/configure.zcml	1970-01-01 00:00:00 +0000
+++ lib/lp/services/session/configure.zcml	2011-03-07 15:37:44 +0000
@@ -0,0 +1,12 @@
+<!-- Copyright 2011 Canonical Ltd.  This software is licensed under the
+     GNU Affero General Public License version 3 (see the file LICENSE).
+-->
+<configure
+    xmlns="http://namespaces.zope.org/zope"
+    xmlns:browser="http://namespaces.zope.org/browser"
+    xmlns:i18n="http://namespaces.zope.org/i18n"
+    i18n_domain="launchpad">
+    <adapter factory=".adapters.session_master_store" />
+    <adapter factory=".adapters.session_slave_store" />
+    <adapter factory=".adapters.session_default_store" />
+</configure>

=== added file 'lib/lp/services/session/interfaces.py'
--- lib/lp/services/session/interfaces.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/session/interfaces.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,15 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Session interfaces."""
+
+__metaclass__ = type
+__all__ = ['IUseSessionStore']
+
+
+from zope.interface import Interface
+
+
+class IUseSessionStore(Interface):
+    """Marker interface for Session Storm database classes and instances."""
+    pass

=== added file 'lib/lp/services/session/model.py'
--- lib/lp/services/session/model.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/session/model.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,47 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Session Storm database classes"""
+
+__metaclass__ = type
+__all__ = ['SessionData', 'SessionPkgData']
+
+from storm.locals import (
+    Pickle,
+    Storm,
+    Unicode,
+    )
+from zope.interface import (
+    classProvides,
+    implements,
+    )
+
+from canonical.database.datetimecol import UtcDateTimeCol
+from lp.services.session.interfaces import IUseSessionStore
+
+
+class SessionData(Storm):
+    """A user's Session."""
+
+    classProvides(IUseSessionStore)
+    implements(IUseSessionStore)
+
+    __storm_table__ = 'SessionData'
+    client_id = Unicode(primary=True)
+    created = UtcDateTimeCol()
+    last_accessed = UtcDateTimeCol()
+
+
+class SessionPkgData(Storm):
+    """Data storage for a Session."""
+
+    classProvides(IUseSessionStore)
+    implements(IUseSessionStore)
+
+    __storm_table__ = 'SessionPkgData'
+    __storm_primary__ = 'client_id', 'product_id', 'key'
+
+    client_id = Unicode()
+    product_id = Unicode()
+    key = Unicode()
+    pickle = Pickle()

=== added directory 'lib/lp/services/session/tests'
=== added file 'lib/lp/services/session/tests/__init__.py'
=== added file 'lib/lp/services/session/tests/test_session.py'
--- lib/lp/services/session/tests/test_session.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/session/tests/test_session.py	2011-03-07 15:37:44 +0000
@@ -0,0 +1,32 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Session tests."""
+
+__metaclass__ = type
+
+from canonical.launchpad.interfaces.lpstorm import (
+    IMasterStore,
+    ISlaveStore,
+    IStore,
+    )
+from canonical.testing.layers import DatabaseFunctionalLayer
+from lp.services.session.model import (
+    SessionData,
+    SessionPkgData,
+    )
+from lp.testing import TestCase
+
+
+class TestSessionModelAdapters(TestCase):
+    layer = DatabaseFunctionalLayer
+
+    def test_adapters(self):
+        for adapter in [IMasterStore, ISlaveStore, IStore]:
+            for cls in [SessionData, SessionPkgData]:
+                for obj in [cls, cls()]:
+                    store = adapter(obj)
+                    self.assert_(
+                        'session' in store.get_database()._dsn,
+                        'Unknown store returned adapting %r to %r'
+                        % (obj, adapter))

=== modified file 'lib/lp/testing/factory.py'
--- lib/lp/testing/factory.py	2011-03-06 23:15:05 +0000
+++ lib/lp/testing/factory.py	2011-03-07 15:37:44 +0000
@@ -101,6 +101,7 @@
     )
 from canonical.launchpad.webapp.sorting import sorted_version_numbers
 from lp.app.enums import ServiceUsage
+from lp.archivepublisher.interfaces.publisherconfig import IPublisherConfigSet
 from lp.archiveuploader.dscfile import DSCFile
 from lp.archiveuploader.uploadpolicy import BuildDaemonUploadPolicy
 from lp.blueprints.enums import (
@@ -3850,6 +3851,20 @@
             description = self.getUniqueString()
         return getUtility(ICveSet).new(sequence, description, cvestate)
 
+    def makePublisherConfig(self, distribution=None, root_dir=None,
+                            base_url=None, copy_base_url=None):
+        """Create a new `PublisherConfig` record."""
+        if distribution is None:
+            distribution = self.makeDistribution()
+        if root_dir is None:
+            root_dir = self.getUniqueString()
+        if base_url is None:
+            base_url = self.getUniqueString()
+        if copy_base_url is None:
+            copy_base_url = self.getUniqueString()
+        return getUtility(IPublisherConfigSet).new(
+            distribution, root_dir, base_url, copy_base_url)
+
 
 # Some factory methods return simple Python types. We don't add
 # security wrappers for them, as well as for objects created by

=== modified file 'lib/lp/testing/tests/test_standard_test_template.py'
--- lib/lp/testing/tests/test_standard_test_template.py	2011-02-02 13:19:02 +0000
+++ lib/lp/testing/tests/test_standard_test_template.py	2011-03-07 15:37:44 +0000
@@ -6,15 +6,16 @@
 __metaclass__ = type
 
 from canonical.testing.layers import DatabaseFunctionalLayer
-from lp.testing import TestCase # or TestCaseWithFactory
+# or TestCaseWithFactory
+from lp.testing import TestCase
 
 
 class TestSomething(TestCase):
     # XXX: Sample test class.  Replace with your own test class(es).
 
-    # XXX: Optional layer--see lib/canonical/testing/layers.py
-    # Get the simplest layer that your test will work on, or if you
-    # don't even use the database, don't set it at all.
+    # XXX: layer--see lib/canonical/testing/layers.py
+    # Get the simplest layer that your test will work on. For unit tests
+    # requiring no resources, this is BaseLayer.
     layer = DatabaseFunctionalLayer
 
     # XXX: Sample test.  Replace with your own test methods.