Merge lp:~mwhudson/launchpad/no-hosted-area into lp:launchpad

[Michael Hudson-Doyle]

Hi Tim,

This branch makes the ssh codehosting server only use one area (the mirrored area) rather than the mirrored and hosted area.

In addition, on branch unlock, rather than requesting the branch be mirrored, the codehosting server calls a method that updates the fields the puller updates for hosted branches, hopefully reducing latency.

In the next pipe I extend this branchChanged endpoint to also record the branch format.

There's obviously no way this branch can land on its own, it will break many many tests.

Cheers,
mwh

[Tim Penhey (thumper)]

lib/lp/code/xmlrpc/codehosting.py
in: def branchChanged(self, branch_id, stacked_on_location, last_revision_id):
  + branch.last_mirrored = datetime.datetime.now(pytz.UTC)
should probably be using UTC_NOW

lib/lp/codehosting/inmemory.py branchChanged event should use UTC_NOW too.

then:
test_branchChanged_sets_last_mirrored
can use:
     self.assertSqlAttributeEqualsDate(
        branch, 'last_mirrored', UTC_NOW)

def test_branchChanged_doesnt_create_scan_job_for_noop_change(self):
    # XXX Is this even the right thing to do?
It will be with your next pipe.

lib/lp/codehosting/vfs/branchfs.py
class LaunchpadServer (I think - around line 558)
the __init__ method still refers to the authserver, also there is a
XXX comment that I'm wondering whether we can remove it or not.

lib/lp/codehosting/vfs/tests/test_branchfs.py - still refers to an
authserver too.

# XXX Maaaybe we could complain on stderr here?
  - Not following our XXX format, and is this something we want to do?

# XXX: JonathanLange 2007-05-29: The 'chroot' line lacks a unit test.
I don't suppose you want to add a unit test for this?

General Note: we probably want to rename config.codehosting.mirrored_branches
at some stage.

[Michael Hudson-Doyle (mwhudson)]

On 19/04/10 16:20, Tim Penhey wrote:
> lib/lp/code/xmlrpc/codehosting.py
> in: def branchChanged(self, branch_id, stacked_on_location, last_revision_id):
> + branch.last_mirrored = datetime.datetime.now(pytz.UTC)
> should probably be using UTC_NOW
>
> lib/lp/codehosting/inmemory.py branchChanged event should use UTC_NOW too.
>
> then:
> test_branchChanged_sets_last_mirrored
> can use:
> self.assertSqlAttributeEqualsDate(
> branch, 'last_mirrored', UTC_NOW)

Yeah OK.

> def test_branchChanged_doesnt_create_scan_job_for_noop_change(self):
> # XXX Is this even the right thing to do?
> It will be with your next pipe.

Right, I'll delete the comment.

> lib/lp/codehosting/vfs/branchfs.py
> class LaunchpadServer (I think - around line 558)
> the __init__ method still refers to the authserver, also there is a
> XXX comment that I'm wondering whether we can remove it or not.

Grar, can I fix this in the later pipe that combines the two endpoints?

> lib/lp/codehosting/vfs/tests/test_branchfs.py - still refers to an
> authserver too.

This too.

> # XXX Maaaybe we could complain on stderr here?
> - Not following our XXX format, and is this something we want to do?

Nah, I'll delete the comment.

> # XXX: JonathanLange 2007-05-29: The 'chroot' line lacks a unit test.
> I don't suppose you want to add a unit test for this?

OK. It's a bit terrible though.

> General Note: we probably want to rename config.codehosting.mirrored_branches
> at some stage.

Given recent edge rollout funnies, I don't know how we'd do this :/ But
yeah, it doesn't really make sense.

Interdiff attached.

Cheers,
mwh

[Tim Penhey (thumper)]

 merge approved

On Tue, 20 Apr 2010 10:39:16 you wrote:
> On 19/04/10 16:20, Tim Penhey wrote:
> > lib/lp/codehosting/vfs/branchfs.py
> > class LaunchpadServer (I think - around line 558)
> > the __init__ method still refers to the authserver, also there is a
> > XXX comment that I'm wondering whether we can remove it or not.
>
> Grar, can I fix this in the later pipe that combines the two endpoints?

Yep, sure.

> > lib/lp/codehosting/vfs/tests/test_branchfs.py - still refers to an
> > authserver too.
>
> This too.

Yes.

Tim

review: Approve

=== modified file 'Makefile'
--- Makefile	2010-04-20 19:10:35 +0000
+++ Makefile	2010-04-27 02:13:38 +0000
@@ -209,9 +209,9 @@
 ftest_inplace: inplace
 	bin/test -f $(TESTFLAGS) $(TESTOPTS)
 
-mpcreationjobs:
-	# Handle merge proposal creations.
-	$(PY) cronscripts/mpcreationjobs.py
+merge-proposal-jobs:
+	# Handle merge proposal email jobs.
+	$(PY) cronscripts/merge-proposal-jobs.py -v
 
 run: check_schema inplace stop
 	$(RM) thread*.request
@@ -255,8 +255,7 @@
 	# Scan branches from the filesystem into the database.
 	$(PY) cronscripts/scan_branches.py
 
-
-sync_branches: pull_branches scan_branches mpcreationjobs
+sync_branches: pull_branches scan_branches merge-proposal-jobs
 
 $(BZR_VERSION_INFO):
 	scripts/update-bzr-version-info.sh
 
=== modified file 'bzrplugins/lpserve.py'
--- bzrplugins/lpserve.py	2010-03-24 00:43:45 +0000
+++ bzrplugins/lpserve.py	2010-04-27 02:13:38 +0000
@@ -85,8 +85,8 @@
         finally:
             ui.ui_factory = old_factory
 
-    def run(self, user_id, port=None, upload_directory=None,
-            mirror_directory=None, branchfs_endpoint_url=None, inet=False):
+    def run(self, user_id, port=None, branch_directory=None,
+            branchfs_endpoint_url=None, inet=False):
         from lp.codehosting.bzrutils import install_oops_handler
         from lp.codehosting.vfs import get_lp_server, hooks
         install_oops_handler(user_id)
@@ -94,8 +94,8 @@
         resource.setrlimit(resource.RLIMIT_AS, (four_gig, four_gig))
         seen_new_branch = hooks.SetProcTitleHook()
         lp_server = get_lp_server(
-            int(user_id), branchfs_endpoint_url,
-            upload_directory, mirror_directory, seen_new_branch.seen)
+            int(user_id), branchfs_endpoint_url, branch_directory,
+            seen_new_branch.seen)
         lp_server.start_server()
 
         old_lockdir_timeout = lockdir._DEFAULT_TIMEOUT_SECONDS
 
=== modified file 'configs/development/launchpad-lazr.conf'
--- configs/development/launchpad-lazr.conf	2010-04-19 03:44:27 +0000
+++ configs/development/launchpad-lazr.conf	2010-04-27 02:13:38 +0000
@@ -201,9 +201,9 @@
 port: 11217
 memory_size: 1
 
-[mpcreationjobs]
+[merge_proposal_jobs]
 error_dir: /var/tmp/codehosting.test
-oops_prefix: DMPCR
+oops_prefix: DMPJ
 
 [personalpackagearchive]
 root: /var/tmp/ppa/
 
=== modified file 'configs/testrunner/launchpad-lazr.conf'
--- configs/testrunner/launchpad-lazr.conf	2010-04-19 03:44:27 +0000
+++ configs/testrunner/launchpad-lazr.conf	2010-04-27 02:13:38 +0000
@@ -34,7 +34,7 @@
 bzr_lp_prefix: lp://dev/
 hosted_branches_root: /tmp/sftp-test/branches
 host_key_pair_path: lib/lp/codehosting/sshserver/tests/keys
-port: tcp:22222:interface=127.0.0.1
+port: tcp:22222:interface=bazaar.launchpad.dev
 error_dir: /var/tmp/codehosting.test
 oops_prefix: SMPSSH
 access_log: /tmp/test-codehosting-access.log
@@ -171,12 +171,8 @@
 # processes spawned through some other mechanism.
 port: 11242
 
-[mpcreationjobs]
-oops_prefix: TMPCJ
-error_dir: /var/tmp/codehosting.test
-
-[update_preview_diffs]
-oops_prefix: TUPD
+[merge_proposal_jobs]
+oops_prefix: TMPJ
 error_dir: /var/tmp/codehosting.test
 
 [upgrade_branches]
 
=== renamed file 'cronscripts/mpcreationjobs.py' => 'cronscripts/merge-proposal-jobs.py'
--- cronscripts/mpcreationjobs.py	2010-02-16 15:25:52 +0000
+++ cronscripts/merge-proposal-jobs.py	2010-04-27 02:13:38 +0000
@@ -1,48 +1,40 @@
 #!/usr/bin/python2.5 -S
 #
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009, 2010 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 # pylint: disable-msg=W0403
 
-"""Handle new BranchMergeProposals.
+"""Handle jobs for BranchMergeProposals.
 
-This script generates a diff for the merge proposal if needed, then notifies
-all interested parties about the merge proposal.
+This script handles all job types for branch merge proposals.
 """
 
 __metaclass__ = type
 
 import _pythonpath
-from zope.component import getUtility
 
-from canonical.config import config
-from lp.codehosting.vfs import get_scanner_server
-from lp.services.job.runner import JobRunner
+# The following line is a horrible hack, but unfortunately necessary right now
+# to stop import errors from circular imports.
+import canonical.launchpad.interfaces
 from lp.code.interfaces.branchmergeproposal import (
-    IMergeProposalCreatedJobSource,)
-from lp.services.scripts.base import LaunchpadCronScript
-from canonical.launchpad.webapp.errorlog import globalErrorUtility
-
-
-class RunMergeProposalCreatedJobs(LaunchpadCronScript):
-    """Run merge proposal creation jobs."""
-
-    def main(self):
-        globalErrorUtility.configure('mpcreationjobs')
-        job_source = getUtility(IMergeProposalCreatedJobSource)
-        runner = JobRunner.fromReady(job_source, self.logger)
-        server = get_scanner_server()
-        server.start_server()
-        try:
-            runner.runAll()
-        finally:
-            server.stop_server()
-        self.logger.info(
-            'Ran %d MergeProposalCreatedJobs.', len(runner.completed_jobs))
+    IBranchMergeProposalJobSource,
+    )
+from lp.services.job.runner import JobCronScript, TwistedJobRunner
+
+
+class RunMergeProposalJobs(JobCronScript):
+    """Run all merge proposal jobs."""
+
+    config_name = 'merge_proposal_jobs'
+    source_interface = IBranchMergeProposalJobSource
+
+    def __init__(self):
+        super(RunMergeProposalJobs, self).__init__(
+            runner_class=TwistedJobRunner,
+            script_name='merge-proposal-jobs')
 
 
 if __name__ == '__main__':
-    script = RunMergeProposalCreatedJobs(
-        'mpcreationjobs', config.mpcreationjobs.dbuser)
+    script = RunMergeProposalJobs()
     script.lock_and_run()
 
=== modified file 'cronscripts/mirror-prober.sh'
--- cronscripts/mirror-prober.sh	2009-10-17 14:11:40 +0000
+++ cronscripts/mirror-prober.sh	2010-04-27 02:13:38 +0000
@@ -39,10 +39,10 @@
 cd /srv/launchpad.net/production/launchpad/cronscripts
 
 echo '== Distribution mirror prober (archive)' `date` ==
-python2.5 distributionmirror-prober.py --content-type=archive --max-mirrors=20
+python2.5 -S distributionmirror-prober.py --content-type=archive --max-mirrors=20
 
 echo '== Distribution mirror prober (cdimage)' `date` ==
-python2.5 distributionmirror-prober.py --content-type=cdimage --max-mirrors=30
+python2.5 -S distributionmirror-prober.py --content-type=cdimage --max-mirrors=30
 
 rm -f $LOCK
 
 
=== modified file 'cronscripts/nightly.sh'
--- cronscripts/nightly.sh	2009-10-17 14:11:40 +0000
+++ cronscripts/nightly.sh	2010-04-27 02:13:38 +0000
@@ -3,11 +3,11 @@
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-# This script performs nightly chores. It should be run from 
+# This script performs nightly chores. It should be run from
 # cron as the launchpad user once a day. Typically the output
 # will be sent to an email address for inspection.
 
-# Note that http/ftp proxies are needed by the product 
+# Note that http/ftp proxies are needed by the product
 # release finder
 
 # Only run this script on loganberry
@@ -42,41 +42,41 @@
 cd /srv/launchpad.net/production/launchpad/cronscripts
 
 echo == Expiring memberships `date` ==
-python2.5 flag-expired-memberships.py -q
+python2.5 -S flag-expired-memberships.py -q
 
 echo == Allocating revision karma `date` ==
-python2.5 allocate-revision-karma.py -q
+python2.5 -S allocate-revision-karma.py -q
 
 echo == Recalculating karma `date` ==
-python2.5 foaf-update-karma-cache.py -q
+python2.5 -S foaf-update-karma-cache.py -q
 
 echo == Updating cached statistics `date` ==
-python2.5 update-stats.py -q
+python2.5 -S update-stats.py -q
 
 echo == Expiring questions `date` ==
-python2.5 expire-questions.py
+python2.5 -S expire-questions.py
 
 ### echo == Expiring bugs `date` ==
-### python2.5 expire-bugtasks.py
+### python2.5 -S expire-bugtasks.py
 
 # checkwatches.py is scheduled in the /code/pqm/launchpad_crontabs branch.
 ### echo == Updating bug watches `date` ==
-### python2.5 checkwatches.py
+### python2.5 -S checkwatches.py
 
 echo == Updating bugtask target name caches `date` ==
-python2.5 update-bugtask-targetnamecaches.py -q
+python2.5 -S update-bugtask-targetnamecaches.py -q
 
 echo == Updating personal standings `date` ==
-python2.5 update-standing.py -q
+python2.5 -S update-standing.py -q
 
 echo == Updating CVE database `date` ==
-python2.5 update-cve.py -q
+python2.5 -S update-cve.py -q
 
 echo == Updating package cache `date` ==
-python2.5 update-pkgcache.py -q
+python2.5 -S update-pkgcache.py -q
 
 echo == Product Release Finder `date` ==
-python2.5 product-release-finder.py -q
+python2.5 -S product-release-finder.py -q
 
 
 rm -f $LOCK
 
=== modified file 'cronscripts/publishing/cron.germinate'
--- cronscripts/publishing/cron.germinate	2010-03-02 09:58:34 +0000
+++ cronscripts/publishing/cron.germinate	2010-04-27 02:13:38 +0000
@@ -127,7 +127,7 @@
 echo " done."
 
 # now generate the Supported extra overrides
-$MAINTAINCE_CHECK $suite > "$MISCROOT/more-extra.override.$suite.main.supported"
+$MAINTAINCE_CHECK $suite > "$MISCROOT/more-extra.override.$suite.main.supported" 2> _maintenance-check.stderr
 if [ $? -eq 0 ]; then
     cat "$MISCROOT/more-extra.override.$suite.main.supported" >> "$MISCROOT/more-extra.override.$suite.main.new"
 fi
 
=== modified file 'cronscripts/publishing/maintenance-check.py'
--- cronscripts/publishing/maintenance-check.py	2010-01-22 13:57:45 +0000
+++ cronscripts/publishing/maintenance-check.py	2010-04-27 02:13:38 +0000
@@ -6,9 +6,19 @@
 #  https://code.edge.launchpad.net/~mvo/ubuntu-maintenance-check/python-port
 # (where it will vanish once taken here)
 
+# this warning filter is only needed on older versions of python-apt,
+# once the machine runs lucid it can be removed
+import warnings
+warnings.filterwarnings("ignore","apt API not stable yet")
+import apt
+warnings.resetwarnings()
+
+import apt_pkg
 import logging
+import os
 import sys
 import urllib2
+import urlparse
 
 from optparse import OptionParser
 
@@ -31,8 +41,8 @@
 SUPPORTED_ARCHES = PRIMARY_ARCHES + ["armel"]
 
 # what defines the seeds is documented in wiki.ubuntu.com/SeedManagement
-SERVER_SEEDS = [ "supported-server"]
-DESKTOP_SEEDS = ["ship", "supported-desktop"]
+SERVER_SEEDS = [ "supported-server", "server-ship"]
+DESKTOP_SEEDS = ["ship", "supported-desktop", "supported-desktop-extra"]
 SUPPORTED_SEEDS = [ "all" ]
 
 # normal support timeframe
@@ -51,32 +61,111 @@
 
 # distro names and if they get LTS support (order is important)
 DISTRO_NAMES_AND_LTS_SUPPORT = [ ("ubuntu",   True),
-                                 ("kubuntu",  False),
-                                 ("edubuntu", False),
+                                 ("kubuntu",  True),
                                  ("netbook",  False),
                                ]
 
 # germinate output base directory
 BASE_URL = "http://people.canonical.com/~ubuntu-archive/germinate-output/"
 
+# hints dir url, hints file is "$distro.hints" by default
+# (e.g. lucid.hints)
+HINTS_DIR_URL = "http://people.canonical.com/~ubuntu-archive/seeds/platform.%s/SUPPORTED_HINTS"
+
+# we need the archive root to parse the Sources file to support
+# by-source hints
+ARCHIVE_ROOT = "http://archive.ubuntu.com/ubuntu"
+
 # support timeframe tag used in the Packages file
 SUPPORT_TAG = "Supported"
 
-
-def get_structure(name, version):
-    """ 
-    get structure file for named distro and distro version 
-    (e.g. kubuntu, lucid)
-    """
-    f = urllib2.urlopen("%s/%s.%s/structure" % (BASE_URL, name, version))
+def get_binaries_for_source_pkg(srcname):
+    """ Return all binary package names for the given source package name.
+
+    :param srcname: The source package name.
+    :return: A list of binary package names.
+    """
+    pkgnames = set()
+    recs = apt_pkg.GetPkgSrcRecords()
+    while recs.Lookup(srcname):
+        for binary in recs.Binaries:
+            pkgnames.add(binary)
+    return pkgnames
+
+def expand_src_pkgname(pkgname):
+    """ Expand a package name if it is prefixed with src.
+
+    If the package name is prefixed with src it will be expanded
+    to a list of binary package names. Otherwise the original
+    package name will be returned.
+    
+    :param pkgname: The package name (that may include src:prefix).
+    :return: A list of binary package names (the list may be one element long).
+    """
+    if not pkgname.startswith("src:"):
+        return [pkgname]
+    return get_binaries_for_source_pkg(pkgname.split("src:")[1])
+
+def create_and_update_deb_src_source_list(distroseries):
+    """ Create sources.list and update cache.
+
+    This creates a sources.list file with deb-src entries for a given 
+    distroseries and apt.Cache.update() to make sure the data is up-to-date.
+    :param distro: The code name of the distribution series (e.g. lucid).
+    :return: None
+    :raises: IOError: When cache update fails.
+    """
+    # apt root dir
+    rootdir="./aptroot.%s" % distroseries
+    sources_list_dir = os.path.join(rootdir, "etc","apt")
+    if not os.path.exists(sources_list_dir):
+        os.makedirs(sources_list_dir)
+    sources_list = open(os.path.join(sources_list_dir, "sources.list"),"w")
+    for pocket in [
+        "%s" % distroseries, 
+        "%s-updates" % distroseries, 
+        "%s-security" % distroseries]:
+        sources_list.write(
+            "deb-src %s %s main restricted\n" % (
+                ARCHIVE_ROOT, pocket))
+        sources_list.write(
+            "deb %s %s main restricted\n" % (
+                ARCHIVE_ROOT, pocket))
+    sources_list.close()
+    # create required dirs/files for apt.Cache(rootdir) to work on older
+    # versions of python-apt. once lucid is used it can be removed
+    for d in  ["var/lib/dpkg", 
+               "var/cache/apt/archives/partial",
+               "var/lib/apt/lists/partial"]:
+        if not os.path.exists(os.path.join(rootdir,d)):
+            os.makedirs(os.path.join(rootdir,d))
+    if not os.path.exists(os.path.join(rootdir,"var/lib/dpkg/status")):
+        open(os.path.join(rootdir,"var/lib/dpkg/status"),"w")
+    # open cache with our just prepared rootdir
+    cache = apt.Cache(rootdir=rootdir)
+    try:
+        cache.update(apt.progress.FetchProgress())
+    except SystemError:
+        logging.exception("cache.update() failed")
+
+def get_structure(distroname, version):
+    """ Get structure file conent for named distro and distro version.
+    
+    :param name: Name of the distribution (e.g. kubuntu, ubuntu, xubuntu).
+    :param version: Code name of the distribution version (e.g. lucid).
+    :return: List of strings with the structure file content
+    """
+    f = urllib2.urlopen("%s/%s.%s/structure" % (BASE_URL, distroname, version))
     structure = f.readlines()
     f.close()
     return structure
 
 def expand_seeds(structure, seedname):
-    """ 
-    expand seed by its dependencies using the strucure file
-    returns a set() for the seed dependencies (excluding the original seedname)
+    """ Expand seed by its dependencies using the strucure file.
+
+    :param structure: The content of the STRUCTURE file as string list.
+    :param seedname: The name of the seed as string that needs to be expanded.
+    :return: a set() for the seed dependencies (excluding the original seedname)
     """
     seeds = []
     for line in structure:
@@ -122,6 +211,28 @@
             in_seeds.add(s)
     return in_seeds
 
+def compare_support_level(x, y):
+    """
+    compare two support level strings of the form 18m, 3y etc
+    :parm x: the first support level
+    :parm y: the second support level
+    :return: negative if x < y, zero if x==y, positive if x > y
+    """
+    def support_to_int(support_time):
+        """
+        helper that takes a support time string and converts it to 
+        a integer for cmp()
+        """
+        # allow strings like "5y (kubuntu-common)
+        x = support_time.split()[0]
+        if x.endswith("y"):
+            return 12 * int(x[0:-1])
+        elif x.endswith("m"):
+            return int(x[0:-1])
+        else:
+            raise ValueError("support time '%s' has to end with y or m" % x)
+    return cmp(support_to_int(x), support_to_int(y))
+
 def get_packages_support_time(structure, name, pkg_support_time, support_timeframe_list):
     """
     input a structure file and a list of pair<timeframe, seedlist>
@@ -137,8 +248,15 @@
             for pkg in pkgs_in_seeds[seed]:
                 if not pkg in pkg_support_time:
                     pkg_support_time[pkg] = timeframe
-                    if options.with_seeds:
-                        pkg_support_time[pkg] += " (%s)" % ", ".join(what_seeds(pkg, pkgs_in_seeds))
+                else:
+                    old_timeframe = pkg_support_time[pkg]
+                    if compare_support_level(old_timeframe, timeframe) < 0:
+                        logging.debug("overwriting %s from %s to %s" % (
+                                pkg, old_timeframe, timeframe))
+                        pkg_support_time[pkg] = timeframe
+                if options.with_seeds:
+                    pkg_support_time[pkg] += " (%s)" % ", ".join(what_seeds(pkg, pkgs_in_seeds))
+
 
     return pkg_support_time
 
@@ -150,6 +268,8 @@
     parser.add_option("--source-packages", "", default=False,
                       action="store_true", 
                       help="show as source pkgs")
+    parser.add_option("--hints-file", "", default=None,
+                      help="use diffenrt use hints file location")
     (options, args) = parser.parse_args()
 
     # init
@@ -160,6 +280,17 @@
             sys.exit(1)
     else:
         distro = "lucid"
+
+    # make sure our deb-src information is up-to-date
+    create_and_update_deb_src_source_list(distro)
+
+    if options.hints_file:
+        hints_file = options.hints_file
+        (schema, netloc, path, query, fragment) = urlparse.urlsplit(hints_file)
+        if not schema:
+            hints_file = "file:%s" % path
+    else:
+        hints_file = HINTS_DIR_URL % distro
         
     # go over the distros we need to check
     pkg_support_time = {}
@@ -175,20 +306,77 @@
         else:
             support_timeframe = SUPPORT_TIMEFRAME
         get_packages_support_time(structure, name, pkg_support_time, support_timeframe)
+
+    # now go over the bits in main that we have not seen (because
+    # they are not in any seed and got added manually into "main"
+    for arch in PRIMARY_ARCHES:
+        rootdir="./aptroot.%s" % distro
+        apt_pkg.Config.Set("APT::Architecture", arch)
+        cache = apt.Cache(rootdir=rootdir)
+        try:
+            cache.update(apt.progress.FetchProgress())
+        except SystemError:
+            logging.exception("cache.update() failed")
+        cache.open(apt.progress.OpProgress())
+        for pkg in cache:
+            if not pkg.name in pkg_support_time:
+                pkg_support_time[pkg.name] = support_timeframe[-1][0]
+                logging.warn("add package in main but not in seeds %s with %s" % 
+                             (pkg.name, pkg_support_time[pkg.name]))
+
+    # now check the hints file that is used to overwrite 
+    # the default seeds
+    try:
+        for line in urllib2.urlopen(hints_file):
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            try:
+                (raw_pkgname, support_time) = line.split()
+                for pkgname in expand_src_pkgname(raw_pkgname):
+                    if support_time == 'unsupported':
+                        try:
+                            del pkg_support_time[pkgname]
+                            sys.stderr.write("hints-file: marking %s unsupported\n" % pkgname)
+                        except KeyError:
+                            pass
+                    else:
+                        if pkg_support_time.get(pkgname) != support_time:
+                            sys.stderr.write(
+                                "hints-file: changing %s from %s to %s\n" % (
+                                    pkgname,  pkg_support_time.get(pkgname), 
+                                    support_time))
+                            pkg_support_time[pkgname] = support_time
+            except:
+                logging.exception("can not parse line '%s'" % line)
+    except urllib2.HTTPError, e:
+        if e.getcode() != 404:
+            raise
+        sys.stderr.write("hints-file: %s gave 404 error\n" % hints_file)
     
     # output suitable for the extra-override file
     for pkgname in sorted(pkg_support_time.keys()):
-        # go over the supported arches, they are divided in 
-        # first-class (PRIMARY) and second-class with different
-        # support levels
-        for arch in SUPPORTED_ARCHES:
-            # full LTS support
-            if arch in PRIMARY_ARCHES:
-                print "%s/%s %s %s" % (
-                    pkgname, arch, SUPPORT_TAG, pkg_support_time[pkgname])
-            else:
-                # not a LTS supported architecture, gets only regular
-                # support_timeframe
-                print "%s/%s %s %s" % (
-                    pkgname, arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])
+        # special case, the hints file may contain overrides that
+        # are arch-specific (like zsh-doc/armel)
+        if "/" in pkgname:
+            print "%s %s %s" % (
+                pkgname, SUPPORT_TAG, pkg_support_time[pkgname])
+        else:
+            # go over the supported arches, they are divided in 
+            # first-class (PRIMARY) and second-class with different
+            # support levels
+            for arch in SUPPORTED_ARCHES:
+                # ensure we do not overwrite arch-specific overwrites
+                pkgname_and_arch = "%s/%s" % (pkgname, arch)
+                if pkgname_and_arch in pkg_support_time:
+                    break
+                if arch in PRIMARY_ARCHES:
+                    # arch with full LTS support
+                    print "%s %s %s" % (
+                        pkgname_and_arch, SUPPORT_TAG, pkg_support_time[pkgname])
+                else:
+                    # not a LTS supported architecture, gets only regular
+                    # support_timeframe
+                    print "%s %s %s" % (
+                        pkgname_and_arch, SUPPORT_TAG, SUPPORT_TIMEFRAME[0][0])
                 
 
=== removed file 'cronscripts/update_preview_diffs.py'
--- cronscripts/update_preview_diffs.py	2010-02-16 15:25:52 +0000
+++ cronscripts/update_preview_diffs.py	1970-01-01 00:00:00 +0000
@@ -1,38 +0,0 @@
-#!/usr/bin/python2.5 -S
-#
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-# pylint: disable-msg=W0403
-
-"""Update or create previews diffs for branch merge proposals."""
-
-__metaclass__ = type
-
-import _pythonpath
-
-from lp.services.job.runner import JobCronScript, JobRunner, TwistedJobRunner
-from lp.code.interfaces.branchmergeproposal import (
-    IUpdatePreviewDiffJobSource,)
-
-
-class RunUpdatePreviewDiffJobs(JobCronScript):
-    """Run UpdatePreviewDiff jobs."""
-
-    config_name = 'update_preview_diffs'
-    source_interface = IUpdatePreviewDiffJobSource
-
-    def __init__(self):
-        super(RunUpdatePreviewDiffJobs, self).__init__()
-        if self.options.twisted:
-            self.runner_class = TwistedJobRunner
-        else:
-            self.runner_class = JobRunner
-
-    def add_my_options(self):
-        self.parser.add_option('--twisted', action='store_true')
-
-
-if __name__ == '__main__':
-    script = RunUpdatePreviewDiffJobs()
-    script.lock_and_run()
 
=== modified file 'database/replication/Makefile'
--- database/replication/Makefile	2010-03-26 08:43:39 +0000
+++ database/replication/Makefile	2010-04-27 02:13:38 +0000
@@ -44,14 +44,16 @@
 
 PGMASSACRE=../../utilities/pgmassacre.py
 
-# Turn off silencing for now so we details on staging deployments.
+CREATEDB_83=createdb --encoding=UTF8
+CREATEDB_84=createdb --encoding=UTF8 --locale=C --template=template0
+CREATEDB=${CREATEDB_83}
+
+# Turn off output silencing so we can see details of staging deployments.
+# Without the timestamps, we are unable to estimate production deployment
+# times.
 #SHHH=../../utilities/shhh.py
 SHHH=
 
-AUTHDB_TABLES=\
-	account accountpassword authkoken emailaddress \
-	openidassociation openidauthorization openidnonce openidrpsummary
-
 default:
 	echo Usage: make [start|stop|restart]
 
@@ -76,7 +78,7 @@
 	
 	# Replicate it again, so we can test with multiple slaves.
 	-${PGMASSACRE} launchpad_dev_slave2
-	createdb --encoding=UTF8 launchpad_dev_slave2
+	${CREATEDB} launchpad_dev_slave2
 	LPCONFIG=${DEV_CONFIG} ./slon_ctl.py start \
 		 node3_node 'dbname=launchpad_dev_slave2 user=slony'
 	LPCONFIG=${DEV_CONFIG} ./new-slave.py 3 launchpad_dev_slave2
@@ -96,23 +98,12 @@
 	    _MASTER=lpmain_staging_new _SLAVE=lpmain_staging_slave_new \
 	    LAG="0 seconds"
 	# Create the DB with the desired default tablespace.
-	createdb --encoding UTF8 --tablespace ${STAGING_TABLESPACE} \
-	    lpmain_staging_new
-	# Restore the DB schema. We need to restore permissions, despite
+	${CREATEDB} --tablespace ${STAGING_TABLESPACE} lpmain_staging_new
+	# Restore the database. We need to restore permissions, despite
 	# later running security.py, to pull in permissions granted on
 	# production to users not maintained by security.py.
 	pg_restore --dbname=lpmain_staging_new \
 	    --no-owner --exit-on-error ${STAGING_DUMP}
-	psql -q -d lpmain_staging_new -f authdb_drop.sql
-	psql -q -d lpmain_staging_new -f authdb_create.sql \
-	    2>&1 | grep -v _sl || true
-	# Restore the authdb data.
-	for table in ${AUTHDB_TABLES}; do \
-	    pg_restore --dbname=lpmain_staging_new \
-		--no-acl --no-owner --disable-triggers --data-only \
-		--table=$$table ${STAGING_DUMP}; \
-	done
-	psql -q -d lpmain_staging_new -f authdb_sequences.sql
 	# Uninstall Slony-I if it is installed - a pg_dump of a DB with
 	# Slony-I installed isn't usable without this step.
 	LPCONFIG=${NEW_STAGING_CONFIG} ./repair-restored-db.py
@@ -144,17 +135,9 @@
 	LPCONFIG=${STAGING_CONFIG} ./slon_ctl.py --lag="${LAG}" start
 
 dogfood:
-	createdb --encoding UTF8 ${DOGFOOD_DBNAME}
+	${CREATEDB} ${DOGFOOD_DBNAME}
 	pg_restore --dbname=${DOGFOOD_DBNAME} --no-acl --no-owner \
 	    --exit-on-error ${DOGFOOD_DUMP}
-	psql -q -d ${DOGFOOD_DBNAME} -f authdb_drop.sql
-	psql -q -d ${DOGFOOD_DBNAME} -f authdb_create.sql \
-	    2>&1 | grep -v _sl || true
-	for table in ${AUTHDB_TABLES}; do \
-	    pg_restore --dbname=${DOGFOOD_DBNAME} \
-		--no-acl --no-owner --disable-triggers --data-only \
-		--table=$$table ${DOGFOOD_DUMP}; \
-	done
 	./repair-restored-db.py -d ${DOGFOOD_DBNAME}
 	../schema/upgrade.py -d ${DOGFOOD_DBNAME}
 	../schema/fti.py -d ${DOGFOOD_DBNAME}
@@ -174,14 +157,15 @@
 _replicate:
 	@echo LPCONFIG currently ${LPCONFIG}
 	# Start the slon daemon for the master.
-	./slon_ctl.py start \
+	./slon_ctl.py --lag="0 seconds" start \
 		 node1_node "dbname=${_MASTER} user=slony"
 	# Initialize the cluster and create replication sets.
 	./initialize.py
 	# Create the soon-to-be-slave database, empty at this point.
-	createdb --encoding=UTF8 --tablespace=${_SLAVE_TABLESPACE} ${_SLAVE}
+	${CREATEDB} --tablespace=${_SLAVE_TABLESPACE} ${_SLAVE}
 	# Start the slon daemon for the slave
-	./slon_ctl.py start node2_node "dbname=${_SLAVE} user=slony"
+	./slon_ctl.py --lag="0 seconds" start \
+	    node2_node "dbname=${_SLAVE} user=slony"
 	# Setup the slave
 	./new-slave.py 2 "dbname=${_SLAVE}"
 	# Upgrade all databases in the cluster and reset security.
@@ -192,12 +176,7 @@
 	@echo Running security.py `date`
 	./slon_ctl.py stop # security.py can deadlock with slony
 	${SHHH} ../schema/security.py --cluster -U slony
-	./slon_ctl.py --lag="0 seconds" start
-	# Migrate tables to the authdb replication set, creating the set
-	# and subscribing nodes to it as necessary.
-	./populate_auth_replication_set.py -U slony
 	# Restart slon daemons with default lag setting.
-	./slon_ctl.py stop
 	./slon_ctl.py --lag="${LAG}" start
 	# Generate a preamble for manual slonik(1) usage.
 	./preamble.py > preamble.sk
 
=== removed file 'database/replication/authdb_create.sql'
--- database/replication/authdb_create.sql	2010-03-30 05:51:30 +0000
+++ database/replication/authdb_create.sql	1970-01-01 00:00:00 +0000
@@ -1,885 +0,0 @@
--- Copyright 2009 Canonical Ltd.  This software is licensed under the
--- GNU Affero General Public License version 3 (see the file LICENSE).
-
--- Generated by:
--- pg_dump --format=p --schema-only --no-owner --no-privileges \
---     --table=Account --table=AccountPassword --table=AuthToken \
---     --table=EmailAddress --table=OpenIDAssociation \
---     --table=OpenIDAuthorization --table=OpenIDNonce \
---     --table=OpenIDRPSummary --table=ValidPersonCache \
---     --table=ValidPersonOrTeamCache launchpad_prod_4
-
---
--- PostgreSQL database dump
---
-
-SET client_encoding = 'UTF8';
-SET standard_conforming_strings = off;
-SET check_function_bodies = false;
-SET client_min_messages = warning;
-SET escape_string_warning = off;
-
-SET search_path = public, pg_catalog;
-
-SET default_tablespace = '';
-
-SET default_with_oids = false;
-
---
--- Name: account; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE account (
-    id integer NOT NULL,
-    date_created timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    creation_rationale integer NOT NULL,
-    status integer NOT NULL,
-    date_status_set timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    displayname text NOT NULL,
-    openid_identifier text DEFAULT generate_openid_identifier() NOT NULL,
-    status_comment text,
-    old_openid_identifier text
-);
-
-
---
--- Name: TABLE account; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON TABLE account IS 'An account that may be used for authenticating to Canonical or other systems.';
-
-
---
--- Name: COLUMN account.status; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.status IS 'The status of the account.';
-
-
---
--- Name: COLUMN account.date_status_set; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.date_status_set IS 'When the status was last changed.';
-
-
---
--- Name: COLUMN account.displayname; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.displayname IS 'Name to display when rendering information about this account.';
-
-
---
--- Name: COLUMN account.openid_identifier; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.openid_identifier IS 'The key used to construct an OpenID identity URL for this account.';
-
-
---
--- Name: COLUMN account.status_comment; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.status_comment IS 'The comment on the status of the account.';
-
-
---
--- Name: COLUMN account.old_openid_identifier; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN account.old_openid_identifier IS 'The previous openid_identifier, used for transitions to the current openid_identifier.';
-
-
---
--- Name: account_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE account_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: account_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE account_id_seq OWNED BY account.id;
-
-
---
--- Name: accountpassword; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE accountpassword (
-    id integer NOT NULL,
-    account integer NOT NULL,
-    password text NOT NULL
-);
-
-
---
--- Name: TABLE accountpassword; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON TABLE accountpassword IS 'A password used to authenticate an Account.';
-
-
---
--- Name: COLUMN accountpassword.password; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN accountpassword.password IS 'SSHA digest encrypted password.';
-
-
---
--- Name: accountpassword_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE accountpassword_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: accountpassword_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE accountpassword_id_seq OWNED BY accountpassword.id;
-
-
---
--- Name: authtoken; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE authtoken (
-    id integer NOT NULL,
-    date_created timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    date_consumed timestamp without time zone,
-    token_type integer NOT NULL,
-    token text NOT NULL,
-    requester integer,
-    requester_email text,
-    email text NOT NULL,
-    redirection_url text
-);
-
-
---
--- Name: TABLE authtoken; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON TABLE authtoken IS 'AuthToken stores one time tokens used by the authentication service for validating email addresses and other tasks that require verifying an email address is valid such as password recovery and account merging. This table will be cleaned occasionally to remove expired tokens. Expiry time is not yet defined.';
-
-
---
--- Name: COLUMN authtoken.date_created; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.date_created IS 'The timestamp that this request was made.';
-
-
---
--- Name: COLUMN authtoken.date_consumed; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.date_consumed IS 'The date and time when this token was consumed. It''s NULL if it hasn''t been consumed yet.';
-
-
---
--- Name: COLUMN authtoken.token_type; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.token_type IS 'The type of request, as per dbschema.TokenType.';
-
-
---
--- Name: COLUMN authtoken.token; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.token IS 'The token (not the URL) emailed used to uniquely identify this request. This token will be used to generate a URL that when clicked on will continue a workflow.';
-
-
---
--- Name: COLUMN authtoken.requester; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.requester IS 'The Account that made this request. This will be null for password recovery requests.';
-
-
---
--- Name: COLUMN authtoken.requester_email; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.requester_email IS 'The email address that was used to login when making this request. This provides an audit trail to help the end user confirm that this is a valid request. It is not a link to the EmailAddress table as this may be changed after the request is made. This field will be null for password recovery requests.';
-
-
---
--- Name: COLUMN authtoken.email; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN authtoken.email IS 'The email address that this request was sent to.';
-
-
---
--- Name: authtoken_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE authtoken_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: authtoken_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE authtoken_id_seq OWNED BY authtoken.id;
-
-
---
--- Name: emailaddress; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE emailaddress (
-    id integer NOT NULL,
-    email text NOT NULL,
-    person integer,
-    status integer NOT NULL,
-    date_created timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    account integer,
-    CONSTRAINT emailaddress__is_linked__chk CHECK (((person IS NOT NULL) OR (account IS NOT NULL)))
-);
-
-
---
--- Name: COLUMN emailaddress.email; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN emailaddress.email IS 'An email address used by a Person. The email address is stored in a casesensitive way, but must be case insensitivly unique.';
-
-
---
--- Name: emailaddress_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE emailaddress_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: emailaddress_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE emailaddress_id_seq OWNED BY emailaddress.id;
-
-
---
--- Name: openidassociation; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE openidassociation (
-    server_url character varying(2047) NOT NULL,
-    handle character varying(255) NOT NULL,
-    secret bytea,
-    issued integer,
-    lifetime integer,
-    assoc_type character varying(64),
-    CONSTRAINT secret_length_constraint CHECK ((length(secret) <= 128))
-);
-
-
---
--- Name: openidauthorization; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE openidauthorization (
-    id integer NOT NULL,
-    account integer NOT NULL,
-    client_id text,
-    date_created timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    date_expires timestamp without time zone NOT NULL,
-    trust_root text NOT NULL
-);
-
-
---
--- Name: openidauthorization_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE openidauthorization_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: openidauthorization_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE openidauthorization_id_seq OWNED BY openidauthorization.id;
-
-
---
--- Name: openidnonce; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE openidnonce (
-    server_url character varying(2047) NOT NULL,
-    "timestamp" integer NOT NULL,
-    salt character(40) NOT NULL
-);
-
-
---
--- Name: TABLE openidnonce; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON TABLE openidnonce IS 'Nonces for our OpenID consumer.';
-
-
---
--- Name: openidrpsummary; Type: TABLE; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE TABLE openidrpsummary (
-    id integer NOT NULL,
-    account integer NOT NULL,
-    openid_identifier text NOT NULL,
-    trust_root text NOT NULL,
-    date_created timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    date_last_used timestamp without time zone DEFAULT timezone('UTC'::text, now()) NOT NULL,
-    total_logins integer DEFAULT 1 NOT NULL
-);
-
-
---
--- Name: TABLE openidrpsummary; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON TABLE openidrpsummary IS 'The summary of the activity between a person and an RP.';
-
-
---
--- Name: COLUMN openidrpsummary.account; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.account IS 'The account who used the RP.';
-
-
---
--- Name: COLUMN openidrpsummary.openid_identifier; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.openid_identifier IS 'The OpenID identifier used to login.';
-
-
---
--- Name: COLUMN openidrpsummary.trust_root; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.trust_root IS 'The trust root for the RP';
-
-
---
--- Name: COLUMN openidrpsummary.date_created; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.date_created IS 'The creation date of this summary; the first time the person used the RP.';
-
-
---
--- Name: COLUMN openidrpsummary.date_last_used; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.date_last_used IS 'The date the RP was last used.';
-
-
---
--- Name: COLUMN openidrpsummary.total_logins; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON COLUMN openidrpsummary.total_logins IS 'The total number of times the RP was used by the person.';
-
-
---
--- Name: openidrpsummary_id_seq; Type: SEQUENCE; Schema: public; Owner: -
---
-
-CREATE SEQUENCE openidrpsummary_id_seq
-    INCREMENT BY 1
-    NO MAXVALUE
-    NO MINVALUE
-    CACHE 1;
-
-
---
--- Name: openidrpsummary_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: -
---
-
-ALTER SEQUENCE openidrpsummary_id_seq OWNED BY openidrpsummary.id;
-
-
---
--- Name: validpersoncache; Type: VIEW; Schema: public; Owner: -
---
-
-CREATE VIEW validpersoncache AS
-    SELECT emailaddress.person AS id FROM emailaddress, account WHERE ((((emailaddress.account = account.id) AND (emailaddress.person IS NOT NULL)) AND (emailaddress.status = 4)) AND (account.status = 20));
-
-
---
--- Name: VIEW validpersoncache; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON VIEW validpersoncache IS 'A materialized view listing the Person.ids of all valid people (but not teams).';
-
-
---
--- Name: validpersonorteamcache; Type: VIEW; Schema: public; Owner: -
---
-
-CREATE VIEW validpersonorteamcache AS
-    SELECT person.id FROM ((person LEFT JOIN emailaddress ON ((person.id = emailaddress.person))) LEFT JOIN account ON ((emailaddress.account = account.id))) WHERE ((person.teamowner IS NOT NULL) OR ((account.status = 20) AND (emailaddress.status = 4)));
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE account ALTER COLUMN id SET DEFAULT nextval('account_id_seq'::regclass);
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE accountpassword ALTER COLUMN id SET DEFAULT nextval('accountpassword_id_seq'::regclass);
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE authtoken ALTER COLUMN id SET DEFAULT nextval('authtoken_id_seq'::regclass);
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE emailaddress ALTER COLUMN id SET DEFAULT nextval('emailaddress_id_seq'::regclass);
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE openidauthorization ALTER COLUMN id SET DEFAULT nextval('openidauthorization_id_seq'::regclass);
-
-
---
--- Name: id; Type: DEFAULT; Schema: public; Owner: -
---
-
-ALTER TABLE openidrpsummary ALTER COLUMN id SET DEFAULT nextval('openidrpsummary_id_seq'::regclass);
-
-
---
--- Name: account_openid_identifier_key; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY account
-    ADD CONSTRAINT account_openid_identifier_key UNIQUE (openid_identifier);
-
-
---
--- Name: account_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY account
-    ADD CONSTRAINT account_pkey PRIMARY KEY (id);
-
-
---
--- Name: accountpassword_account_key; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY accountpassword
-    ADD CONSTRAINT accountpassword_account_key UNIQUE (account);
-
-
---
--- Name: accountpassword_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY accountpassword
-    ADD CONSTRAINT accountpassword_pkey PRIMARY KEY (id);
-
-
---
--- Name: authtoken__token__key; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY authtoken
-    ADD CONSTRAINT authtoken__token__key UNIQUE (token);
-
-
---
--- Name: authtoken_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY authtoken
-    ADD CONSTRAINT authtoken_pkey PRIMARY KEY (id);
-
-
---
--- Name: emailaddress_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY emailaddress
-    ADD CONSTRAINT emailaddress_pkey PRIMARY KEY (id);
-
-
---
--- Name: openidassociation_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY openidassociation
-    ADD CONSTRAINT openidassociation_pkey PRIMARY KEY (server_url, handle);
-
-
---
--- Name: openidauthorization_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY openidauthorization
-    ADD CONSTRAINT openidauthorization_pkey PRIMARY KEY (id);
-
-
---
--- Name: openidnonce_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY openidnonce
-    ADD CONSTRAINT openidnonce_pkey PRIMARY KEY (server_url, "timestamp", salt);
-
-
---
--- Name: openidrpsummary__account__trust_root__openid_identifier__key; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY openidrpsummary
-    ADD CONSTRAINT openidrpsummary__account__trust_root__openid_identifier__key UNIQUE (account, trust_root, openid_identifier);
-
-
---
--- Name: openidrpsummary_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
---
-
-ALTER TABLE ONLY openidrpsummary
-    ADD CONSTRAINT openidrpsummary_pkey PRIMARY KEY (id);
-
-
---
--- Name: account__old_openid_identifier__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX account__old_openid_identifier__idx ON account USING btree (old_openid_identifier);
-
-
---
--- Name: authtoken__date_consumed__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX authtoken__date_consumed__idx ON authtoken USING btree (date_consumed);
-
-
---
--- Name: authtoken__date_created__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX authtoken__date_created__idx ON authtoken USING btree (date_created);
-
-
---
--- Name: authtoken__requester__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX authtoken__requester__idx ON authtoken USING btree (requester);
-
-
---
--- Name: emailaddress__account__key; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE UNIQUE INDEX emailaddress__account__key ON emailaddress USING btree (account) WHERE ((status = 4) AND (account IS NOT NULL));
-
-
---
--- Name: INDEX emailaddress__account__key; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON INDEX emailaddress__account__key IS 'Ensures that an Account only has one preferred email address';
-
-
---
--- Name: emailaddress__lower_email__key; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX emailaddress__lower_email__key ON emailaddress USING btree (lower(email));
-
-
---
--- Name: emailaddress__person__key; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE UNIQUE INDEX emailaddress__person__key ON emailaddress USING btree (person) WHERE ((status = 4) AND (person IS NOT NULL));
-
-
---
--- Name: INDEX emailaddress__person__key; Type: COMMENT; Schema: public; Owner: -
---
-
-COMMENT ON INDEX emailaddress__person__key IS 'Ensures that a Person only has one preferred email address';
-
-
---
--- Name: emailaddress__person__status__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX emailaddress__person__status__idx ON emailaddress USING btree (person, status);
-
-
---
--- Name: openidauthorixation__account__troot__expires__client_id__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX openidauthorixation__account__troot__expires__client_id__idx ON openidauthorization USING btree (account, trust_root, date_expires, client_id);
-
-
---
--- Name: openidauthorixation__account__trust_root__key; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE UNIQUE INDEX openidauthorixation__account__trust_root__key ON openidauthorization USING btree (account, trust_root) WHERE (client_id IS NULL);
-
-
---
--- Name: openidauthorization__account__client_id__trust_root__key; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE UNIQUE INDEX openidauthorization__account__client_id__trust_root__key ON openidauthorization USING btree (account, client_id, trust_root) WHERE (client_id IS NOT NULL);
-
-
---
--- Name: openidrpsummary__openid_identifier__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX openidrpsummary__openid_identifier__idx ON openidrpsummary USING btree (openid_identifier);
-
-
---
--- Name: openidrpsummary__trust_root__idx; Type: INDEX; Schema: public; Owner: -; Tablespace: 
---
-
-CREATE INDEX openidrpsummary__trust_root__idx ON openidrpsummary USING btree (trust_root);
-
-
---
--- Name: _sl_logtrigger_200; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_200
-    AFTER INSERT OR DELETE OR UPDATE ON account
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '200', 'kvvvvvvvv');
-
-
---
--- Name: _sl_logtrigger_201; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_201
-    AFTER INSERT OR DELETE OR UPDATE ON accountpassword
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '201', 'kvv');
-
-
---
--- Name: _sl_logtrigger_274; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_274
-    AFTER INSERT OR DELETE OR UPDATE ON emailaddress
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '274', 'kvvvvv');
-
-
---
--- Name: _sl_logtrigger_335; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_335
-    AFTER INSERT OR DELETE OR UPDATE ON openidauthorization
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '335', 'kvvvvv');
-
-
---
--- Name: _sl_logtrigger_337; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_337
-    AFTER INSERT OR DELETE OR UPDATE ON openidrpsummary
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '337', 'kvvvvvv');
-
-
---
--- Name: _sl_logtrigger_438; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_438
-    AFTER INSERT OR DELETE OR UPDATE ON authtoken
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '438', 'kvvvvvvvv');
-
-
---
--- Name: _sl_logtrigger_439; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_439
-    AFTER INSERT OR DELETE OR UPDATE ON openidassociation
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '439', 'kkvvvv');
-
-
---
--- Name: _sl_logtrigger_445; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER _sl_logtrigger_445
-    AFTER INSERT OR DELETE OR UPDATE ON openidnonce
-    FOR EACH ROW
-    EXECUTE PROCEDURE _sl.logtrigger('_sl', '445', 'kkk');
-
-
---
--- Name: set_date_status_set_t; Type: TRIGGER; Schema: public; Owner: -
---
-
-CREATE TRIGGER set_date_status_set_t
-    BEFORE UPDATE ON account
-    FOR EACH ROW
-    EXECUTE PROCEDURE set_date_status_set();
-
-
---
--- Name: accountpassword_account_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY accountpassword
-    ADD CONSTRAINT accountpassword_account_fkey FOREIGN KEY (account) REFERENCES account(id) ON DELETE CASCADE;
-
-
---
--- Name: authtoken__requester__fk; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY authtoken
-    ADD CONSTRAINT authtoken__requester__fk FOREIGN KEY (requester) REFERENCES account(id);
-
-
---
--- Name: emailaddress__account__fk; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY emailaddress
-    ADD CONSTRAINT emailaddress__account__fk FOREIGN KEY (account) REFERENCES account(id) ON DELETE SET NULL;
-
-
---
--- Name: openidauthorization__account__fk; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY openidauthorization
-    ADD CONSTRAINT openidauthorization__account__fk FOREIGN KEY (account) REFERENCES account(id);
-
-
---
--- Name: openidrpsummary_account_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY openidrpsummary
-    ADD CONSTRAINT openidrpsummary_account_fkey FOREIGN KEY (account) REFERENCES account(id);
-
-
---
--- PostgreSQL database dump complete
---
-
-CREATE INDEX emailaddress__account__status__idx
-    ON EmailAddress(account, status);
-
-
--- Permissions for Ubuntu SSO server testing on staging.
-
--- Mirrored from sso_auth user 2010-01-12.
--- These tables will eventually not be available.
---
-GRANT SELECT,INSERT,DELETE,UPDATE ON TABLE account TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE account_id_seq TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE,UPDATE ON TABLE accountpassword TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE accountpassword_id_seq TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE,UPDATE ON TABLE authtoken TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE authtoken_id_seq TO ubuntu_sso;
-GRANT SELECT ON TABLE person TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE,UPDATE ON TABLE emailaddress TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE emailaddress_id_seq TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE ON TABLE openidassociation TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE,UPDATE ON TABLE openidauthorization TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE openidauthorization_id_seq TO ubuntu_sso;
-GRANT SELECT,INSERT,DELETE ON TABLE openidnonce TO ubuntu_sso;
-GRANT SELECT,INSERT,UPDATE ON TABLE openidrpsummary TO ubuntu_sso;
-GRANT USAGE ON SEQUENCE openidrpsummary_id_seq TO ubuntu_sso;
-GRANT SELECT ON SEQUENCE person_id_seq TO ubuntu_sso;
-GRANT SELECT ON TABLE personlocation TO ubuntu_sso;
-GRANT SELECT ON SEQUENCE personlocation_id_seq TO ubuntu_sso;
-GRANT SELECT ON TABLE teamparticipation TO ubuntu_sso;
-GRANT SELECT ON SEQUENCE teamparticipation_id_seq TO ubuntu_sso;
-
--- Permissions on the Ubuntu SSO tables.
---
-GRANT SELECT, INSERT, DELETE, UPDATE ON TABLE auth_permission,
-auth_group_permissions, auth_group, auth_user, auth_user_groups,
-auth_user_user_permissions, auth_message, django_content_type,
-django_session, django_site, django_admin_log,
-ssoopenidrpconfig TO ubuntu_sso;
-
-GRANT USAGE ON SEQUENCE auth_group_id_seq,
-auth_group_permissions_id_seq, auth_message_id_seq,
-auth_permission_id_seq, auth_user_groups_id_seq, auth_user_id_seq,
-auth_user_user_permissions_id_seq, django_admin_log_id_seq,
-django_content_type_id_seq, django_site_id_seq,
-ssoopenidrpconfig_id_seq TO ubuntu_sso;
-
--- Permissions on the lpmirror tables (mirrors of relevant Launchpad
--- information, available even when Launchpad database upgrades are in
--- progress).
-GRANT SELECT
-ON TABLE
-    lp_person, lp_personlocation, lp_teamparticipation, lp_account
-TO ubuntu_sso;
-
 
=== removed file 'database/replication/authdb_drop.sql'
--- database/replication/authdb_drop.sql	2009-11-11 10:32:35 +0000
+++ database/replication/authdb_drop.sql	1970-01-01 00:00:00 +0000
@@ -1,14 +0,0 @@
--- Copyright 2009 Canonical Ltd.  This software is licensed under the
--- GNU Affero General Public License version 3 (see the file LICENSE).
-
-SET client_min_messages=ERROR;
-
--- Drop everything in the authdb replication set.
-DROP TABLE IF EXISTS Account CASCADE;
-DROP TABLE IF EXISTS AccountPassword CASCADE;
-DROP TABLE IF EXISTS AuthToken CASCADE;
-DROP TABLE IF EXISTS EmailAddress CASCADE;
-DROP TABLE IF EXISTS OpenIDAssociation CASCADE;
-DROP TABLE IF EXISTS OpenIDAuthorization CASCADE;
-DROP TABLE IF EXISTS OpenIDNonce CASCADE;
-DROP TABLE IF EXISTS OpenIDRPSummary;
 
=== removed file 'database/replication/authdb_sequences.sql'
--- database/replication/authdb_sequences.sql	2010-01-13 06:54:32 +0000
+++ database/replication/authdb_sequences.sql	1970-01-01 00:00:00 +0000
@@ -1,22 +0,0 @@
--- Repair sequences in the authdb replication set. We need to do this because
--- we cannot restore the sequence values from the dump when restoring the
--- data using pg_restore --data-only.
-
-SELECT setval('account_id_seq', max(id)) AS Account
-FROM Account;
-
-SELECT setval('accountpassword_id_seq', max(id)) AS AccountPassword
-FROM AccountPassword;
-
-SELECT setval('authtoken_id_seq', max(id)) AS AuthToken
-FROM AuthToken;
-
-SELECT setval('emailaddress_id_seq', max(id)) AS EmailAddress
-FROM EmailAddress;
-
-SELECT setval('openidauthorization_id_seq', max(id)) AS OpenIDAuthorization
-FROM OpenIDAuthorization;
-
-SELECT setval('openidrpsummary_id_seq', max(id)) AS OpenIDRPSummary
-FROM OpenIDRPSummary;
-
 
=== modified file 'database/replication/helpers.py'
--- database/replication/helpers.py	2010-02-26 03:34:49 +0000
+++ database/replication/helpers.py	2010-04-27 02:13:38 +0000
@@ -28,20 +28,15 @@
 
 # Replication set id constants. Don't change these without DBA help.
 LPMAIN_SET_ID = 1
-AUTHDB_SET_ID = 2
 HOLDING_SET_ID = 666
+LPMIRROR_SET_ID = 4
 
-# Seed tables for the authdb replication set to be passed to
+# Seed tables for the lpmain replication set to be passed to
 # calculate_replication_set().
-AUTHDB_SEED = frozenset([
+LPMAIN_SEED = frozenset([
     ('public', 'account'),
+    ('public', 'openidnonce'),
     ('public', 'openidassociation'),
-    ('public', 'openidnonce'),
-    ])
-
-# Seed tables for the lpmain replication set to be passed to
-# calculate_replication_set().
-LPMAIN_SEED = frozenset([
     ('public', 'person'),
     ('public', 'launchpaddatabaserevision'),
     ('public', 'databasereplicationlag'),
@@ -57,7 +52,6 @@
     ('public', 'launchpadstatistic'),
     ('public', 'parsedapachelog'),
     ('public', 'shipitsurvey'),
-    ('public', 'openidassociations'), # Remove this in April 2009 or later.
     ('public', 'databasereplicationlag'),
     ])
 
@@ -70,6 +64,7 @@
     'public.secret', 'public.sessiondata', 'public.sessionpkgdata',
     # Mirror tables, per Bug #489078. These tables have their own private
     # replication set that is setup manually.
+    'public.lp_account',
     'public.lp_person',
     'public.lp_personlocation',
     'public.lp_teamparticipation',
@@ -176,12 +171,13 @@
         script = preamble() + script
 
     if sync is not None:
-        script = script + dedent("""\
+        sync_script = dedent("""\
             sync (id = @master_node);
             wait for event (
-                origin = ALL, confirmed = ALL,
+                origin = @master_node, confirmed = ALL,
                 wait on = @master_node, timeout = %d);
             """ % sync)
+        script = script + sync_script
 
     # Copy the script to a NamedTemporaryFile rather than just pumping it
     # to slonik via stdin. This way it can be examined if slonik appears
@@ -192,7 +188,7 @@
 
     # Run slonik
     log.debug("Executing slonik script %s" % script_on_disk.name)
-    log.log(DEBUG2, script)
+    log.log(DEBUG2, 'Running script:\n%s' % script)
     returncode = subprocess.call(['slonik', script_on_disk.name])
 
     if returncode != 0:
@@ -323,10 +319,10 @@
         cluster name = sl;
 
         # Symbolic ids for replication sets.
-        define lpmain_set  %d;
-        define authdb_set  %d;
-        define holding_set %d;
-        """ % (LPMAIN_SET_ID, AUTHDB_SET_ID, HOLDING_SET_ID))]
+        define lpmain_set   %d;
+        define holding_set  %d;
+        define lpmirror_set %d;
+        """ % (LPMAIN_SET_ID, HOLDING_SET_ID, LPMIRROR_SET_ID))]
 
     if master_node is not None:
         preamble.append(dedent("""\
@@ -503,19 +499,6 @@
         raise ReplicationConfigError(
             "Unreplicated sequences: %s" % repr(unrepl_sequences))
 
-    authdb_tables, authdb_sequences = calculate_replication_set(
-        cur, AUTHDB_SEED)
     lpmain_tables, lpmain_sequences = calculate_replication_set(
         cur, LPMAIN_SEED)
 
-    confused_tables = authdb_tables.intersection(lpmain_tables)
-    if confused_tables:
-        raise ReplicationConfigError(
-            "Tables exist in multiple replication sets: %s"
-            % repr(confused_tables))
-    confused_sequences = authdb_sequences.intersection(lpmain_sequences)
-    if confused_sequences:
-        raise ReplicationConfigError(
-            "Sequences exist in multiple replication sets: %s"
-            % repr(confused_sequences))
-
 
=== modified file 'database/replication/initialize.py'
--- database/replication/initialize.py	2010-01-22 06:25:48 +0000
+++ database/replication/initialize.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
@@ -88,54 +88,13 @@
     helpers.sync(120) # Will exit on failure.
 
 
-def create_replication_sets(
-    authdb_tables, authdb_sequences, lpmain_tables, lpmain_sequences):
+def create_replication_sets(lpmain_tables, lpmain_sequences):
     """Create the replication sets."""
     log.info('Creating Slony-I replication sets.')
 
-    # Instead of creating both the authdb and lpmain replication sets,
-    # we just create the lpmain replication set containing everything.
-    # This way, we can then test the populate_auth_replication_set.py
-    # migration script that moves the relevant tables from the lpmain
-    # replication set to the authdb replication set.
-    # We will turn this behavior off once we are running two
-    # replication sets in production and remove the migration script.
-    lpmain_tables = lpmain_tables.union(authdb_tables)
-    lpmain_sequences = lpmain_sequences.union(authdb_sequences)
-
     script = ["try {"]
-    # script,append("""
-    #     echo 'Creating AuthDB replication set (@authdb_set)';
-    #     create set (
-    #         id=@authdb_set, origin=@master_node,
-    #         comment='AuthDB tables and sequences');
-    #     """)
 
-    # entry_id = 1
-    # for table in sorted(authdb_tables):
-    #     script.append("""
-    #         echo 'Adding %(table)s to replication set @authdb_set';
-    #         set add table (
-    #             set id=@authdb_set,
-    #             origin=@master_node,
-    #             id=%(entry_id)d,
-    #             fully qualified name='%(table)s');
-    #         """ % vars())
-    #     entry_id += 1
-    # entry_id = 1
-    # for sequence in sorted(authdb_sequences):
-    #     script.append("""
-    #         echo 'Adding %(sequence)s to replication set @authdb_set';
-    #         set add sequence (
-    #             set id=@authdb_set,
-    #             origin=@master_node,
-    #             id=%(entry_id)d,
-    #             fully qualified name='%(sequence)s');
-    #         """ % vars())
-    #     entry_id += 1
-    #
-    # assert entry_id < 200, 'authdb replcation set has > 200 objects???'
-    entry_id = 200
+    entry_id = 1
 
     script.append("""
         echo 'Creating LPMain replication set (@lpmain_set)';
@@ -157,7 +116,7 @@
             """ % vars())
         entry_id += 1
 
-    entry_id = 200
+    entry_id = 1
     script.append(
         "echo 'Adding %d sequences to replication set @lpmain_set';"
         % len(lpmain_sequences))
@@ -199,9 +158,6 @@
     con.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
     global cur
     cur = con.cursor()
-    log.debug("Calculating authdb replication set.")
-    authdb_tables, authdb_sequences = helpers.calculate_replication_set(
-        cur, helpers.AUTHDB_SEED)
     log.debug("Calculating lpmain replication set.")
     lpmain_tables, lpmain_sequences = helpers.calculate_replication_set(
         cur, helpers.LPMAIN_SEED)
@@ -212,8 +168,7 @@
     fails = 0
     for table in all_tables_in_schema(cur, 'public'):
         times_seen = 0
-        for table_set in [
-            authdb_tables, lpmain_tables, helpers.IGNORED_TABLES]:
+        for table_set in [lpmain_tables, helpers.IGNORED_TABLES]:
             if table in table_set:
                 times_seen += 1
         if times_seen == 0:
@@ -224,8 +179,7 @@
             fails += 1
     for sequence in all_sequences_in_schema(cur, 'public'):
         times_seen = 0
-        for sequence_set in [
-            authdb_sequences, lpmain_sequences, helpers.IGNORED_SEQUENCES]:
+        for sequence_set in [lpmain_sequences, helpers.IGNORED_SEQUENCES]:
             if sequence in sequence_set:
                 times_seen += 1
         if times_seen == 0:
@@ -242,8 +196,7 @@
 
     ensure_live()
 
-    create_replication_sets(
-        authdb_tables, authdb_sequences, lpmain_tables, lpmain_sequences)
+    create_replication_sets(lpmain_tables, lpmain_sequences)
 
     helpers.sync(0)
 
 
=== modified file 'database/replication/new-slave.py'
--- database/replication/new-slave.py	2010-01-13 08:40:48 +0000
+++ database/replication/new-slave.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
@@ -22,10 +22,9 @@
 from canonical.database.sqlbase import (
     connect_string, ISOLATION_LEVEL_AUTOCOMMIT)
 from canonical.launchpad.scripts import db_options, logger_options, logger
-from canonical.launchpad.webapp.adapter import _auth_store_tables
 
 import replication.helpers
-from replication.helpers import AUTHDB_SET_ID, LPMAIN_SET_ID
+from replication.helpers import LPMAIN_SET_ID
 
 def main():
     parser = OptionParser(
@@ -77,8 +76,6 @@
     # Get the connection string for masters.
     lpmain_connection_string = get_master_connection_string(
         source_connection, parser, LPMAIN_SET_ID) or source_connection_string
-    authdb_connection_string = get_master_connection_string(
-        source_connection, parser, AUTHDB_SET_ID) or source_connection_string
 
     # Sanity check the target connection string.
     target_connection_string = ConnectionString(raw_target_connection_string)
@@ -130,31 +127,6 @@
         log.error("Failed to duplicate database schema.")
         return 1
 
-    # Drop the authdb replication set tables we just restored, as they
-    # will be broken if the authdb master is a seperate database to the
-    # lpmain master.
-    log.debug("Dropping (possibly corrupt) authdb tables.")
-    cur = target_con.cursor()
-    for table_name in _auth_store_tables:
-        cur.execute("DROP TABLE IF EXISTS %s CASCADE" % table_name)
-    target_con.commit()
-
-    # Duplicate the authdb schema.
-    log.info("Duplicating authdb schema from '%s' to '%s'" % (
-        authdb_connection_string, target_connection_string))
-    table_args = ["--table=%s" % table for table in _auth_store_tables]
-    # We need to restore the two cross-replication-set views that where
-    # dropped as a side effect of dropping the auth store tables.
-    table_args.append("--table=ValidPersonCache")
-    table_args.append("--table=ValidPersonOrTeamCache")
-    cmd = "pg_dump --schema-only --no-privileges %s %s | psql -1 -q %s" % (
-        ' '.join(table_args),
-        source_connection_string.asPGCommandLineArgs(),
-        target_connection_string.asPGCommandLineArgs())
-    if subprocess.call(cmd, shell=True) != 0:
-        log.error("Failed to duplicate database schema.")
-        return 1
-
     # Trash the broken Slony tables we just duplicated.
     log.debug("Removing slony cruft.")
     cur = target_con.cursor()
@@ -163,21 +135,30 @@
     del target_con
 
     # Get a list of existing set ids that can be subscribed too. This
-    # is all sets where the origin is the master_node, and set 2 if
-    # the master happens to be configured as a forwarding slave. We
+    # is all sets where the origin is the master_node. We
     # don't allow other sets where the master is configured as a
     # forwarding slave as we have to special case rebuilding the database
-    # schema (such as we do for the authdb replication set 2).
+    # schema, and we want to avoid cascading slave configurations anyway
+    # since we are running an antique Slony-I at the moment - keep it
+    # simple!
+    # We order the sets smallest to largest by number of tables.
+    # This should let us subscribe the quickest sets first for more
+    # immediate feedback.
     source_connection.rollback()
     master_node = replication.helpers.get_master_node(source_connection)
     cur = source_connection.cursor()
     cur.execute("""
-        SELECT set_id FROM _sl.sl_set WHERE set_origin=%d
-        UNION
-        SELECT sub_set AS set_id FROM _sl.sl_subscribe
-        WHERE sub_receiver=%d AND sub_forward IS TRUE AND sub_active IS TRUE
-            AND sub_set=2
-        """ % (master_node.node_id, master_node.node_id))
+        SELECT set_id
+        FROM _sl.sl_set, (
+            SELECT tab_set, count(*) AS tab_count
+            FROM _sl.sl_table GROUP BY tab_set
+            ) AS TableCounts
+        WHERE
+            set_origin=%d
+            AND tab_set = set_id
+        ORDER BY tab_count
+        """
+        % (master_node.node_id,))
     set_ids = [set_id for set_id, in cur.fetchall()]
     log.debug("Discovered set ids %s" % repr(list(set_ids)))
 
@@ -209,19 +190,32 @@
         } on error { echo 'Failed.'; exit 1; }
         """)
 
+    full_sync = []
+    sync_nicknames = [node.nickname for node in existing_nodes]
+    sync_nicknames.append('new_node');
+    for nickname in sync_nicknames:
+        full_sync.append(dedent("""\
+            echo 'Waiting for %(nickname)s sync.';
+            sync (id=@%(nickname)s);
+            wait for event (
+                origin = @%(nickname)s, confirmed=ALL,
+                wait on = @%(nickname)s, timeout=0);
+            """ % {'nickname': nickname}))
+    full_sync = '\n'.join(full_sync)
+    script += full_sync
+
     for set_id in set_ids:
-
         script += dedent("""\
         echo 'Subscribing new node to set %d.';
         subscribe set (
             id=%d, provider=@master_node, receiver=@new_node, forward=yes);
-
-        echo 'Waiting for sync... this might take a while...';
+        echo 'Waiting for subscribe to start processing.';
         sync (id = @master_node);
         wait for event (
-            origin = ALL, confirmed = ALL,
+            origin = @master_node, confirmed = ALL,
             wait on = @master_node, timeout = 0);
         """ % (set_id, set_id))
+        script += full_sync
 
     replication.helpers.execute_slonik(script)
 
 
=== removed file 'database/replication/populate_auth_replication_set.py'
--- database/replication/populate_auth_replication_set.py	2009-10-17 14:06:03 +0000
+++ database/replication/populate_auth_replication_set.py	1970-01-01 00:00:00 +0000
@@ -1,177 +0,0 @@
-#!/usr/bin/python2.5
-#
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Populate the auth replication set.
-
-This script moves the the SSO tables from the main replication set to
-the auth replication set.
-
-Once it has been run on production, these tables can no longer be
-maintained using the Launchpad database maintenance scripts
-(upgrade.py, security.py etc.).
-
-We do this so Launchpad database upgrades do not lock the SSO tables,
-allowing the SSO service to continue to operate.
-
-This is a single shot script.
-"""
-
-__metaclass__ = type
-__all__ = []
-
-import _pythonpath
-
-import sys
-from textwrap import dedent
-from optparse import OptionParser
-
-from canonical.database.sqlbase import (
-    connect, ISOLATION_LEVEL_AUTOCOMMIT, sqlvalues)
-from canonical.launchpad.scripts import db_options, logger_options, logger
-
-import replication.helpers
-
-def create_auth_set(cur):
-    """Create the auth replication set if it doesn't already exist."""
-    cur.execute("SELECT TRUE FROM _sl.sl_set WHERE set_id=2")
-    if cur.fetchone() is not None:
-        log.info("Auth set already exists.")
-        return
-    slonik_script = dedent("""\
-        create set (
-            id=@authdb_set, origin=@master_node,
-            comment='SSO service tables');
-        """)
-    log.info("Creating authdb replication set.")
-    replication.helpers.execute_slonik(slonik_script, sync=0)
-
-
-def subscribe_auth_set(cur):
-    """The authdb set subscription much match the lpmain set subscription.
-
-    This is a requirement to move stuff between replication sets. It
-    is also what we want (all nodes replicating everything).
-    """
-    cur.execute("""
-        SELECT sub_receiver FROM _sl.sl_subscribe WHERE sub_set = 1
-        EXCEPT
-        SELECT sub_receiver FROM _sl.sl_subscribe WHERE sub_set = 2
-        """)
-    for node_id in (node_id for node_id, in cur.fetchall()):
-        log.info("Subscribing Node #%d to authdb replication set" % node_id)
-        success = replication.helpers.execute_slonik(dedent("""\
-            subscribe set (
-                id = @authdb_set, provider = @master_node,
-                receiver = %d, forward = yes);
-            """ % node_id), sync=0)
-        if not success:
-            log.error("Slonik failed. Exiting.")
-            sys.exit(1)
-
-
-def migrate_tables_and_sequences(cur):
-    auth_tables, auth_sequences = (
-        replication.helpers.calculate_replication_set(
-            cur, replication.helpers.AUTHDB_SEED))
-
-    slonik_script = ["try {"]
-    for table_fqn in auth_tables:
-        namespace, table_name = table_fqn.split('.')
-        cur.execute("""
-            SELECT tab_id, tab_set
-            FROM _sl.sl_table
-            WHERE tab_nspname = %s AND tab_relname = %s
-            """ % sqlvalues(namespace, table_name))
-        try:
-            table_id, set_id = cur.fetchone()
-        except IndexError:
-            log.error("Table %s not found in _sl.sl_tables" % table_fqn)
-            sys.exit(1)
-        if set_id == 1:
-            slonik_script.append("echo 'Moving table %s';" % table_fqn)
-            slonik_script.append(
-                "set move table "
-                "(origin=@master_node, id=%d, new set=@authdb_set);"
-                % table_id)
-        elif set_id == 2:
-            log.warn(
-                "Table %s already in authdb replication set"
-                % table_fqn)
-        else:
-            log.error("Unknown replication set %s" % set_id)
-            sys.exit(1)
-
-    for sequence_fqn in auth_sequences:
-        namespace, sequence_name = sequence_fqn.split('.')
-        cur.execute("""
-            SELECT seq_id, seq_set
-            FROM _sl.sl_sequence
-            WHERE seq_nspname = %s AND seq_relname = %s
-            """ % sqlvalues(namespace, sequence_name))
-        try:
-            sequence_id, set_id = cur.fetchone()
-        except IndexError:
-            log.error(
-                "Sequence %s not found in _sl.sl_sequences" % sequence_fqn)
-            sys.exit(1)
-        if set_id == 1:
-            slonik_script.append("echo 'Moving sequence %s';" % sequence_fqn)
-            slonik_script.append(
-                "set move sequence "
-                "(origin=@master_node, id=%d, new set=@authdb_set);"
-                % sequence_id)
-        elif set_id ==2:
-            log.warn(
-                "Sequence %s already in authdb replication set."
-                % sequence_fqn)
-        else:
-            log.error("Unknown replication set %s" % set_id)
-            sys.exit(1)
-
-    if len(slonik_script) == 1:
-        log.warn("No tables or sequences to migrate.")
-        return
-
-    slonik_script.append(dedent("""\
-        } on error {
-            echo 'Failed to move one or more tables or sequences.';
-            exit 1;
-        }
-        """))
-
-    slonik_script = "\n".join(slonik_script)
-
-    log.info("Running migration script...")
-    if not replication.helpers.execute_slonik(slonik_script, sync=0):
-        log.error("Slonik failed. Exiting.")
-        sys.exit(1)
-
-
-def main():
-    parser = OptionParser()
-    db_options(parser)
-    logger_options(parser)
-    options, args = parser.parse_args()
-
-    global log
-    log = logger(options)
-
-    con = connect('slony', isolation=ISOLATION_LEVEL_AUTOCOMMIT)
-    cur = con.cursor()
-
-    # Don't start until cluster is synced.
-    log.info("Waiting for sync.")
-    replication.helpers.sync(0)
-
-    create_auth_set(cur)
-    subscribe_auth_set(cur)
-    migrate_tables_and_sequences(cur)
-
-
-log = None # Global log
-
-
-if __name__ == '__main__':
-    main()
 
=== modified file 'database/replication/preamble.py'
--- database/replication/preamble.py	2009-10-17 14:06:03 +0000
+++ database/replication/preamble.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/replication/repair-restored-db.py'
--- database/replication/repair-restored-db.py	2010-01-22 06:25:48 +0000
+++ database/replication/repair-restored-db.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/replication/report.py'
--- database/replication/report.py	2009-10-17 14:06:03 +0000
+++ database/replication/report.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/replication/slon_ctl.py'
--- database/replication/slon_ctl.py	2009-10-17 14:06:03 +0000
+++ database/replication/slon_ctl.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/replication/sync.py'
--- database/replication/sync.py	2010-01-06 15:52:31 +0000
+++ database/replication/sync.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2010 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/diagram.py'
--- database/schema/diagram.py	2010-02-09 01:31:05 +0000
+++ database/schema/diagram.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/emptytables.py'
--- database/schema/emptytables.py	2009-10-17 14:06:03 +0000
+++ database/schema/emptytables.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/fti.py'
--- database/schema/fti.py	2009-11-06 21:58:50 +0000
+++ database/schema/fti.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/online_fti_updater.py'
--- database/schema/online_fti_updater.py	2009-10-17 14:06:03 +0000
+++ database/schema/online_fti_updater.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== added file 'database/schema/patch-2207-47-0.sql'
--- database/schema/patch-2207-47-0.sql	1970-01-01 00:00:00 +0000
+++ database/schema/patch-2207-47-0.sql	2010-04-27 02:13:38 +0000
@@ -0,0 +1,6 @@
+SET client_min_messages=ERROR;
+
+UPDATE BugWatchActivity SET result = 9 WHERE result IS NULL;
+ALTER TABLE BugWatchActivity ALTER COLUMN result SET NOT NULL;
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 47, 0);
 
=== added file 'database/schema/patch-2207-48-0.sql'
--- database/schema/patch-2207-48-0.sql	1970-01-01 00:00:00 +0000
+++ database/schema/patch-2207-48-0.sql	2010-04-27 02:13:38 +0000
@@ -0,0 +1,27 @@
+SET client_min_messages=ERROR;
+
+ALTER TABLE EmailAddress ADD CONSTRAINT emailaddress__person__fk
+    FOREIGN KEY (person) REFERENCES Person;
+
+CREATE TEMPORARY TABLE DudAccountLinks AS
+SELECT Person.id
+FROM Person
+LEFT OUTER JOIN Account ON Person.account = Account.id
+WHERE Person.account IS NOT NULL AND Account.id IS NULL;
+
+UPDATE Person SET account = NULL
+FROM DudAccountLinks
+WHERE Person.id = DudAccountLinks.id;
+
+DROP TABLE DudAccountLinks;
+
+ALTER TABLE Person ADD CONSTRAINT person__account__fk
+    FOREIGN KEY (account) REFERENCES Account;
+
+ALTER TABLE MailingListSubscription
+    ADD CONSTRAINT mailinglistsubscription__email_address_fk
+    FOREIGN KEY (email_address) REFERENCES EmailAddress
+    ON DELETE CASCADE;
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 48, 0);
+
 
=== modified file 'database/schema/pending/add-mailing-list-experts.py'
--- database/schema/pending/add-mailing-list-experts.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/add-mailing-list-experts.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/create-openid-rp-configs.py'
--- database/schema/pending/create-openid-rp-configs.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/create-openid-rp-configs.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/gnu-savannah-celebrity.py'
--- database/schema/pending/gnu-savannah-celebrity.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/gnu-savannah-celebrity.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/migrate_kde_potemplates.py'
--- database/schema/pending/migrate_kde_potemplates.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/migrate_kde_potemplates.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/new-person-columns.py'
--- database/schema/pending/new-person-columns.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/new-person-columns.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== added file 'database/schema/pending/patch-2207-49-0.sql'
--- database/schema/pending/patch-2207-49-0.sql	1970-01-01 00:00:00 +0000
+++ database/schema/pending/patch-2207-49-0.sql	2010-04-27 02:13:38 +0000
@@ -0,0 +1,16 @@
+SET client_min_messages=ERROR;
+
+DROP VIEW RevisionNumber;
+
+ALTER TABLE BranchRevision DROP COLUMN id;
+ALTER TABLE BranchRevision
+    ADD CONSTRAINT branchrevision_pkey
+        PRIMARY KEY (branch, revision);
+ALTER TABLE BranchRevision
+    DROP CONSTRAINT revision__branch__revision__key,
+    DROP CONSTRAINT revision__revision__branch__key,
+    DROP CONSTRAINT revisionnumber_branch_sequence_unique;
+CREATE UNIQUE INDEX branchrevision__branch__sequence__key
+    ON BranchRevision (branch, sequence) WHERE sequence IS NOT NULL;
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2207, 49, 0);
 
=== modified file 'database/schema/pending/prune-nonce.py'
--- database/schema/pending/prune-nonce.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/prune-nonce.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/update-shippingrequest-types.py'
--- database/schema/pending/update-shippingrequest-types.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/update-shippingrequest-types.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/pending/update-translation-credits.py'
--- database/schema/pending/update-translation-credits.py	2009-10-17 14:06:03 +0000
+++ database/schema/pending/update-translation-credits.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
@@ -54,7 +54,7 @@
              pomsgid
         WHERE
             posubmission.active IS TRUE AND
-            posubmission.pomsgset=pomsgset.id AND 
+            posubmission.pomsgset=pomsgset.id AND
             potmsgset=potmsgset.id AND
             primemsgid=pomsgid.id AND
             published IS NOT TRUE AND
@@ -77,7 +77,7 @@
              pomsgid
         WHERE
             posubmission.active IS FALSE AND
-            posubmission.pomsgset=pomsgset.id AND 
+            posubmission.pomsgset=pomsgset.id AND
             pomsgset.potmsgset=potmsgset.id AND
             potmsgset.primemsgid=pomsgid.id AND
             posubmission.published IS TRUE AND
 
=== modified file 'database/schema/reset_sequences.py'
--- database/schema/reset_sequences.py	2009-10-17 14:06:03 +0000
+++ database/schema/reset_sequences.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/security.cfg'
--- database/schema/security.cfg	2010-04-21 19:41:18 +0000
+++ database/schema/security.cfg	2010-04-27 02:13:38 +0000
@@ -35,20 +35,27 @@
 public.is_printable_ascii(text)            = EXECUTE
 public.launchpaddatabaserevision           = SELECT
 public.name_blacklist_match(text)          = EXECUTE
-public.fticache                            =
 public.pillarname                          = SELECT
 public.ulower(text)                        = EXECUTE
-public._killall_backends(text)             =
 public.generate_openid_identifier()        = EXECUTE
 public.getlocalnodeid()                    = EXECUTE
 public.replication_lag()                   = EXECUTE
 public.replication_lag(integer)            = EXECUTE
 public.assert_patch_applied(integer, integer, integer) = EXECUTE
+# Explicitly state 'no permissions on these objects' to silence
+# security.py warnings.
+public.fticache                            =
+public._killall_backends(text)             =
 public.exclusivelocks                      =
 public.alllocks                            =
 public.pgstattuple(oid)                    =
 public.pgstattuple(text)                   =
 public.bugnotificationarchive              =
+public.lp_account                          =
+public.lp_personlocation                   =
+public.lp_person                           =
+public.lp_teamparticipation                =
+public.bug_update_latest_patch_uploaded(integer) =
 
 [ro]
 # A user with full readonly access to the database. Generally used for
@@ -61,59 +68,6 @@
 type=user
 groups=admin
 
-[sso_auth]
-# authdb replication set write access from the SSO service.
-type=user
-public.account                          = SELECT, INSERT, UPDATE, DELETE
-public.accountpassword                  = SELECT, INSERT, UPDATE, DELETE
-public.authtoken                        = SELECT, INSERT, UPDATE, DELETE
-public.emailaddress                     = SELECT, INSERT, UPDATE, DELETE
-public.openidrpsummary                  = SELECT, INSERT, UPDATE
-public.openidassociation                = SELECT, INSERT, DELETE
-public.openidnonce                      = SELECT, INSERT, DELETE
-public.openidauthorization              = SELECT, INSERT, UPDATE, DELETE
-public.person                           = SELECT
-public.personlocation                   = SELECT
-public.teamparticipation                = SELECT
-
-[sso_main]
-# main replication set access from the SSO service login.launchpad.net
-type=user
-public.language                         = SELECT
-public.openidrpconfig                   = SELECT
-public.person                           = SELECT
-public.personlanguage                   = SELECT
-public.personlocation                   = SELECT
-public.shippingrequest                  = SELECT
-public.teammembership                   = SELECT
-public.teamparticipation                = SELECT
-public.validpersoncache                 = SELECT
-# Needed for person.preferredemail to work.
-public.emailaddress                     = SELECT
-# Needed for OpenID login to work - Bug #352727
-public.country                          = SELECT
-# Needed for display of OpenID consumer logo per Bug #353926
-public.libraryfilealias                 = SELECT
-public.libraryfilecontent               = SELECT
-
-[launchpad_auth]
-# authdb replication set access from the main Z3 application.
-type=user
-public.account                          = SELECT, INSERT, UPDATE, DELETE
-public.accountpassword                  = SELECT, INSERT, UPDATE, DELETE
-public.authtoken                        = SELECT, INSERT, UPDATE
-public.emailaddress                     = SELECT, INSERT, UPDATE, DELETE
-public.language                         = SELECT
-public.openidrpconfig                   = SELECT
-public.openidrpsummary                  = SELECT
-public.person                           = SELECT
-public.personlanguage                   = SELECT
-public.teammembership                   = SELECT
-public.teamparticipation                = SELECT
-# XXX 2009-05-07 stub bug=373252: SELECT and DELETE permissions required
-# for garbo.py. INSERT permission needed for the tests.
-public.openidassociation                = SELECT, INSERT, DELETE
-
 [launchpad_main]
 # lpmain replication set access from the main Z3 application.
 type=user
@@ -884,34 +838,6 @@
 public.country                                  = SELECT
 public.parsedapachelog                          = SELECT, INSERT, UPDATE
 
-[sourcerer]
-type=user
-groups=script
-public.archive                                  = SELECT
-public.archivearch                              = SELECT
-public.branch                                   = SELECT, INSERT, UPDATE
-public.revision                                 = SELECT, INSERT, UPDATE
-# Karma
-public.karma                                    = SELECT, INSERT
-public.karmaaction                              = SELECT
-# To get at a source package's manifest
-public.distribution                             = SELECT
-public.distroseries                             = SELECT
-public.sourcepackagename                        = SELECT
-public.sourcepackagepublishinghistory           = SELECT
-public.sourcepackagerelease                     = SELECT, UPDATE
-public.sourcepackagereleasefile                 = SELECT
-# To get at an upstream product's manifest
-public.product                                  = SELECT
-public.productseries                            = SELECT
-public.productrelease                           = SELECT, UPDATE
-public.productreleasefile                       = SELECT
-# To get from source package to upstream
-public.packaging                                = SELECT
-# To get stuff from the librarian
-public.libraryfilealias                         = SELECT
-public.libraryfilecontent                       = SELECT
-
 [write]
 type=group
 # Full access except for tables that are exclusively updated by
@@ -1688,7 +1614,7 @@
 public.teamparticipation                = SELECT
 public.validpersoncache                 = SELECT
 
-[mp-creation-job]
+[merge-proposal-jobs]
 type=user
 groups=script
 public.account                          = SELECT
@@ -1725,18 +1651,6 @@
 public.teamparticipation                = SELECT
 public.validpersoncache                 = SELECT
 
-[update-preview-diffs]
-type=user
-groups=script
-public.branch                           = SELECT
-public.branchmergeproposal              = SELECT, UPDATE
-public.branchmergeproposaljob           = SELECT
-public.diff                             = SELECT, INSERT
-public.job                              = SELECT, UPDATE
-public.libraryfilealias                 = SELECT, INSERT
-public.libraryfilecontent               = SELECT, INSERT
-public.previewdiff                      = SELECT, INSERT
-
 [upgrade-branches]
 type=user
 groups=script
 
=== modified file 'database/schema/security.py'
--- database/schema/security.py	2010-02-09 01:31:05 +0000
+++ database/schema/security.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/sort_sql.py'
--- database/schema/sort_sql.py	2009-10-17 14:06:03 +0000
+++ database/schema/sort_sql.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/trusted.sql'
--- database/schema/trusted.sql	2010-03-26 08:25:24 +0000
+++ database/schema/trusted.sql	2010-04-27 02:13:38 +0000
@@ -1470,36 +1470,36 @@
 
 -- Update the (redundant) column bug.latest_patch_uploaded when a
 -- a bug attachment is added or removed or if its type is changed.
-CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded(integer) RETURNS VOID
-    SECURITY DEFINER LANGUAGE plpgsql AS
-    $$
-    BEGIN
-        UPDATE bug SET latest_patch_uploaded =
-            (SELECT max(message.datecreated)
-                FROM message, bugattachment
-                WHERE bugattachment.message=message.id AND
-                    bugattachment.bug=$1 AND
-                    bugattachment.type=1)
-            WHERE bug.id=$1;
-    END;
-    $$;
-
-
-CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded_on_insert_update() RETURNS trigger
-    SECURITY DEFINER LANGUAGE plpgsql AS
-    $$
-    BEGIN
-        PERFORM bug_update_latest_patch_uploaded(NEW.bug);
-        RETURN NULL; -- Ignored - this is an AFTER trigger
-    END;
-    $$;
-
-
-CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded_on_delete() RETURNS trigger
-    SECURITY DEFINER LANGUAGE plpgsql AS
-    $$
-    BEGIN
-        PERFORM bug_update_latest_patch_uploaded(OLD.bug);
-        RETURN NULL; -- Ignored - this is an AFTER trigger
-    END;
-    $$;
+CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded(integer)
+RETURNS VOID SECURITY DEFINER LANGUAGE plpgsql AS
+$$
+BEGIN
+    UPDATE bug SET latest_patch_uploaded =
+        (SELECT max(message.datecreated)
+            FROM message, bugattachment
+            WHERE bugattachment.message=message.id AND
+                bugattachment.bug=$1 AND
+                bugattachment.type=1)
+        WHERE bug.id=$1;
+END;
+$$;
+
+
+CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded_on_insert_update()
+RETURNS trigger SECURITY DEFINER LANGUAGE plpgsql AS
+$$
+BEGIN
+    PERFORM bug_update_latest_patch_uploaded(NEW.bug);
+    RETURN NULL; -- Ignored - this is an AFTER trigger
+END;
+$$;
+
+
+CREATE OR REPLACE FUNCTION bug_update_latest_patch_uploaded_on_delete()
+RETURNS trigger SECURITY DEFINER LANGUAGE plpgsql AS
+$$
+BEGIN
+    PERFORM bug_update_latest_patch_uploaded(OLD.bug);
+    RETURN NULL; -- Ignored - this is an AFTER trigger
+END;
+$$;
 
=== modified file 'database/schema/unautovacuumable.py'
--- database/schema/unautovacuumable.py	2009-11-06 21:58:50 +0000
+++ database/schema/unautovacuumable.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
=== modified file 'database/schema/upgrade.py'
--- database/schema/upgrade.py	2009-10-17 14:06:03 +0000
+++ database/schema/upgrade.py	2010-04-27 02:13:38 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/python2.5
+#!/usr/bin/python2.5 -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
@@ -249,9 +249,10 @@
                     id=@holding_set,
                     provider=@master_node, receiver=@node%d_node, forward=yes);
                 echo 'Waiting for sync';
-                sync (id=1);
+                sync (id=@master_node);
                 wait for event (
-                    origin=ALL, confirmed=ALL, wait on=@master_node, timeout=0
+                    origin=@master_node, confirmed=ALL,
+                    wait on=@master_node, timeout=0
                     );
                 """ % (slave_node.node_id, slave_node.node_id))
 
@@ -281,7 +282,7 @@
         (fqn(nspname, relname), tab_id)
         for nspname, relname, tab_id in cur.fetchall())
 
-    # Generate a slonik script to remove tables from the replication set, 
+    # Generate a slonik script to remove tables from the replication set,
     # and a DROP TABLE/DROP SEQUENCE sql script to run after.
     if tabs_to_drop:
         log.info("Dropping tables: %s" % ', '.join(
 
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf	2010-04-19 03:44:27 +0000
+++ lib/canonical/config/schema-lazr.conf	2010-04-27 02:13:38 +0000
@@ -402,6 +402,11 @@
 # mapping done by branch-rewrite.py for.
 branch_rewrite_cache_lifetime: 10
 
+# Update Preview diff ready timeout
+#
+# How long, in minutes, we wait for a branch to be ready in order to
+# generate a diff for a merge proposal (in the UpdatePreviewDiffJob).
+update_preview_diff_ready_timeout: 15
 
 [codeimport]
 # Where the Bazaar imports are stored.
@@ -1419,6 +1424,25 @@
 port: 11217
 
 
+[merge_proposal_jobs]
+# The database user which will be used by this process.
+# datatype: string
+dbuser: merge-proposal-jobs
+storm_cache: generational
+storm_cache_size: 500
+
+# See [error_reports].
+error_dir: none
+
+# See [error_reports].
+oops_prefix: none
+
+# See [error_reports].
+copy_to_zlog: false
+
+##
+## TODO: delete mpcreationjobs section after 10.04 rollout.
+##
 [mpcreationjobs]
 # The database user which will be used by this process.
 # datatype: string
@@ -1435,6 +1459,9 @@
 # See [error_reports].
 copy_to_zlog: false
 
+##
+## TODO: delete update_preview_diffs section after 10.04 rollout.
+##
 [update_preview_diffs]
 dbuser: update-preview-diffs
 
@@ -1447,6 +1474,7 @@
 # See [error_reports].
 copy_to_zlog: false
 
+
 [upgrade_branches]
 dbuser: upgrade-branches
 
 
=== added directory 'lib/canonical/launchpad/apidoc'
=== removed directory 'lib/canonical/launchpad/apidoc'
=== modified file 'lib/canonical/launchpad/daemons/tachandler.py'
--- lib/canonical/launchpad/daemons/tachandler.py	2010-04-05 09:22:54 +0000
+++ lib/canonical/launchpad/daemons/tachandler.py	2010-04-27 02:13:38 +0000
@@ -164,8 +164,8 @@
     def _waitForDaemonStartup(self):
         """ Wait for the daemon to fully start.
 
-        Times out after 20 seconds.  If that happens, the log file will
-        not be cleaned up so the user can post-mortem it.
+        Times out after 20 seconds.  If that happens, the log file content
+        will be included in the exception message for debugging purpose.
 
         :raises TacException: Timeout.
         """
@@ -178,8 +178,8 @@
             now = time.time()
 
         if now >= deadline:
-            raise TacException('Unable to start %s. Check %s.' % (
-                self.tacfile, self.logfile))
+            raise TacException('Unable to start %s. Content of %s:\n%s' % (
+                self.tacfile, self.logfile, open(self.logfile).read()))
 
     def tearDown(self):
         self.killTac()
 
=== modified file 'lib/canonical/launchpad/doc/product-update-remote-product-script.txt'
--- lib/canonical/launchpad/doc/product-update-remote-product-script.txt	2009-03-27 03:29:31 +0000
+++ lib/canonical/launchpad/doc/product-update-remote-product-script.txt	2010-04-27 02:13:38 +0000
@@ -14,7 +14,7 @@
     0
 
     >>> print err
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-updateremoteproduct.lock
     INFO    0 projects using BUGZILLA needing updating.
     ...
     INFO    0 projects using RT needing updating.
 
=== modified file 'lib/canonical/launchpad/scripts/garbo.py'
--- lib/canonical/launchpad/scripts/garbo.py	2010-04-08 08:55:10 +0000
+++ lib/canonical/launchpad/scripts/garbo.py	2010-04-27 02:13:38 +0000
@@ -26,9 +26,7 @@
 from canonical.launchpad.database.openidconsumer import OpenIDConsumerNonce
 from canonical.launchpad.interfaces import IMasterStore
 from canonical.launchpad.interfaces.emailaddress import EmailAddressStatus
-from canonical.launchpad.interfaces.looptuner import ITunableLoop
-from canonical.launchpad.utilities.looptuner import (
-    DBLoopTuner, TunableLoop)
+from canonical.launchpad.utilities.looptuner import TunableLoop
 from canonical.launchpad.webapp.interfaces import (
     IStoreSelector, MAIN_STORE, MASTER_FLAVOR)
 from lp.bugs.interfaces.bug import IBugSet
@@ -42,7 +40,6 @@
 from lp.code.model.branchjob import BranchJob
 from lp.code.model.codeimportresult import CodeImportResult
 from lp.code.model.revision import RevisionAuthor, RevisionCache
-from lp.registry.model.mailinglist import MailingListSubscription
 from lp.registry.model.person import Person
 from lp.services.job.model.job import Job
 from lp.services.scripts.base import (
@@ -346,150 +343,6 @@
         transaction.commit()
 
 
-class MailingListSubscriptionPruner(TunableLoop):
-    """Prune `MailingListSubscription`s pointing at deleted email addresses.
-
-    Users subscribe to mailing lists with one of their verified email
-    addresses.  When they remove an address, the mailing list
-    subscription should go away too.
-    """
-
-    maximum_chunk_size = 1000
-
-    def __init__(self, log, abort_time=None):
-        super(MailingListSubscriptionPruner, self).__init__(log, abort_time)
-        self.subscription_store = IMasterStore(MailingListSubscription)
-        self.email_store = IMasterStore(EmailAddress)
-
-        (self.min_subscription_id,
-         self.max_subscription_id) = self.subscription_store.find(
-            (Min(MailingListSubscription.id),
-             Max(MailingListSubscription.id))).one()
-
-        self.next_subscription_id = self.min_subscription_id
-
-    def isDone(self):
-        return (self.min_subscription_id is None or
-                self.next_subscription_id > self.max_subscription_id)
-
-    def __call__(self, chunk_size):
-        result = self.subscription_store.find(
-            MailingListSubscription,
-            MailingListSubscription.id >= self.next_subscription_id,
-            MailingListSubscription.id < (self.next_subscription_id +
-                                          chunk_size))
-        used_ids = set(result.values(MailingListSubscription.email_addressID))
-        existing_ids = set(self.email_store.find(
-                EmailAddress.id, EmailAddress.id.is_in(used_ids)))
-        deleted_ids = used_ids - existing_ids
-
-        self.subscription_store.find(
-            MailingListSubscription,
-            MailingListSubscription.id >= self.next_subscription_id,
-            MailingListSubscription.id < (self.next_subscription_id +
-                                          chunk_size),
-            MailingListSubscription.email_addressID.is_in(deleted_ids)
-            ).remove()
-
-        self.next_subscription_id += chunk_size
-        transaction.commit()
-
-
-class PersonEmailAddressLinkChecker(TunableLoop):
-    """Report invalid references between the authdb and main replication sets.
-
-    We can't use referential integrity to ensure references remain valid,
-    so we have to check regularly for any bugs that creep into our code.
-
-    We don't repair links yet, but could add this feature. I'd
-    rather track down the source of problems and fix problems there
-    and avoid automatic repair, which might be dangerous. In particular,
-    replication lag introduces a number of race conditions that would
-    need to be addressed.
-    """
-    maximum_chunk_size = 1000
-
-    def __init__(self, log, abort_time=None):
-        super(PersonEmailAddressLinkChecker, self).__init__(log, abort_time)
-
-        self.person_store = IMasterStore(Person)
-        self.email_store = IMasterStore(EmailAddress)
-
-        # This query detects invalid links between Person and EmailAddress.
-        # The first part detects difference in opionion about what Account
-        # is linked to. The second part detects EmailAddresses linked to
-        # non existent Person records.
-        query = """
-            SELECT Person.id, EmailAddress.id
-            FROM EmailAddress, Person
-            WHERE EmailAddress.person = Person.id
-                AND Person.account IS DISTINCT FROM EmailAddress.account
-            UNION
-            SELECT NULL, EmailAddress.id
-            FROM EmailAddress LEFT OUTER JOIN Person
-                ON EmailAddress.person = Person.id
-            WHERE EmailAddress.person IS NOT NULL
-                AND Person.id IS NULL
-            """
-        # We need to issue this query twice, waiting between calls
-        # for all pending database changes to replicate. The known
-        # bad set are the entries common in both results.
-        bad_links_1 = set(self.person_store.execute(query))
-        transaction.abort()
-
-        self.blockForReplication()
-
-        bad_links_2 = set(self.person_store.execute(query))
-        transaction.abort()
-
-        self.bad_links = bad_links_1.intersection(bad_links_2)
-
-    def blockForReplication(self):
-        start = time.time()
-        while True:
-            lag = self.person_store.execute(
-                "SELECT COALESCE(EXTRACT(EPOCH FROM replication_lag()), 0);"
-                ).get_one()[0]
-            if lag < (time.time() - start):
-                return
-            # Guestimate on how long we should wait for. We cap
-            # it as several hours of lag can clear in an instant
-            # in some cases.
-            naptime = min(300, lag)
-            self.log.debug(
-                "Waiting for replication. Lagged %s secs. Napping %s secs."
-                % (lag, naptime))
-            time.sleep(naptime)
-
-    def isDone(self):
-        return not self.bad_links
-
-    def __call__(self, chunksize):
-        for counter in range(0, int(chunksize)):
-            if not self.bad_links:
-                return
-            person_id, emailaddress_id = self.bad_links.pop()
-            if person_id is None:
-                person = None
-            else:
-                person = self.person_store.get(Person, person_id)
-            emailaddress = self.email_store.get(EmailAddress, emailaddress_id)
-            self.report(person, emailaddress)
-            # We don't repair... yet.
-            # self.repair(person, emailaddress)
-        transaction.abort()
-
-    def report(self, person, emailaddress):
-        if person is None:
-            self.log.error(
-                "Corruption - '%s' is linked to a non-existant Person."
-                % emailaddress.email)
-        else:
-            self.log.error(
-                "Corruption - '%s' and '%s' reference different Accounts."
-                % (emailaddress.email, person.name))
-
-
 class PersonPruner(TunableLoop):
 
     maximum_chunk_size = 1000
@@ -662,7 +515,7 @@
     def __call__(self, chunk_size):
         chunk_size = int(chunk_size)
         ids_to_remove = list(self._ids_to_remove()[:chunk_size])
-        num_removed = self.job_store.find(
+        self.job_store.find(
             BranchJob,
             In(BranchJob.id, ids_to_remove)).remove()
         transaction.commit()
@@ -918,8 +771,6 @@
         CodeImportResultPruner,
         RevisionAuthorEmailLinker,
         HWSubmissionEmailLinker,
-        MailingListSubscriptionPruner,
-        PersonEmailAddressLinkChecker,
         BugNotificationPruner,
         BranchJobPruner,
         BugWatchActivityPruner,
 
=== modified file 'lib/canonical/launchpad/scripts/tests/test_garbo.py'
--- lib/canonical/launchpad/scripts/tests/test_garbo.py	2010-04-13 01:49:42 +0000
+++ lib/canonical/launchpad/scripts/tests/test_garbo.py	2010-04-27 02:13:38 +0000
@@ -351,26 +351,6 @@
         LaunchpadZopelessLayer.switchDbUser('testadmin')
         self.assertEqual(sub3.owner, person3)
 
-    def test_MailingListSubscriptionPruner(self):
-        LaunchpadZopelessLayer.switchDbUser('testadmin')
-        team, mailing_list = self.factory.makeTeamAndMailingList(
-            'mlist-team', 'mlist-owner')
-        person = self.factory.makePerson(email='[e-mail]')
-        email = self.factory.makeEmail('[e-mail]', person)
-        transaction.commit()
-        mailing_list.subscribe(person, email)
-
-        # User remains subscribed if we run the garbage collector.
-        self.runDaily()
-        self.assertNotEqual(mailing_list.getSubscription(person), None)
-
-        # If we remove the email address that was subscribed, the
-        # garbage collector removes the subscription.
-        LaunchpadZopelessLayer.switchDbUser('testadmin')
-        Store.of(email).remove(email)
-        self.runDaily()
-        self.assertEqual(mailing_list.getSubscription(person), None)
-
     def test_PersonPruner(self):
         personset = getUtility(IPersonSet)
         # Switch the DB user because the garbo_daily user isn't allowed to
@@ -466,42 +446,6 @@
                 BugNotification.date_emailed < THIRTY_DAYS_AGO).count(),
             0)
 
-    def test_PersonEmailAddressLinkChecker(self):
-        LaunchpadZopelessLayer.switchDbUser('testadmin')
-
-        # Make an EmailAddress record reference a non-existant Person.
-        emailaddress = IMasterStore(EmailAddress).get(EmailAddress, 16)
-        emailaddress.personID = -1
-
-        # Make a Person record reference a different Account to its
-        # EmailAddress records.
-        person = IMasterStore(Person).get(Person, 1)
-        person_email = Store.of(person).find(
-            EmailAddress, person=person).any()
-        person.accountID = -1
-
-        # Run the garbage collector. We should get two ERROR reports
-        # about the corrupt data.
-        collector = self.runDaily()
-
-        # The PersonEmailAddressLinkChecker is not intelligent enough
-        # to repair corruption. It is only there to alert us to the
-        # issue so data can be manually repaired and the cause
-        # tracked down and fixed.
-        self.assertEqual(emailaddress.personID, -1)
-        self.assertNotEqual(person.accountID, person_email.accountID)
-
-        # The corruption has been reported though as a ERROR messages.
-        log_output = collector.logger.output_file.getvalue()
-        error_message_1 = (
-            "ERROR Corruption - "
-            "'[e-mail]' is linked to a non-existant Person.")
-        self.assertNotEqual(log_output.find(error_message_1), -1)
-        error_message_2 = (
-            "ERROR Corruption - "
-            "'[e-mail]' and 'mark' reference different Accounts")
-        self.assertNotEqual(log_output.find(error_message_2), -1)
-
     def test_BranchJobPruner(self):
         # Garbo should remove jobs completed over 30 days ago.
         self.useBzrBranches()
 
=== modified file 'lib/contrib/glock.py'
--- lib/contrib/glock.py	2007-01-29 18:48:21 +0000
+++ lib/contrib/glock.py	2010-04-27 02:13:38 +0000
@@ -157,7 +157,7 @@
                        the caller decided not to block.
         """
         if self.logger:
-            self.logger.info('creating lockfile')
+            self.logger.info('Creating lockfile: %s', self.fpath)
         if _windows:
             if blocking:
                 timeout = win32event.INFINITE
 
=== modified file 'lib/lp/answers/doc/expiration.txt'
--- lib/lp/answers/doc/expiration.txt	2009-07-23 17:49:31 +0000
+++ lib/lp/answers/doc/expiration.txt	2010-04-27 02:13:38 +0000
@@ -132,7 +132,7 @@
     ...     stderr=subprocess.PIPE)
     >>> (out, err) = process.communicate()
     >>> print err
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-expire-questions.lock
     INFO    Expiring OPEN and NEEDSINFO questions without activity for the
             last 15 days.
     INFO    Found 5 questions to expire.
 
=== modified file 'lib/lp/archivepublisher/publishing.py'
--- lib/lp/archivepublisher/publishing.py	2010-02-09 00:17:40 +0000
+++ lib/lp/archivepublisher/publishing.py	2010-04-27 02:13:38 +0000
@@ -12,6 +12,7 @@
 import hashlib
 import logging
 import os
+import shutil
 
 from datetime import datetime
 
@@ -29,7 +30,7 @@
 from canonical.database.sqlbase import sqlvalues
 from lp.registry.interfaces.pocket import (
     PackagePublishingPocket, pocketsuffix)
-from lp.soyuz.interfaces.archive import ArchivePurpose
+from lp.soyuz.interfaces.archive import ArchivePurpose, ArchiveStatus
 from lp.soyuz.interfaces.binarypackagerelease import (
     BinaryPackageFormat)
 from lp.soyuz.interfaces.component import IComponentSet
@@ -596,3 +597,32 @@
             in_file.close()
 
         out_file.write(" %s % 16d %s\n" % (checksum, length, file_name))
+
+    def deleteArchive(self):
+        """Delete the archive.
+        
+        Physically remove the entire archive from disk and set the archive's 
+        status to DELETED.
+
+        Any errors encountered while removing the archive from disk will 
+        be caught and an OOPS report generated.
+        """
+
+        root_dir = os.path.join(
+            self._config.distroroot, self.archive.owner.name,
+            self.archive.name)
+
+        self.log.info(
+            "Attempting to delete archive '%s/%s' at '%s'." % (
+                self.archive.owner.name, self.archive.name, root_dir))
+
+        try:
+            shutil.rmtree(root_dir)
+        except (shutil.Error, OSError), e:
+            self.log.warning(
+                "Failed to delete directory '%s' for archive '%s/%s'\n%s" % (
+                    root_dir, self.archive.owner.name, 
+                    self.archive.name, e))
+
+        self.archive.status = ArchiveStatus.DELETED
+        self.archive.publish = False
 
=== modified file 'lib/lp/archivepublisher/scripts/generate_ppa_htaccess.py'
--- lib/lp/archivepublisher/scripts/generate_ppa_htaccess.py	2009-10-17 14:06:03 +0000
+++ lib/lp/archivepublisher/scripts/generate_ppa_htaccess.py	2010-04-27 02:13:38 +0000
@@ -17,7 +17,7 @@
 from canonical.launchpad.webapp import canonical_url
 
 from lp.archivepublisher.config import getPubConfig
-from lp.soyuz.interfaces.archive import IArchiveSet
+from lp.soyuz.interfaces.archive import IArchiveSet, ArchiveStatus
 from lp.soyuz.interfaces.archiveauthtoken import (
     IArchiveAuthTokenSet)
 from lp.soyuz.interfaces.archivesubscriber import (
@@ -241,6 +241,13 @@
                         ppa.name,
                         ppa.owner.displayname)
                 continue
+            elif ppa.status == ArchiveStatus.DELETED or ppa.enabled is False:
+                self.logger.info(
+                    "Skipping htacess updates for deleted or disabled PPA "
+                    " '%s' owned by %s.",
+                        ppa.name,
+                        ppa.owner.displayname)
+                continue
 
             self.ensureHtaccess(ppa)
             temp_htpasswd = self.generateHtpasswd(ppa, valid_tokens)
 
=== modified file 'lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py'
--- lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py	2010-03-09 07:29:18 +0000
+++ lib/lp/archivepublisher/tests/test_generate_ppa_htaccess.py	2010-04-27 02:13:38 +0000
@@ -24,6 +24,7 @@
 from lp.archivepublisher.config import getPubConfig
 from lp.archivepublisher.scripts.generate_ppa_htaccess import (
     HtaccessTokenGenerator)
+from lp.soyuz.interfaces.archive import ArchiveStatus
 from lp.soyuz.interfaces.archivesubscriber import (
     ArchiveSubscriberStatus)
 from lp.testing import TestCaseWithFactory
@@ -476,6 +477,47 @@
         self.assertFalse(os.path.isfile(htaccess))
         self.assertFalse(os.path.isfile(htpasswd))
 
+    def testSkippingOfDisabledPPAs(self):
+        """Test that the htaccess for disabled PPAs are not touched."""
+        subs, tokens = self.setupDummyTokens()
+        htaccess, htpasswd = self.ensureNoFiles()
+
+        # Setup subscription so that htaccess/htpasswd is pending generation.
+        now = datetime.now(pytz.UTC)
+        subs[0].date_expires = now + timedelta(minutes=3)
+        self.assertEqual(subs[0].status, ArchiveSubscriberStatus.CURRENT)
+
+        # Set the PPA as disabled.
+        self.ppa.disable()
+        self.assertFalse(self.ppa.enabled)
+
+        script = self.getScript()
+        script.main()
+
+        # The htaccess and htpasswd files should not be generated.
+        self.assertFalse(os.path.isfile(htaccess))
+        self.assertFalse(os.path.isfile(htpasswd))
+
+    def testSkippingOfDeletedPPAs(self):
+        """Test that the htaccess for deleted PPAs are not touched."""
+        subs, tokens = self.setupDummyTokens()
+        htaccess, htpasswd = self.ensureNoFiles()
+
+        # Setup subscription so that htaccess/htpasswd is pending generation.
+        now = datetime.now(pytz.UTC)
+        subs[0].date_expires = now + timedelta(minutes=3)
+        self.assertEqual(subs[0].status, ArchiveSubscriberStatus.CURRENT)
+
+        # Set the PPA as deleted.
+        self.ppa.status = ArchiveStatus.DELETED
+
+        script = self.getScript()
+        script.main()
+
+        # The htaccess and htpasswd files should not be generated.
+        self.assertFalse(os.path.isfile(htaccess))
+        self.assertFalse(os.path.isfile(htpasswd))
+
     def testSendingCancellationEmail(self):
         """Test that when a token is deactivated, its user gets an email.
 
 
=== modified file 'lib/lp/archivepublisher/tests/test_publisher.py'
--- lib/lp/archivepublisher/tests/test_publisher.py	2010-02-10 00:25:55 +0000
+++ lib/lp/archivepublisher/tests/test_publisher.py	2010-04-27 02:13:38 +0000
@@ -8,7 +8,6 @@
 
 import bz2
 import gzip
-import hashlib
 import os
 import shutil
 import stat
@@ -26,7 +25,7 @@
 from canonical.database.constants import UTC_NOW
 from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir
 from lp.soyuz.interfaces.archive import (
-    ArchivePurpose, IArchiveSet)
+    ArchivePurpose, ArchiveStatus, IArchiveSet)
 from lp.soyuz.interfaces.binarypackagerelease import (
     BinaryPackageFormat)
 from lp.registry.interfaces.distribution import IDistributionSet
@@ -37,7 +36,6 @@
 from lp.soyuz.interfaces.publishing import PackagePublishingStatus
 from lp.archivepublisher.interfaces.archivesigningkey import (
     IArchiveSigningKey)
-from lp.testing import get_lsb_information
 from lp.soyuz.tests.test_publishing import TestNativePublishingBase
 from canonical.zeca.ftests.harness import ZecaTestSetup
 
@@ -95,6 +93,32 @@
         foo_path = "%s/main/f/foo/foo_666.dsc" % self.pool_dir
         self.assertEqual(open(foo_path).read().strip(), 'Hello world')
 
+    def testDeletingPPA(self):
+        """Test deleting a PPA"""
+        ubuntu_team = getUtility(IPersonSet).getByName('ubuntu-team')
+        test_archive = getUtility(IArchiveSet).new(
+            distribution=self.ubuntutest, owner=ubuntu_team,
+            purpose=ArchivePurpose.PPA)
+        publisher = getPublisher(test_archive, None, self.logger)
+
+        self.assertTrue(os.path.exists(publisher._config.archiveroot))
+
+        # Create a file inside archiveroot to ensure we're recursive.
+        open(os.path.join(
+            publisher._config.archiveroot, 'test_file'), 'w').close()
+
+        publisher.deleteArchive()
+        root_dir = os.path.join(
+            publisher._config.distroroot, test_archive.owner.name,
+            test_archive.name)
+        self.assertFalse(os.path.exists(root_dir))
+        self.assertEqual(test_archive.status, ArchiveStatus.DELETED)
+        self.assertEqual(test_archive.publish, False)
+
+        # Trying to delete it again won't fail, in the corner case where
+        # some admin manually deleted the repo.
+        publisher.deleteArchive()
+
     def testPublishPartner(self):
         """Test that a partner package is published to the right place."""
         archive = self.ubuntutest.getArchiveByComponent('partner')
@@ -104,8 +128,7 @@
             pub_config.poolroot, pub_config.temproot, self.logger)
         publisher = Publisher(
             self.logger, pub_config, disk_pool, archive)
-        pub_source = self.getPubSource(archive=archive,
-            filecontent="I am partner")
+        self.getPubSource(archive=archive, filecontent="I am partner")
 
         publisher.A_publish(False)
 
@@ -143,7 +166,7 @@
         disk_pool = DiskPool(
             pub_config.poolroot, pub_config.temproot, self.logger)
         publisher = Publisher(self.logger, pub_config, disk_pool, archive)
-        pub_source = self.getPubSource(
+        self.getPubSource(
             archive=archive, filecontent="I am partner",
             status=PackagePublishingStatus.PENDING)
 
@@ -230,8 +253,7 @@
             self.logger, self.config, self.disk_pool,
             self.ubuntutest.main_archive)
 
-        pub_source = self.getPubSource(
-            status=PackagePublishingStatus.PUBLISHED)
+        self.getPubSource(status=PackagePublishingStatus.PUBLISHED)
 
         # a new non-careful publisher won't find anything to publish, thus
         # no pockets will be *dirtied*.
@@ -251,7 +273,7 @@
             self.logger, self.config, self.disk_pool,
             self.ubuntutest.main_archive)
 
-        pub_source = self.getPubSource(
+        self.getPubSource(
             filecontent='Hello world',
             status=PackagePublishingStatus.PUBLISHED)
 
@@ -394,17 +416,17 @@
         ubuntu = getUtility(IDistributionSet)['ubuntu']
 
         spiv = person_set.getByName('spiv')
-        spiv_archive = archive_set.new(
+        archive_set.new(
             owner=spiv, distribution=ubuntu, purpose=ArchivePurpose.PPA)
         name16 = person_set.getByName('name16')
-        name16_archive = archive_set.new(
+        archive_set.new(
             owner=name16, distribution=ubuntu, purpose=ArchivePurpose.PPA)
 
-        pub_source = self.getPubSource(
+        self.getPubSource(
             sourcename="foo", filename="foo_1.dsc", filecontent='Hello world',
             status=PackagePublishingStatus.PENDING, archive=spiv.archive)
 
-        pub_source = self.getPubSource(
+        self.getPubSource(
             sourcename="foo", filename="foo_1.dsc", filecontent='Hello world',
             status=PackagePublishingStatus.PUBLISHED, archive=name16.archive)
 
@@ -467,7 +489,7 @@
         pub_source = self.getPubSource(
             sourcename="foo", filename="foo_1.dsc", filecontent='Hello world',
             status=PackagePublishingStatus.PENDING, archive=cprov.archive)
-        pub_bin = self.getPubBinaries(
+        self.getPubBinaries(
             pub_source=pub_source,
             description="   My leading spaces are normalised to a single "
                         "space but not trailing.  \n    It does nothing, "
@@ -478,7 +500,7 @@
         ignored_source = self.getPubSource(
             status=PackagePublishingStatus.DELETED,
             archive=cprov.archive)
-        pub_udeb = self.getPubBinaries(
+        self.getPubBinaries(
             pub_source=ignored_source, binaryname='bingo',
             description='nice udeb', format=BinaryPackageFormat.UDEB)[0]
 
@@ -609,27 +631,27 @@
         # waiting to be deleted, each in different pockets.  The deleted
         # source in the release pocket should not be processed.  We'll
         # also have a binary waiting to be deleted.
-        published_source = self.getPubSource(
+        self.getPubSource(
             pocket=PackagePublishingPocket.RELEASE,
             status=PackagePublishingStatus.PUBLISHED)
 
-        deleted_source_in_release_pocket = self.getPubSource(
+        self.getPubSource(
             pocket=PackagePublishingPocket.RELEASE,
             status=PackagePublishingStatus.DELETED)
 
-        removed_source = self.getPubSource(
+        self.getPubSource(
             scheduleddeletiondate=UTC_NOW,
             dateremoved=UTC_NOW,
             pocket=PackagePublishingPocket.UPDATES,
             status=PackagePublishingStatus.DELETED)
 
-        deleted_source = self.getPubSource(
+        self.getPubSource(
             pocket=PackagePublishingPocket.SECURITY,
             status=PackagePublishingStatus.DELETED)
 
-        deleted_binary = self.getPubBinaries(
+        self.getPubBinaries(
             pocket=PackagePublishingPocket.BACKPORTS,
-            status=PackagePublishingStatus.DELETED)[0]
+            status=PackagePublishingStatus.DELETED)
 
         # Run the deletion detection.
         publisher.A2_markPocketsWithDeletionsDirty()
@@ -681,19 +703,19 @@
 
         # Create pending deletions in RELEASE, BACKPORTS, SECURITY and
         # UPDATES pockets.
-        deleted_source = self.getPubSource(
+        self.getPubSource(
             pocket=PackagePublishingPocket.RELEASE,
             status=PackagePublishingStatus.DELETED)
 
-        deleted_binary = self.getPubBinaries(
+        self.getPubBinaries(
             pocket=PackagePublishingPocket.BACKPORTS,
             status=PackagePublishingStatus.DELETED)[0]
 
-        allowed_source_deletion = self.getPubSource(
+        self.getPubSource(
             pocket=PackagePublishingPocket.SECURITY,
             status=PackagePublishingStatus.DELETED)
 
-        allowed_binary_deletion = self.getPubBinaries(
+        self.getPubBinaries(
             pocket=PackagePublishingPocket.UPDATES,
             status=PackagePublishingStatus.DELETED)[0]
 
@@ -763,7 +785,7 @@
             self.logger, self.config, self.disk_pool,
             self.ubuntutest.main_archive)
 
-        pub_source = self.getPubSource(filecontent='Hello world')
+        self.getPubSource(filecontent='Hello world')
 
         publisher.A_publish(False)
         publisher.C_doFTPArchive(False)
@@ -848,8 +870,7 @@
         archive_publisher = getPublisher(
             cprov.archive, allowed_suites, self.logger)
 
-        pub_source = self.getPubSource(
-            filecontent='Hello world', archive=cprov.archive)
+        self.getPubSource(filecontent='Hello world', archive=cprov.archive)
 
         archive_publisher.A_publish(False)
         self.layer.txn.commit()
@@ -952,8 +973,7 @@
         allowed_suites = []
         archive_publisher = getPublisher(
             named_ppa, allowed_suites, self.logger)
-        pub_source = self.getPubSource(
-            filecontent='Hello world', archive=named_ppa)
+        self.getPubSource(filecontent='Hello world', archive=named_ppa)
 
         archive_publisher.A_publish(False)
         self.layer.txn.commit()
@@ -1062,7 +1082,7 @@
         Publish files in pool, generate archive indexes and release files.
         """
         self.setupPublisher(archive)
-        pub_source = self.getPubSource(archive=archive)
+        self.getPubSource(archive=archive)
 
         self.archive_publisher.A_publish(False)
         transaction.commit()
 
=== modified file 'lib/lp/bugs/browser/bugwatch.py'
--- lib/lp/bugs/browser/bugwatch.py	2010-01-15 03:32:46 +0000
+++ lib/lp/bugs/browser/bugwatch.py	2010-04-27 02:13:38 +0000
@@ -6,12 +6,15 @@
 __metaclass__ = type
 __all__ = [
     'BugWatchSetNavigation',
+    'BugWatchActivityPortletView',
     'BugWatchEditView',
     'BugWatchView']
 
+
 from zope.component import getUtility
 from zope.interface import Interface
 
+from canonical.database.constants import UTC_NOW
 from canonical.widgets.textwidgets import URIWidget
 
 from canonical.launchpad import _
@@ -21,7 +24,8 @@
 from canonical.launchpad.fields import URIField
 from canonical.launchpad.webapp.interfaces import ILaunchBag
 from lp.bugs.interfaces.bugwatch import (
-    IBugWatch, IBugWatchSet, NoBugTrackerFound, UnrecognizedBugTrackerURL)
+    BUG_WATCH_ACTIVITY_SUCCESS_STATUSES, IBugWatch, IBugWatchSet,
+    NoBugTrackerFound, UnrecognizedBugTrackerURL)
 from canonical.launchpad.webapp import (
     action, canonical_url, custom_widget, GetitemNavigation,
     LaunchpadFormView, LaunchpadView)
@@ -99,6 +103,11 @@
         """See `LaunchpadFormView.`"""
         return {'url' : self.context.url}
 
+    @property
+    def watch_has_activity(self):
+        """Return True if there has been activity on the bug watch."""
+        return not self.context.activity.is_empty()
+
     def validate(self, data):
         """See `LaunchpadFormView.`"""
         if 'url' not in data:
@@ -136,3 +145,53 @@
         return canonical_url(getUtility(ILaunchBag).bug)
 
     cancel_url = next_url
+
+
+class BugWatchActivityPortletView(LaunchpadFormView):
+    """A portlet for displaying the activity of a bug watch."""
+
+    schema = BugWatchEditForm
+
+    def userCanReschedule(self, action=None):
+        """Return True if the current user can reschedule the bug watch."""
+        return self.context.can_be_rescheduled
+
+    @action('Update Now', name='reschedule', condition=userCanReschedule)
+    def reschedule_action(self, action, data):
+        """Schedule the current bug watch for immediate checking."""
+        bugwatch = self.context
+        bugwatch.setNextCheck(UTC_NOW)
+        self.request.response.addInfoNotification(
+            structured(
+                'The <a href="%(url)s">%(bugtracker)s #%(remote_bug)s</a> '
+                'bug watch has been scheduled for immediate checking.',
+                url=bugwatch.url, bugtracker=bugwatch.bugtracker.name,
+                remote_bug=bugwatch.remotebug))
+
+    @property
+    def next_url(self):
+        return canonical_url(getUtility(ILaunchBag).bug)
+
+    cancel_url = next_url
+
+    @property
+    def recent_watch_activity(self):
+        """Return a list of dicts representing recent watch activity."""
+        activity_items = []
+        for activity in self.context.activity:
+            if activity.result in BUG_WATCH_ACTIVITY_SUCCESS_STATUSES:
+                icon = "/@@/yes"
+                completion_message = "completed successfully"
+            else:
+                icon = "/@@/no"
+                completion_message = (
+                    "failed with error '%s'" % activity.result.title)
+
+            activity_items.append({
+                'icon': icon,
+                'date': activity.activity_date,
+                'completion_message': completion_message,
+                'result_text': activity.result.title,
+                })
+
+        return activity_items
 
=== modified file 'lib/lp/bugs/browser/configure.zcml'
--- lib/lp/bugs/browser/configure.zcml	2010-03-11 01:39:25 +0000
+++ lib/lp/bugs/browser/configure.zcml	2010-04-27 02:13:38 +0000
@@ -1103,6 +1103,12 @@
             class="lp.bugs.browser.bugwatch.BugWatchEditView"
             permission="launchpad.AnyPerson"
             template="../templates/bugwatch-editform.pt"/>
+        <browser:page
+            for="lp.bugs.interfaces.bugwatch.IBugWatch"
+            name="+portlet-activity"
+            class="lp.bugs.browser.bugwatch.BugWatchActivityPortletView"
+            permission="launchpad.AnyPerson"
+            template="../templates/bugwatch-portlet-activity.pt"/>
         <browser:pages
             for="lp.bugs.interfaces.bugwatch.IBugWatch"
             permission="launchpad.AnyPerson">
 
=== modified file 'lib/lp/bugs/browser/tests/bugwatch-views.txt'
--- lib/lp/bugs/browser/tests/bugwatch-views.txt	2009-10-22 11:55:51 +0000
+++ lib/lp/bugs/browser/tests/bugwatch-views.txt	2010-04-27 02:13:38 +0000
@@ -1,4 +1,5 @@
-= Bug Watch Edit Page =
+Bug Watch Edit Page
+===================
 
 It's possible to edit a bug watch on +edit, as well as deleting it.
 Deleting a bug watch is only possible when the bug watch isn't linked to
@@ -28,3 +29,84 @@
     >>> [action.label for action in unlinked_bugwatch_view.actions
     ...  if action.available()]
     ['Change', 'Delete Bug Watch']
+
+
+Recent activity
+---------------
+
+The Bug Watch +edit page displays a list of the recent activity for the
+watch. This is provided by the BugWatch activity portlet view and can be
+accessed via the recent_watch_activity property of BugWatchView.
+
+We'll create a new watch in order to demonstrate this.
+
+    >>> from canonical.launchpad.ftests import login
+    >>> login('[e-mail]')
+    >>> new_watch = factory.makeBugWatch()
+
+The view for the new watch will have an empty recent_watch_activity list
+since it hasn't been updated yet.
+
+    >>> new_watch_view = create_initialized_view(
+    ...     new_watch, '+portlet-activity')
+    >>> len(new_watch_view.recent_watch_activity)
+    0
+
+The BugWatch +edit view has a watch_has_activity property, which is used
+to determine whether the recent activity portlet should be displayed.
+
+    >>> new_watch_edit_view = create_initialized_view(
+    ...     new_watch, '+edit')
+    >>> print new_watch_edit_view.watch_has_activity
+    False
+
+Adding a successful activity entry for the watch will cause it to show
+up on the BugWatchView's recent_watch_activity property.
+
+    >>> new_watch.addActivity()
+    >>> len(new_watch_view.recent_watch_activity)
+    1
+
+The BugWatch +edit view's watch_has_activity property will also have
+changed.
+
+    >>> new_watch_edit_view = create_initialized_view(
+    ...     new_watch, '+edit')
+    >>> print new_watch_edit_view.watch_has_activity
+    True
+
+Each entry in the recent_watch_activity list is a dict containing data
+about the activity.
+
+    >>> from pprint import pprint
+    >>> for activity_dict in new_watch_view.recent_watch_activity:
+    ...     pprint(activity_dict)
+    {'completion_message': 'completed successfully',
+     'date': datetime.datetime(...tzinfo=<UTC>),
+     'icon': '/@@/yes',
+     'result_text': 'Synchronisation succeeded'}
+
+If an activity entry records a failure, the 'icon' entry in the dict
+will point to the 'no' icon and the completion_message will explain the
+failure.
+
+We'll commit the transaction to make sure that the two activities have
+different dates.
+
+    >>> import transaction
+    >>> transaction.commit()
+
+    >>> from lp.bugs.interfaces.bugwatch import BugWatchActivityStatus
+    >>> new_watch.addActivity(result=BugWatchActivityStatus.BUG_NOT_FOUND)
+    >>> for activity_dict in new_watch_view.recent_watch_activity:
+    ...     pprint(activity_dict)
+    {'completion_message': "failed with error 'Bug Not Found'",
+     'date': datetime.datetime(...tzinfo=<UTC>),
+     'icon': '/@@/no',
+     'result_text': 'Bug Not Found'}
+    {'completion_message': 'completed successfully',
+     'date': datetime.datetime(...tzinfo=<UTC>),
+     'icon': '/@@/yes',
+     'result_text': 'Synchronisation succeeded'}
+
+
 
=== modified file 'lib/lp/bugs/configure.zcml'
--- lib/lp/bugs/configure.zcml	2010-04-14 12:55:44 +0000
+++ lib/lp/bugs/configure.zcml	2010-04-27 02:13:38 +0000
@@ -849,7 +849,9 @@
                     bug
                     bugtasks
                     bugtracker
+                    can_be_rescheduled
                     datecreated
+                    failed_activity
                     getLastErrorMessage
                     hasComment
                     unpushed_comments
@@ -870,7 +872,8 @@
                 permission="launchpad.AnyPerson"
                 attributes="
                     destroySelf
-                    addActivity"
+                    addActivity
+                    setNextCheck"
                 set_attributes="bugtracker remotebug"/>
             <require
                 permission="launchpad.Admin"
 
=== modified file 'lib/lp/bugs/doc/bug-watch-activity.txt'
--- lib/lp/bugs/doc/bug-watch-activity.txt	2010-04-21 10:30:24 +0000
+++ lib/lp/bugs/doc/bug-watch-activity.txt	2010-04-27 02:13:38 +0000
@@ -50,11 +50,14 @@
     >>> activity.activity_date
     datetime.datetime...
 
+The BugWatchActivity's result will be BugWatchActivityStatus.SYNC_SUCCEEDED.
+
+    >>> print activity.result.title
+    Synchronisation succeeded
+
 The other fields on the BugWatchActivity record, which aren't required,
 will all be None.
 
-    >>> print activity.result
-    None
     >>> print activity.message
     None
     >>> print activity.oops_id
@@ -83,12 +86,13 @@
     >>> print bug_watch.activity.count()
     2
 
-The most recent activity entry will have a result of None since it was
+The most recent activity entry will have a result of
+BugWatchActivityStatus.SYNC_SUCCEEDED since it was
 successful.
 
     >>> most_recent_activity = bug_watch.activity.first()
-    >>> print most_recent_activity.result
-    None
+    >>> print most_recent_activity.result.title
+    Synchronisation succeeded
 
 Its message will also be empty
 
 
=== modified file 'lib/lp/bugs/doc/bugnotification-sending.txt'
--- lib/lp/bugs/doc/bugnotification-sending.txt	2010-04-15 10:58:02 +0000
+++ lib/lp/bugs/doc/bugnotification-sending.txt	2010-04-27 02:13:38 +0000
@@ -984,7 +984,7 @@
     >>> process.returncode
     0
     >>> print err
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-send-bug-notifications.lock
     INFO    Notifying [e-mail] about bug 2.
     ...
     INFO    Notifying [e-mail] about bug 2.
 
=== modified file 'lib/lp/bugs/doc/bugtask-expiration.txt'
--- lib/lp/bugs/doc/bugtask-expiration.txt	2010-04-14 13:23:02 +0000
+++ lib/lp/bugs/doc/bugtask-expiration.txt	2010-04-27 02:13:38 +0000
@@ -445,7 +445,7 @@
     ...     stderr=subprocess.PIPE)
     >>> (out, err) = process.communicate()
     >>> print err
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-expire-bugtasks.lock
     INFO    Expiring unattended, INCOMPLETE bugtasks older than
             60 days for projects that use Launchpad Bugs.
     INFO    Found 3 bugtasks to expire.
 
=== modified file 'lib/lp/bugs/doc/bugtask.txt'
--- lib/lp/bugs/doc/bugtask.txt	2010-01-21 17:40:23 +0000
+++ lib/lp/bugs/doc/bugtask.txt	2010-04-27 02:13:38 +0000
@@ -1104,7 +1104,7 @@
     >>> (out, err) = process.communicate()
 
     >>> print err
-    INFO creating lockfile
+    INFO Creating lockfile: /var/lock/launchpad-launchpad-targetnamecacheupdater.lock
     INFO Updating targetname cache of bugtasks.
     INFO Updating 1 BugTasks (starting id: 2).
     INFO Updating ...BugTasks...
 
=== modified file 'lib/lp/bugs/doc/bugwatch.txt'
--- lib/lp/bugs/doc/bugwatch.txt	2010-04-21 10:30:24 +0000
+++ lib/lp/bugs/doc/bugwatch.txt	2010-04-27 02:13:38 +0000
@@ -513,3 +513,87 @@
     >>> bug.removeWatch(bug_watch, factory.makePerson())
     >>> [bug_watch.remotebug for bug_watch in bug.watches]
     []
+
+
+Checking if a watch can be rescheduled
+--------------------------------------
+
+IBugWatch provides an attribute, can_be_rescheduled, which indicates
+whether or not the watch can be rescheduled. For a new bug watch this
+will be False.
+
+    >>> schedulable_watch = factory.makeBugWatch()
+    >>> schedulable_watch.next_check = None
+    >>> schedulable_watch.can_be_rescheduled
+    False
+
+If there's been activity on the watch but it's always been successful,
+can_be_rescheduled will be False.
+
+    >>> schedulable_watch.addActivity()
+    >>> schedulable_watch.can_be_rescheduled
+    False
+
+If the watch's updates have failed less than 60% of the time,
+can_be_rescheduled will be True
+
+    >>> from lp.bugs.interfaces.bugwatch import BugWatchActivityStatus
+    >>> schedulable_watch.addActivity(
+    ...     result=BugWatchActivityStatus.BUG_NOT_FOUND)
+    >>> schedulable_watch.can_be_rescheduled
+    True
+
+If the watch is rescheduled, can_be_rescheduled will be False, since the
+next_check time for the watch will be in the past (or in this case is
+now) and therefore it will be checked with the next checkwatches run.
+
+    >>> from pytz import utc
+    >>> from datetime import datetime
+    >>> schedulable_watch.next_check = datetime.now(utc)
+    >>> schedulable_watch.can_be_rescheduled
+    False
+
+However, if the watch has failed more than 60% of the time
+can_be_rescheduled will be False, since it's assumed that the watch
+needs attention in order for it to be able to work again.
+
+    >>> schedulable_watch.next_check = None
+    >>> schedulable_watch.addActivity(
+    ...     result=BugWatchActivityStatus.BUG_NOT_FOUND)
+    >>> schedulable_watch.can_be_rescheduled
+    False
+
+
+Rescheduling a watch
+--------------------
+
+The rescheduling of a watch is done via IBugWatch.setNextCheck(). This
+is to ensure that watches are only rescheduled when can_be_rescheduled
+is True (note that the BugWatch Scheduler bypasses setNextCheck() and
+sets next_check directly because it has admin privileges).
+
+The schedulable_watch that we used in the previous test cannot currently
+be rescheduled.
+
+    >>> schedulable_watch.can_be_rescheduled
+    False
+
+Calling setNextCheck() on this watch will cause an Exception,
+BugWatchCannotBeRescheduled, to be raised.
+
+    >>> schedulable_watch.setNextCheck(datetime.now(utc))
+    Traceback (most recent call last):
+      ...
+    BugWatchCannotBeRescheduled...
+
+If we add some activity to the watch, to make its can_be_rescheduled
+property become True, setNextCheck() will succeed.
+
+    >>> schedulable_watch.addActivity()
+    >>> schedulable_watch.can_be_rescheduled
+    True
+
+    >>> next_check = datetime.now(utc)
+    >>> schedulable_watch.setNextCheck(next_check)
+    >>> schedulable_watch.next_check == next_check
+    True
 
=== modified file 'lib/lp/bugs/doc/checkwatches.txt'
--- lib/lp/bugs/doc/checkwatches.txt	2010-04-21 10:30:24 +0000
+++ lib/lp/bugs/doc/checkwatches.txt	2010-04-27 02:13:38 +0000
@@ -44,7 +44,7 @@
     0
 
     >>> print err
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-checkwatches.lock
     DEBUG   No global batch size specified.
     DEBUG   Skipping updating Ubuntu Bugzilla watches.
     DEBUG   No watches to update on http://bugs.debian.org
 
=== modified file 'lib/lp/bugs/doc/cve-update.txt'
--- lib/lp/bugs/doc/cve-update.txt	2009-06-12 16:36:02 +0000
+++ lib/lp/bugs/doc/cve-update.txt	2010-04-27 02:13:38 +0000
@@ -37,7 +37,7 @@
     ...     )
     >>> (output, empty) = process.communicate()
     >>> print output
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock
     ...
     INFO    CVE-1999-0002 created
     INFO    Creating new SGI reference for 1999-0002
@@ -91,7 +91,7 @@
     ...     )
     >>> (output, empty) = process.communicate()
     >>> print output
-    INFO    creating lockfile
+    INFO    Creating lockfile: /var/lock/launchpad-updatecve.lock
     ...
     INFO    Creating new CERT reference for 1999-0002
     INFO    Creating new CIAC reference for 1999-0002
 
=== modified file 'lib/lp/bugs/interfaces/bugwatch.py'
--- lib/lp/bugs/interfaces/bugwatch.py	2010-03-23 12:55:05 +0000
+++ lib/lp/bugs/interfaces/bugwatch.py	2010-04-27 02:13:38 +0000
@@ -8,7 +8,9 @@
 __metaclass__ = type
 
 __all__ = [
+    'BUG_WATCH_ACTIVITY_SUCCESS_STATUSES',
     'BugWatchActivityStatus',
+    'BugWatchCannotBeRescheduled',
     'IBugWatch',
     'IBugWatchActivity',
     'IBugWatchSet',
@@ -93,6 +95,42 @@
         Launchpad cannot import the status of private remote bugs.
         """)
 
+    SYNC_SUCCEEDED = DBItem(9, """
+        Synchronisation succeeded
+
+        The remote bug's status was successfully synchronized to Launchpad.
+        """)
+
+    COMMENT_IMPORT_FAILED = DBItem(10, """
+        Unable to import comments
+
+        The remote bug's status was synchronized successfully but
+        comments could not be imported from the remote bug.
+        """)
+
+    COMMENT_PUSH_FAILED = DBItem(11, """
+        Unable to push comments
+
+        The remote bug's status was synchronized successfully and
+        its comments were successfully imported but Launchpad was unable
+        to push comments back to the remote bug.
+        """)
+
+    BACKLINK_FAILED = DBItem(12, """
+        Unable to set link remote bug to Launchpad
+
+        The remote bug's status and comments were synchronized
+        sucessfully with Launchpad but Launchpad was unable to set the
+        remote bug's link back to the relevant Launchpad bug.
+        """)
+
+
+# The set of BugWatchActivityStatuses that are considered to indicate
+# success.
+BUG_WATCH_ACTIVITY_SUCCESS_STATUSES = [
+    BugWatchActivityStatus.SYNC_SUCCEEDED,
+    ]
+
 
 class IBugWatch(IHasBug):
     """A bug on a remote system."""
@@ -173,6 +211,10 @@
         Text(title=_('The URL at which to view the remote bug.'),
              readonly=True))
 
+    can_be_rescheduled = Attribute(
+        "A True or False indicator of whether or not this watch can be "
+        "rescheduled.")
+
     def updateImportance(remote_importance, malone_importance):
         """Update the importance of the bug watch and any linked bug task.
 
@@ -213,6 +255,13 @@
     def addActivity(result=None, message=None, oops_id=None):
         """Add an `IBugWatchActivity` record for this BugWatch."""
 
+    def setNextCheck(next_check):
+        """Set the next_check time of the watch.
+
+        :raises: `BugWatchCannotBeRescheduled` if
+                 `IBugWatch.can_be_rescheduled` is False.
+        """
+
 
 # Defined here because of circular imports.
 IBugTracker['watches'].value_type.schema = IBugWatch
@@ -326,3 +375,6 @@
         title=_('OOPS ID'), readonly=True,
         description=_("The OOPS ID associated with this activity."))
 
+
+class BugWatchCannotBeRescheduled(Exception):
+    """The current `IBugWatch` can't be rescheduled."""
 
=== modified file 'lib/lp/bugs/model/bugwatch.py'
--- lib/lp/bugs/model/bugwatch.py	2010-04-09 15:04:19 +0000
+++ lib/lp/bugs/model/bugwatch.py	2010-04-27 02:13:38 +0000
@@ -12,6 +12,9 @@
 
 import re
 import urllib
+
+from datetime import datetime
+from pytz import utc
 from urlparse import urlunsplit
 
 from zope.event import notify
@@ -44,8 +47,9 @@
 
 from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet
 from lp.bugs.interfaces.bugwatch import (
-    BugWatchActivityStatus, IBugWatch, IBugWatchActivity, IBugWatchSet,
-    NoBugTrackerFound, UnrecognizedBugTrackerURL)
+    BUG_WATCH_ACTIVITY_SUCCESS_STATUSES, BugWatchActivityStatus,
+    BugWatchCannotBeRescheduled, IBugWatch, IBugWatchActivity,
+    IBugWatchSet, NoBugTrackerFound, UnrecognizedBugTrackerURL)
 from lp.bugs.model.bugmessage import BugMessage
 from lp.bugs.model.bugset import BugSetBase
 from lp.bugs.model.bugtask import BugTask
@@ -66,6 +70,9 @@
     }
 
 
+WATCH_RESCHEDULE_THRESHOLD = 0.6
+
+
 class BugWatch(SQLBase):
     """See `IBugWatch`."""
     implements(IBugWatch)
@@ -212,6 +219,8 @@
 
         if self.last_error_type in error_message_mapping:
             message = error_message_mapping[self.last_error_type]
+        elif self.last_error_type != BugWatchActivityStatus.UNKNOWN:
+            message = self.last_error_type.description
         else:
             message = ("Launchpad couldn't import bug #%(bug)s from "
                 "%(bugtracker)s.")
@@ -284,7 +293,12 @@
         """See `IBugWatch`."""
         activity = BugWatchActivity()
         activity.bug_watch = self
-        activity.result = result
+        if result is None:
+            # If no result is passed we assume that the activity
+            # succeded and set the result field accordingly.
+            activity.result = BugWatchActivityStatus.SYNC_SUCCEEDED
+        else:
+            activity.result = result
         if message is not None:
             activity.message = unicode(message)
         if oops_id is not None:
@@ -300,6 +314,52 @@
             BugWatchActivity.bug_watch == self).order_by(
                 Desc('activity_date'))
 
+    @property
+    def can_be_rescheduled(self):
+        """See `IBugWatch`."""
+        if (self.next_check is not None and
+            self.next_check <= datetime.now(utc)):
+            # If the watch is already scheduled for a time in the past
+            # (or for right now) it can't be rescheduled, since it
+            # should be be checked by the next checkwatches run anyway.
+            return False
+
+        if self.activity.is_empty():
+            # Don't show the reschedule button if the watch has never
+            # been checked.
+            return False
+
+        if self.failed_activity.is_empty():
+            # Don't show the reschedule button if the watch has never
+            # failed.
+            return False
+
+        # If the ratio is lower than the reschedule threshold, we
+        # can show the button.
+        failure_ratio = (
+            float(self.failed_activity.count()) /
+            self.activity.count())
+        return failure_ratio <= WATCH_RESCHEDULE_THRESHOLD
+
+    @property
+    def failed_activity(self):
+        store = Store.of(self)
+        success_status_ids = [
+            status.value for status in BUG_WATCH_ACTIVITY_SUCCESS_STATUSES]
+
+        return store.find(
+            BugWatchActivity,
+            BugWatchActivity.bug_watch == self,
+            Not(In(BugWatchActivity.result, success_status_ids))).order_by(
+                Desc('activity_date'))
+
+    def setNextCheck(self, next_check):
+        """See `IBugWatch`."""
+        if not self.can_be_rescheduled:
+            raise BugWatchCannotBeRescheduled()
+
+        self.next_check = next_check
+
 
 class BugWatchSet(BugSetBase):
     """A set for BugWatch"""
 
=== modified file 'lib/lp/bugs/scripts/bugheat.py'
--- lib/lp/bugs/scripts/bugheat.py	2010-04-14 12:55:44 +0000
+++ lib/lp/bugs/scripts/bugheat.py	2010-04-27 02:13:38 +0000
@@ -79,12 +79,12 @@
             self._getHeatFromSubscribers(),
             ])
 
-        # Bugs decay over time. Every month the bug isn't touched its heat
-        # decreases by 10%.
-        months = (
+        # Bugs decay over time. Every day the bug isn't touched its heat
+        # decreases by 1%.
+        days = (
             datetime.utcnow() -
-            self.bug.date_last_updated.replace(tzinfo=None)).days / 30
-        total_heat = int(total_heat * (0.9 ** months))
+            self.bug.date_last_updated.replace(tzinfo=None)).days
+        total_heat = int(total_heat * (0.99 ** days))
 
         return total_heat
 
 
=== modified file 'lib/lp/bugs/scripts/checkwatches/scheduler.py'
--- lib/lp/bugs/scripts/checkwatches/scheduler.py	2010-03-26 14:33:46 +0000
+++ lib/lp/bugs/scripts/checkwatches/scheduler.py	2010-04-27 02:13:38 +0000
@@ -10,12 +10,11 @@
 
 import transaction
 
-from storm.expr import Not
-
 from canonical.database.sqlbase import sqlvalues
 from canonical.launchpad.utilities.looptuner import TunableLoop
 from canonical.launchpad.interfaces import IMasterStore
 
+from lp.bugs.interfaces.bugwatch import BUG_WATCH_ACTIVITY_SUCCESS_STATUSES
 from lp.bugs.model.bugwatch import BugWatch
 
 
@@ -70,7 +69,7 @@
                         FROM (SELECT 1
                             FROM bugwatchactivity
                            WHERE bugwatchactivity.bug_watch = bug_watch.id
-                             AND bugwatchactivity.result IS NOT NULL
+                             AND bugwatchactivity.result NOT IN (%s)
                            ORDER BY bugwatchactivity.id DESC
                            LIMIT %s) AS recent_failures
                     ) AS recent_failure_count
@@ -80,7 +79,8 @@
             ) AS counts
         WHERE BugWatch.id = counts.id
         """ % sqlvalues(
-            self.delay_coefficient, self.max_sample_size, chunk_size)
+            self.delay_coefficient, BUG_WATCH_ACTIVITY_SUCCESS_STATUSES,
+            self.max_sample_size, chunk_size)
         self.transaction.begin()
         result = self.store.execute(query)
         self.log.debug("Scheduled %s watches" % result.rowcount)
 
=== modified file 'lib/lp/bugs/scripts/tests/test_bugheat.py'
--- lib/lp/bugs/scripts/tests/test_bugheat.py	2010-04-14 12:55:44 +0000
+++ lib/lp/bugs/scripts/tests/test_bugheat.py	2010-04-27 02:13:38 +0000
@@ -215,12 +215,12 @@
             "Expected %s, got %s" % (0, heat))
 
     def test_getBugHeat_decay(self):
-        # Every month, a bug that wasn't touched has its heat reduced by 10%.
+        # Every day, a bug that wasn't touched has its heat reduced by 1%.
         aging_bug = self.factory.makeBug()
         fresh_heat = BugHeatCalculator(aging_bug).getBugHeat()
         aging_bug.date_last_updated = (
-            aging_bug.date_last_updated - timedelta(days=32))
-        expected = int(fresh_heat * 0.9)
+            aging_bug.date_last_updated - timedelta(days=1))
+        expected = int(fresh_heat * 0.99)
         heat = BugHeatCalculator(aging_bug).getBugHeat()
         self.assertEqual(
             expected, heat,
 
=== modified file 'lib/lp/bugs/stories/bugwatches/xx-bugwatch-errors.txt'
--- lib/lp/bugs/stories/bugwatches/xx-bugwatch-errors.txt	2010-04-16 11:20:33 +0000
+++ lib/lp/bugs/stories/bugwatches/xx-bugwatch-errors.txt	2010-04-27 02:13:38 +0000
@@ -62,11 +62,17 @@
     The Mozilla.org Bug Tracker bug #900 appears not to exist. Check
     that the bug number is correct.
 
-We can observe this for each of the BugWatchActivityStatus values:
+We can observe this for each of the BugWatchActivityStatus failure values:
 
+    >>> from lp.bugs.interfaces.bugwatch import (
+    ...     BUG_WATCH_ACTIVITY_SUCCESS_STATUSES)
     >>> from lp.bugs.tests.externalbugtracker import (
     ...     set_bugwatch_error_type)
-    >>> for item in sorted(BugWatchActivityStatus.items):
+
+    >>> failure_values = [
+    ...     value for value in sorted(BugWatchActivityStatus.items) if
+    ...     value not in BUG_WATCH_ACTIVITY_SUCCESS_STATUSES]
+    >>> for item in failure_values:
     ...     set_bugwatch_error_type(watch, item)
     ...     user_browser.open('http://bugs.launchpad.dev/thunderbird/+bug/12')
     ...     for tag in find_tags_by_class(user_browser.contents,
@@ -86,6 +92,14 @@
     Launchpad doesn't support importing bugs from Bugzilla bug trackers.
     The bug is marked as private on the remote bug tracker. Launchpad
     cannot import the status of private remote bugs.
+    The remote bug's status was synchronized successfully but comments
+    could not be imported from the remote bug.
+    The remote bug's status was synchronized successfully and its
+    comments were successfully imported but Launchpad was unable to push
+    comments back to the remote bug.
+    The remote bug's status and comments were synchronized sucessfully
+    with Launchpad but Launchpad was unable to set the remote bug's link
+    back to the relevant Launchpad bug.
 
 Finally, if the error gets solved (or no error occurs), the error
 message will go away.
 
=== modified file 'lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt'
--- lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt	2010-04-09 12:00:54 +0000
+++ lib/lp/bugs/stories/bugwatches/xx-edit-bugwatch.txt	2010-04-27 02:13:38 +0000
@@ -77,3 +77,126 @@
     ...     admin_browser.contents, 'bugwatch-next_check')
     >>> print extract_text(data_tag.renderContents())
     Next check: 2010-04-08...
+
+
+Recent activity
+---------------
+
+Recent activity on a bug watch is shown on the page as a list of
+activity entries. When a watch has not been checked, no activity is
+shown.
+
+    >>> admin_browser.open('http://bugs.launchpad.dev/bugs/1/+watch/2')
+    >>> recent_activity_list = find_tag_by_id(
+    ...     admin_browser.contents, 'recent-watch-activity')
+    >>> print recent_activity_list
+    None
+
+Adding some activity to the watch will cause it to show up in the recent
+activity list.
+
+    >>> login('[e-mail]')
+    >>> watch = getUtility(IBugWatchSet).get(2)
+    >>> watch.addActivity()
+    >>> logout()
+
+    >>> admin_browser.open('http://bugs.launchpad.dev/bugs/1/+watch/2')
+    >>> recent_activity_list = find_tag_by_id(
+    ...     admin_browser.contents, 'recent-watch-activity')
+    >>> print extract_text(recent_activity_list)
+    Update completed successfully ... ago
+
+If an update fails, that too will be reflected in the list.
+
+    >>> from lp.bugs.interfaces.bugwatch import BugWatchActivityStatus
+    >>> login('[e-mail]')
+    >>> watch = getUtility(IBugWatchSet).get(2)
+    >>> watch.addActivity(result=BugWatchActivityStatus.BUG_NOT_FOUND)
+    >>> logout()
+
+    >>> admin_browser.open('http://bugs.launchpad.dev/bugs/1/+watch/2')
+    >>> recent_activity_list = find_tag_by_id(
+    ...     admin_browser.contents, 'recent-watch-activity')
+    >>> print extract_text(recent_activity_list)
+    Update failed with error 'Bug Not Found' ... ago
+    Update completed successfully ... ago
+
+
+Rescheduling a watch
+--------------------
+
+It's possible to reschedule a failing watch via the BugWatch +edit page
+by clicking the "Update Now" button.
+
+For a new watch, the "Update Now" button isn't shown.
+
+    >>> from pytz import utc
+    >>> from datetime import datetime, timedelta
+    >>> login('[e-mail]')
+    >>> bug_watch = factory.makeBugWatch()
+    >>> bug_watch.next_check = None
+    >>> watch_url = (
+    ...     'http://bugs.launchpad.dev/bugs/%s/+watch/%s' %
+    ...     (bug_watch.bug.id, bug_watch.id))
+    >>> logout()
+
+    >>> user_browser.open(watch_url)
+    >>> user_browser.getControl('Update Now')
+    Traceback (most recent call last):
+      ...
+    LookupError: label 'Update Now'
+
+If the watch has been checked but has never failed, the button will
+remain hidden.
+
+    >>> login('[e-mail]')
+    >>> bug_watch.addActivity()
+    >>> logout()
+
+    >>> user_browser.open(watch_url)
+    >>> user_browser.getControl('Update Now')
+    Traceback (most recent call last):
+      ...
+    LookupError: label 'Update Now'
+
+If the watch has failed less than 60% of its recent checks, the button
+will appear on the page.
+
+    >>> login('[e-mail]')
+    >>> bug_watch.addActivity(result=BugWatchActivityStatus.BUG_NOT_FOUND)
+    >>> logout()
+
+    >>> user_browser.open(watch_url)
+    >>> reschedule_button = user_browser.getControl('Update Now')
+
+    >>> data_tag = find_tag_by_id(
+    ...     user_browser.contents, 'bugwatch-next_check')
+    >>> print extract_text(data_tag.renderContents())
+    Next check: Not yet scheduled
+
+Clicking the Update Now button will schedule it to be checked
+immediately.
+
+    >>> reschedule_button.click()
+
+    >>> for message in find_tags_by_class(
+    ...     user_browser.contents, 'informational message'):
+    ...     print extract_text(message)
+    The ... bug watch has been scheduled for immediate checking.
+
+Looking at the watch +edit page again, we can see that the watch has
+been scheduled.
+
+    >>> user_browser.open(watch_url)
+    >>> data_tag = find_tag_by_id(
+    ...     user_browser.contents, 'bugwatch-next_check')
+    >>> print extract_text(data_tag.renderContents())
+    Next check: 2...
+
+The button will no longer be shown on the page.
+
+    >>> reschedule_button = user_browser.getControl('Update Now')
+    Traceback (most recent call last):
+      ...
+    LookupError: label 'Update Now'
+
 
=== modified file 'lib/lp/bugs/templates/bugwatch-editform.pt'
--- lib/lp/bugs/templates/bugwatch-editform.pt	2009-09-03 12:43:53 +0000
+++ lib/lp/bugs/templates/bugwatch-editform.pt	2010-04-27 02:13:38 +0000
@@ -21,6 +21,10 @@
         </div>
       </div>
 
+      <div class="yui-g" tal:condition="view/watch_has_activity">
+        <div tal:replace="structure context/@@+portlet-activity" />
+      </div>
+
       <div class="yui-g">
         <div metal:use-macro="context/@@launchpad_form/form"/>
       </div>
 
=== added file 'lib/lp/bugs/templates/bugwatch-portlet-activity.pt'
--- lib/lp/bugs/templates/bugwatch-portlet-activity.pt	1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/templates/bugwatch-portlet-activity.pt	2010-04-27 02:13:38 +0000
@@ -0,0 +1,44 @@
+<div
+    xmlns:tal="http://xml.zope.org/namespaces/tal"
+    xmlns:metal="http://xml.zope.org/namespaces/metal"
+    xmlns:i18n="http://xml.zope.org/namespaces/i18n"
+    class="portlet" id="portlet-watches">
+  <h2>Recent activity</h2>
+  <div id="recent-watch-activity">
+    <div>
+      <form
+          tal:attributes="action view/action_url;"
+          tal:condition="view/userCanReschedule"
+          name="launchpadform"
+          id="reschedule-form"
+          method="post"
+          enctype="multipart/form-data"
+          accept-charset="UTF-8">
+        <div>
+          This watch has failed to update at
+          <tal:fail-count
+              replace="context/failed_activity/count" />
+          out of the last
+          <tal:fail-count
+              replace="context/activity/count" />
+          attempts.
+        </div>
+        <div>
+          The next update will occur
+          <tal:next-check
+              replace="view/context/next_check/fmt:approximatedate" />
+          <tal:reschedule-button
+              replace="structure view/reschedule_action/render" />
+        </div>
+      </form>
+    </div>
+    <tal:activity repeat="activity view/recent_watch_activity">
+      <div>
+        <img tal:attributes="src activity/icon; title activity/result_text" />
+        Update
+        <tal:message replace="activity/completion_message" />
+        <tal:time replace="activity/date/fmt:displaydate" />
+      </div>
+    </tal:activity>
+  </div>
+</div>
 
=== modified file 'lib/lp/bugs/tests/test_apportjob.py'
--- lib/lp/bugs/tests/test_apportjob.py	2010-04-08 13:26:26 +0000
+++ lib/lp/bugs/tests/test_apportjob.py	2010-04-27 02:13:38 +0000
@@ -273,7 +273,7 @@
             expect_returncode=0)
         self.assertEqual('', stdout)
         self.assertIn(
-            'INFO    Ran 1 IProcessApportBlobJobSource jobs.\n', stderr)
+            'INFO    Ran 1 ProcessApportBlobJob jobs.\n', stderr)
 
     def test_getFileBugData(self):
         # The IProcessApportBlobJobSource.getFileBugData() method
 
=== modified file 'lib/lp/bugs/tests/test_bugheat.py'
--- lib/lp/bugs/tests/test_bugheat.py	2010-02-25 21:37:02 +0000
+++ lib/lp/bugs/tests/test_bugheat.py	2010-04-27 02:13:38 +0000
@@ -96,7 +96,7 @@
             expect_returncode=0)
         self.assertEqual('', stdout)
         self.assertIn(
-            'INFO    Ran 1 ICalculateBugHeatJobSource jobs.\n', stderr)
+            'INFO    Ran 1 CalculateBugHeatJob jobs.\n', stderr)
 
     def test_getOopsVars(self):
         # BugJobDerived.getOopsVars() returns the variables to be used
 
=== modified file 'lib/lp/code/configure.zcml'
--- lib/lp/code/configure.zcml	2010-04-23 02:35:47 +0000
+++ lib/lp/code/configure.zcml	2010-04-27 02:13:38 +0000
@@ -38,7 +38,7 @@
   <subscriber
       for="lp.code.interfaces.codereviewvote.ICodeReviewVoteReference
            lp.code.interfaces.event.IReviewerNominatedEvent"
-      handler="lp.code.mail.branchmergeproposal.send_review_requested_notifications"/>
+      handler="lp.code.subscribers.branchmergeproposal.review_requested"/>
 
   <!-- CodeImportMachine -->
 
@@ -272,19 +272,84 @@
       factory="lp.code.browser.branchmergeproposal.text_xhtml_representation"
       name="description"/>
 
+  <!-- Branch Merge Proposal Jobs -->
 
   <class class="lp.code.model.branchmergeproposaljob.CreateMergeProposalJob">
     <allow interface="canonical.launchpad.interfaces.IMessageJob"/>
     <allow interface="lp.code.interfaces.branchmergeproposal.ICreateMergeProposalJob"/>
   </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.CreateMergeProposalJob"
+      provides="lp.code.interfaces.branchmergeproposal.ICreateMergeProposalJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.ICreateMergeProposalJobSource"/>
+  </securedutility>
+
+  <class class="lp.code.model.branchmergeproposaljob.MergeProposalCreatedJob">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob"/>
+    <allow interface="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJob"/>
+  </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.MergeProposalCreatedJob"
+      provides="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJobSource"/>
+  </securedutility>
+
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.BranchMergeProposalJobSource"
+      provides="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJobSource"/>
+  </securedutility>
+
+  <class class="lp.code.model.branchmergeproposaljob.UpdatePreviewDiffJob">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IUpdatePreviewDiffJob" />
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob" />
+  </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.UpdatePreviewDiffJob"
+      provides="lp.code.interfaces.branchmergeproposal.IUpdatePreviewDiffJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IUpdatePreviewDiffJobSource"/>
+  </securedutility>
+
+  <class class="lp.code.model.branchmergeproposaljob.CodeReviewCommentEmailJob">
+    <allow interface="lp.code.interfaces.branchmergeproposal.ICodeReviewCommentEmailJob" />
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob" />
+  </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.CodeReviewCommentEmailJob"
+      provides="lp.code.interfaces.branchmergeproposal.ICodeReviewCommentEmailJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.ICodeReviewCommentEmailJobSource"/>
+  </securedutility>
+
+  <class class="lp.code.model.branchmergeproposaljob.ReviewRequestedEmailJob">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IReviewRequestedEmailJob" />
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob" />
+  </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.ReviewRequestedEmailJob"
+      provides="lp.code.interfaces.branchmergeproposal.IReviewRequestedEmailJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IReviewRequestedEmailJobSource"/>
+  </securedutility>
+
+  <class class="lp.code.model.branchmergeproposaljob.MergeProposalUpdatedEmailJob">
+    <allow interface="lp.services.job.interfaces.job.IRunnableJob" />
+    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob" />
+  </class>
+  <securedutility
+      component="lp.code.model.branchmergeproposaljob.MergeProposalUpdatedEmailJob"
+      provides="lp.code.interfaces.branchmergeproposal.IMergeProposalUpdatedEmailJobSource">
+    <allow interface="lp.code.interfaces.branchmergeproposal.IMergeProposalUpdatedEmailJobSource"/>
+  </securedutility>
+
+  <!-- Branch Merge Proposal Subscribers -->
+
   <subscriber
       for="lp.code.interfaces.branchmergeproposal.IBranchMergeProposal
            lp.code.interfaces.event.INewBranchMergeProposalEvent"
-      handler="lp.code.mail.branchmergeproposal.send_merge_proposal_created_notifications"/>
+      handler="lp.code.subscribers.branchmergeproposal.merge_proposal_created"/>
   <subscriber
       for="lp.code.interfaces.branchmergeproposal.IBranchMergeProposal
            lazr.lifecycle.interfaces.IObjectModifiedEvent"
-      handler="lp.code.mail.branchmergeproposal.send_merge_proposal_modified_notifications"/>
+      handler="lp.code.subscribers.branchmergeproposal.merge_proposal_modified"/>
   <subscriber
       for="lp.code.interfaces.branchmergeproposal.IBranchMergeProposal
            lp.code.interfaces.event.INewBranchMergeProposalEvent"
@@ -297,10 +362,6 @@
       for="lp.code.interfaces.branchmergeproposal.IBranchMergeProposal"
       provides="canonical.launchpad.webapp.interfaces.IPrimaryContext"
       factory="lp.code.browser.branchmergeproposal.BranchMergeProposalPrimaryContext"/>
-  <class class="lp.code.model.branchmergeproposaljob.MergeProposalCreatedJob">
-    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob"/>
-    <allow interface="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJob"/>
-  </class>
 
   <!-- hierarchy -->
 
@@ -309,16 +370,6 @@
       provides="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalGetter">
     <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalGetter"/>
   </securedutility>
-  <securedutility
-      component="lp.code.model.branchmergeproposaljob.CreateMergeProposalJob"
-      provides="lp.code.interfaces.branchmergeproposal.ICreateMergeProposalJobSource">
-    <allow interface="lp.code.interfaces.branchmergeproposal.ICreateMergeProposalJobSource"/>
-  </securedutility>
-  <securedutility
-      component="lp.code.model.branchmergeproposaljob.MergeProposalCreatedJob"
-      provides="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJobSource">
-    <allow interface="lp.code.interfaces.branchmergeproposal.IMergeProposalCreatedJobSource"/>
-  </securedutility>
   <class
       class="lp.code.model.seriessourcepackagebranch.SeriesSourcePackageBranch">
     <allow interface="lp.code.interfaces.seriessourcepackagebranch.ISeriesSourcePackageBranch"/>
@@ -907,15 +958,6 @@
     <allow interface="lp.code.interfaces.revision.IRevisionSet" />
   </securedutility>
 
-  <securedutility
-      component="lp.code.model.branchmergeproposaljob.UpdatePreviewDiffJob"
-      provides="lp.code.interfaces.branchmergeproposal.IUpdatePreviewDiffJobSource">
-    <allow interface="lp.code.interfaces.branchmergeproposal.IUpdatePreviewDiffJobSource"/>
-  </securedutility>
-  <class class="lp.code.model.branchmergeproposaljob.UpdatePreviewDiffJob">
-    <allow interface="lp.services.job.interfaces.job.IRunnableJob" />
-    <allow interface="lp.code.interfaces.branchmergeproposal.IBranchMergeProposalJob" />
-  </class>
 
   <securedutility
       component="lp.code.model.branchjob.BranchUpgradeJob"
 
=== modified file 'lib/lp/code/doc/branch-merge-proposal-notifications.txt'
--- lib/lp/code/doc/branch-merge-proposal-notifications.txt	2010-02-19 02:15:27 +0000
+++ lib/lp/code/doc/branch-merge-proposal-notifications.txt	2010-04-27 02:13:38 +0000
@@ -15,7 +15,7 @@
     ...     BranchSubscriptionDiffSize, BranchSubscriptionNotificationLevel,
     ...     CodeReviewNotificationLevel)
     >>> from lp.code.interfaces.branchmergeproposal import (
-    ...     IMergeProposalCreatedJobSource)
+    ...     IBranchMergeProposalJobSource)
     >>> from lp.code.model.diff import PreviewDiff
     >>> from lp.testing.mail_helpers import pop_notifications
     >>> import transaction
@@ -103,8 +103,11 @@
     >>> bmp = source_branch.addLandingTarget(
     ...     registrant, target_branch)
     >>> removeSecurityProxy(bmp).preview_diff = preview_diff
-    >>> [job,] = list(getUtility(IMergeProposalCreatedJobSource).iterReady())
-    >>> job.run(_create_preview=False)
+    >>> # Fake the update preview diff as done.
+    >>> bmp.next_preview_diff_job.start()
+    >>> bmp.next_preview_diff_job.complete()
+    >>> [job] = list(getUtility(IBranchMergeProposalJobSource).iterReady())
+    >>> job.run()
     >>> notifications = pop_notifications(
     ...     sort_key=lambda n: n.get('X-Envelope-To'))
 
@@ -155,8 +158,11 @@
     ...     registrant, target_branch,
     ...     description=initial_comment, review_requests=reviewers)
     >>> removeSecurityProxy(bmp).preview_diff = preview_diff
-    >>> [job,] = list(getUtility(IMergeProposalCreatedJobSource).iterReady())
-    >>> job.run(_create_preview=False)
+    >>> # Fake the update preview diff as done.
+    >>> bmp.next_preview_diff_job.start()
+    >>> bmp.next_preview_diff_job.complete()
+    >>> [job] = list(getUtility(IBranchMergeProposalJobSource).iterReady())
+    >>> job.run()
     >>> notifications = pop_notifications(
     ...     sort_key=lambda n: n.get('X-Envelope-To'))
     >>> for notification in notifications:
 
=== modified file 'lib/lp/code/doc/codereviewcomment.txt'
--- lib/lp/code/doc/codereviewcomment.txt	2010-02-18 00:35:39 +0000
+++ lib/lp/code/doc/codereviewcomment.txt	2010-04-27 02:13:38 +0000
@@ -45,6 +45,15 @@
     >>> comment3.message.parent == comment2.message
     True
 
+When comments are added, a job is created to send the emails to the
+subscribers of the merge proposal.
+
+    >>> # Needed for now to make the iterReady show the jobs.
+    >>> factory.makeRevisionsForBranch(merge_proposal.source_branch)
+    >>> factory.makeRevisionsForBranch(merge_proposal.target_branch)
+    >>> from lp.code.tests.helpers import mark_all_merge_proposal_jobs_done
+    >>> mark_all_merge_proposal_jobs_done()
+
 If there is a subscriber to any of the branches involved in the merge,
 a notification is produced when the comment is created.
 
@@ -60,9 +69,18 @@
     >>> from lp.testing.mail_helpers import (
     ...     pop_notifications, print_emails)
     >>> _unused = pop_notifications()
+    >>> merge_proposal.root_message_id = '<[e-mail]>'
     >>> comment = merge_proposal.createComment(
     ...     sender, 'Please merge', 'This patch is very nice.',
     ...     vote=CodeReviewVote.APPROVE, review_type='DB')
+
+Now run the pending job to send the email.
+
+    >>> from lp.code.interfaces.branchmergeproposal import (
+    ...     IBranchMergeProposalJobSource)
+    >>> [job] = list(getUtility(IBranchMergeProposalJobSource).iterReady())
+    >>> job.run()
+
     >>> notifications = pop_notifications()
     >>> notifications = [email for email in notifications if
     ...                  email['X-Launchpad-Message-Rationale'] == 'Owner']
 
=== modified file 'lib/lp/code/interfaces/branchmergeproposal.py'
--- lib/lp/code/interfaces/branchmergeproposal.py	2010-03-05 03:35:10 +0000
+++ lib/lp/code/interfaces/branchmergeproposal.py	2010-04-27 02:13:38 +0000
@@ -11,11 +11,19 @@
     'IBranchMergeProposal',
     'IBranchMergeProposalGetter',
     'IBranchMergeProposalJob',
+    'IBranchMergeProposalJobSource',
     'IBranchMergeProposalListingBatchNavigator',
+    'ICodeReviewCommentEmailJob',
+    'ICodeReviewCommentEmailJobSource',
     'ICreateMergeProposalJob',
     'ICreateMergeProposalJobSource',
     'IMergeProposalCreatedJob',
     'IMergeProposalCreatedJobSource',
+    'IMergeProposalUpdatedEmailJob',
+    'IMergeProposalUpdatedEmailJobSource',
+    'IReviewRequestedEmailJob',
+    'IReviewRequestedEmailJobSource',
+    'IUpdatePreviewDiffJob',
     'IUpdatePreviewDiffJobSource',
     'notify_modified',
     ]
@@ -29,12 +37,13 @@
 
 from canonical.launchpad import _
 from canonical.launchpad.fields import PublicPersonChoice, Summary, Whiteboard
-from canonical.launchpad.interfaces import IBug, IPrivacy
+from canonical.launchpad.interfaces.launchpad import IPrivacy
+from lp.bugs.interfaces.bug import IBug
 from lp.code.enums import BranchMergeProposalStatus, CodeReviewVote
 from lp.code.interfaces.branch import IBranch
 from lp.registry.interfaces.person import IPerson
 from lp.code.interfaces.diff import IPreviewDiff, IStaticDiff
-from lp.services.job.interfaces.job import IJob, IRunnableJob
+from lp.services.job.interfaces.job import IJob, IJobSource, IRunnableJob
 from canonical.launchpad.webapp.interfaces import ITableBatchNavigator
 from lazr.restful.fields import CollectionField, Reference
 from lazr.restful.declarations import (
@@ -500,6 +509,10 @@
         """Destroy this object."""
 
 
+class IBranchMergeProposalJobSource(IJobSource):
+    """A job source that will get all supported merge proposal jobs."""
+
+
 class IBranchMergeProposalListingBatchNavigator(ITableBatchNavigator):
     """A marker interface for registering the appropriate listings."""
 
@@ -570,15 +583,12 @@
     """
 
 
-class ICreateMergeProposalJobSource(Interface):
+class ICreateMergeProposalJobSource(IJobSource):
     """Acquire MergeProposalJobs."""
 
     def create(message_bytes):
         """Return a CreateMergeProposalJob for this message."""
 
-    def iterReady():
-        """Iterate through jobs that are ready to run."""
-
 
 class IMergeProposalCreatedJob(IRunnableJob):
     """Interface for review diffs."""
@@ -590,8 +600,12 @@
     def create(bmp):
         """Create a MergeProposalCreatedJob for the specified Job."""
 
-    def iterReady():
-        """Iterate through all ready MergeProposalCreatedJobs."""
+
+class IUpdatePreviewDiffJob(IRunnableJob):
+    """Interface for the job to update the diff for a merge proposal."""
+
+    def checkReady():
+        """Check to see if this job is ready to run."""
 
 
 class IUpdatePreviewDiffJobSource(Interface):
@@ -603,11 +617,55 @@
     def get(id):
         """Return the UpdatePreviewDiffJob with this id."""
 
-    def iterReady():
-        """Iterate through jobs ready to update preview diffs."""
-
-    def contextManager():
-        """Get a context for running this kind of job in."""
+
+class ICodeReviewCommentEmailJob(IRunnableJob):
+    """Interface for the job to send code review comment email."""
+
+    code_review_comment = Attribute('The code review comment.')
+
+
+class ICodeReviewCommentEmailJobSource(Interface):
+    """Create or retrieve jobs that update preview diffs."""
+
+    def create(code_review_comment):
+        """Create a job to email subscribers about the comment."""
+
+
+class IReviewRequestedEmailJob(IRunnableJob):
+    """Interface for the job to sends review request emails."""
+
+    reviewer = Attribute('The person or team asked to do the review.')
+    requester = Attribute('The person who has asked for the review.')
+
+
+class IReviewRequestedEmailJobSource(Interface):
+    """Create or retrieve jobs that email review requests."""
+
+    def create(review_request):
+        """Create a job to email a review request.
+
+        :param review_request: A vote reference for the requested review.
+        """
+
+
+class IMergeProposalUpdatedEmailJob(IRunnableJob):
+    """Interface for the job to sends email about merge proposal updates."""
+
+    editor = Attribute('The person that did the editing.')
+    delta_text = Attribute(
+        'The textual representation of the changed fields.')
+
+
+class IMergeProposalUpdatedEmailJobSource(Interface):
+    """Create or retrieve jobs that email about merge proposal updates."""
+
+    def create(merge_proposal, delta_text, editor):
+        """Create a job to email merge proposal updates to subscribers.
+
+        :param merge_proposal: The merge proposal that has been edited.
+        :param delta_text: The text representation of the changed fields.
+        :param editor: The person who did the editing.
+        """
 
 
 # XXX: JonathanLange 2010-01-06: This is only used in the scanner, perhaps it
 
=== modified file 'lib/lp/code/interfaces/codehosting.py'
--- lib/lp/code/interfaces/codehosting.py	2010-02-24 04:25:38 +0000
+++ lib/lp/code/interfaces/codehosting.py	2010-04-27 02:13:38 +0000
@@ -179,6 +179,18 @@
         :param branchID: a branch ID.
         """
 
+    def branchChanged(branch_id, stacked_on_url, last_revision_id):
+        """Record that a branch has been changed.
+
+        This method records the stacked on branch and tip revision id of the
+        branch and creates a scan job if the tip revision id has changed.
+
+        :param branchID: The database id of the branch to operate on.
+        :param stacked_on_url: The unique name of the branch this branch is
+            stacked on, or '' if this branch is not stacked.
+        :param last_revision_id: The tip revision ID of the branch.
+        """
+
     def translatePath(requester_id, path):
         """Translate 'path' so that the codehosting transport can access it.
 
 
=== modified file 'lib/lp/code/interfaces/codereviewcomment.py'
--- lib/lp/code/interfaces/codereviewcomment.py	2010-04-22 04:40:15 +0000
+++ lib/lp/code/interfaces/codereviewcomment.py	2010-04-27 02:13:38 +0000
@@ -74,12 +74,16 @@
             attachments.
         """
 
+    def getOriginalEmail():
+        """An email object of the original raw email if there was one."""
+
     as_quoted_email = exported(
         TextLine(
             title=_('The message as quoted in email.'),
             readonly=True))
 
 
+
 class ICodeReviewCommentDeletion(Interface):
     """This interface provides deletion of CodeReviewComments.
 
 
=== modified file 'lib/lp/code/mail/branch.py'
--- lib/lp/code/mail/branch.py	2010-03-09 16:58:30 +0000
+++ lib/lp/code/mail/branch.py	2010-04-27 02:13:38 +0000
@@ -69,23 +69,23 @@
             review_level=subscription.review_level)
 
     @classmethod
-    def forReviewer(cls, vote_reference, recipient,
+    def forReviewer(cls, branch_merge_proposal, pending_review, reviewer,
                     branch_identity_cache=None):
         """Construct RecipientReason for a reviewer.
 
         The reviewer will be the sole recipient.
         """
-        merge_proposal = vote_reference.branch_merge_proposal
-        branch = merge_proposal.source_branch
-        if vote_reference.comment is None:
+        branch = branch_merge_proposal.source_branch
+        if pending_review:
             reason_template = (
                 '%(entity_is)s requested to review %(merge_proposal)s.')
         else:
             reason_template = (
                 '%(entity_is)s reviewing %(merge_proposal)s.')
-        return cls(vote_reference.reviewer, recipient, branch,
-                     'Reviewer', reason_template, merge_proposal,
-                     branch_identity_cache=branch_identity_cache)
+        return cls(reviewer, reviewer, branch,
+                   cls.makeRationale('Reviewer', reviewer),
+                   reason_template, branch_merge_proposal,
+                   branch_identity_cache=branch_identity_cache)
 
     @classmethod
     def forRegistrant(cls, merge_proposal, branch_identity_cache=None):
@@ -93,7 +93,6 @@
 
         The registrant will be the sole recipient.
         """
-        branch = merge_proposal.source_branch
         reason_template = 'You proposed %(branch_name)s for merging.'
         return cls(merge_proposal.registrant, merge_proposal.registrant,
                      merge_proposal.source_branch,
@@ -124,16 +123,16 @@
         The owner will be the sole recipient.
         """
         return cls(branch.owner, recipient, branch,
-                     cls.makeRationale('Owner', branch.owner, recipient),
+                     cls.makeRationale('Owner', branch.owner),
                      'You are getting this email as %(lc_entity_is)s the'
                      ' owner of the branch and someone has edited the'
                      ' details.',
                      branch_identity_cache=branch_identity_cache)
 
     @staticmethod