Merge lp:~allenap/launchpad/dynamic-batch-size-bug-546085 into lp:launchpad/db-devel

[Gavin Panella]

In checkwatches, calculate a sensible batch size at run-time based on the number of watches there are for a bug tracker.

When checkwatches runs it observes a batch_size attribute on the object representing the remote system. This is fine, but for bug trackers with an especially large number of bug watches the batch size is a bit too low. This branch makes batch_size equal to 2% of the number of bug watches recorded for a bug tracker, or 100, whichever is greater. For GNOME Bugs this works out at about 300 (GNOME Bugs is the largest active bug tracker registered in Launchpad, with >15000 bug watches).

I've also moved the test for BugWatchScheduler into the lp.bugs.scripts.checkwatches.tests package, and the tests for the updater too.

[Edwin Grubbs (edwin-grubbs)]

Hi Gavin,

This is a nice branch. I just have two comments besides the lint errors.

merge-conditional

-Edwin

== Pyflakes notices ==

lib/lp/bugs/externalbugtracker/debbugs.py
    124: local variable 'severity' is assigned to but never used

lib/lp/bugs/scripts/checkwatches/tests/test_updater.py
    157: local variable 'bug' is assigned to but never used

>=== modified file 'lib/lp/bugs/externalbugtracker/debbugs.py'
>--- lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-16 16:52:42 +0000
>+++ lib/lp/bugs/externalbugtracker/debbugs.py 2010-03-31 15:33:31 +0000
>@@ -61,7 +61,7 @@
> # Because we keep a local copy of debbugs, we remove the batch_size
> # limit so that all debbugs watches that need checking will be
> # checked each time checkwatches runs.
>- batch_size = None
>+ batch_size = 0

Zero is a magic number. It would be good to define it as a constant
as opposed to relying on comments to explain what it means. You might
want to define the constant as a unique object such as:
    UNLIMITED = object()
so that there is never the possiblity of someone using zero and getting
the opposite results than they expect. That would allow you to get
rid of this if-statement in _getRemoteIdsToCheck() where you change
the meaning of the magic values.
    if batch_size == 0:
        batch_size = None

>
> def __init__(self, baseurl, db_location=None):
> super(DebBugs, self).__init__(baseurl)
>
>=== modified file 'lib/lp/bugs/scripts/checkwatches/updater.py'
>--- lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-26 21:03:30 +0000
>+++ lib/lp/bugs/scripts/checkwatches/updater.py 2010-03-31 15:33:31 +0000
>@@ -242,6 +247,22 @@
> yield item
>
>
>+def suggest_batch_size(remote_system, num_watches):
>+ """Suggest a value for batch_size if it's not set.
>+
>+ Givend the number of bug watches for an `remote_system`, this sets
>+ a suggested batch size on it. If `remote_system` already has a
>+ batch size set, this does not override it.
>+
>+ :param remote_system: An `ExternalBugTracker`.
>+ :param num_watches: The number of watches for `remote_system`.
>+ """
>+ if remote_system.batch_size is None:
>+ remote_system.batch_size = max(
>+ SUGGESTED_BATCH_SIZE_MIN, int(
>+ SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))

I think it would be easier to read with this wrapping.
        remote_system.batch_size = max(
            SUGGESTED_BATCH_SIZE_MIN,
            int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))

>+
> class BugWatchUpdater(object):
> """Takes responsibility for updating remote bug watches."""
>

review: Approve (code)

[Gavin Panella (allenap)]

Post-review diff, constructed from revisions 9177, 9178 and 9180.

[Gavin Panella (allenap)]

Conversation re. additional changes from IRC:

<EdwinGrubbs> allenap: I'm surprised that you added a line to tests/externalbugtracker.py to set the batch_size to BATCH_SIZE_UNLIMITED, but it didn't change the results of any of the tests.
<allenap> EdwinGrubbs: It kind of does the same as lines 153 to 157 used to do.
<allenap> EdwinGrubbs: ... because most calls into updateBugWatches() and _getRemoteIdsToCheck() didn't specify batch_size.
<EdwinGrubbs> allenap: ok, that makes sense. Looks good.

=== modified file 'lib/lp/bugs/doc/checkwatches.txt'
--- lib/lp/bugs/doc/checkwatches.txt	2010-04-06 21:36:16 +0000
+++ lib/lp/bugs/doc/checkwatches.txt	2010-04-08 15:38:28 +0000
@@ -45,7 +45,7 @@
 
     >>> print err
     INFO    Creating lockfile: /var/lock/launchpad-checkwatches.lock
-    DEBUG   Using a global batch size of None
+    DEBUG   No global batch size specified.
     DEBUG   Skipping updating Ubuntu Bugzilla watches.
     DEBUG   No watches to update on http://bugs.debian.org
     DEBUG   No watches to update on mailto:[e-mail]
@@ -274,7 +274,7 @@
     ...     updater.log = original_log
     ...     externalbugtracker.Roundup.batch_size = batch_size
     ...     urllib2.urlopen = urlopen
-    DEBUG Using a global batch size of None
+    DEBUG No global batch size specified.
     INFO Updating 5 watches for 5 bugs on http://bugs.example.com
     ERROR Connection timed out when updating ... (OOPS-...)
 
@@ -412,7 +412,8 @@
     >>> from lp.bugs.interfaces.externalbugtracker import (
     ...     ISupportsCommentImport, ISupportsCommentPushing,
     ...     ISupportsBackLinking)
-    >>> from lp.bugs.externalbugtracker.base import ExternalBugTracker
+    >>> from lp.bugs.externalbugtracker.base import (
+    ...     BATCH_SIZE_UNLIMITED, ExternalBugTracker)
 
     >>> nowish = datetime.now(utc)
     >>> class UselessExternalBugTracker(ExternalBugTracker):
@@ -421,6 +422,8 @@
     ...         ISupportsBackLinking, ISupportsCommentImport,
     ...         ISupportsCommentPushing)
     ...
+    ...     batch_size = BATCH_SIZE_UNLIMITED
+    ...
     ...     def initializeRemoteBugDB(self, bug_ids):
     ...         # This just exists to stop errors from being raised.
     ...         pass

=== modified file 'lib/lp/bugs/doc/externalbugtracker-debbugs.txt'
--- lib/lp/bugs/doc/externalbugtracker-debbugs.txt	2010-03-26 13:48:53 +0000
+++ lib/lp/bugs/doc/externalbugtracker-debbugs.txt	2010-04-08 15:38:28 +0000
@@ -14,7 +14,7 @@
 You can specify the db_location explicitly:
 
     >>> from lp.bugs.externalbugtracker import (
-    ...     DebBugs)
+    ...     BATCH_SIZE_UNLIMITED, DebBugs)
     >>> from canonical.testing import LaunchpadZopelessLayer
     >>> txn = LaunchpadZopelessLayer.txn
     >>> external_debbugs = DebBugs(
@@ -42,8 +42,8 @@
 worry about batching bug watch updates for performance reasons, so
 DebBugs instances don't have a batch_size limit.
 
-    >>> print external_debbugs.batch_size
-    None
+    >>> external_debbugs.batch_size == BATCH_SIZE_UNLIMITED
+    True
 
 
 == Retrieving bug status from the debbugs database ==

=== modified file 'lib/lp/bugs/doc/externalbugtracker.txt'
--- lib/lp/bugs/doc/externalbugtracker.txt	2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/doc/externalbugtracker.txt	2010-04-08 15:38:28 +0000
@@ -1042,13 +1042,16 @@
 
 _getRemoteIdsToCheck() will interpret a batch_size parameter of 0 as an
 instruction to ignore the batch size limitation altogether and just return all
-the IDs that need checking.
+the IDs that need checking. The constant BATCH_SIZE_UNLIMITED should
+be used in place of using 0 verbatim.
+
+    >>> from lp.bugs.externalbugtracker import BATCH_SIZE_UNLIMITED
 
     >>> ids = bug_watch_updater._getRemoteIdsToCheck(
     ...     external_bugtracker,
     ...     unchecked_watches + watches_with_comments + old_watches,
     ...     external_bugtracker.getCurrentDBTime(), utc_now,
-    ...     batch_size=0)
+    ...     batch_size=BATCH_SIZE_UNLIMITED)
     >>> print sorted(ids['remote_ids_to_check'])
     [u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9']
 

=== modified file 'lib/lp/bugs/externalbugtracker/__init__.py'
--- lib/lp/bugs/externalbugtracker/__init__.py	2009-06-25 00:40:31 +0000
+++ lib/lp/bugs/externalbugtracker/__init__.py	2010-04-08 15:38:28 +0000
@@ -7,7 +7,7 @@
 
 __metaclass__ = type
 __all__ = [
-    'get_external_bugtracker',
+    'BATCH_SIZE_UNLIMITED',
     'BugNotFound',
     'BugTrackerConnectError',
     'BugWatchUpdateError',
@@ -30,6 +30,7 @@
     'UnparseableBugData',
     'UnparseableBugTrackerVersion',
     'UnsupportedBugTrackerVersion',
+    'get_external_bugtracker',
     ]
 
 from lp.bugs.externalbugtracker.base import *
@@ -41,6 +42,8 @@
 from lp.bugs.externalbugtracker.rt import *
 from lp.bugs.externalbugtracker.trac import *
 from lp.bugs.interfaces.bugtracker import BugTrackerType
+
+
 BUG_TRACKER_CLASSES = {
     BugTrackerType.BUGZILLA: Bugzilla,
     BugTrackerType.DEBBUGS: DebBugs,

=== modified file 'lib/lp/bugs/externalbugtracker/base.py'
--- lib/lp/bugs/externalbugtracker/base.py	2010-03-26 15:12:59 +0000
+++ lib/lp/bugs/externalbugtracker/base.py	2010-04-08 15:38:28 +0000
@@ -5,6 +5,7 @@
 
 __metaclass__ = type
 __all__ = [
+    'BATCH_SIZE_UNLIMITED',
     'BugNotFound',
     'BugTrackerAuthenticationError',
     'BugTrackerConnectError',
@@ -39,6 +40,9 @@
 # The user agent we send in our requests
 LP_USER_AGENT = "Launchpad Bugscraper/0.2 (https://bugs.launchpad.net/)"
 
+# To signify that all bug watches should be checked in a single run.
+BATCH_SIZE_UNLIMITED = 0
+
 
 #
 # Errors.
@@ -133,7 +137,7 @@
 
     implements(IExternalBugTracker)
 
-    batch_size = 100
+    batch_size = None
     batch_query_threshold = config.checkwatches.batch_query_threshold
     comment_template = 'default_remotecomment_template.txt'
 

=== modified file 'lib/lp/bugs/externalbugtracker/debbugs.py'
--- lib/lp/bugs/externalbugtracker/debbugs.py	2010-03-31 20:09:55 +0000
+++ lib/lp/bugs/externalbugtracker/debbugs.py	2010-04-08 15:38:28 +0000
@@ -22,18 +22,19 @@
 
 from canonical.config import config
 from canonical.database.sqlbase import commit
+from canonical.launchpad.interfaces.message import IMessageSet
+from canonical.launchpad.mail import simple_sendmail
+from canonical.launchpad.webapp import urlsplit
+
 from lp.bugs.externalbugtracker import (
-    BugNotFound, BugTrackerConnectError, ExternalBugTracker,
-    InvalidBugId, UnknownRemoteStatusError)
-from canonical.launchpad.interfaces.message import IMessageSet
+    BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
+    ExternalBugTracker, InvalidBugId, UnknownRemoteStatusError)
+from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
 from lp.bugs.interfaces.bugtask import BugTaskImportance, BugTaskStatus
 from lp.bugs.interfaces.externalbugtracker import (
     ISupportsBugImport, ISupportsCommentImport, ISupportsCommentPushing,
     UNKNOWN_REMOTE_IMPORTANCE)
-from canonical.launchpad.mail import simple_sendmail
 from lp.bugs.scripts import debbugs
-from canonical.launchpad.webapp import urlsplit
-from lp.bugs.externalbugtracker.isolation import ensure_no_transaction
 
 
 debbugsstatusmap = {'open':      BugTaskStatus.NEW,
@@ -59,7 +60,7 @@
     # Because we keep a local copy of debbugs, we remove the batch_size
     # limit so that all debbugs watches that need checking will be
     # checked each time checkwatches runs.
-    batch_size = None
+    batch_size = BATCH_SIZE_UNLIMITED
 
     def __init__(self, baseurl, db_location=None):
         super(DebBugs, self).__init__(baseurl)

=== added directory 'lib/lp/bugs/scripts/checkwatches/tests'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/__init__.py'
=== added file 'lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py'
--- lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py	1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_scheduler.py	2010-04-08 15:38:28 +0000
@@ -0,0 +1,110 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""XXX: Module docstring goes here."""
+
+__metaclass__ = type
+
+import transaction
+import unittest
+
+from datetime import datetime, timedelta
+from pytz import utc
+
+from zope.component import getUtility
+
+from canonical.launchpad.scripts.logger import QuietFakeLogger
+from canonical.testing import DatabaseFunctionalLayer
+
+from lp.bugs.interfaces.bugwatch import (
+    BugWatchActivityStatus, IBugWatchSet)
+from lp.bugs.scripts.checkwatches.scheduler import BugWatchScheduler
+
+from lp.testing import TestCaseWithFactory
+
+
+class TestBugWatchScheduler(TestCaseWithFactory):
+    """Tests for the BugWatchScheduler, which runs as part of garbo."""
+
+    layer = DatabaseFunctionalLayer
+
+    def setUp(self):
+        super(TestBugWatchScheduler, self).setUp('[e-mail]')
+        # We'll make sure that all the other bug watches look like
+        # they've been scheduled so that only our watch gets scheduled.
+        for watch in getUtility(IBugWatchSet).search():
+            watch.next_check = datetime.now(utc)
+        self.bug_watch = self.factory.makeBugWatch()
+        self.scheduler = BugWatchScheduler(QuietFakeLogger())
+        transaction.commit()
+
+    def test_scheduler_schedules_unchecked_watches(self):
+        # The BugWatchScheduler will schedule a BugWatch that has never
+        # been checked to be checked immediately.
+        self.bug_watch.next_check = None
+        self.scheduler(1)
+
+        self.assertNotEqual(None, self.bug_watch.next_check)
+        self.assertTrue(
+            self.bug_watch.next_check <= datetime.now(utc))
+
+    def test_scheduler_schedules_working_watches(self):
+        # If a watch has been checked and has never failed its next
+        # check will be scheduled for 24 hours after its last check.
+        now = datetime.now(utc)
+        # Add some succesful activity to ensure that successful activity
+        # is handled correctly.
+        self.bug_watch.addActivity()
+        self.bug_watch.lastchecked = now
+        self.bug_watch.next_check = None
+        transaction.commit()
+        self.scheduler(1)
+
+        self.assertEqual(
+            now + timedelta(hours=24), self.bug_watch.next_check)
+
+    def test_scheduler_schedules_failing_watches(self):
+        # If a watch has failed once, it will be scheduled more than 24
+        # hours after its last check.
+        now = datetime.now(utc)
+        self.bug_watch.lastchecked = now
+
+        # The delay depends on the number of failures that the watch has
+        # had.
+        for failure_count in range(1, 6):
+            self.bug_watch.next_check = None
+            self.bug_watch.addActivity(
+                result=BugWatchActivityStatus.BUG_NOT_FOUND)
+            transaction.commit()
+            self.scheduler(1)
+
+            coefficient = self.scheduler.delay_coefficient * failure_count
+            self.assertEqual(
+                now + timedelta(days=1 + coefficient),
+                self.bug_watch.next_check)
+
+        # The scheduler only looks at the last 5 activity items, so even
+        # if there have been more failures the maximum delay will be 7
+        # days.
+        for count in range(10):
+            self.bug_watch.addActivity(
+                result=BugWatchActivityStatus.BUG_NOT_FOUND)
+        self.bug_watch.next_check = None
+        transaction.commit()
+        self.scheduler(1)
+        self.assertEqual(
+            now + timedelta(days=7), self.bug_watch.next_check)
+
+    def test_scheduler_doesnt_schedule_scheduled_watches(self):
+        # The scheduler will ignore watches whose next_check has been
+        # set.
+        next_check_date = datetime.now(utc) + timedelta(days=1)
+        self.bug_watch.next_check = next_check_date
+        transaction.commit()
+        self.scheduler(1)
+
+        self.assertEqual(next_check_date, self.bug_watch.next_check)
+
+
+def test_suite():
+    return unittest.TestLoader().loadTestsFromName(__name__)

=== renamed file 'lib/lp/bugs/scripts/tests/test_checkwatches.py' => 'lib/lp/bugs/scripts/checkwatches/tests/test_updater.py'
--- lib/lp/bugs/scripts/tests/test_checkwatches.py	2010-03-25 21:59:48 +0000
+++ lib/lp/bugs/scripts/checkwatches/tests/test_updater.py	2010-04-08 15:38:28 +0000
@@ -167,6 +167,21 @@
             # stop SQL statemnet logging.
             clear_request_started()
 
+    def test_suggest_batch_size(self):
+        class RemoteSystem: pass
+        remote_system = RemoteSystem()
+        # When the batch_size is None, suggest_batch_size() will set
+        # it accordingly.
+        remote_system.batch_size = None
+        checkwatches.updater.suggest_batch_size(remote_system, 1)
+        self.failUnlessEqual(100, remote_system.batch_size)
+        remote_system.batch_size = None
+        checkwatches.updater.suggest_batch_size(remote_system, 12350)
+        self.failUnlessEqual(247, remote_system.batch_size)
+        # If the batch_size is already set, it will not be changed.
+        checkwatches.updater.suggest_batch_size(remote_system, 99999)
+        self.failUnlessEqual(247, remote_system.batch_size)
+
 
 class TestUpdateBugsWithLinkedQuestions(unittest.TestCase):
     """Tests for updating bugs with linked questions."""

=== modified file 'lib/lp/bugs/scripts/checkwatches/updater.py'
--- lib/lp/bugs/scripts/checkwatches/updater.py	2010-03-26 21:03:30 +0000
+++ lib/lp/bugs/scripts/checkwatches/updater.py	2010-04-08 15:38:28 +0000
@@ -61,16 +61,15 @@
 
 from lp.bugs import externalbugtracker
 from lp.bugs.externalbugtracker import (
-    BugNotFound, BugTrackerConnectError, BugWatchUpdateError,
-    BugWatchUpdateWarning, InvalidBugId, PrivateRemoteBug,
-    UnknownBugTrackerTypeError, UnknownRemoteStatusError, UnparseableBugData,
-    UnparseableBugTrackerVersion, UnsupportedBugTrackerVersion)
-from lp.bugs.externalbugtracker.bugzilla import (
-    BugzillaAPI)
+    BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError,
+    BugWatchUpdateError, BugWatchUpdateWarning, InvalidBugId,
+    PrivateRemoteBug, UnknownBugTrackerTypeError, UnknownRemoteStatusError,
+    UnparseableBugData, UnparseableBugTrackerVersion,
+    UnsupportedBugTrackerVersion)
+from lp.bugs.externalbugtracker.bugzilla import BugzillaAPI
 from lp.bugs.externalbugtracker.isolation import check_no_transaction
 from lp.bugs.interfaces.bug import IBugSet
-from lp.bugs.interfaces.externalbugtracker import (
-    ISupportsBackLinking)
+from lp.bugs.interfaces.externalbugtracker import ISupportsBackLinking
 from lp.services.limitedlist import LimitedList
 from lp.services.scripts.base import LaunchpadCronScript
 
@@ -78,6 +77,11 @@
 SYNCABLE_GNOME_PRODUCTS = []
 MAX_SQL_STATEMENTS_LOGGED = 10000
 
+# The minimum batch size to suggest to an IExternalBugTracker.
+SUGGESTED_BATCH_SIZE_MIN = 100
+# The proportion of all watches to suggest as a batch size.
+SUGGESTED_BATCH_SIZE_PROPORTION = 0.02
+
 
 class TooMuchTimeSkew(BugWatchUpdateError):
     """Time difference between ourselves and the remote server is too much."""
@@ -242,6 +246,22 @@
             yield item
 
 
+def suggest_batch_size(remote_system, num_watches):
+    """Suggest a value for batch_size if it's not set.
+
+    Given the number of bug watches for a `remote_system`, this sets a
+    suggested batch size on it. If `remote_system` already has a batch
+    size set, this does not override it.
+
+    :param remote_system: An `ExternalBugTracker`.
+    :param num_watches: The number of watches for `remote_system`.
+    """
+    if remote_system.batch_size is None:
+        remote_system.batch_size = max(
+            SUGGESTED_BATCH_SIZE_MIN,
+            int(SUGGESTED_BATCH_SIZE_PROPORTION * num_watches))
+
+
 class BugWatchUpdater(object):
     """Takes responsibility for updating remote bug watches."""
 
@@ -381,7 +401,12 @@
         `BaseScheduler`. If no scheduler is given, `SerialScheduler`
         will be used, which simply runs the jobs in order.
         """
-        self.log.debug("Using a global batch size of %s" % batch_size)
+        if batch_size is None:
+            self.log.debug("No global batch size specified.")
+        elif batch_size == BATCH_SIZE_UNLIMITED:
+            self.log.debug("Using an unlimited global batch size.")
+        else:
+            self.log.debug("Using a global batch size of %s" % batch_size)
 
         # Default to using the very simple SerialScheduler.
         if scheduler is None:
@@ -497,6 +522,7 @@
     def _getExternalBugTrackersAndWatches(self, bug_tracker, bug_watches):
         """Return an `ExternalBugTracker` instance for `bug_tracker`."""
         with self.transaction:
+            num_watches = bug_tracker.watches.count()
             remotesystem = (
                 externalbugtracker.get_external_bugtracker(bug_tracker))
             # We special-case the Gnome Bugzilla.
@@ -506,6 +532,9 @@
         # Probe the remote system for additional capabilities.
         remotesystem_to_use = remotesystem.getExternalBugTrackerToUse()
 
+        # Try to hint at how many bug watches to check each time.
+        suggest_batch_size(remotesystem_to_use, num_watches)
+
         if (is_gnome_bugzilla and
             isinstance(remotesystem_to_use, BugzillaAPI) and
             len(self._syncable_gnome_products) > 0):
@@ -656,11 +685,6 @@
             # by the ExternalBugTracker.
             batch_size = remotesystem.batch_size
 
-        if batch_size == 0:
-            # A batch_size of 0 means that there's no batch size limit
-            # for this bug tracker.
-            batch_size = None
-
         with self.transaction:
             old_bug_watches = set(
                 bug_watch for bug_watch in bug_watches
@@ -693,7 +717,7 @@
         # are actually some bugs that we're interested in so as to
         # avoid unnecessary network traffic.
         if server_time is not None and len(remote_old_ids) > 0:
-            if batch_size is None:
+            if batch_size == BATCH_SIZE_UNLIMITED:
                 remote_old_ids_to_check = (
                     remotesystem.getModifiedRemoteBugs(
                         remote_old_ids, oldest_lastchecked))
@@ -721,7 +745,7 @@
         remote_ids_to_check = chain(
             remote_ids_with_comments, remote_new_ids, remote_old_ids_to_check)
 
-        if batch_size is not None:
+        if batch_size != BATCH_SIZE_UNLIMITED:
             # Some remote bug IDs may appear in more than one list so
             # we must filter the list before slicing.
             remote_ids_to_check = islice(

=== modified file 'lib/lp/bugs/tests/externalbugtracker.py'
--- lib/lp/bugs/tests/externalbugtracker.py	2009-12-23 12:14:59 +0000
+++ lib/lp/bugs/tests/externalbugtracker.py	2010-04-08 15:38:28 +0000
@@ -24,8 +24,8 @@
 from canonical.config import config
 from canonical.database.sqlbase import commit, ZopelessTransactionManager
 from lp.bugs.externalbugtracker import (
-    BugNotFound, BugTrackerConnectError, Bugzilla, DebBugs,
-    ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,
+    BATCH_SIZE_UNLIMITED, BugNotFound, BugTrackerConnectError, Bugzilla,
+    DebBugs, ExternalBugTracker, Mantis, RequestTracker, Roundup, SourceForge,
     Trac)
 from lp.bugs.externalbugtracker.trac import (
     FAULT_TICKET_NOT_FOUND, LP_PLUGIN_BUG_IDS_ONLY, LP_PLUGIN_FULL,
@@ -152,6 +152,8 @@
     implementation, though it doesn't actually do anything.
     """
 
+    batch_size = BATCH_SIZE_UNLIMITED
+
     def __init__(self, baseurl='http://example.com/'):
         super(TestExternalBugTracker, self).__init__(baseurl)
 

=== modified file 'lib/lp/bugs/tests/test_bugwatch.py'
--- lib/lp/bugs/tests/test_bugwatch.py	2010-04-07 12:46:15 +0000
+++ lib/lp/bugs/tests/test_bugwatch.py	2010-04-08 15:38:28 +0000
@@ -8,9 +8,6 @@
 import transaction
 import unittest
 
-from datetime import datetime, timedelta
-from pytz import utc
-
 from urlparse import urlunsplit
 
 from zope.component import getUtility
@@ -24,10 +21,8 @@
 
 from lp.bugs.interfaces.bugtracker import BugTrackerType, IBugTrackerSet
 from lp.bugs.interfaces.bugwatch import (
-    BugWatchActivityStatus, IBugWatchSet, NoBugTrackerFound,
-    UnrecognizedBugTrackerURL)
-from lp.bugs.scripts.checkwatches.scheduler import (
-    BugWatchScheduler, MAX_SAMPLE_SIZE)
+    IBugWatchSet, NoBugTrackerFound, UnrecognizedBugTrackerURL)
+from lp.bugs.scripts.checkwatches.scheduler import MAX_SAMPLE_SIZE
 from lp.registry.interfaces.person import IPersonSet
 
 from lp.testing import TestCaseWithFactory
@@ -466,88 +461,5 @@
             self.failUnless("Activity %s" % i in messages)
 
 
-class TestBugWatchScheduler(TestCaseWithFactory):
-    """Tests for the BugWatchScheduler, which runs as part of garbo."""
-
-    layer = DatabaseFunctionalLayer
-
-    def setUp(self):
-        super(TestBugWatchScheduler, self).setUp('[e-mail]')
-        # We'll make sure that all the other bug watches look like
-        # they've been scheduled so that only our watch gets scheduled.
-        for watch in getUtility(IBugWatchSet).search():
-            watch.next_check = datetime.now(utc)
-        self.bug_watch = self.factory.makeBugWatch()
-        self.scheduler = BugWatchScheduler(QuietFakeLogger())
-        transaction.commit()
-
-    def test_scheduler_schedules_unchecked_watches(self):
-        # The BugWatchScheduler will schedule a BugWatch that has never
-        # been checked to be checked immediately.
-        self.bug_watch.next_check = None
-        self.scheduler(1)
-
-        self.assertNotEqual(None, self.bug_watch.next_check)
-        self.assertTrue(
-            self.bug_watch.next_check <= datetime.now(utc))
-
-    def test_scheduler_schedules_working_watches(self):
-        # If a watch has been checked and has never failed its next
-        # check will be scheduled for 24 hours after its last check.
-        now = datetime.now(utc)
-        # Add some succesful activity to ensure that successful activity
-        # is handled correctly.
-        self.bug_watch.addActivity()
-        self.bug_watch.lastchecked = now
-        self.bug_watch.next_check = None
-        transaction.commit()
-        self.scheduler(1)
-
-        self.assertEqual(
-            now + timedelta(hours=24), self.bug_watch.next_check)
-
-    def test_scheduler_schedules_failing_watches(self):
-        # If a watch has failed once, it will be scheduled more than 24
-        # hours after its last check.
-        now = datetime.now(utc)
-        self.bug_watch.lastchecked = now
-
-        # The delay depends on the number of failures that the watch has
-        # had.
-        for failure_count in range(1, 6):
-            self.bug_watch.next_check = None
-            self.bug_watch.addActivity(
-                result=BugWatchActivityStatus.BUG_NOT_FOUND)
-            transaction.commit()
-            self.scheduler(1)
-
-            coefficient = self.scheduler.delay_coefficient * failure_count
-            self.assertEqual(
-                now + timedelta(days=1 + coefficient),
-                self.bug_watch.next_check)
-
-        # The scheduler only looks at the last 5 activity items, so even
-        # if there have been more failures the maximum delay will be 7
-        # days.
-        for count in range(10):
-            self.bug_watch.addActivity(
-                result=BugWatchActivityStatus.BUG_NOT_FOUND)
-        self.bug_watch.next_check = None
-        transaction.commit()
-        self.scheduler(1)
-        self.assertEqual(
-            now + timedelta(days=7), self.bug_watch.next_check)
-
-    def test_scheduler_doesnt_schedule_scheduled_watches(self):
-        # The scheduler will ignore watches whose next_check has been
-        # set.
-        next_check_date = datetime.now(utc) + timedelta(days=1)
-        self.bug_watch.next_check = next_check_date
-        transaction.commit()
-        self.scheduler(1)
-
-        self.assertEqual(next_check_date, self.bug_watch.next_check)
-
-
 def test_suite():
     return unittest.TestLoader().loadTestsFromName(__name__)