Merge lp:~mbp/bzr/597791-http-tests into lp:bzr

[Martin Pool]

cleanup test_http to use scenarios; add load_tests_from_scenarios

TestCase classes can now just have a `variations` attribute to describe how they should be multiplied. This lets us entirely delete the load_tests function complained of in bug 597791 and to my mind it makes the code _much_ nicer by eliminating spooky action at a distance. Now each class says what kind of variation it wants, without needing to know all the specific values, which may be dynamically determined.

I tested this doesn't change the parameterization by running 'selftest --list' on this and trunk, and ignoring ordering they're the same.

It could be nice to move or copy this into testscenarios.

I haven't yet rolled this out to other modules but I expect in many cases we could delete their custom load_tests methods in favour of just using load_tests_from_their_variations. (The name is a bit explicit because the behaviour's a bit magical.) I think this may also help with cases where we've used subclassing to parameterize because the official way was a bit hard.

[Martin Pool (mbp)]

After some discussion with Robert and bug 658044, I think I can make this converge a bit more with what's already done in testscenarios:

 * name the attribute 'scenarios'
 * make it an iterable of scenarios directly, rather than a list of TestVariations to multiply
 * generate the cross product by calling a function rather than implicitly; to work well with scenario generators that come from registries this function should be quite lazy
 * change the variations to generators or functions rather than classes

Also stylistically
 * clearer tests
 * remove extra VWS
 * module docstring

[lp:~mbp/bzr/597791-http-tests]

5476. By Martin Pool on 2010-10-11

Unify varations with scenario protocol

5477. By Martin Pool on 2010-10-11

Turn variations into scenario lists

5478. By Martin Pool on 2010-10-11

pyflakes cleanups

5479. By Martin Pool on 2010-10-11

Updated test scenario docs

5480. By Martin Pool on 2010-10-11

merge news

[Vincent Ladeuil (vila)]

Lovely result :)

I had a look at that strange assertionError and... it looks like an uncaught regression *long* ago.
Or even worse, this looks like a test that has *never* failed :-(
I'll look at it again tomorrow.

One question: why is it required to do:

  load_tests = load_tests_from_scenarios

Can't that be triggered automatically if a 'scenarios' attribute is defined on the test class ?

It's true that load_tests is generally required when you want to parametrize but... is it still possible to define a load_tests function and call load_tests_from_scenarios for some classes only ?

I can't put my finger on it as I write this but I somehow feel there is a hole somewhere... imbw, you may just have fill the hole instead...

review: Approve

[Martin Pool (mbp)]

I added (obviously) the raise AssertionError() just to make it really obvious that the thing does indeed fail. My guess is that an error here is sent back to the client as an http 500 error, and it accepts that as a reasonable response.

Is load_tests needed? I think Robert added that as a semi-standard extension point where the test loader calls back in to our code. If we want the test suite to work with other runners, we can't hardcode a check for scenarios. Perhaps we can simplify it more. otoh explicit is better than implicit, and now it's only one line (plus an import...)

[Robert Collins (lifeless)]

load_tests is in python 2.7 (with a slightly different signature); it would be nice to adapt to that.

Doing load_tests = load_tests_from_scenarios (perhaps it would be better to say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and not terrible.

Doing a global is fine to - you could simply use generate_scenarios in bzrlib.test_suite()

381 + # XXX: this is strange; the 'random' name below seems undefined and
382 + # yet the tests pass -- mbp 2010-10-11
383 + raise AssertionError()

Looks like you can't land it like this?

This looks lovely. I would love to see a patch with the missing bits put into testscenarios.

review: Approve

[Martin Pool (mbp)]

> load_tests is in python 2.7 (with a slightly different signature); it would be
> nice to adapt to that.

Right, bug 607412.

> Doing load_tests = load_tests_from_scenarios (perhaps it would be better to
> say 'load_tests_apply_scenarios', but - *blue*) is pretty pithy, explicit and
> not terrible.
>
> Doing a global is fine to - you could simply use generate_scenarios in
> bzrlib.test_suite()
>
>
> 381 + # XXX: this is strange; the 'random' name below seems
> undefined and
> 382 + # yet the tests pass -- mbp 2010-10-11
> 383 + raise AssertionError()
>
> Looks like you can't land it like this?

I can; the bug (bug 658773) was latent before I came here, and I just noticed it by running pyflakes.

>
> This looks lovely. I would love to see a patch with the missing bits put into
> testscenarios.

Right, I'll do that under bug 658044.

[lp:~mbp/bzr/597791-http-tests]

5481. By Martin Pool on 2010-10-11

Mention bug 658773 in comment

5482. By Martin Pool on 2010-10-11

doc

5483. By Martin Pool on 2010-10-11

Rename to load_tests_apply_scenarios

[Martin Pool (mbp)]

sent to pqm by email

=== modified file 'NEWS'
--- NEWS	2010-10-11 15:18:38 +0000
+++ NEWS	2010-10-11 22:43:41 +0000
@@ -158,6 +158,12 @@
 Testing
 *******
 
+* Add a new simpler way to generate multiple test variations, by setting
+  the `scenarios` attribute of a test class to a list of scenarios
+  descriptions, then using `load_tests_apply_scenarios`.  (See the testing
+  guide and `bzrlib.tests.scenarios`.) Simplify `test_http` using this.
+  (Martin Pool, #597791)
+
 * Add ``tests/ssl_certs/ca.crt`` to the required test files list. Test
   involving the pycurl https test server fail otherwise when running
   selftest from an installed version. (Vincent Ladeuil, #651706)

=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py	2010-10-08 10:16:12 +0000
+++ bzrlib/tests/__init__.py	2010-10-11 22:43:41 +0000
@@ -3809,6 +3809,7 @@
         'bzrlib.tests.test_rio',
         'bzrlib.tests.test_rules',
         'bzrlib.tests.test_sampler',
+        'bzrlib.tests.test_scenarios',
         'bzrlib.tests.test_script',
         'bzrlib.tests.test_selftest',
         'bzrlib.tests.test_serializer',
@@ -3991,7 +3992,19 @@
     return suite
 
 
-def multiply_scenarios(scenarios_left, scenarios_right):
+def multiply_scenarios(*scenarios):
+    """Multiply two or more iterables of scenarios.
+
+    It is safe to pass scenario generators or iterators.
+
+    :returns: A list of compound scenarios: the cross-product of all 
+        scenarios, with the names concatenated and the parameters
+        merged together.
+    """
+    return reduce(_multiply_two_scenarios, map(list, scenarios))
+
+
+def _multiply_two_scenarios(scenarios_left, scenarios_right):
     """Multiply two sets of scenarios.
 
     :returns: the cartesian product of the two sets of scenarios, that is

=== added file 'bzrlib/tests/scenarios.py'
--- bzrlib/tests/scenarios.py	1970-01-01 00:00:00 +0000
+++ bzrlib/tests/scenarios.py	2010-10-11 22:43:41 +0000
@@ -0,0 +1,61 @@
+# Copyright (C) 2005-2010 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+
+"""Generate multiple variations in different scenarios.
+
+For a class whose tests should be repeated in varying scenarios, set a
+`scenarios` member to a list of scenarios where it should be repeated.
+
+This is similar to the interface provided by
+<http://launchpad.net/testscenarios/>.
+"""
+
+
+from bzrlib.tests import (
+    iter_suite_tests,
+    multiply_scenarios,
+    multiply_tests,
+    )
+
+
+def load_tests_apply_scenarios(standard_tests, module, loader):
+    """Multiply tests depending on their 'scenarios' attribute.
+
+    This can be assigned to 'load_tests' in any test module to make this
+    automatically work across tests in the module.
+    """
+    result = loader.suiteClass()
+    multiply_tests_by_their_scenarios(standard_tests, result)
+    return result
+
+
+def multiply_tests_by_their_scenarios(some_tests, into_suite):
+    """Multiply the tests in the given suite by their declared scenarios.
+
+    Each test must have a 'scenarios' attribute which is a list of 
+    (name, params) pairs.
+
+    :param some_tests: TestSuite or Test.
+    :param into_suite: A TestSuite into which the resulting tests will be
+        inserted.
+    """
+    for test in iter_suite_tests(some_tests):
+        scenarios = getattr(test, 'scenarios', None)
+        if scenarios is None:
+            into_suite.addTest(test)
+        else:
+            multiply_tests(test, test.scenarios, into_suite)

=== modified file 'bzrlib/tests/test_http.py'
--- bzrlib/tests/test_http.py	2010-08-24 13:03:18 +0000
+++ bzrlib/tests/test_http.py	2010-10-11 22:43:41 +0000
@@ -23,10 +23,7 @@
 # TODO: Should be renamed to bzrlib.transport.http.tests?
 # TODO: What about renaming to bzrlib.tests.transport.http ?
 
-from cStringIO import StringIO
 import httplib
-import os
-import select
 import SimpleHTTPServer
 import socket
 import sys
@@ -42,7 +39,6 @@
     tests,
     transport,
     ui,
-    urlutils,
     )
 from bzrlib.tests import (
     features,
@@ -50,6 +46,10 @@
     http_utils,
     test_server,
     )
+from bzrlib.tests.scenarios import (
+    load_tests_apply_scenarios,
+    multiply_scenarios,
+    )
 from bzrlib.transport import (
     http,
     remote,
@@ -64,17 +64,11 @@
     from bzrlib.transport.http._pycurl import PyCurlTransport
 
 
-def load_tests(standard_tests, module, loader):
-    """Multiply tests for http clients and protocol versions."""
-    result = loader.suiteClass()
-
-    # one for each transport implementation
-    t_tests, remaining_tests = tests.split_suite_by_condition(
-        standard_tests, tests.condition_isinstance((
-                TestHttpTransportRegistration,
-                TestHttpTransportUrls,
-                Test_redirected_to,
-                )))
+load_tests = load_tests_apply_scenarios
+
+
+def vary_by_http_client_implementation():
+    """Test the two libraries we can use, pycurl and urllib."""
     transport_scenarios = [
         ('urllib', dict(_transport=_urllib.HttpTransport_urllib,
                         _server=http_server.HttpServer_urllib,
@@ -85,85 +79,48 @@
             ('pycurl', dict(_transport=PyCurlTransport,
                             _server=http_server.HttpServer_PyCurl,
                             _url_protocol='http+pycurl',)))
-    tests.multiply_tests(t_tests, transport_scenarios, result)
-
-    protocol_scenarios = [
-            ('HTTP/1.0',  dict(_protocol_version='HTTP/1.0')),
-            ('HTTP/1.1',  dict(_protocol_version='HTTP/1.1')),
-            ]
-
-    # some tests are parametrized by the protocol version only
-    p_tests, remaining_tests = tests.split_suite_by_condition(
-        remaining_tests, tests.condition_isinstance((
-                TestAuthOnRedirected,
-                )))
-    tests.multiply_tests(p_tests, protocol_scenarios, result)
-
-    # each implementation tested with each HTTP version
-    tp_tests, remaining_tests = tests.split_suite_by_condition(
-        remaining_tests, tests.condition_isinstance((
-                SmartHTTPTunnellingTest,
-                TestDoCatchRedirections,
-                TestHTTPConnections,
-                TestHTTPRedirections,
-                TestHTTPSilentRedirections,
-                TestLimitedRangeRequestServer,
-                TestPost,
-                TestProxyHttpServer,
-                TestRanges,
-                TestSpecificRequestHandler,
-                )))
-    tp_scenarios = tests.multiply_scenarios(transport_scenarios,
-                                            protocol_scenarios)
-    tests.multiply_tests(tp_tests, tp_scenarios, result)
-
-    # proxy auth: each auth scheme on all http versions on all implementations.
-    tppa_tests, remaining_tests = tests.split_suite_by_condition(
-        remaining_tests, tests.condition_isinstance((
-                TestProxyAuth,
-                )))
-    proxy_auth_scheme_scenarios = [
+    return transport_scenarios
+
+
+def vary_by_http_protocol_version():
+    """Test on http/1.0 and 1.1"""
+    return [
+        ('HTTP/1.0',  dict(_protocol_version='HTTP/1.0')),
+        ('HTTP/1.1',  dict(_protocol_version='HTTP/1.1')),
+        ]
+
+
+def vary_by_http_proxy_auth_scheme():
+    return [
         ('basic', dict(_auth_server=http_utils.ProxyBasicAuthServer)),
         ('digest', dict(_auth_server=http_utils.ProxyDigestAuthServer)),
         ('basicdigest',
-         dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),
+            dict(_auth_server=http_utils.ProxyBasicAndDigestAuthServer)),
         ]
-    tppa_scenarios = tests.multiply_scenarios(tp_scenarios,
-                                              proxy_auth_scheme_scenarios)
-    tests.multiply_tests(tppa_tests, tppa_scenarios, result)
-
-    # auth: each auth scheme on all http versions on all implementations.
-    tpa_tests, remaining_tests = tests.split_suite_by_condition(
-        remaining_tests, tests.condition_isinstance((
-                TestAuth,
-                )))
-    auth_scheme_scenarios = [
+
+
+def vary_by_http_auth_scheme():
+    return [
         ('basic', dict(_auth_server=http_utils.HTTPBasicAuthServer)),
         ('digest', dict(_auth_server=http_utils.HTTPDigestAuthServer)),
         ('basicdigest',
-         dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),
+            dict(_auth_server=http_utils.HTTPBasicAndDigestAuthServer)),
         ]
-    tpa_scenarios = tests.multiply_scenarios(tp_scenarios,
-                                             auth_scheme_scenarios)
-    tests.multiply_tests(tpa_tests, tpa_scenarios, result)
-
-    # activity: on all http[s] versions on all implementations
-    tpact_tests, remaining_tests = tests.split_suite_by_condition(
-        remaining_tests, tests.condition_isinstance((
-                TestActivity,
-                )))
+
+
+def vary_by_http_activity():
     activity_scenarios = [
         ('urllib,http', dict(_activity_server=ActivityHTTPServer,
-                             _transport=_urllib.HttpTransport_urllib,)),
+                            _transport=_urllib.HttpTransport_urllib,)),
         ]
     if tests.HTTPSServerFeature.available():
         activity_scenarios.append(
             ('urllib,https', dict(_activity_server=ActivityHTTPSServer,
-                                  _transport=_urllib.HttpTransport_urllib,)),)
+                                _transport=_urllib.HttpTransport_urllib,)),)
     if features.pycurl.available():
         activity_scenarios.append(
             ('pycurl,http', dict(_activity_server=ActivityHTTPServer,
-                                 _transport=PyCurlTransport,)),)
+                                _transport=PyCurlTransport,)),)
         if tests.HTTPSServerFeature.available():
             from bzrlib.tests import (
                 ssl_certs,
@@ -181,16 +138,8 @@
 
             activity_scenarios.append(
                 ('pycurl,https', dict(_activity_server=ActivityHTTPSServer,
-                                      _transport=HTTPS_pycurl_transport,)),)
-
-    tpact_scenarios = tests.multiply_scenarios(activity_scenarios,
-                                               protocol_scenarios)
-    tests.multiply_tests(tpact_tests, tpact_scenarios, result)
-
-    # No parametrization for the remaining tests
-    result.addTests(remaining_tests)
-
-    return result
+                                    _transport=HTTPS_pycurl_transport,)),)
+    return activity_scenarios
 
 
 class FakeManager(object):
@@ -401,6 +350,8 @@
 class TestHttpTransportUrls(tests.TestCase):
     """Test the http urls."""
 
+    scenarios = vary_by_http_client_implementation()
+
     def test_abs_url(self):
         """Construction of absolute http URLs"""
         t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
@@ -413,7 +364,7 @@
 
     def test_invalid_http_urls(self):
         """Trap invalid construction of urls"""
-        t = self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
+        self._transport('http://bazaar-vcs.org/bzr/bzr.dev/')
         self.assertRaises(errors.InvalidURL,
                           self._transport,
                           'http://http://bazaar-vcs.org/bzr/bzr.dev/')
@@ -475,6 +426,11 @@
 class TestHTTPConnections(http_utils.TestCaseWithWebserver):
     """Test the http connections."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         http_utils.TestCaseWithWebserver.setUp(self)
         self.build_tree(['foo/', 'foo/bar'], line_endings='binary',
@@ -525,6 +481,8 @@
 class TestHttpTransportRegistration(tests.TestCase):
     """Test registrations of various http implementations"""
 
+    scenarios = vary_by_http_client_implementation()
+
     def test_http_registered(self):
         t = transport.get_transport('%s://foo.com/' % self._url_protocol)
         self.assertIsInstance(t, transport.Transport)
@@ -533,6 +491,11 @@
 
 class TestPost(tests.TestCase):
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def test_post_body_is_received(self):
         server = RecordingServer(expect_body_tail='end-of-body',
                                  scheme=self._url_protocol)
@@ -585,6 +548,11 @@
     Daughter classes are expected to override _req_handler_class
     """
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     # Provide a useful default
     _req_handler_class = http_server.TestingHTTPRequestHandler
 
@@ -841,7 +809,7 @@
         t = self.get_readonly_transport()
         # force transport to issue multiple requests
         t._get_max_size = 2
-        l = list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))
+        list(t.readv('a', ((0, 1), (1, 1), (2, 4), (6, 4))))
         # The server should have issued 3 requests
         self.assertEqual(3, server.GET_request_nb)
         self.assertEqual('0123456789', t.get_bytes('a'))
@@ -924,6 +892,8 @@
     def get_multiple_ranges(self, file, file_size, ranges):
         self.send_response(206)
         self.send_header('Accept-Ranges', 'bytes')
+        # XXX: this is strange; the 'random' name below seems undefined and
+        # yet the tests pass -- mbp 2010-10-11 bug 658773
         boundary = "%d" % random.randint(0,0x7FFFFFFF)
         self.send_header("Content-Type",
                          "multipart/byteranges; boundary=%s" % boundary)
@@ -1055,6 +1025,11 @@
 class TestLimitedRangeRequestServer(http_utils.TestCaseWithWebserver):
     """Tests readv requests against a server erroring out on too much ranges."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     # Requests with more range specifiers will error out
     range_limit = 3
 
@@ -1134,6 +1109,11 @@
     to the file names).
     """
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     # FIXME: We don't have an https server available, so we don't
     # test https connections. --vila toolongago
 
@@ -1236,6 +1216,11 @@
 class TestRanges(http_utils.TestCaseWithWebserver):
     """Test the Range header in GET methods."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         http_utils.TestCaseWithWebserver.setUp(self)
         self.build_tree_contents([('a', '0123456789')],)
@@ -1281,6 +1266,11 @@
 class TestHTTPRedirections(http_utils.TestCaseWithRedirectedWebserver):
     """Test redirection between http servers."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         super(TestHTTPRedirections, self).setUp()
         self.build_tree_contents([('a', '0123456789'),
@@ -1349,6 +1339,11 @@
     -- vila 20070212
     """
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         if (features.pycurl.available()
             and self._transport == PyCurlTransport):
@@ -1399,6 +1394,11 @@
 class TestDoCatchRedirections(http_utils.TestCaseWithRedirectedWebserver):
     """Test transport.do_catching_redirections."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         super(TestDoCatchRedirections, self).setUp()
         self.build_tree_contents([('a', '0123456789'),],)
@@ -1446,6 +1446,12 @@
 class TestAuth(http_utils.TestCaseWithWebserver):
     """Test authentication scheme"""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(),
+        vary_by_http_protocol_version(),
+        vary_by_http_auth_scheme(),
+        )
+
     _auth_header = 'Authorization'
     _password_prompt_prefix = ''
     _username_prompt_prefix = ''
@@ -1655,6 +1661,12 @@
 class TestProxyAuth(TestAuth):
     """Test proxy authentication schemes."""
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(),
+        vary_by_http_protocol_version(),
+        vary_by_http_proxy_auth_scheme(),
+        )
+
     _auth_header = 'Proxy-authorization'
     _password_prompt_prefix = 'Proxy '
     _username_prompt_prefix = 'Proxy '
@@ -1716,6 +1728,11 @@
 
 class SmartHTTPTunnellingTest(tests.TestCaseWithTransport):
 
+    scenarios = multiply_scenarios(
+        vary_by_http_client_implementation(), 
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         super(SmartHTTPTunnellingTest, self).setUp()
         # We use the VFS layer as part of HTTP tunnelling tests.
@@ -1810,6 +1827,8 @@
 
 class Test_redirected_to(tests.TestCase):
 
+    scenarios = vary_by_http_client_implementation()
+
     def test_redirected_to_subdir(self):
         t = self._transport('http://www.example.com/foo')
         r = t._redirected_to('http://www.example.com/foo',
@@ -2061,6 +2080,11 @@
 
 class TestActivity(tests.TestCase, TestActivityMixin):
 
+    scenarios = multiply_scenarios(
+        vary_by_http_activity(),
+        vary_by_http_protocol_version(),
+        )
+
     def setUp(self):
         TestActivityMixin.setUp(self)
 
@@ -2087,6 +2111,8 @@
 class TestAuthOnRedirected(http_utils.TestCaseWithRedirectedWebserver):
     """Test authentication on the redirected http server."""
 
+    scenarios = vary_by_http_protocol_version()
+
     _auth_header = 'Authorization'
     _password_prompt_prefix = ''
     _username_prompt_prefix = ''

=== added file 'bzrlib/tests/test_scenarios.py'
--- bzrlib/tests/test_scenarios.py	1970-01-01 00:00:00 +0000
+++ bzrlib/tests/test_scenarios.py	2010-10-11 22:43:41 +0000
@@ -0,0 +1,110 @@
+# Copyright (C) 2005-2010 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+
+"""Tests for generating multiple tests for scenarios."""
+
+from bzrlib.tests import (
+    TestCase,
+    TestLoader,
+    iter_suite_tests,
+    multiply_tests,
+    )
+
+from bzrlib.tests.scenarios import (
+    load_tests_apply_scenarios,
+    multiply_scenarios,
+    multiply_tests_by_their_scenarios,
+    )
+
+
+# There aren't any actually parameterized tests here, but this exists as a
+# demonstration; so that you can interactively observe them being multiplied;
+# and so that we check everything hooks up properly.
+load_tests = load_tests_apply_scenarios
+
+
+def vary_by_color():
+    """Very simple static variation example"""
+    for color in ['red', 'green', 'blue']:
+        yield (color, {'color': color})
+
+
+def vary_named_attribute(attr_name):
+    """More sophisticated: vary a named parameter"""
+    yield ('a', {attr_name: 'a'})
+    yield ('b', {attr_name: 'b'})
+
+
+def get_generated_test_attributes(suite, attr_name):
+    """Return the `attr_name` attribute from all tests in the suite"""
+    return sorted([
+        getattr(t, attr_name) for t in iter_suite_tests(suite)])
+
+
+class TestTestScenarios(TestCase):
+
+    def test_multiply_tests(self):
+        loader = TestLoader()
+        suite = loader.suiteClass()
+        multiply_tests(
+            self,
+            vary_by_color(),
+            suite)
+        self.assertEquals(
+            ['blue', 'green', 'red'],
+            get_generated_test_attributes(suite, 'color'))
+
+    def test_multiply_scenarios_from_generators(self):
+        """It's safe to multiply scenarios that come from generators"""
+        s = multiply_scenarios(
+            vary_named_attribute('one'),
+            vary_named_attribute('two'),
+            )
+        self.assertEquals(
+            2*2,
+            len(s),
+            s)
+
+    def test_multiply_tests_by_their_scenarios(self):
+        loader = TestLoader()
+        suite = loader.suiteClass()
+        test_instance = PretendVaryingTest('test_nothing')
+        multiply_tests_by_their_scenarios(
+            test_instance,
+            suite)
+        self.assertEquals(
+            ['a', 'a', 'b', 'b'],
+            get_generated_test_attributes(suite, 'value'))
+
+    def test_multiply_tests_no_scenarios(self):
+        """Tests with no scenarios attribute aren't multiplied"""
+        suite = TestLoader().suiteClass()
+        multiply_tests_by_their_scenarios(self,
+            suite)
+        self.assertLength(1, list(iter_suite_tests(suite)))
+
+
+class PretendVaryingTest(TestCase):
+    
+    scenarios = multiply_scenarios(
+        vary_named_attribute('value'), 
+        vary_named_attribute('other'),
+        )
+
+    def test_nothing(self):
+        """This test exists just so it can be multiplied"""
+        pass

=== modified file 'doc/developers/testing.txt'
--- doc/developers/testing.txt	2010-10-07 07:51:54 +0000
+++ doc/developers/testing.txt	2010-10-11 22:43:41 +0000
@@ -811,17 +811,11 @@
 whether a test should be added for that particular implementation,
 or for all implementations of the interface.
 
-The multiplication of tests for different implementations is normally
-accomplished by overriding the ``load_tests`` function used to load tests
-from a module.  This function typically loads all the tests, then applies
-a TestProviderAdapter to them, which generates a longer suite containing
-all the test variations.
-
 See also `Per-implementation tests`_ (above).
 
 
-Test scenarios
---------------
+Test scenarios and variations
+-----------------------------
 
 Some utilities are provided for generating variations of tests.  This can
 be used for per-implementation tests, or other cases where the same test
@@ -832,8 +826,24 @@
 values to which the test should be applied.  The test suite should then
 also provide a list of scenarios in which to run the tests.
 
-Typically ``multiply_tests_from_modules`` should be called from the test
-module's ``load_tests`` function.
+A single *scenario* is defined by a `(name, parameter_dict)` tuple.  The
+short string name is combined with the name of the test method to form the
+test instance name.  The parameter dict is merged into the instance's
+attributes.
+
+For example::
+
+    load_tests = load_tests_apply_scenarios
+
+    class TestCheckout(TestCase):
+
+    variations = multiply_scenarios(
+        VaryByRepositoryFormat(), 
+        VaryByTreeFormat(),
+        )
+
+The `load_tests` declaration or definition should be near the top of the
+file so its effect can be seen.
 
 
 Test support