>> In summary, the 2 registrations are replaced by 10 test classes each with
    >> their own setUp() to replace the parametrization. As mentioned on IRC, this
    >> open the door to divergence if more classes are added.

    > Yeah, I agree it's also far from ideal, but I like it better than
    > having to register test helpers in a plugins' __init__.

Reminder: the registration have to be done there before either of
bp.svn.tests.test_config or bt.test_config is loaded. No additional file
is loaded by this lazy registration though, so it shouldn't have any
impact on startup times (which is the point of lazy loading).

What exactly do you dislike with registration here ? I'm not trying to
make you take the "blue pill", registration is not meant to be
mandatory. In fact, it defines a very narrow API that should be easy to
implement. I'm not trying to *reduce* the flexibility for plugin authors
(quite the contrary), I'm just trying to make it easy to reuse the
tests.

I'm Ok with investigating alternate ways to get there though.

    >> This means the tests are now part of -s bp.svn but all the debug
    >> still happen in the base class, not super sexy to use.

    > I think to do this right, we need to fix
    > bzrlib.tests.test_config. Perhaps it can provide some sort of
    > multiply method? E.g.

That's what parametrization is about. I don't understand the kind of fix
you're talking about here ?

    > multiply_store_tests(build_store_fn)

more like:

  multiply_store_tests(build_store)
  multiply_matcher_tests(build_store, build_matcher)
  multiply_stack_tests(build_stack, build_store)

which still scales worse than parametrization, while still requiring
implementing the same helpers.

It seems to reduce the flexibility for all parties involved, I'm fine
with plugins not reusing the bzrlib tests but it seems we're
bike-shedding about how they can reuse them.

There are already 2 ways to reuse these tests:

- lazy registration, the simplest, required to be used in __init__ to be
  taken into account in all cases (-s and all that jazz) and be seen as
  additional parametrization of bt.test_config. No performance impact
  should occur here even if the tests are not run.

- subclassing which allow specific additional tests to be run or base
  tests to be skipped or whatever the plugin want to test.

The former guarantees that the plugin conforms to the required API. The
second allows the plugin to diverge from this API or extend it if
needed. Both can or should be used by the plugins depending on what they
implement.

subclassing also requires a more intimate knowledge of what the base
tests are about.

    > This method could then be called inside
    > bzrlib.plugins.svn.tests.test_config.load_tests.

    > It would remove the need to do test helper registration in
    > bp.svn.__init___.

This introduces an additional constraint on bt.test_config. What are the
benefits there ?

ISTM we are walking on the path that led to poolie's introduction of:

  load_tests = scenarios.load_tests_apply_scenarios

Using a registry targeted at tests is the same logic IMHO: the test
class defines an API (the parameters) and we need a way to add some
scenarios (to simplify: from plugins which is the most constrained
case).

http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/revision/5483?start_revid=5483
is the best example I can think of to explain that.

Even subclassing as I did in the last revisions shows some limits: the
setUp() methods requires some care to set some attributes (as parameters
rendering the reuse more brittle) after or before calling the base
setUp() which you don't need to care about when registering.

I think it's fair to pay this price for greater control, but making it
the preferred way to reuse the tests makes plugin authors life harder.