Merge lp:~mbp/testscenarios/module-scenarios into lp:~testtools-committers/testscenarios/trunk

[Martin Pool]

This separates out something we do in bzr, which is to multiply tests by both Python and Pyrex/C implementations of the same interface.

[Robert Collins (lifeless)]

For some reason this has generated a merge proposal with a bunch of conflicts :(

[Martin Pool (mbp)]

conflicts fixed (at least locally)

[Martin Pool (mbp)]

robert could you review this please?

[Robert Collins (lifeless)]

I think this is good but IIRC John has a desire to sometimes require that the other implementation is present. Perhaps rather than eliding implements we cannot import, we emit synthetic skips for them, would permit the test runner to decide on policy (e.g. to fail, or to just report the test as skipped).

What do you think?

Other than that this seems very nice.

[Martin Pool (mbp)]

On 29 November 2011 13:13, Robert Collins <email address hidden> wrote:
> I think this is good but IIRC John has a desire to sometimes require that the other implementation is present. Perhaps rather than eliding implements we cannot import, we emit synthetic skips for them, would permit the test runner to decide on policy (e.g. to fail, or to just report the test as skipped).

I think that's reasonable to want. Also, squashing ImportError
altogether is probably too likely to hide real problems.

[Martin Pool (mbp)]

On consideration, although those improvements would be useful, I think
we could land this and add them as enhancements when they're wanted -
maybe when we change bzr to use this.

[Robert Collins (lifeless)]

On Tue, Nov 29, 2011 at 7:19 PM, Martin Pool <email address hidden> wrote:
> On consideration, although those improvements would be useful, I think
> we could land this and add them as enhancements when they're wanted -
> maybe when we change bzr to use this.

I'd really rather not be hiding ImportError at all - It seems to me
all users are going to want that, so rather than changing the
behaviour in a later release, I'd rather we tweak this now.

[Robert Collins (lifeless)]

I think this needs a (shallow) patch to testtools to make it trivial to wrap a test with a skip.

[Robert Collins (lifeless)]

https://code.launchpad.net/~lifeless/testtools/placeholderskip/+merge/100564

will provide such a tool.

[Martin Pool (mbp)]

Thanks!
On Apr 4, 2012 8:05 PM, <email address hidden> wrote:

> The proposal to merge lp:~mbp/testscenarios/module-scenarios into
> lp:testscenarios has been updated.
>
> Status: Needs review => Merged
>
> For more details, see:
> https://code.launchpad.net/~mbp/testscenarios/module-scenarios/+merge/80287
> --
> https://code.launchpad.net/~mbp/testscenarios/module-scenarios/+merge/80287
> You are the owner of lp:~mbp/testscenarios/module-scenarios.
>

=== modified file 'NEWS'
--- NEWS	2010-10-12 06:57:23 +0000
+++ NEWS	2011-10-25 04:00:30 +0000
@@ -9,6 +9,12 @@
 0.2
 ~~~
 
+NEW FEATURES:
+
+* New function ``per_module_scenarios`` for tests that should be applied across 
+  multiple modules providing the same interface, some of which may not be 
+  available at run time.  (Martin Pool)
+
 CHANGES:
 
 * Adjust the cloned tests ``shortDescription`` if one is present. (Ben Finney)

=== modified file 'README'
--- README	2010-10-12 06:57:23 +0000
+++ README	2011-10-25 04:00:30 +0000
@@ -258,6 +258,35 @@
 selection.
 
 
+Generating Scenarios
+====================
+
+Some functions (currently one :-) are available to ease generation of scenario
+lists for common situations.
+
+Testing Per Implementation Module
++++++++++++++++++++++++++++++++++
+
+It is reasonably common to have multiple Python modules that provide the same
+capabilities and interface, and to want apply the same tests to all of them.
+
+In some cases, not all of the statically defined implementations will be able
+to be used in a particular testing environment.  For example, there may be both
+a C and a pure-Python implementation of a module.  You want to test the C
+module if it can be loaded, but also to have the tests pass if the C module has
+not been compiled.
+
+The ``per_module_scenarios`` function generates a scenario for each named
+module, omitting those that raise an ``ImportError``.  For each test object,
+the implementation to be tested is installed into a given attribute.
+
+Note that for the test to be valid, all access to the module under test must go
+through the relevant attribute of the test object.  If one of the
+implementations is also directly imported by the test module or any other,
+testscenarios will not magically stop it being used.
+
+
+
 Advice on Writing Scenarios
 ===========================
 

=== modified file 'lib/testscenarios/scenarios.py'
--- lib/testscenarios/scenarios.py	2010-10-12 06:57:23 +0000
+++ lib/testscenarios/scenarios.py	2011-10-25 04:00:30 +0000
@@ -2,7 +2,7 @@
 #  dependency injection ('scenarios') by tests.
 #
 # Copyright (c) 2009, Robert Collins <[e-mail]>
-# Copyright (c) 2010 Martin Pool <[e-mail]>
+# Copyright (c) 2010, 2011 Martin Pool <[e-mail]>
 # 
 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
 # license at the users choice. A copy of both licenses are available in the
@@ -129,3 +129,37 @@
             scenario_parameters.update(parameter)
         result.append((scenario_name, scenario_parameters))
     return result
+
+
+def per_module_scenarios(attribute_name, modules):
+    """Generate scenarios for available implementation modules.
+
+    This is typically used when there is a subsystem implemented, for
+    example, in both Python and C, and we want to apply the same tests to
+    both, but the C module may sometimes not be available.
+
+    Note: if the module can't be loaded, it's silently omitted from
+    testing.
+
+    :param attribute_name: A name to be set in the scenario parameter
+        dictionary (and thence onto the test instance) pointing to the 
+        implementation module for this scenario.
+
+    :param modules: An iterable of (short_name, module_name), where 
+        the short name is something like 'python' to put in the
+        scenario name, and the long name is a fully-qualified Python module
+        name.
+    """
+    scenarios = []
+    for short_name, module_name in modules:
+        try:
+            mod = __import__(module_name, {}, {}, [''])
+        except ImportError:
+            # TODO: optionally pass this back through a callback, so it can be
+            # logged etc?
+            pass
+        else:
+            scenarios.append((
+                short_name, 
+                {attribute_name: mod}))
+    return scenarios

=== modified file 'lib/testscenarios/tests/test_scenarios.py'
--- lib/testscenarios/tests/test_scenarios.py	2010-10-12 06:57:23 +0000
+++ lib/testscenarios/tests/test_scenarios.py	2011-10-25 04:00:30 +0000
@@ -2,7 +2,7 @@
 #  dependency injection ('scenarios') by tests.
 #
 # Copyright (c) 2009, Robert Collins <[e-mail]>
-# Copyright (c) 2010 Martin Pool <[e-mail]>
+# Copyright (c) 2010, 2011 Martin Pool <[e-mail]>
 # 
 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
 # license at the users choice. A copy of both licenses are available in the
@@ -239,3 +239,19 @@
         self.assertEqual(
             'a,a,a,a',
             scenarios[0][0])
+
+
+class TestPerModuleScenarios(testtools.TestCase):
+
+    def test_per_module_scenarios(self):
+        """Generate scenarios for available modules"""
+        s = testscenarios.scenarios.per_module_scenarios(
+            'the_module', [
+                ('Python', 'testscenarios'),
+                ('unittest', 'unittest'),
+                ('nonexistent', 'nonexistent'),
+                ])
+        self.assertEqual(s, [
+            ('Python', {'the_module': testscenarios}),
+            ('unittest', {'the_module': unittest}),
+            ])