Merge lp:~mbp/testscenarios/module-scenarios into lp:~testtools-committers/testscenarios/trunk

Proposed by Martin Pool
Status: Merged
Merged at revision: 19
Proposed branch: lp:~mbp/testscenarios/module-scenarios
Merge into: lp:~testtools-committers/testscenarios/trunk
Diff against target: 140 lines (+87/-2)
4 files modified
NEWS (+6/-0)
README (+29/-0)
lib/testscenarios/scenarios.py (+35/-1)
lib/testscenarios/tests/test_scenarios.py (+17/-1)
To merge this branch: bzr merge lp:~mbp/testscenarios/module-scenarios
Reviewer Review Type Date Requested Status
Robert Collins Pending
Review via email: mp+80287@code.launchpad.net

This proposal supersedes a proposal from 2011-07-08.

Description of the change

This separates out something we do in bzr, which is to multiply tests by both Python and Pyrex/C implementations of the same interface.

To post a comment you must log in.
Revision history for this message
Robert Collins (lifeless) wrote : Posted in a previous version of this proposal

For some reason this has generated a merge proposal with a bunch of conflicts :(

Revision history for this message
Martin Pool (mbp) wrote :

conflicts fixed (at least locally)

Revision history for this message
Martin Pool (mbp) wrote :

robert could you review this please?

Revision history for this message
Robert Collins (lifeless) wrote :

I think this is good but IIRC John has a desire to sometimes require that the other implementation is present. Perhaps rather than eliding implements we cannot import, we emit synthetic skips for them, would permit the test runner to decide on policy (e.g. to fail, or to just report the test as skipped).

What do you think?

Other than that this seems very nice.

Revision history for this message
Martin Pool (mbp) wrote :

On 29 November 2011 13:13, Robert Collins <email address hidden> wrote:
> I think this is good but IIRC John has a desire to sometimes require that the other implementation is present. Perhaps rather than eliding implements we cannot import, we emit synthetic skips for them, would permit the test runner to decide on policy (e.g. to fail, or to just report the test as skipped).

I think that's reasonable to want. Also, squashing ImportError
altogether is probably too likely to hide real problems.

Revision history for this message
Martin Pool (mbp) wrote :

On consideration, although those improvements would be useful, I think
we could land this and add them as enhancements when they're wanted -
maybe when we change bzr to use this.

Revision history for this message
Robert Collins (lifeless) wrote :

On Tue, Nov 29, 2011 at 7:19 PM, Martin Pool <email address hidden> wrote:
> On consideration, although those improvements would be useful, I think
> we could land this and add them as enhancements when they're wanted -
> maybe when we change bzr to use this.

I'd really rather not be hiding ImportError at all - It seems to me
all users are going to want that, so rather than changing the
behaviour in a later release, I'd rather we tweak this now.

Revision history for this message
Robert Collins (lifeless) wrote :

I think this needs a (shallow) patch to testtools to make it trivial to wrap a test with a skip.

Revision history for this message
Robert Collins (lifeless) wrote :
Revision history for this message
Martin Pool (mbp) wrote :

Thanks!
On Apr 4, 2012 8:05 PM, <email address hidden> wrote:

> The proposal to merge lp:~mbp/testscenarios/module-scenarios into
> lp:testscenarios has been updated.
>
> Status: Needs review => Merged
>
> For more details, see:
> https://code.launchpad.net/~mbp/testscenarios/module-scenarios/+merge/80287
> --
> https://code.launchpad.net/~mbp/testscenarios/module-scenarios/+merge/80287
> You are the owner of lp:~mbp/testscenarios/module-scenarios.
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'NEWS'
2--- NEWS 2010-10-12 06:57:23 +0000
3+++ NEWS 2011-10-25 04:00:30 +0000
4@@ -9,6 +9,12 @@
5 0.2
6 ~~~
7
8+NEW FEATURES:
9+
10+* New function ``per_module_scenarios`` for tests that should be applied across
11+ multiple modules providing the same interface, some of which may not be
12+ available at run time. (Martin Pool)
13+
14 CHANGES:
15
16 * Adjust the cloned tests ``shortDescription`` if one is present. (Ben Finney)
17
18=== modified file 'README'
19--- README 2010-10-12 06:57:23 +0000
20+++ README 2011-10-25 04:00:30 +0000
21@@ -258,6 +258,35 @@
22 selection.
23
24
25+Generating Scenarios
26+====================
27+
28+Some functions (currently one :-) are available to ease generation of scenario
29+lists for common situations.
30+
31+Testing Per Implementation Module
32++++++++++++++++++++++++++++++++++
33+
34+It is reasonably common to have multiple Python modules that provide the same
35+capabilities and interface, and to want apply the same tests to all of them.
36+
37+In some cases, not all of the statically defined implementations will be able
38+to be used in a particular testing environment. For example, there may be both
39+a C and a pure-Python implementation of a module. You want to test the C
40+module if it can be loaded, but also to have the tests pass if the C module has
41+not been compiled.
42+
43+The ``per_module_scenarios`` function generates a scenario for each named
44+module, omitting those that raise an ``ImportError``. For each test object,
45+the implementation to be tested is installed into a given attribute.
46+
47+Note that for the test to be valid, all access to the module under test must go
48+through the relevant attribute of the test object. If one of the
49+implementations is also directly imported by the test module or any other,
50+testscenarios will not magically stop it being used.
51+
52+
53+
54 Advice on Writing Scenarios
55 ===========================
56
57
58=== modified file 'lib/testscenarios/scenarios.py'
59--- lib/testscenarios/scenarios.py 2010-10-12 06:57:23 +0000
60+++ lib/testscenarios/scenarios.py 2011-10-25 04:00:30 +0000
61@@ -2,7 +2,7 @@
62 # dependency injection ('scenarios') by tests.
63 #
64 # Copyright (c) 2009, Robert Collins <robertc@robertcollins.net>
65-# Copyright (c) 2010 Martin Pool <mbp@sourcefrog.net>
66+# Copyright (c) 2010, 2011 Martin Pool <mbp@sourcefrog.net>
67 #
68 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
69 # license at the users choice. A copy of both licenses are available in the
70@@ -129,3 +129,37 @@
71 scenario_parameters.update(parameter)
72 result.append((scenario_name, scenario_parameters))
73 return result
74+
75+
76+def per_module_scenarios(attribute_name, modules):
77+ """Generate scenarios for available implementation modules.
78+
79+ This is typically used when there is a subsystem implemented, for
80+ example, in both Python and C, and we want to apply the same tests to
81+ both, but the C module may sometimes not be available.
82+
83+ Note: if the module can't be loaded, it's silently omitted from
84+ testing.
85+
86+ :param attribute_name: A name to be set in the scenario parameter
87+ dictionary (and thence onto the test instance) pointing to the
88+ implementation module for this scenario.
89+
90+ :param modules: An iterable of (short_name, module_name), where
91+ the short name is something like 'python' to put in the
92+ scenario name, and the long name is a fully-qualified Python module
93+ name.
94+ """
95+ scenarios = []
96+ for short_name, module_name in modules:
97+ try:
98+ mod = __import__(module_name, {}, {}, [''])
99+ except ImportError:
100+ # TODO: optionally pass this back through a callback, so it can be
101+ # logged etc?
102+ pass
103+ else:
104+ scenarios.append((
105+ short_name,
106+ {attribute_name: mod}))
107+ return scenarios
108
109=== modified file 'lib/testscenarios/tests/test_scenarios.py'
110--- lib/testscenarios/tests/test_scenarios.py 2010-10-12 06:57:23 +0000
111+++ lib/testscenarios/tests/test_scenarios.py 2011-10-25 04:00:30 +0000
112@@ -2,7 +2,7 @@
113 # dependency injection ('scenarios') by tests.
114 #
115 # Copyright (c) 2009, Robert Collins <robertc@robertcollins.net>
116-# Copyright (c) 2010 Martin Pool <mbp@sourcefrog.net>
117+# Copyright (c) 2010, 2011 Martin Pool <mbp@sourcefrog.net>
118 #
119 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
120 # license at the users choice. A copy of both licenses are available in the
121@@ -239,3 +239,19 @@
122 self.assertEqual(
123 'a,a,a,a',
124 scenarios[0][0])
125+
126+
127+class TestPerModuleScenarios(testtools.TestCase):
128+
129+ def test_per_module_scenarios(self):
130+ """Generate scenarios for available modules"""
131+ s = testscenarios.scenarios.per_module_scenarios(
132+ 'the_module', [
133+ ('Python', 'testscenarios'),
134+ ('unittest', 'unittest'),
135+ ('nonexistent', 'nonexistent'),
136+ ])
137+ self.assertEqual(s, [
138+ ('Python', {'the_module': testscenarios}),
139+ ('unittest', {'the_module': unittest}),
140+ ])

Subscribers

People subscribed via source and target branches