Merge lp:~mbp/testscenarios/mbp-doc into lp:~testtools-committers/testscenarios/trunk

[Martin Pool]

[Martin Pool (mbp)]

Documentation is now more correct and tested

=== modified file 'Makefile'
--- Makefile	2009-03-07 09:58:55 +0000
+++ Makefile	2009-03-09 06:45:33 +0000
@@ -1,9 +1,13 @@
 PYTHONPATH:=$(shell pwd)/lib:${PYTHONPATH}
+PYTHON ?= python
 
 all:
 
+# it would be nice to use doctest directly to run the README, but that's 
+# only supported from python2.6 onwards, so we need a script
 check:
-	PYTHONPATH=$(PYTHONPATH) python ./test_all.py $(TESTRULE)
+	PYTHONPATH=$(PYTHONPATH):.:./lib $(PYTHON) run_doctest.py README
+	PYTHONPATH=$(PYTHONPATH) $(PYTHON) ./test_all.py $(TESTRULE)
 
 clean:
 	find . -name '*.pyc' -print0 | xargs -0 rm -f

=== modified file 'README'
--- README	2009-03-08 04:37:33 +0000
+++ README	2009-03-09 06:45:33 +0000
@@ -1,5 +1,6 @@
-testscenarios: extensions to python unittest to allow declarative
-dependency injection ('scenarios') by tests.
+*****************************************************************
+testscenarios: extensions to python unittest to support scenarios
+*****************************************************************
 
 Copyright (C) 2009  Robert Collins <[e-mail]>
 
@@ -24,11 +25,13 @@
 dependencies externally to the test code itself, allowing easy testing in
 different situations).
 
-Dependencies:
-=============
+Dependencies
+============
 
 * Python 2.4+
-* testtools
+* testtools <https://launchpad.net/testtools>
+
+  >>> import testtools
 
 
 Why TestScenarios
@@ -51,19 +54,33 @@
 in multiple scenarios clear, easy to debug and work with even when the list
 of scenarios is dynamically generated.
 
-Getting Scenarios applied:
-==========================
+
+Defining Scenarios
+==================
+
+A **scenario** is a tuple of a string name for the scenario, and a dict of
+parameters describing the scenario.  The name is appended to the test name, and
+the parameters are made available to the test instance when it's run.
+
+Scenarios are presented in **scenario lists** which are typically Python lists
+but may be any iterable.
+
+
+Getting Scenarios applied
+=========================
 
 At its heart the concept is simple. For a given test object with a list of
 scenarios we prepare a new test object for each scenario. This involves:
- * Clone the test to a new test with a new id uniquely distinguishing it.
- * Apply the scenario to the test by setting each key, value in the scenario
-   as attributes on the test object.
+
+* Clone the test to a new test with a new id uniquely distinguishing it.
+* Apply the scenario to the test by setting each key, value in the scenario
+  as attributes on the test object.
 
 There are some complicating factors around making this happen seamlessly. These
 factors are in two areas:
- * Choosing what scenarios to use. (See Setting Scenarios For A Test).
- * Getting the multiplication to happen. 
+
+* Choosing what scenarios to use. (See Setting Scenarios For A Test).
+* Getting the multiplication to happen. 
 
 Subclasssing
 ++++++++++++
@@ -84,10 +101,21 @@
 useful test base classes, or need to override run() or __call__ yourself) then 
 you can cause scenario application to happen later by calling
 ``testscenarios.generate_scenarios()``. For instance::
-  >>> mytests = loader.loadTestsFromNames([...])
-  >>> test_suite = TestSuite()
+
+  >>> import unittest
+  >>> from testscenarios.scenarios import generate_scenarios
+
+This can work with loaders and runners from the standard library, or possibly other
+implementations::
+
+  >>> loader = unittest.TestLoader()
+  >>> test_suite = unittest.TestSuite()
+  >>> runner = unittest.TextTestRunner()
+
+  >>> mytests = loader.loadTestsFromNames(['example.test_sample'])
   >>> test_suite.addTests(generate_scenarios(mytests))
   >>> runner.run(test_suite)
+  <unittest._TextTestResult run=1 errors=0 failures=0>
 
 Testloaders
 +++++++++++
@@ -100,35 +128,37 @@
 With ``load_tests``::
 
   >>> def load_tests(standard_tests, module, loader):
-  >>>     result = loader.suiteClass()
-  >>>     result.addTests(generate_scenarios(standard_tests))
-  >>>     return result
+  ...     result = loader.suiteClass()
+  ...     result.addTests(generate_scenarios(standard_tests))
+  ...     return result
 
 With ``test_suite``::
 
   >>> def test_suite():
-  >>>     loader = TestLoader()
-  >>>     tests = loader.loadTestsFromName(__name__)
-  >>>     result = loader.suiteClass()
-  >>>     result.addTests(generate_scenarios(tests))
-  >>>     return result
-
-
-Setting Scenarios for a test:
-=============================
+  ...     loader = TestLoader()
+  ...     tests = loader.loadTestsFromName(__name__)
+  ...     result = loader.suiteClass()
+  ...     result.addTests(generate_scenarios(tests))
+  ...     return result
+
+
+Setting Scenarios for a test
+============================
 
 A sample test using scenarios can be found in the doc/ folder.
 
-See pydoc testscenarios for details.
+See `pydoc testscenarios` for details.
 
 On the TestCase
 +++++++++++++++
 
 You can set a scenarios attribute on the test case::
 
-  >>> class MyTest(TestCase):
-  >>>
-  >>>     scenarios = [scenario1, scenario2, ...]
+  >>> class MyTest(unittest.TestCase):
+  ...
+  ...     scenarios = [
+  ...         ('scenario1', dict(param=1)),
+  ...         ('scenario2', dict(param=2)),]
 
 This provides the main interface by which scenarios are found for a given test.
 Subclasses will inherit the scenarios (unless they override the attribute).
@@ -139,23 +169,41 @@
 Test scenarios can also be generated arbitrarily later, as long as the test has
 not yet run. Simply replace (or alter, but be aware that many tests may share a
 single scenarios attribute) the scenarios attribute. For instance in this
-example some third party tests are extended to run with a custom scenario.
+example some third party tests are extended to run with a custom scenario. ::
 
-  >>> for test in iterate_tests(stock_library_tests):
-  >>>     if isinstance(test, TestVFS):
-  >>>         test.scenarios = test.scenarios + [my_vfs_scenario]
-  >>> ...
+  >>> class TestTransport:
+  ...     """Hypothetical test case for bzrlib transport tests"""
+  ...     pass
+  ...
+  >>> stock_library_tests = unittest.TestLoader().loadTestsFromNames(
+  ...     ['example.test_sample'])
+  ...
+  >>> for test in testtools.iterate_tests(stock_library_tests):
+  ...     if isinstance(test, TestTransport):
+  ...         test.scenarios = test.scenarios + [my_vfs_scenario]
+  ...
+  >>> suite = unittest.TestSuite()
   >>> suite.addTests(generate_scenarios(stock_library_tests))
 
-Note that adding scenarios to a test that has already been parameterised via
-generate_scenarios generates a cross product::
-  >>> class CrossProductDemo(TestCase):
-  >>>     scenarios = [scenario_0_0, scenario_0_1]
-  >>>     def test_foo(self):
-  >>>         return
+Generated tests don't have a ``scenarios`` list, because they don't normally
+require any more expansion.  However, you can add a ``scenarios`` list back on
+to them, and then run them through ``generate_scenarios`` again to generate the
+cross product of tests. ::
+
+  >>> class CrossProductDemo(unittest.TestCase):
+  ...     scenarios = [('scenario_0_0', {}),
+  ...                  ('scenario_0_1', {})]
+  ...     def test_foo(self):
+  ...         return
+  ...
+  >>> suite = unittest.TestSuite()
   >>> suite.addTests(generate_scenarios(CrossProductDemo("test_foo")))
-  >>> for test in iterate_tests(suite):
-  >>>     test.scenarios = test.scenarios + [scenario_1_0, scenario_1_1]
+  >>> for test in testtools.iterate_tests(suite):
+  ...     test.scenarios = [
+  ...         ('scenario_1_0', {}), 
+  ...         ('scenario_1_1', {})]
+  ...
+  >>> suite2 = unittest.TestSuite()
   >>> suite2.addTests(generate_scenarios(suite))
   >>> print suite2.countTestCases()
   4
@@ -168,27 +216,28 @@
 scenarios somewhere relevant to the tests that will use it, and then that can
 be customised, or dynamically populate your scenarios from a registry etc.
 For instance::
+
   >>> hash_scenarios = []
   >>> try:
-  >>>     import md5
-  >>> except ImportError:
-  >>>     pass
-  >>> else:
-  >>>     hash_scenarios.append(("md5", "hash": md5.new))
+  ...     from hashlib import md5
+  ... except ImportError:
+  ...     pass
+  ... else:
+  ...     hash_scenarios.append(("md5", dict(hash=md5)))
   >>> try:
-  >>>     import sha1
-  >>> except ImportError:
-  >>>     pass
-  >>> else:
-  >>>     hash_scenarios.append(("sha1", "hash": sha1.new))
-  >>>
-  >>> class TestHashContract(TestCase):
-  >>>
-  >>>     scenarios = hash_scenarios
-  >>>
-  >>> class TestHashPerformance(TestCase):
-  >>>
-  >>>     scenarios = hash_scenarios
+  ...     from hashlib import sha1
+  ... except ImportError:
+  ...     pass
+  ... else:
+  ...     hash_scenarios.append(("sha1", dict(hash=sha1)))
+  ...
+  >>> class TestHashContract(unittest.TestCase):
+  ...
+  ...     scenarios = hash_scenarios
+  ...
+  >>> class TestHashPerformance(unittest.TestCase):
+  ...
+  ...     scenarios = hash_scenarios
 
 
 Forcing Scenarios
@@ -201,3 +250,10 @@
 ``apply_scenarios`` function does not reset the test scenarios attribute,
 allowing it to be used to layer scenarios without affecting existing scenario
 selection.
+
+
+Advice on Writing Scenarios
+===========================
+
+If a parameterised test is because of a bug run without being parameterized,
+it should fail rather than running with defaults, because this can hide bugs.

=== added directory 'example'
=== added file 'example/__init__.py'
--- example/__init__.py	1970-01-01 00:00:00 +0000
+++ example/__init__.py	2009-03-09 06:45:33 +0000
@@ -0,0 +1,1 @@
+# contractual obligation

=== added file 'example/test_sample.py'
--- example/test_sample.py	1970-01-01 00:00:00 +0000
+++ example/test_sample.py	2009-03-09 06:45:33 +0000
@@ -0,0 +1,6 @@
+import unittest
+
+class TestSample(unittest.TestCase):
+
+    def test_so_easy(self):
+        pass

=== added file 'run_doctest.py'
--- run_doctest.py	1970-01-01 00:00:00 +0000
+++ run_doctest.py	2009-03-09 06:45:33 +0000
@@ -0,0 +1,5 @@
+import doctest
+import sys
+
+for n in sys.argv:
+    print doctest.testfile(n, raise_on_error=False, )