Merge lp:~sylvain-pineau/checkbox/nested_tp_doc into lp:checkbox

[Sylvain Pineau]

Finally, the doc about nested test plans and all their possible usages.

[Paul Larson (pwlars)]

A question and a small fix below. Otherwise it looks pretty good. Also, after we have a lot of these in place, do we think it would be useful to go back and do some editing for consistency of titles, format, etc?

review: Needs Information

[lp:~sylvain-pineau/checkbox/nested_tp_doc]

4481. By Sylvain Pineau on 2016-09-06

checkbox-ng:doc: Added a tutorial to start with nested test plans

[Sylvain Pineau (sylvain-pineau)]

Fixed and repushed (debian->Debian + a link to the overrides section in the test plan man page)

review: Needs Resubmitting

[Paul Larson (pwlars)]

+1

review: Approve

=== modified file 'checkbox-ng/docs/index.rst'
--- checkbox-ng/docs/index.rst	2016-08-25 08:56:43 +0000
+++ checkbox-ng/docs/index.rst	2016-09-06 15:48:18 +0000
@@ -58,6 +58,7 @@
    stack.rst
    launcher-tutorial.rst
    configs.rst
+   nested-test-plan.rst
    snappy.rst
    testing-snappy.rst
 

=== added file 'checkbox-ng/docs/nested-test-plan.rst'
--- checkbox-ng/docs/nested-test-plan.rst	1970-01-01 00:00:00 +0000
+++ checkbox-ng/docs/nested-test-plan.rst	2016-09-06 15:48:18 +0000
@@ -0,0 +1,379 @@
+Checkbox/plainbox nested test plans tutorial
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+We designed checkbox to consume test providers. Hence the test harness and the
+tests are completely separated. Checkbox can load tests from multiple providers.
+They can be installed as Debian packages or loaded from source to build a snap.
+
+To load the tests and run them we need a test plan. Test plans for checkbox are
+a collection of job (test) ids meant to be run one by one.
+
+Most of the time when we create a new test plan, there's a need to include a 
+generic section, common to several other test plans. But the test plan unit was 
+not allowing such feature and we ended up having a lot of duplication across 
+our projects. And duplication means duplicated efforts to maintain all those 
+test plan sections in sync and up-to-date.
+
+What if it could be possible now to have nested test plans. One being built by
+aggregating sections from one or more "base test plans"?
+
+Let's review in detail this new feature available in checkbox since plainbox 0.29
+
+Quick start
+===========
+
+The only thing to add to your test plan is the identifier of the test plan you
+want to include, as follow:
+
+::
+
+    nested_part:
+        2013.com.canonical.certification::my_base_test_plan
+
+The test plan order will then be test plan ``include`` + all nested test plan 
+``include``, in that order.
+
+Loading nested parts will load the ``include``, ``mandatory_include`` and 
+``bootstrap_include`` sections and all the overrides (``category``, 
+``certification status``).
+
+Note: All mandatory includes will always be run first.
+
+Note: Job and test plan ids can be listed in their abbreviated form (without 
+the namespace prefix) if the job definitions reside in the same namespace as 
+the provider that is defining the test plan.
+
+Use cases
+=========
+
+All the following examples are available here: 
+https://github.com/yphus/nested_testplan_demo To test them locally you just 
+need to develop the 3 providers and run one of the demo launchers:
+
+::
+
+    git clone https://github.com/yphus/nested_testplan_demo.git
+    cd nested_testplan_demo/
+    find . -name manage.py -exec {} develop \;
+    ./demo1 # or demo2, 3, 4, 5, 6.
+
+How to use a base test plan?
+----------------------------
+
+Let's use two providers, both belonging to the same namespace, ``2016.com.ubuntu``:
+
+``2016.com.ubuntu:foo`` and ``2016.com.ubuntu:baz``
+
+Baz provider contains the following units, 4 jobs and a test plan (our base 
+test plan):
+
+::
+
+    id: hello
+    command: echo hello
+    flags: simple
+    
+    id: bye
+    command: echo bye
+    flags: simple
+    
+    id: mandatory
+    command: true
+    flags: simple
+    
+    id: bootstrap
+    command: echo os: ubuntu
+    plugin: resource
+    flags: simple
+    
+    unit: test plan
+    id: baz_tp
+    _name: Generic baz test plan
+    _description: This test plan contains generic test cases
+    estimated_duration: 1m
+    include:
+        hello       certification-status=blocker
+        bye         certification-status=non-blocker
+    mandatory_include:
+        mandatory   certification-status=blocker
+    bootstrap_include:
+        bootstrap
+
+Foo provider contains two new tests:
+
+::
+
+    id: always-pass
+    command: true
+    flags: simple
+    
+    id: always-fail
+    command: true
+    flags: simple
+
+We want to reuse the ``baz_tp`` in a new test plan (in the Foo provider) with 
+the two new tests. Such test plan will look like this:
+
+
+::
+
+    unit: test plan
+    id: foo_tp_1
+    _name: Foo test plan 1
+    _description: This test plan contains generic tests + 2 new tests
+    include:
+        always-pass       certification-status=blocker
+        always-fail
+    nested_part:
+        baz_tp
+
+The jobs execution order is:
+
+- ``bootstrap``
+- ``mandatory``
+- ``always-pass``
+- ``always-fail``
+- ``hello``
+- ``bye``
+
+How to use a base test plan, but without running them last?
+-----------------------------------------------------------
+
+Let's keep the previous providers, Foo and Baz. This time we want to run the 
+base test plan between ``always-pass`` and ``always-fail``. In order to change 
+the job execution order, the new test plan will be made of several nested 
+parts, since they will follow the list order. Let's create in the Foo provider
+2 new test plans that we'll use as nested parts to fine tune the job ordering:
+
+::
+
+    unit: test plan
+    id: foo_tp_part1
+    _name: Foo test plan part 1
+    _description: This test plan contains part 1
+    estimated_duration: 1m
+    include:
+        always-pass       certification-status=blocker
+    
+    unit: test plan
+    id: foo_tp_part2
+    _name: Foo test plan part 2
+    _description: This test plan contains part 2
+    estimated_duration: 1m
+    include:
+        always-fail
+
+The final test plan will only contain nested parts:
+
+::
+
+    unit: test plan
+    id: foo_tp_2
+    _name: Foo test plan 2
+    _description:
+     This test plan contains generic tests + 2 new tests (but ordered differently)
+    include:
+    nested_part:
+        foo_tp_part1
+        baz_tp
+        foo_tp_part2
+
+Note: Always keep the ``include`` section (even empty) as this field is 
+mandatory and validation would failed otherwise (and the test plan never loaded 
+by checkbox)
+
+The jobs execution order is:
+
+- ``bootstrap``
+- ``mandatory``
+- ``always-pass``
+- ``hello``
+- ``bye``
+- ``always-fail``
+
+How to change category or certification status of jobs coming from nested parts?
+--------------------------------------------------------------------------------
+
+The `test plan override mechanism
+<http://plainbox.readthedocs.io/en/latest/manpages/plainbox-test-plan-units.html?highlight=category-overrides>`_
+still works with nested parts. For example the ``hello`` job from the Baz
+provider was defined as a blocker and did not have a category.
+
+Let's update the previous use case:
+
+::
+
+    unit: test plan
+    id: foo_tp_3
+    _name: Foo test plan 3
+    _description: This test plan contains generic tests + 2 new tests + overrides
+    include:
+        always-pass       certification-status=blocker
+        always-fail
+    nested_part:
+        baz_tp
+    certification_status_overrides:
+        apply non-blocker to hello
+    category_overrides:
+        apply 2013.com.canonical.plainbox::audio to hello
+
+To check that overrides worked as expected, you can open the json exporter 
+report:
+
+::
+
+    "result_map": {
+        "2016.com.ubuntu::hello": {
+            "summary": "hello",
+            "category_id": "2013.com.canonical.plainbox::audio",
+            "certification_status": "non-blocker"
+    [...]
+
+How to include a nested part from another namespace?
+----------------------------------------------------
+
+You can include a nested part from another namespace, just prefix the test plan
+identifier with the provider namespace.
+
+Let's use a third provider (Bar, under the ``2013.com.ubuntu`` namespace) as an 
+example:
+
+::
+
+    id: sleep
+    command: sleep 1
+    flags: simple
+    
+    id: uname
+    command: uname -a
+    flags: simple
+    
+    unit: test plan
+    id: bar_tp
+    _name: bar test plan
+    _description: This test plan contains bar test cases
+    estimated_duration: 1m
+    include:
+        sleep
+        uname
+
+Now in provider Foo, a test plan including a part from provider Bar will look 
+like this:
+
+::
+
+    unit: test plan
+    id: foo_tp_4
+    _name: Foo test plan 4
+    _description:
+     This test plan contains generic tests + 2 new tests + 2 tests from a
+     different namespace provider
+    include:
+        always-pass       certification-status=blocker
+        always-fail
+    nested_part:
+        baz_tp
+        2013.com.ubuntu::bar_tp
+
+The jobs execution order is:
+
+- ``bootstrap``
+- ``mandatory``
+- ``always-pass``
+- ``always-fail``
+- ``hello``
+- ``bye``
+- ``sleep``
+- ``uname``
+
+Is it possible to have multiple levels of nesting?
+--------------------------------------------------
+
+Yes, it's possible to have multiple levels of nesting, a nested part being 
+built from another nested parts, each level bringing its own set of new tests.
+
+Let's add a new test plan to provider Baz:
+
+::
+
+    unit: test plan
+    id: baz_tp_2
+    _name: Generic baz test plan 2
+    _description: This test plan contains generic test cases + a nested part
+    include:
+        hello       certification-status=blocker
+        bye         certification-status=non-blocker
+    mandatory_include:
+        mandatory   certification-status=blocker
+    bootstrap_include:
+        bootstrap
+    nested_part:
+        2013.com.ubuntu::bar_tp
+
+As you can see this test plan includes a part from provider Bar (the same used 
+in the previous example). In provider Foo, we can create a new test plan 
+including `baz_tp_2`:
+
+::
+
+    unit: test plan
+    id: foo_tp_5
+    _name: Foo test plan 5
+    _description: This test plan is built from multiple level of nested test plans
+    include:
+        always-pass       certification-status=blocker
+        always-fail
+    nested_part:
+        baz_tp_2
+
+The jobs execution order is still:
+
+- ``bootstrap``
+- ``mandatory``
+- ``always-pass``
+- ``always-fail``
+- ``hello``
+- ``bye``
+- ``sleep``
+- ``uname``
+
+How to use a base test plan except a few jobs?
+----------------------------------------------
+
+The test plan units support an optional field - ``exclude`` - that we can use
+to remove jobs from a nested part ``include`` section. 
+
+Note: The ``exclude`` ids cannot remove jobs that are parts of the 
+``mandatory_include`` sections (nested or not).
+
+The test plan below (from provider Foo) won't run the ``hello`` job of provider 
+Baz:
+
+::
+
+    unit: test plan
+    id: foo_tp_6
+    _name: Foo test plan 6
+    _description: This test plan contains generic tests + 2 new tests - hello job
+    include:
+        always-pass       certification-status=blocker
+        always-fail
+    exclude:
+        hello
+    nested_part:
+        baz_tp
+
+The jobs execution order is:
+
+- ``bootstrap``
+- ``mandatory``
+- ``always-pass``
+- ``always-fail``
+- ``bye``
+
+Known limitations
+=================
+
+You can create infinite loops if a nested part is calling itself or if 
+somewhere in the nested chain such a loop exists. Checkbox won't like that and 
+so far there's no validation to prevent it, be warned!