Merge lp:~vila/ubuntu-ci-services-itself/1290401-doc-subunit into lp:ubuntu-ci-services-itself

[Vincent Ladeuil]

Document subunit usage from the streams produced by the test runner.

Document subunit usage from the streams produced by the test runner.

[Francis Ginther (fginther)]

76 +Converting to the junixml format (used by jenkins)

One typo in this line (junixml -> junitxml). I also would remove the '(used by jenkins)', as it might lead an inexperience user to think the they need to manually load this into jenkins. I think 'junitxml format' stands well enough on it's own.

Otherwise, this is perfect. Thanks.

review: Needs Fixing

[lp:~vila/ubuntu-ci-services-itself/1290401-doc-subunit]

383. By Vincent Ladeuil on 2014-03-13

Fix review comment.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:382
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/408/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/408/rebuild

review: Approve (continuous-integration)

[Francis Ginther (fginther)]

Looks good. I see one more mention of jenkins at the very end, but as I other changes to make to this exact same file. I'll just do it then.

review: Approve

=== modified file 'docs/usage.rst'
--- docs/usage.rst	2014-01-24 18:33:12 +0000
+++ docs/usage.rst	2014-03-13 18:35:59 +0000
@@ -52,3 +52,80 @@
 
 Once a ticket is completed, it's status will be updated appropriately and all
 build and test results and logs will be available via the web interface.
+
+
+Exploiting Test Results
+-----------------------
+
+The test runner produces results in the subunit v1 format.
+
+The subunit stream can be downloaded from the web interface and processed
+locally in different ways.
+
+Note that the subunit stream contains the dep-8 test results.
+
+The subunit v1 format is text only and as such can be read but it's not
+especially user-friendly:
+
+.. code-block:: none
+
+  test: dsc0t-build
+  successful: dsc0t-build [ multipart
+  Content-Type: text/plain;charset=utf8
+  stderr
+  0^M
+  Content-Type: text/plain;charset=utf8
+  stdout
+  12^M
+  build: OK
+  run: OK
+  0^M
+  ]
+
+subunit provide filters to convert a stream into more readable outputs.
+
+
+Converting to the python unittest format
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: none
+
+  $ subunit-1to2  <subunit.stream | subunit2pyunit
+
+This filter is appropriate to inspect failures in tests and will produce
+(for example):
+
+.. code-block:: none
+
+  $ subunit-1to2  <fail.stream | subunit2pyunit
+  tests.test_pass
+  tests.test_pass ... ok
+  tests.test_fail
+  tests.test_fail ... FAIL
+  ======================================================================
+  FAIL: tests.test_fail
+  tests.test_fail
+  ----------------------------------------------------------------------
+  testtools.testresult.real._StringException: Traceback (most recent call last):
+    File "tests.py", line 31, in test_fail
+      self.assertTrue(False)
+    File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
+      raise self.failureException(msg)
+  AssertionError: False is not true
+   
+   
+  ----------------------------------------------------------------------
+  Ran 2 tests in 0.001s
+   
+  FAILED (failures=1)
+
+
+Converting to the junitxml format
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: none
+
+  $ subunit-1to2  <subunit.stream | subunit2junitxml >results.xml
+
+This produces a 'results.xml' file suitable for jenkins.
+