Merge lp:~pwlars/checkbox/junit-exporter into lp:checkbox

[Paul Larson]

This adds an exporter for junit xml. I find this particularly useful for working with jenkins, which has some native support for taking the output in junit and producing nice graphs and trends for test case pass/fail rates.

[Maciej Kisielewski (kissiel)]

+1.

Tested with a few scenarios, including non-matching plainbox run -i.
Everything worked beautifully.

I'm wondering if we can template name="" with something (see inline)

review: Approve

[Paul Larson (pwlars)]

I thought about that, but couldn't see where to get some definitive name that would work for all cases. Maybe just "checkbox" but that's just as generic as nothing at all. I looked back at some older jenkins runs that pulled results from autopilot and it had nothing in that field also. It seems to be optional and jenkins certainly doesn't care, but we can easily fill it in later if we want.

=== modified file 'checkbox-ng/launchers/checkbox-cli'
--- checkbox-ng/launchers/checkbox-cli	2016-07-21 17:41:44 +0000
+++ checkbox-ng/launchers/checkbox-cli	2016-07-25 17:58:00 +0000
@@ -693,6 +693,7 @@
             if not os.path.exists(self.base_dir):
                 os.makedirs(self.base_dir)
             for exporter, file_ext in [('hexr', '.xml'), ('html', '.html'),
+                                       ('junit', '.junit.xml'),
                                        ('xlsx', '.xlsx'), ('tar', '.tar.xz')]:
                 path = os.path.join(self.base_dir, ''.join(
                     ['submission_', timestamp, file_ext]))

=== added file 'plainbox/plainbox/impl/providers/exporters/data/junit.xml'
--- plainbox/plainbox/impl/providers/exporters/data/junit.xml	1970-01-01 00:00:00 +0000
+++ plainbox/plainbox/impl/providers/exporters/data/junit.xml	2016-07-25 17:58:00 +0000
@@ -0,0 +1,28 @@
+{%- set state = manager.default_device_context.state -%}
+{%- set job_state_map = state.job_state_map -%}
+{%- set passes = manager.state.get_outcome_stats()["pass"] -%}
+{%- set fails = manager.state.get_outcome_stats()["fail"] -%}
+{%- set skips = manager.state.get_outcome_stats()["skip"] + 
+                manager.state.get_outcome_stats()["not-supported"] -%} 
+{%- set errors = manager.state.get_outcome_stats()["crash"] -%}
+<?xml version="1.0" encoding="UTF-8"?>
+  <testsuites failures="{{ fails }}" name="" tests="{{ passes }}" skipped="{{ skips }}" errors="{{ errors }}">
+    {%- for job_id in job_state_map|sort %}
+    {%- set job_state = job_state_map[job_id] %}
+    {%- if job_state.job.plugin not in ("resource", "local", "attachment") and job_state.result.outcome %}
+    <testcase classname="{{ job_state.job.id }}" name="{{ job_state.job.id }}" time="{{ job_state.result.execution_duration }}">
+    {%- if job_state.result.outcome == 'skip' or job_state.result.outcome == 'not-supported' -%}
+      <skipped />
+    {%- elif job_state.result.outcome == 'fail' -%}
+      <failure type="">
+      {{ job_state.result.io_log_as_flat_text }}
+      </failure>
+    {%- elif job_state.result.outcome == 'crash' -%}
+      <error type="">
+      {{ job_state.result.io_log_as_flat_text }}
+      </error>
+    {%- endif %}
+    </testcase>
+    {%- endif -%}
+    {%- endfor %}
+  </testsuites>

=== modified file 'plainbox/plainbox/impl/providers/exporters/units/exporter.pxu'
--- plainbox/plainbox/impl/providers/exporters/units/exporter.pxu	2016-07-01 09:08:55 +0000
+++ plainbox/plainbox/impl/providers/exporters/units/exporter.pxu	2016-07-25 17:58:00 +0000
@@ -50,3 +50,10 @@
 entry_point: jinja2
 file_extension: txt
 data: {"template": "global.txt"}
+
+unit: exporter
+id: junit
+_summary: Generate junit XML
+entry_point: jinja2
+file_extension: junit.xml
+data: {"template": "junit.xml"}

=== modified file 'plainbox/plainbox/impl/session/manager.py'
--- plainbox/plainbox/impl/session/manager.py	2016-07-04 09:51:32 +0000
+++ plainbox/plainbox/impl/session/manager.py	2016-07-25 17:58:00 +0000
@@ -480,6 +480,7 @@
             '2013.com.canonical.plainbox::hexr': 'xml',
             '2013.com.canonical.plainbox::html': 'html',
             '2013.com.canonical.plainbox::json': 'json',
+            '2013.com.canonical.plainbox::junit': 'junit',
             '2013.com.canonical.plainbox::rfc822': 'rfc822',
             '2013.com.canonical.plainbox::tar': 'tar',
             '2013.com.canonical.plainbox::text': 'text',