Merge ~pwlars/checkbox-ng:get-test-outcome-stats into checkbox-ng:master

[Paul Larson]

It seems that the stats we generate for junit don't match up with the stats we generate for the c3 submission. This is because the junit stats do not filter out the resource and attachment jobs. This restricts junit output to only look at those, and adds a new helper method in states.py to facilitate getting the right set of stats for only actual tests.

[Paul Larson (pwlars)]

I almost forgot to mention... I tested this by creating a small dummy provider with 2 simple tests (one pass, one fail) and 2 attachment jobs (1 pass, 1 fail). Without the patch, I see 4 tests, 2 fails. With the pass, I only see the two actual tests, with 1 failed.

[Maciej Kisielewski (kissiel)]

Very nice! +1

review: Approve

[Sylvain Pineau (sylvain-pineau)]

I'd keep resources and attachments in the big picture as they can fail (generating less jobs for example or lead to other tricky side effects). This patchset ignores/filters them and I don't think it's a good idea if we are basing automated reviews exclusively on "real" jobs.

Where should we tweak reports to add the missing resource/attachments and keep the existing junit.xml?

review: Needs Information

[Paul Larson (pwlars)]

The problem isn't with the results summary, as that uses the submission.json and results for artifacts/resource already don't show up in the submission.json. The problem I was trying to solve with this is that when you look at c3 results and at jenkins results for the same test, they often differ greatly in the number of tests executed, failed, etc. The goal is to make those numbers match so that it's less confusing. I don't think jenkins (nor junit) give us a good ability to set up another class of tests that gets filtered from the main results

diff --git a/plainbox/impl/providers/exporters/data/junit.xml b/plainbox/impl/providers/exporters/data/junit.xml
index 2a7e64a..2ac1115 100644
--- a/plainbox/impl/providers/exporters/data/junit.xml
+++ b/plainbox/impl/providers/exporters/data/junit.xml
@@ -1,15 +1,13 @@
 {%- set state = manager.default_device_context.state -%}
 {%- set job_state_map = state.job_state_map -%}
-{%- set passes = manager.state.get_outcome_stats()["pass"] -%}
-{%- set fails = manager.state.get_outcome_stats()["fail"] -%}
-{%- set skips = manager.state.get_outcome_stats()["skip"] + 
-                manager.state.get_outcome_stats()["not-supported"] -%} 
-{%- set errors = manager.state.get_outcome_stats()["crash"] -%}
+{%- set passes = manager.state.get_test_outcome_stats()["pass"] -%}
+{%- set fails = manager.state.get_test_outcome_stats()["fail"] -%}
+{%- set skips = manager.state.get_test_outcome_stats()["skip"] + 
+                manager.state.get_test_outcome_stats()["not-supported"] -%} 
+{%- set errors = manager.state.get_test_outcome_stats()["crash"] -%}
 <?xml version="1.0" encoding="UTF-8"?>
-  <testsuites failures="{{ fails }}" name="" tests="{{ passes }}" skipped="{{ skips }}" errors="{{ errors }}">
-    {%- for job_id in job_state_map|sort %}
-    {%- set job_state = job_state_map[job_id] %}
-    {%- if job_state.result.outcome %}
+  <testsuites failures="{{ fails }}" name="" tests="{{ passes+fails+skips+errors }}" skipped="{{ skips }}" errors="{{ errors }}">
+    {%- for job_id, job_state in job_state_map|dictsort if job_state.result.outcome != None and job_state.job.plugin not in ("resource", "attachment") %}
     <testcase classname="{{ job_state.job.id }}" name="{{ job_state.job.id }}" time="{{ job_state.result.execution_duration }}">
     {%- if job_state.result.outcome == 'skip' or job_state.result.outcome == 'not-supported' -%}
       <skipped />
@@ -23,6 +21,5 @@
       </error>
     {%- endif %}
     </testcase>
-    {%- endif -%}
     {%- endfor %}
   </testsuites>
diff --git a/plainbox/impl/session/state.py b/plainbox/impl/session/state.py
index 9993689..63238e4 100644
--- a/plainbox/impl/session/state.py
+++ b/plainbox/impl/session/state.py
@@ -1297,6 +1297,26 @@ class SessionState:
             stats[job_state.result.outcome] += 1
         return stats
 
+    def get_test_outcome_stats(self):
+        """
+        Process the JobState map to get stats about the job outcomes
+        excluding attachment and resource jobs.
+
+        :returns:
+            a mapping of "outcome": "total" key/value pairs
+
+        .. note::
+            Only the outcomes seen during this session are reported, not all
+            possible values (such as crash, not implemented, ...).
+        """
+        stats = collections.defaultdict(int)
+        for job_id, job_state in self.job_state_map.items():
+            if (not job_state.result.outcome or
+                job_state.job.plugin in ("resource", "attachment")):
+                continue
+            stats[job_state.result.outcome] += 1
+        return stats
+
     @property
     def category_map(self):
         """Map from category id to their corresponding translated names."""