few fields missing when merging submissions with merge-submission tool

Bug #1817029 reported by Pierre Equoy
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Next Generation Checkbox (CLI)
Fix Released
High
Sylvain Pineau

Bug Description

Version of Checkbox-ng: 1.4

Steps to reproduce:
===================

1. Download a recent submission (save it as sub1.tar.xz):

https://certification.canonical.com/hardware/201812-26755/submission/138731/

2. Try to "merge" the same submission twice:

$ checkbox-cli merge-submissions -o /tmp/final.tar.xz sub1.tar.xz sub1.tar.xz

3. Compare submission.json in both final.tar.xz and sub1.tar.xz

Expected result:
================

We end up with virtually the same JSON file in final.tar.xz and sub1.tar.xz

Actual result:
==============

Some fields are missing:

- description
- certification_status is marked as "unspecified" whereas it was "blocker" before
- category (for attachment jobs)
- kernel-cmdline
- buildstamp
- dkms_info_json
- (others might be missing such as bto-info)

Looking at checkbox-ng/plainbox/impl/providers/exporters/data/checkbox.json, it seems most of the missing fields are using `state.job_state_map`

Rationale
=========

I'm looking to reuse the same logic to create a small webapp to let users modify the content of a session (e.g. add/modify comment of a specific job), so I need the output session to be as close as possible from the original (minus the changes made by the end user, of course).

Related branches

Pierre Equoy (pieq)
Changed in checkbox-ng:
assignee: nobody → Sylvain Pineau (sylvain-pineau)
importance: Undecided → Medium
importance: Medium → High
description: updated
Pierre Equoy (pieq)
Changed in checkbox-ng:
assignee: Sylvain Pineau (sylvain-pineau) → Pierre Equoy (pieq)
status: New → In Progress
Revision history for this message
Pierre Equoy (pieq) wrote :

The reason why so many fields are missing is because when exporting a Checkbox session to a json submission, the jinja2 exporter removes the namespaces from the job id:

https://git.launchpad.net/checkbox-ng/tree/plainbox/impl/providers/exporters/data/checkbox.json#n48

When using this json submission to create a Checkbox session (before re-exporting it in json), the job ids are truncated, so when re-exporting to json, the ids "com.canonical.certification::<job_id>" cannot be found; for example, for info/buildstamp:

https://git.launchpad.net/checkbox-ng/tree/plainbox/impl/providers/exporters/data/checkbox.json#n152

the job is actually still there in the json file, but it's called 'info/buildstamp', not 'com.canonical.certification::info/buildstamp', so the jinja template doesn't find it.

Pierre Equoy (pieq)
Changed in checkbox-ng:
status: In Progress → Confirmed
Revision history for this message
Pierre Equoy (pieq) wrote :

I've tried the MP by spineau [1] with the 18.04 cert client test plan (only a subset of tests).

1. Run checkbox-cli, select 18.04 test plan, then a subset of jobs to run (see attached submission for list of jobs run)
2. Once finished, use the merge-submissions feature to merge this submission archive with itself, in order to check if the resulting submission is identical:

checkbox-cli merge-submissions -o after.tar.xz submission_2019-07-15T07.09.17.189148.tar.xz submission_2019-07-15T07.09.17.189148.tar.xz

3. Compare submission.json and submission.html files

Expected results: both files are identical
Actual results: json files are identical, but HTML report is not: some jobs (such as optical/detect) go from "not supported" to "skipped".

[1] https://code.launchpad.net/~sylvain-pineau/checkbox-ng/+git/checkbox-ng/+merge/369818

Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote :

The lost of status in the HTML report is a limitation of HEXR as it only supports pass,fail and skip statuses. When exporting to json checkbox maps for example 'not-supported' to just 'skip'.

I've fixed the export to also record the job outcome, leaving the status as is for hexr compatibility.

Changed in checkbox-ng:
milestone: none → 1.5.0
status: Confirmed → In Progress
Changed in checkbox-ng:
status: In Progress → Fix Committed
Changed in checkbox-ng:
assignee: Pierre Equoy (pieq) → Sylvain Pineau (sylvain-pineau)
Changed in checkbox-ng:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.