Code review comment for ~sylvain-pineau/checkbox-ng:fix-1817029

Revision history for this message
Pierre Equoy (pieq) wrote :

Tested the latest fix like so:

1. checkbox-cli, select 18.04 test plan, then a few jobs from it (e.g. optical/detect ...)
2. during the execution, mark one of the manual jobs as Failed
→ You get a submission (let's call it sub1.tar.xz)
3. re-launch checkbox-cli, select the same test plans and the same few jobs
4. during the execution, mark the job you marked as Failed in the previous run as Passed
→ You get a submission (let's call it sub2.tar.xz)
5. Let's try to merge one submission with itself to make sure we get the same results:

$ checkbox-cli merge-submissions -o /tmp/selfmerge.tar.xz sub1.tar.xz sub1.tar.xz

→ Both HTML and JSON files are the same in selfmerge.tar.xz and sub1.tar.xz (OK)

6. Now let's merge 2 different submissions to make sure the feature actually works as expected :)

$ checkbox-cli merge-submissions -o /tmp/selfmerge.tar.xz sub1.tar.xz sub2.tar.xz

→ Since I have selected exactly the same jobs for sub1 and sub2, and marked the job as Passed in sub2, selfmerge.tar.xz and sub2.tar.xz HTML and json files are identical (OK)

7. Let's merge the submissions, but in reverse order:

$ checkbox-cli merge-submissions -o /tmp/selfmerge.tar.xz sub2.tar.xz sub1.tar.xz

This time, selfmerge.tar.xz HTML and json files are similar to the ones in sub1.tar.xz (OK)

+1 to land this.

Thanks a lot Sylvain !

(I've noticed the submission title disappearance, but I think we'll live without it :))

review: Approve

« Back to merge proposal