Merge ~bettyl/plainbox-provider-checkbox:add-after-suspend-manual-thunderbolt-thumderbolt3-tests into plainbox-provider-checkbox:master

[Betty Lin]

I have tested this in a stock Ubuntu 18.04 with adding "after-suspend-thunderbolt-cert" to somerville-iev-full.pxu.
When I do:
$ checkbox-cli list-bootstrapped com.canonical.ce::somerville-iev-full
The thunderbolt/thunderbolt3 related jobs will be listed:
com.canonical.certification::after-suspend-manual-thunderbolt/insert
com.canonical.certification::after-suspend-manual-thunderbolt/storage-test
com.canonical.certification::after-suspend-manual-thunderbolt/remove
com.canonical.certification::after-suspend-manual-thunderbolt3/insert
com.canonical.certification::after-suspend-manual-thunderbolt3/storage-test
com.canonical.certification::after-suspend-manual-thunderbolt3/remove

[Jonathan Cave (jocave)]

Although this will probably work as you have shown in your testing, I think it might be sensible not to rely on the after-suspend flags in this case.

The depends fields in the jobs are trying to ensure the jobs are run in a series with the storage-test and remove jobs depending on the insert job.

For example after-suspend-manual-thunderbolt/storage-test should have a depends field identifying after-suspend-manual-thunderbolt/insert. I think I might consider using a "sibling" job in this case.

The test plan modifications are fine.

review: Needs Fixing

[Betty Lin (bettyl)]

@Jonathan

I add the sibling job, please take a look.
Also, I tested them, here is the test report:
https://certification.canonical.com/hardware/201810-26511/submission/145268/test-results/

I just tried thunderbolt3, I believe that thunderbolt will get the same result, thanks!

review: Needs Resubmitting

[Jonathan Cave (jocave)]

Thanks this is very close to what I was thinking of. One change needed however, can you add an (extra) depends on the job suspend/suspend_advanced to make sure these occur after the suspend event.

[Betty Lin (bettyl)]

@Jonathan

Since thunderbolt/storage-test, thunderbolt/remove, hunderbolt3/storage-test, thunderbolt3/remove depend on thunderbolt/insert and thunderbolt3/insert, I only add suspend/suspend_advanced to them.

Test report:
https://certification.canonical.com/hardware/201810-26511/submission/145321/

review: Needs Resubmitting

[Jonathan Cave (jocave)]

Great, thanks for working through the revisions with me.

review: Approve

diff --git a/units/thunderbolt/jobs.pxu b/units/thunderbolt/jobs.pxu
index 14fa3f8..74481fe 100644
--- a/units/thunderbolt/jobs.pxu
+++ b/units/thunderbolt/jobs.pxu
@@ -5,6 +5,11 @@ imports: from com.canonical.plainbox import manifest
 requires: manifest.has_thunderbolt == 'True'
 estimated_duration: 20.0
 command: removable_storage_watcher insert --timeout 40 scsi
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt/insert",
+      "_summary": "thunderbolt/insert after suspend",
+      "depends": "suspend/suspend_advanced"}
+    ]
 _summary: Storage insert detection on Thunderbolt
 _description:
  PURPOSE:
@@ -27,6 +32,11 @@ requires: manifest.has_thunderbolt == 'True'
 depends: thunderbolt/insert
 estimated_duration: 45.0
 command: removable_storage_test -s 268400000 scsi
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt/storage-test",
+      "_summary": "thunderbolt/storage-test after suspend",
+      "depends": "after-suspend-manual-thunderbolt/insert"}
+    ]
 _summary: Storage test on Thunderbolt
 _description:
  This is an automated test which performs read/write operations on an attached
@@ -41,6 +51,11 @@ depends: thunderbolt/insert
 estimated_duration: 10.0
 command: removable_storage_watcher remove scsi
 _summary: Storage removal detection on Thunderbolt
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt/remove",
+      "_summary": "thunderbolt/remove after suspend",
+      "depends": "after-suspend-manual-thunderbolt/insert"}
+    ]
 _description:
  PURPOSE:
      This test will check the system can detect the removal of a Thunderbolt HDD
@@ -82,6 +97,11 @@ imports: from com.canonical.plainbox import manifest
 requires: manifest.has_thunderbolt3 == 'True'
 estimated_duration: 20.0
 command: removable_storage_watcher insert --timeout 40 scsi
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt3/insert",
+      "_summary": "thunderbolt3/insert after suspend",
+      "depends": "suspend/suspend_advanced"}
+    ]
 _summary: Storage insert detection on Thunderbolt 3 port
 _description:
  PURPOSE:
@@ -104,6 +124,11 @@ requires: manifest.has_thunderbolt3 == 'True'
 depends: thunderbolt3/insert
 estimated_duration: 45.0
 command: removable_storage_test -s 268400000 scsi
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt3/storage-test",
+      "_summary": "thunderbolt3/storage-test after suspend",
+      "depends": "after-suspend-manual-thunderbolt3/insert"}
+    ]
 _summary: Storage test on Thunderbolt 3
 _description:
  This is an automated test which performs read/write operations on an attached
@@ -117,6 +142,11 @@ requires: manifest.has_thunderbolt3 == 'True'
 depends: thunderbolt3/insert
 estimated_duration: 10.0
 command: removable_storage_watcher remove scsi
+_siblings: [
+    { "id": "after-suspend-manual-thunderbolt3/remove",
+      "_summary": "thunderbolt3/remove after suspend",
+      "depends": "after-suspend-manual-thunderbolt3/insert"}
+    ]
 _summary: Storage removal detection on Thunderbolt 3 port
 _description:
  PURPOSE:
diff --git a/units/thunderbolt/test-plan.pxu b/units/thunderbolt/test-plan.pxu
index a0efcda..f80a0ad 100644
--- a/units/thunderbolt/test-plan.pxu
+++ b/units/thunderbolt/test-plan.pxu
@@ -8,6 +8,15 @@ nested_part:
  com.canonical.certification::thunderbolt-cert-manual
  com.canonical.certification::thunderbolt-cert-automated
 
+id: after-suspend-thunderbolt-cert-full
+unit: test plan
+_name: Thunderbolt tests
+_description:
+ Thunderbolt tests
+include:
+nested_part:
+ com.canonical.certification::after-suspend-thunderbolt-cert
+
 id: thunderbolt-cert-manual
 unit: test plan
 _name: Thunderbolt tests (Manual)
@@ -37,3 +46,27 @@ include:
  thunderbolt3/insert                         certification-status=blocker
  thunderbolt3/storage-test                   certification-status=blocker
  thunderbolt3/remove                         certification-status=blocker
+
+id: after-suspend-thunderbolt-cert
+unit: test plan
+_name: Thunderbolt tests (after suspend Manual)
+_description:
+ Thunderbolt tests (after suspend Manual)
+include:
+ after-suspend-manual-thunderbolt/insert                         certification-status=non-blocker
+ after-suspend-manual-thunderbolt/storage-test                   certification-status=non-blocker
+ after-suspend-manual-thunderbolt/remove                         certification-status=non-blocker
+ after-suspend-manual-thunderbolt3/insert                        certification-status=blocker
+ after-suspend-manual-thunderbolt3/storage-test                  certification-status=blocker
+ after-suspend-manual-thunderbolt3/remove                        certification-status=blocker
+
+id: after-suspend-thunderbolt-cert-blockers
+unit: test plan
+_name: Thunderbolt tests (after suspend - certification blockers only)
+_description: Thunderbolt tests (after suspend - certification blockers only)
+include:
+ after-suspend-manual-thunderbolt3/insert                         certification-status=blocker
+ after-suspend-manual-thunderbolt3/storage-test                   certification-status=blocker
+ after-suspend-manual-thunderbolt3/remove                         certification-status=blocker
+
+