Merge ~sylvain-pineau/plainbox-provider-sru:more-nested-parts into plainbox-provider-sru:master

[Sylvain Pineau]

Update the sru (desktop variant) test plan to rely more on nested parts when possible.

Reminder: Include jobs list is run **before** all nested parts !!

Since the previous version was including suspend_advanced_auto, nested parts assuming a fresh boot where running actually after suspend. This commit reorders everything.

Tested using list-bootstrap before and after to ensure no jobs got removed (only additions, mainly attachments).

old version: https://pastebin.ubuntu.com/p/sn4B7dwG4N/
new version: https://pastebin.ubuntu.com/p/rQGKWcJbTN/

Note: sort the previous listings before using diff.

[Sylvain Pineau (sylvain-pineau)]

New test with sd and usb key plugged in:

https://pastebin.ubuntu.com/p/dx5H8W8CxQ/

Note the new templated tests:

com.canonical.certification::mediacard/storage-preinserted-disk/by-uuid/e7da8d1b-74d7-4007-9418-13013621fe6d

com.canonical.certification::usb/storage-preinserted-disk/by-uuid/2019-07-20-15-27-07-00
com.canonical.certification::usb/storage-preinserted-disk/by-uuid/044E-AC17

review: Needs Resubmitting

[Sylvain Pineau (sylvain-pineau)]

self-approved

review: Approve

diff --git a/units/sru.pxu b/units/sru.pxu
index af31086..6c5ef30 100644
--- a/units/sru.pxu
+++ b/units/sru.pxu
@@ -41,6 +41,27 @@ command: [ -f ${PLAINBOX_SESSION_SHARE}/screenshot_opencv_validation.jpg ] && ca
 _description: Attaches the screenshot captured in graphics/screenshot_opencv_validation.
 _summary: Attach the screenshot captured for the automatically validated screenshot test
 
+id: after-suspend-sru
+_name: All SRU Tests (Ubuntu Desktop) to run after suspend
+unit: test plan
+_description:
+    After suspend SRU jobs
+include:
+    suspend/suspend-single-log-attach
+    suspend/audio_after_suspend_auto
+    suspend/network_resume_time_auto
+    suspend/wifi_resume_time_auto
+    suspend/usb_storage_preinserted_after_suspend
+    suspend/record_playback_after_suspend_auto
+    after-suspend-bluetooth4/beacon_eddystone_url_.*
+    suspend/cpu_after_suspend_auto
+    suspend/memory_after_suspend_auto
+    suspend/gpu_lockup_after_suspend
+    suspend/screenshot_after_suspend
+    screenshot_after_suspend.jpg
+    disk/disk_stress_ng_.*
+    stress/cpu_stress_ng_test
+
 id: sru
 _name: All SRU Tests (Ubuntu Desktop)
 unit: test plan
@@ -48,36 +69,14 @@ _description:
     This test plan contains tests that are useful for validating a Stable
     Release Update (SRU) on Ubuntu Certified systems.  This test plan is not
     recommended for, nor will it be accepted for self-testing purposes.
-nested_part:
-    com.canonical.certification::pts
-    com.canonical.certification::stress-pm-graph
-    ethernet-automated
-    wireless-automated
-    usb-automated
 include:
+    # Following jobs all run first before the nested parts
+    # Please keep it short and avoid jobs triggering suspend or reboots
+    # Whenever possible prefer the use of nested parts instead
     miscellanea/submission-resources
-    acpi_sleep_attachment
-    codecs_attachment
-    cpuinfo_attachment
-    dmesg_attachment
-    dmidecode_attachment
-    efi_attachment
-    lspci_attachment
-    meminfo_attachment
-    modprobe_attachment
-    modules_attachment
+    info/systemd-analyze
     net_if_management_attachment
-    sysctl_attachment
-    lsmod_attachment
     audio/alsa_record_playback_automated
-    bluetooth/detect-output
-    camera/detect
-    camera/multiple-resolution-images_.*
-    cpu/scaling_test
-    cpu/scaling_test-log-attach
-    cpu/offlining_test
-    cpu/topology
-    disk/read_performance_.*
     graphics/xorg-version
     graphics/1_compiz_check_.*
     graphics/xorg-failsafe
@@ -87,38 +86,36 @@ include:
     graphics/1_screenshot_fullscreen_video_.*
     1_screenshot_fullscreen_video_.*.jpg
     install/apt-get-gets-updates
-    mediacard/sd-preinserted
-    memory/info
     miscellanea/dkms_build_validation
+    miscellanea/fan_stress_reaction
     networking/http
     networking/gateway_ping
-    power-management/tickless_idle
-    power-management/rtc
-    power-management/fwts_wakealarm
-    power-management/fwts_wakealarm-log-attach
-    bluetooth4/beacon_eddystone_url_.*
-    miscellanea/fan_stress_reaction
-    suspend/audio_before_suspend
-    suspend/cpu_before_suspend
-    suspend/network_before_suspend
-    suspend/memory_before_suspend
-    suspend/suspend_advanced_auto
-    suspend/suspend-single-log-attach
-    suspend/audio_after_suspend_auto
-    suspend/network_resume_time_auto
-    suspend/wifi_resume_time_auto
-    suspend/usb_storage_preinserted_after_suspend
-    suspend/record_playback_after_suspend_auto
-    after-suspend-bluetooth4/beacon_eddystone_url_.*
-    suspend/cpu_after_suspend_auto
-    suspend/memory_after_suspend_auto
+nested_part:
+    info-attachment-cert-full
+    bluetooth-cert-automated
+    camera-cert-automated
+    cpu-cert-automated
+    disk-automated
+    ethernet-automated
+    mediacard-automated
+    memory-automated
+    power-management-precheck-cert-automated
+    usb-automated
+    wireless-automated
+    before-suspend-reference-cert-automated
+    suspend-tp
+    after-suspend-sru
     after-suspend-wireless-automated
-    suspend/gpu_lockup_after_suspend
-    suspend/screenshot_after_suspend
-    screenshot_after_suspend.jpg
-    disk/disk_stress_ng_.*
-    stress/cpu_stress_ng_test
+    pts
+    stress-pm-graph
 bootstrap_include:
     device
     graphics_card
     net_if_management
+exclude:
+    cpu/clocktest
+    disk/stats_.*
+    disk/storage_device_.*
+    suspend/bluetooth_obex_send_before_suspend
+    graphics/1_auto_switch_card_.*
+    installer_debug.gz