Merge lp:~cypressyew/checkbox/fix-S3-S4-keys into lp:checkbox

[Po-Hsu Lin]

Moved the S3 and S4 key tests to where S3 / S4 has already been tested, so we don't have to change their dependency.
Also we could avoid to start S3 test before the "media card before S3" tests

[Zygmunt Krynicki (zyga)]

I don't quite understand why we're reordering the whitelist and not removing the (IMHO bogus) dependency on suspend itself.

While it's true that suspend and hibernate keys may be special and may not be applicable for "does this key work" test that simply intercepts the key and prevents its special function from working I don't understand why that key test actually needs to depend on the suspend job.

review: Needs Information

[Po-Hsu Lin (cypressyew)]

I'm ok with removing the dependecny.
But for whitelist re-ordering,
if we're just removing the dependency without re-ordering the whitelist, the system will be suspended after you start testing the suspend key (if it works).
In that case, our following media card jobs will become the "media-card-test-after-suspend", not the "before-suspend" one that we expected.

The reason why I didn't remove it, is that after the re-ordering, these key tests could safely depends on them.

[Daniel Manrique (roadmr)]

I think this is OK, we could remove the dependency on suspend but the move is still needed because, if the jobs try to honor whitelist ordering, they would run before suspend, effectively messing up a series of other tests as Sam mentioned.

Actually, there's one change I suggest making here: the keys/hibernate should run after all the *-after-suspend_* tests, otherwise it may create some strange conditions, as the before-suspend tests would run "after hibernate". Maybe we can put it before power-management/poweroff, so we cluster those jobs together (hibernate, poweroff, reboot) which should not affect anything in between.

Let me know if this makes sense.

review: Needs Information

[lp:~cypressyew/checkbox/fix-S3-S4-keys]

3083. By Po-Hsu Lin on 2014-06-19

Remove the dependency of S3/S4 keys, re-ordering the S4 and Hybird-Sleep tests

[Po-Hsu Lin (cypressyew)]

I removed the dependency, also re-ordering the whitelist a little bit:
1. Move the __firmware__ back.
2. Move hybrid_sleep to after suspend
3. Move hibernate to run after all the *-after-suspend_* tests

I think the whitelist ordering is worthy of discussion, any suggestion is welcomed.
Thanks!

review: Needs Resubmitting

[Daniel Manrique (roadmr)]

Thanks, this looks OK now.

review: Approve

=== modified file 'providers/plainbox-provider-certification-client/whitelists/client-cert.whitelist'
--- providers/plainbox-provider-certification-client/whitelists/client-cert.whitelist	2014-06-11 10:05:41 +0000
+++ providers/plainbox-provider-certification-client/whitelists/client-cert.whitelist	2014-06-19 02:14:20 +0000
@@ -77,6 +77,8 @@
 firewire/insert
 firewire/storage-test
 firewire/remove
+__firmware__
+# See Automated Tests section at the end of this file
 __graphics__
 graphics/generator_.*
 graphics/1_glxgears_.*
@@ -119,12 +121,10 @@
 keys/media-control
 keys/mute
 keys/volume
-keys/sleep
 keys/video-out
 keys/wireless
 keys/keyboard-backlight
 keys/keyboard-overhead-light
-keys/hibernate
 keys/microphone-mute
 __led__
 led/bluetooth
@@ -201,10 +201,6 @@
 power-management/tickless_idle
 power-management/fwts_wakealarm
 power-management/fwts_wakealarm-log-attach
-power-management/hibernate_advanced
-power-management/hibernate-single-log-attach
-suspend/hybrid_sleep
-suspend/hybrid-sleep-single-log-attach
 __touchpad__
 touchpad/basic
 touchpad/horizontal
@@ -256,6 +252,9 @@
 suspend/suspend_advanced
 suspend/suspend-time-check
 suspend/suspend-single-log-attach
+suspend/hybrid_sleep
+suspend/hybrid-sleep-single-log-attach
+keys/sleep
 led/power-blink-suspend
 led/suspend
 suspend/network_after_suspend
@@ -303,6 +302,9 @@
 graphics/1_switch_card_.*
 suspend/1_suspend_after_switch_to_card_.*
 suspend/1_display_after_suspend.*
+power-management/hibernate_advanced
+power-management/hibernate-single-log-attach
+keys/hibernate
 # The following tests should run BEFORE the automated tests. The reboot and
 # power off tests will also give us a clean system to start the stress run
 # with.
@@ -313,7 +315,6 @@
 # Automated Tests
 # The following tests are purely automated and/or lenghty stress tests. They
 # have been moved to the end of the test run to improve the testing process.
-__firmware__
 firmware/fwts
 firmware/fwts_logs
 firmware/fwts_desktop_diagnosis

=== modified file 'providers/plainbox-provider-certification-client/whitelists/client-selftest.whitelist'
--- providers/plainbox-provider-certification-client/whitelists/client-selftest.whitelist	2014-06-11 10:05:41 +0000
+++ providers/plainbox-provider-certification-client/whitelists/client-selftest.whitelist	2014-06-19 02:14:20 +0000
@@ -74,6 +74,8 @@
 firewire/insert
 firewire/storage-test
 firewire/remove
+__firmware__
+# See Automated Tests section at the end of this file
 __graphics__
 graphics/generator_.*
 graphics/1_glxgears_.*
@@ -116,12 +118,10 @@
 keys/media-control
 keys/mute
 keys/volume
-keys/sleep
 keys/video-out
 keys/wireless
 keys/keyboard-backlight
 keys/keyboard-overhead-light
-keys/hibernate
 keys/microphone-mute
 __led__
 led/wlan-disabled
@@ -186,10 +186,6 @@
 power-management/tickless_idle
 power-management/fwts_wakealarm
 power-management/fwts_wakealarm-log-attach
-power-management/hibernate_advanced
-power-management/hibernate-single-log-attach
-suspend/hybrid_sleep
-suspend/hybrid-sleep-single-log-attach
 __touchpad__
 touchpad/basic
 touchpad/horizontal
@@ -247,6 +243,9 @@
 suspend/suspend_advanced
 suspend/suspend-time-check
 suspend/suspend-single-log-attach
+suspend/hybrid_sleep
+suspend/hybrid-sleep-single-log-attach
+keys/sleep
 led/power-blink-suspend
 led/suspend
 suspend/network_after_suspend
@@ -291,6 +290,9 @@
 suspend/wireless_connection_after_suspend_open_n_manual
 suspend/wireless_connection_after_suspend_wpa_ac_manual
 suspend/wireless_connection_after_suspend_open_ac_manual
+power-management/hibernate_advanced
+power-management/hibernate-single-log-attach
+keys/hibernate
 # The following tests should run BEFORE the automated tests. The reboot and
 # power off tests will also give us a clean system to start the stress run
 # with.
@@ -301,7 +303,6 @@
 # Automated Tests
 # The following tests are purely automated and/or lenghty stress tests. They
 # have been moved to the end of the test run to improve the testing process.
-__firmware__
 firmware/fwts
 firmware/fwts_logs
 firmware/fwts_desktop_diagnosis

=== modified file 'providers/plainbox-provider-checkbox/jobs/keys.txt.in'
--- providers/plainbox-provider-checkbox/jobs/keys.txt.in	2014-05-09 16:53:32 +0000
+++ providers/plainbox-provider-checkbox/jobs/keys.txt.in	2014-06-19 02:14:20 +0000
@@ -61,7 +61,6 @@
 id: keys/sleep
 requires:
  device.category == 'KEYBOARD'
-depends: suspend/suspend_advanced
 _description:
  PURPOSE:
      This test will test the sleep key
@@ -197,7 +196,6 @@
 plugin: manual
 id: keys/hibernate
 requires: dmi.product in ['Notebook','Laptop','Portable']
-depends: power-management/hibernate_advanced
 _description:
  PURPOSE:
      This test will test the hibernate key