Merge lp:~cypressyew/checkbox/fwts-hwe-test into lp:checkbox

[Po-Hsu Lin]

Pull the HWE tests out of the CERT tests into a stand-alone list with an --hwe option to trigger it.
And call the remaining CERT tests as "QA" tests, with --qa option.
(CERT = HWE + QA)

We could run the fwts tests for QA and HWE individually, so that we can know which one is failing.

Now it won't split log for hwe, as we can explicitly assign the test and the log filename. e.g.
fwts --hwe -l fwts_result_hwe.log
fwts --qa -l fwts_result.log

For server, if we don't need to split the extra hwe log, we can simply run:
fwts -l fwts_result.log

This require further changes in test plans.

[Jeff Lane  (bladernr)]

Quick look and it looks good to me. Thanks for re-working this based on my comments about splitting the test lists out rather than the logs. Still needs Sylvain to ack, but I +1.

review: Approve

[Sylvain Pineau (sylvain-pineau)]

Tanks for the feedback Jeff, +1 from me too

review: Approve

=== modified file 'providers/plainbox-provider-checkbox/bin/fwts_test'
--- providers/plainbox-provider-checkbox/bin/fwts_test	2015-08-26 08:01:19 +0000
+++ providers/plainbox-provider-checkbox/bin/fwts_test	2015-09-01 09:46:27 +0000
@@ -18,26 +18,20 @@
                      'power_button',
                      'brightness',
                      'lid']
-# These are usually performed on normal certification runs
-CERT_TESTS = ['acpitests',
-              'acpidump',
-              'acpitables',
-              'apicedge',
-              'apicinstance',
-              'aspm',
-              'bios32',
-              'dmicheck',
-              'ebda',
-              'klog',
-              'mpcheck',
-              'msr',
-              'mtrr',
-              'nx',
-              'oops',
-              'uefibootpath',
-              'uefirtvariable',
-              'version',
-              'virt']
+# These are performed on QA certification runs
+QA_TESTS = ['acpitests',
+            'acpidump',
+            'acpitables',
+            'apicinstance',
+            'aspm',
+            'bios32',
+            'dmicheck',
+            'ebda',
+            'mpcheck',
+            'msr',
+            'nx',
+            'uefirtvariable',
+            'version']
 # These are advanced tests that shouldn't affect certification status
 NON_CERT_TESTS = ['bios_info',
                   'cmosdump',
@@ -67,8 +61,6 @@
                   'uefirttime',
                   'uefivarinfo'
                   ]
-TESTS = sorted(CERT_TESTS + NON_CERT_TESTS)
-
 # The following tests will record logs in a separate file for the HWE team
 HWE_TESTS = ['mtrr',
              'virt',
@@ -76,6 +68,9 @@
              'klog',
              'oops',
              'uefibootpath']
+CERT_TESTS = sorted(QA_TESTS + HWE_TESTS)
+TESTS = sorted(QA_TESTS + NON_CERT_TESTS + HWE_TESTS)
+
 
 def get_sleep_times(start_marker, end_marker, sleep_time, resume_time):
     logfile = '/var/log/syslog'
@@ -166,10 +161,10 @@
                    '--s3-quirks\n'
                    '--s3-sleep-delay\n'
                    '--s3power-sleep-delay\n\n'
-                   'Example: fwts-test --sleep s3 --s3-min-delay 30 '
+                   'Example: fwts_test --sleep s3 --s3-min-delay 30 '
                    '--s3-multiple 10 --s3-device-check\n\n'
                    'For further help with sleep options:\n'
-                   'fwts-test --fwts-help')
+                   'fwts_test --fwts-help')
     parser = ArgumentParser(description=description_text,
                             epilog=epilog_text,
                             formatter_class=RawTextHelpFormatter)
@@ -226,6 +221,12 @@
                              'For info on these extra fwts options, please \n'
                              'see the epilog below and \n'
                              'the --fwts-help option.'))
+    group.add_argument('--hwe',
+                       action='store_true',
+                       help='Run HWE concerned tests in fwts')
+    group.add_argument('--qa',
+                       action='store_true',
+                       help='Run QA concerned tests in fwts')
     group.add_argument('--fwts-help',
                        dest='fwts_help',
                        action='store_true',
@@ -239,6 +240,12 @@
     group.add_argument('--list-advanced',
                        action='store_true',
                        help='List all advanced tests in fwts.')
+    group.add_argument('--list-hwe',
+                       action='store_true',
+                       help='List all HWE concerned tests in fwts')
+    group.add_argument('--list-qa',
+                       action='store_true',
+                       help='List all QA concerned tests in fwts')
     args = parser.parse_args()
 
     tests = []
@@ -279,10 +286,20 @@
     elif args.list_advanced:
         print('\n'.join(NON_CERT_TESTS))
         return 0
+    elif args.list_hwe:
+        print('\n'.join(HWE_TESTS))
+        return 0
+    elif args.list_qa:
+        print('\n'.join(QA_TESTS))
+        return 0
     elif args.test:
         tests.extend(args.test)
     elif args.all:
         tests.extend(['wakealarm', 'cpufreq', 'maxfreq'] + TESTS)
+    elif args.hwe:
+        tests.extend(HWE_TESTS)
+    elif args.qa:
+        tests.extend(QA_TESTS)
     elif args.sleep:
         args.sleep = fix_sleep_args(args.sleep)
         iterations = 1
@@ -382,8 +399,6 @@
             # ACPI tests can now be run with --acpitests (fwts >= 15.07.00)
             log = args.log
             # Split the log file for HWE (only if -t is not used)
-            if not args.test and test in HWE_TESTS:
-                log = log.replace('.log', '_hwe.log')
             if test == 'acpitests':
                 test = '--acpitests'
             command = ('fwts -q --stdout-summary -r %s %s'

=== modified file 'providers/plainbox-provider-checkbox/jobs/firmware.txt.in'
--- providers/plainbox-provider-checkbox/jobs/firmware.txt.in	2015-08-10 17:51:39 +0000
+++ providers/plainbox-provider-checkbox/jobs/firmware.txt.in	2015-09-01 09:46:27 +0000
@@ -41,11 +41,24 @@
   package.name == 'fwts'
 user: root
 _description:
- Run Firmware Test Suite (fwts) desktop-specific diagnosis tests.
-_summary: Run FWTS desktop-specific diagnosis tests.
-environ: PLAINBOX_SESSION_SHARE
-command:
- fwts_test -l $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log
+ Run Firmware Test Suite (fwts) QA-concerned desktop-specific diagnosis tests.
+_summary: Run FWTS QA-concerned desktop-specific diagnosis tests.
+environ: PLAINBOX_SESSION_SHARE
+command:
+ fwts_test --qa -l $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log
+
+plugin:shell
+id: firmware/fwts_desktop_diagnosis_hwe
+estimated_duration: 5.0
+requires:
+  package.name == 'fwts'
+user: root
+_description:
+ Run Firmware Test Suite (fwts) HWE-concerned desktop-specific diagnosis tests.
+_summary: Run FWTS HWE-concerned desktop-specific diagnosis tests.
+environ: PLAINBOX_SESSION_SHARE
+command:
+ fwts_test --hwe -l $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results_hwe.log
 
 plugin: attachment
 estimated_duration: 0.5