Merge lp:~sylvain-pineau/checkbox/fwts_test_split into lp:checkbox

[Sylvain Pineau]

This MR updates the fwts_tests based on HWE recommendations for certification.

The major change is the use of the new option --acpitests, selecting automatically all ACPI related tests.

To ease log review, the logs from firmware/fwts_desktop_diagnosis are split into attachments:

    - firmware/fwts_desktop_diagnosis_results.log
    - firmware/fwts_desktop_diagnosis_results_hwe.log

Again following the HWE recommendation. S3 and S4 were also part of the HWE selection but already log
in dedicated files.

[lp:~sylvain-pineau/checkbox/fwts_test_split]

3922. By Maciej Kisielewski on 2015-07-30

"automatic merge of lp:~kissiel/checkbox/error-logging-in-cbt-fixes/ by tarmac [r=zyga][bug=][author=kissiel]"

3923. By Christopher Lee on 2015-07-30

"Remove question comment from Transports (certificate and launchpad) [r=zyga][bug=][author=veebers]"

3924. By Christopher Lee on 2015-07-30

"Assign to settings once to avoid conditional memory leak [r=sylvain-pineau][bug=][author=veebers]"

[Zygmunt Krynicki (zyga)]

Just one small thing

[lp:~sylvain-pineau/checkbox/fwts_test_split]

3925. By Sylvain Pineau on 2015-07-30

providers:checkbox:fwts_test: Update test selection based on HWE recommendations

Select all ACPI test with --acpitests, removing doubled values from the initial
list. The --acpitests is exposed as just acpitests with the -l option as some
jobs used the --list output as variable name to create new jobs.
Remove the deprecated test hpet_check.
Add dmicheck and uefibootpath based on recommendations.

3926. By Sylvain Pineau on 2015-07-30

providers:checkbox:fwts_test: Save logs meant for HWE analysis into a different file

The selection is mtrr, virt, apicedge, klog, oops and uefibootpath.
S3 and S4 are already saved in dedicated logs.

3927. By Sylvain Pineau on 2015-07-30

providers:checkbox:firmware.txt.in: Remove fwts_desktop_diagnosis selection

The tests to run are defined in fwts_test.
By default all cert test are selected.

3928. By Sylvain Pineau on 2015-07-30

providers:checkbox:firmware.txt.in: Add the fwts logs for HWE.

3929. By Sylvain Pineau on 2015-07-30

providers:certification-client: Add the new fwts log for HWE to 14.04.3 testplan

[Sylvain Pineau (sylvain-pineau)]

I've removed the unneeded braces, reposting

review: Needs Resubmitting

[Zygmunt Krynicki (zyga)]

This looks good, thanks, +1

review: Approve

=== modified file 'providers/plainbox-provider-certification-client/units/client-cert-14-04-3.pxu'
--- providers/plainbox-provider-certification-client/units/client-cert-14-04-3.pxu	2015-07-22 12:46:21 +0000
+++ providers/plainbox-provider-certification-client/units/client-cert-14-04-3.pxu	2015-07-30 12:20:04 +0000
@@ -367,6 +367,7 @@
     firmware/fwts_logs
     firmware/fwts_desktop_diagnosis
     firmware/fwts_desktop_diagnosis_results.log
+    firmware/fwts_desktop_diagnosis_results_hwe.log
     firmware/fwts_wakealarm.*                              certification-status=blocker
     firmware/fwts_uefirtvariable.*                         certification-status=blocker
     firmware/no_ACPI_REV_interface                         certification-status=blocker

=== modified file 'providers/plainbox-provider-checkbox/bin/fwts_test'
--- providers/plainbox-provider-checkbox/bin/fwts_test	2014-10-07 08:47:52 +0000
+++ providers/plainbox-provider-checkbox/bin/fwts_test	2015-07-30 12:20:04 +0000
@@ -19,38 +19,31 @@
                      'brightness',
                      'lid']
 # These are usually performed on normal certification runs
-CERT_TESTS = ['acpidump',
+CERT_TESTS = ['acpitests',
+              'acpidump',
               'acpitables',
               'apicedge',
               'apicinstance',
               'aspm',
               'bios32',
-              'checksum',
-              'cstates',
-              'dmar',
+              'dmicheck',
               'ebda',
-              'fadt',
-              'hpet_check',
               'klog',
-              'mcfg',
-              'method',
               'mpcheck',
               'msr',
               'mtrr',
               'nx',
               'oops',
+              'uefibootpath',
               'uefirtvariable',
               'version',
-              'virt',
-              'wmi']
+              'virt']
 # These are advanced tests that shouldn't affect certification status
-NON_CERT_TESTS = ['acpiinfo',
-                  'bios_info',
+NON_CERT_TESTS = ['bios_info',
                   'cmosdump',
                   'crs',
                   'crsdump',
                   'csm',
-                  'dmicheck',
                   'ebdadump',
                   'fan',
                   'gpedump',
@@ -62,7 +55,6 @@
                   'mpdump',
                   'os2gap',
                   'osilinux',
-                  'pcc',
                   'pciirq',
                   'plddump',
                   'pnp',
@@ -77,6 +69,13 @@
                   ]
 TESTS = sorted(CERT_TESTS + NON_CERT_TESTS)
 
+# The following tests will record logs in a separate file for the HWE team
+HWE_TESTS = ['mtrr',
+             'virt',
+             'apicedge',
+             'klog',
+             'oops',
+             'uefibootpath']
 
 def get_sleep_times(start_marker, end_marker, sleep_time, resume_time):
     logfile = '/var/log/syslog'
@@ -380,8 +379,14 @@
                     results['sleep'] = 'FAILED_CRITICAL'
     else:
         for test in tests:
+            # ACPI tests can now be run with --acpitests (fwts >= 15.07.00)
+            log = args.log
+            if test in HWE_TESTS:
+                log = log.replace('.log', '_hwe.log')
+            if test == 'acpitests':
+                test = '--acpitests'
             command = ('fwts -q --stdout-summary -r %s %s'
-                       % (args.log, test))
+                       % (log, test))
             results[test] = (Popen(command, stdout=PIPE, shell=True)
                              .communicate()[0].strip()).decode()
 

=== modified file 'providers/plainbox-provider-checkbox/jobs/firmware.txt.in'
--- providers/plainbox-provider-checkbox/jobs/firmware.txt.in	2015-07-16 21:25:01 +0000
+++ providers/plainbox-provider-checkbox/jobs/firmware.txt.in	2015-07-30 12:20:04 +0000
@@ -45,16 +45,24 @@
 _summary: Run FWTS desktop-specific diagnosis tests.
 environ: PLAINBOX_SESSION_SHARE
 command:
- fwts_test -l $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log -t acpidump -t version -t acpitables -t apicinstance -t hpet_check -t mcfg -t method -t mpcheck -t msr -t mtrr -t apicedge -t klog
+ fwts_test -l $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log
 
 plugin: attachment
 estimated_duration: 0.5
 id: firmware/fwts_desktop_diagnosis_results.log
 command:
- [ -e ${PLAINBOX_SESSION_SHARE}/fwts_desktop_diagnosis_results.log ] && gzip -c ${PLAINBOX_SESSION_SHARE}/fwts_desktop_diagnosis_results.log
+ [ -e $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log ] && gzip -c $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results.log
 _description: Attaches the FWTS desktop diagnosis results log to the submission
 _summary: Attach FWTS desktop diagnosis log to submission
 
+plugin: attachment
+estimated_duration: 0.5
+id: firmware/fwts_desktop_diagnosis_results_hwe.log
+command:
+ [ -e $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results_hwe.log ] && gzip -c $PLAINBOX_SESSION_SHARE/fwts_desktop_diagnosis_results_hwe.log
+_description: Attaches the FWTS desktop diagnosis results log to the submission (to HWE)
+_summary: Attach FWTS desktop diagnosis log to submission (to HWE)
+
 plugin: shell
 id: firmware/no_ACPI_REV_interface
 _summary: No _REV interface in ACPI [DS]SDT tables