Checkbox

Some HIGH/CRITICAL failures in fwts related tests cannot be detected by checkbox

Bug #1252186 reported by Po-Hsu Lin on 2013-11-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Checkbox	Fix Released	Medium	Brendan Donegan	Checkbox 2014-jan-31 "kingsley-falls"
	Firmware Test Suite	Fix Released	Critical	Colin Ian King	Firmware Test Suite 14.02.00

Bug Description

In this fwts report from 201308-14057 (12.04.3 + 3.8.0-33), there are some CRITICAL failures reported by fwts in method test, but not recognized by the test case parser.
In the final report, the fwts_test was marked as PASSED with only MEDIUM failures recorded.

CRITICAL failures: 14
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D0._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D1._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D0._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D1._STA'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0._INI'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA._INI'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D0._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.PRID.P_D1._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D0._PS0'.
  method: Detected error 'Not exist' when evaluating '\_SB_.PCI0.SATA.SECD.S_D1._PS0'.
  method: Detected error 'Type' when evaluating '\_SB_.PCI0.LPC0.ADP1._PCL'.
  method: Detected error 'Not exist' when evaluating '\_WAK'.

See original description

Tags:

Related branches

lp:ubuntu/trusty-proposed/fwts

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2013-11-18:

fwts_results.log Edit (148.5 KiB, text/plain)

Daniel Manrique (roadmr) on 2013-11-18

tags:

added: scripts

Ara Pulido (ara) on 2013-11-19

Changed in checkbox:
milestone:	none → 2013-dec-06

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2013-11-22:

fwts_results.log Edit (176.0 KiB, text/plain)

A HIGH failure in fadt test cannot be correctly parsed as well

FAILED [HIGH] FADTPM1AInvalidWidth: Test 1, FADT pm1a register has invalid bit width of 0.

In fact, not all errors correctly parsed,
please find the attached fwts_result.log and the submission.xml

(I think this one is also related in fwts_test itself, so I didn't open a new bug.
Please inform me if you think it will be better to open a new one.)

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2013-11-22:

201304-13208-submission.xml Edit (1.1 MiB, text/xml)

According to this submission file, only one Low Failure was detected.
But there are many MEDIUM failures as well.

summary:	- CRITICAL failures in fwts method test cannot be detected by checkbox + Some HIGH/CRITICAL failures in fwts related tests cannot be detected by + checkbox
description:	updated

Sylvain Pineau (sylvain-pineau) on 2013-11-29

Changed in checkbox:
milestone:	2013-dec-06 → 2013-dec-20

Revision history for this message

Brendan Donegan (brendan-donegan) wrote on 2013-12-12:

I'll have a look at this. It will be easiest to work on if I can get access to this system.

Changed in checkbox:
status:	New → In Progress
assignee:	nobody → Brendan Donegan (brendan-donegan)
importance:	Undecided → Medium

Daniel Manrique (roadmr) on 2013-12-20

Changed in checkbox:
milestone:	2013-dec-20 → 2014-01-17

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2014-01-03:

Hello Brendand,
please contact us if you need to access this system
Thanks!

Revision history for this message

Brendan Donegan (brendan-donegan) wrote on 2014-01-17:

This is the current output:

ubuntu@201308-14057:~$ /usr/share/checkbox/scripts/fwts_test -l fwts_results.log

High Failures: 2
WARNING: The following test cases were reported as high
level failures by fwts. Please review the log at
fwts_results.log for more information.
- klog
- oops
Passed: 6
- apicedge
- version
- dmar
- bios_info
- acpiinfo
- cstates
Aborted Tests: 17
WARNING: The following test cases were aborted by fwts
Please review the log at fwts_results.log for more information.
- hpet_check
- fadt
- aspm
- mpcheck
- ebda
- msr
- nx
- wmi
- virt
- mcfg
- uefirtvariable
- mtrr
- apicinstance
- checksum
- method
- bios32
- acpitables

It's curious there are so many aborted tests, but I can't seem to reproduce the initial issue.

Revision history for this message

Brendan Donegan (brendan-donegan) wrote on 2014-01-17:

That was without sudo - my bad. Here's the correct result:

ubuntu@201308-14057:~$ sudo /usr/share/checkbox/scripts/fwts_test -l fwts_results.log
[sudo] password for ubuntu:
High Failures: 2
WARNING: The following test cases were reported as high
level failures by fwts. Please review the log at
fwts_results.log for more information.
- klog
- oops
Passed: 18
- apicedge
- hpet_check
- fadt
- mpcheck
- ebda
- nx
- wmi
- version
- virt
- mcfg
- mtrr
- apicinstance
- checksum
- bios_info
- bios32
- acpitables
- acpiinfo
- cstates
Aborted Tests: 1
WARNING: The following test cases were aborted by fwts
Please review the log at fwts_results.log for more information.
- uefirtvariable

Revision history for this message

Brendan Donegan (brendan-donegan) wrote on 2014-01-17:

It seems to be the case that the test is run but not reported on - unfortunately I lost access to the system in order to be able to figure out why. Once I get access again I can debug it.

Brendan Donegan (brendan-donegan) on 2014-01-17

Changed in checkbox:
milestone:	2014-01-17 → 2014-jan-31

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2014-01-20:

#10

Hello Brendan,

I found the poor connection might be caused by the IP address conflict between two systems, it should be fine now.

Revision history for this message

Brendan Donegan (brendan-donegan) wrote on 2014-01-20:

#11

This seems to be a bug in FWTS. We use the --stdout-summary option of FWTS to get the overall status of each test - for the 'method' test on this system it's reported as SKIPPED, so we don't report on it. According to cking this is erroneous and should be FAILED in this instance. He will fix it and this bug will go away.

Changed in fwts:
status:	New → In Progress

Revision history for this message

Colin Ian King (colin-king) wrote on 2014-01-20:

#12

Fix sent to fwts-devel for review and inclusion at the next release https://lists.ubuntu.com/archives/fwts-devel/2014-January/004330.html

Changed in fwts:
importance:	Undecided → Critical
assignee:	nobody → Colin King (colin-king)
milestone:	none → 14.02.00

Revision history for this message

Colin Ian King (colin-king) wrote on 2014-01-20:

#13

A test version of fwts is available in my ppa, ppa:colin-king - this contains the fix. Is it possible for this to be tested?

Brendan Donegan (brendan-donegan) on 2014-01-29

Changed in checkbox:
status:	In Progress → Fix Committed
Changed in fwts:
status:	In Progress → Fix Committed

Revision history for this message

Colin Ian King (colin-king) wrote on 2014-02-02:

#14

committed, commit e8a1258fc8a5af43964c9c4c424dc6c77d7b9708 and released in 14.01.01

Changed in fwts:
status:	Fix Committed → Fix Released

Zygmunt Krynicki (zyga) on 2014-02-07

Changed in checkbox:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.