Merge lp:~jeffmarcom/opencompute/add-idle-and-syslog-check-tests into lp:opencompute/checkbox

[Jeff Marcom]

Added a couple of idle verification and syslog error check tests to the opencompute ready whitelist

This adds a couple of idle verification and syslog error check tests to the opencompute ready whitelist.

[lp:~jeffmarcom/opencompute/add-idle-and-syslog-check-tests]

2154. By Jeff Marcom on 2013-09-25

Added 12hr idle, idle check, and syslog check tests to opencompute ready whitelist

Signed-off-by: Jeff Marcom <email address hidden>

2155. By Jeff Marcom on 2013-09-25

Added job definitions for 12hr idle and syslog tests

Signed-off-by: Jeff Marcom <email address hidden>

2156. By Jeff Marcom on 2013-09-25

Updated changelog

Signed-off-by: Jeff Marcom <email address hidden>

[Jeff Marcom (jeffmarcom)]

rebased to fix changelog conflict

[Jeff Lane  (bladernr)]

I wonder about the placement of the 12 hour idle test.

Your assumption seems to be that the tester will idle the machine for > 12 hours before running checkbox, as evidinced by the idle check test and syslog check to look for errors. However, what if the tester idles for 12 hours then reboots the machine before starting checkbox? They wont necessarily know to not do that until the idle test itself runs after idle_check and syslog_check.

What about actually putting that at the end of the test run and making it all automatic:
at end of the run do a manual test for 12 hour IDLE that looks like this:
description:
 PURPOSE:
  To test that they system idles for 12 hours minimum
 STEPS:
  Select Test to begin the 12 hour idle test. Wait the prescribed amount of time and the
  testing should complete automatically
 VERIFICATION:
  The verification of this test is automatic
command:
 for x in `seq 1 12`; do echo "12 Hour Idle Test: Hour $x"; sleep 1h; done && True

followed up by idle_check and syslog_check just before finishing the run.

That, at least, guarantees that all the bits will work. My concern is that the way it's currently laid out is brittle (not sure if my idea is that much less brittle, but it seems so to me, feel free to poke holes)

review: Needs Information

[Jeff Marcom (jeffmarcom)]

> I wonder about the placement of the 12 hour idle test.
>
> Your assumption seems to be that the tester will idle the machine for > 12
> hours before running checkbox, as evidinced by the idle check test and syslog
> check to look for errors. However, what if the tester idles for 12 hours then
> reboots the machine before starting checkbox? They wont necessarily know to
> not do that until the idle test itself runs after idle_check and syslog_check.

That's why I tried to include a manual verification test. I assume it may happen.
>
> What about actually putting that at the end of the test run and making it all
> automatic:
> at end of the run do a manual test for 12 hour IDLE that looks like this:
> description:
> PURPOSE:
> To test that they system idles for 12 hours minimum
> STEPS:
> Select Test to begin the 12 hour idle test. Wait the prescribed amount of
> time and the
> testing should complete automatically
> VERIFICATION:
> The verification of this test is automatic
> command:
> for x in `seq 1 12`; do echo "12 Hour Idle Test: Hour $x"; sleep 1h; done &&
> True
>
> followed up by idle_check and syslog_check just before finishing the run.
>
> That, at least, guarantees that all the bits will work. My concern is that
> the way it's currently laid out is brittle (not sure if my idea is that much
> less brittle, but it seems so to me, feel free to poke holes)

The problem is that I believe they want checkbox to actually perform testing for ready in ~ 4 hours.

[Jeff Marcom (jeffmarcom)]

I don't think it's "brittle" in that the test works as advertised...I would agree that the test is extremely bare-bones...but so is the test itself really.

[Jeff Lane  (bladernr)]

Fair enough, though I'd really like them to provide feedback on these things too, we can change it as necessary later on.

review: Approve

=== modified file 'data/whitelists/opencompute-ready-local.whitelist'
--- data/whitelists/opencompute-ready-local.whitelist	2013-09-25 17:30:41 +0000
+++ data/whitelists/opencompute-ready-local.whitelist	2013-09-25 23:51:31 +0000
@@ -68,6 +68,9 @@
 memory/mcelog_check
 __miscellanea__
 miscellanea/ipmi_test
+miscellanea/idle_check
+miscellanea/syslog_check
+miscellanea/12hr_idle_verify
 __networking__
 networking/detect
 networking/bandwidth

=== modified file 'debian/changelog'
--- debian/changelog	2013-09-25 22:41:35 +0000
+++ debian/changelog	2013-09-25 23:51:31 +0000
@@ -5,6 +5,7 @@
   * Updated plainbox based on version 0.4.dev in lp:checkbox (16.12)
   * Updated checkbox OCP intro prompt
   * Updated Open Compute ready whitelist with new power management, cpu stress, and networking tests"
+  * Added 12hr idle verification test and sylog check for PCI/Device errors
 
   [ Jeff Lane ]
   * Updated OCP Checkbox to latest checkbox trunk, 0.16.11 revno 2353

=== modified file 'jobs/miscellanea.txt.in'
--- jobs/miscellanea.txt.in	2013-09-08 04:14:01 +0000
+++ jobs/miscellanea.txt.in	2013-09-25 23:51:31 +0000
@@ -1,3 +1,27 @@
+plugin: shell
+name: miscellanea/syslog_check
+command: SYSLOG_ERROR=`grep -i "${acpi,device} error" /var/log/syslog -A 20 -B 20`; echo $SYSLOG_ERROR >> /dev/stderr; [ -z "$SYSLOG_ERROR" ]
+_description:
+ Checks the system log for any critical pci/pcie or acpi event errors
+
+plugin: shell
+name: miscellanea/idle_check
+command: [ `uptime | awk '{split($0,a,":"); print a[1]}'` -lt 12 ]
+
+plugin: manual
+name: miscellanea/12hr_idle_verify
+depends: miscellanea/idle_check
+_description:
+ PURPOSE:
+   Warning. We could not verify that the system has already completed a 12 hour idle session.
+   This test will ask you to verify the system has remained active and booted to an OS for 12+ hours.
+ STEPS:
+   1. Prior to running this test, you should have attempted to let the machine idle for more than 12 hrs while booted to an OS.
+ VERIFICATION:
+   1. Select Yes if the system remained on and booted to an operating system for 12 hours without error.
+   2. Select No if you have not allowed the system to be on and booted to an operating system for 12 hours.
+   3. Select No if you attempted to let the system remain idle for 12 hours and it failed to do so for any reason/
+
 plugin: manual
 name: miscellanea/tester-info
 _description: