Merge lp:~asac/ubuntu-test-cases/default-systemsettle-test into lp:ubuntu-test-cases

[Alexander Sack]

add systemsettle test to default smoke test suite; we wait until system has settled to 99.25% average idle before continuing further testing.

be aware that the tc_control part is untested, while the script is. should be easy to adjust so please do during merge/commit at best.

[lp:~asac/ubuntu-test-cases/default-systemsettle-test]

14. By Alexander Sack on 2013-08-13

systemsettle: add run-forever option for utah timeout support and improve toplog formatting

15. By Alexander Sack on 2013-08-13

systemsettle: improve tc_control action and expected_results wording

16. By Alexander Sack on 2013-08-13

systemsettle: refactor pass/success exit code logic into trap handler

Unmerged revisions

16. By Alexander Sack on 2013-08-13

systemsettle: refactor pass/success exit code logic into trap handler

15. By Alexander Sack on 2013-08-13

systemsettle: improve tc_control action and expected_results wording

14. By Alexander Sack on 2013-08-13

systemsettle: add run-forever option for utah timeout support and improve toplog formatting

13. By Alexander Sack on 2013-08-13

rename quiesce_ variables to idle_avg_

12. By Alexander Sack on 2013-08-13

add systemsettle to earliest sensible position in tslist.run

11. By Alexander Sack on 2013-08-13

systemsettle: clean up temp files

10. By Alexander Sack on 2013-08-13

add systemsettle test that waits for idle avg >= 99.25%

9. By Paul Larson on 2013-06-18

Add a smoke test for detecting presence of the UI on ubuntu-touch

8. By Paul Larson on 2013-06-18

Fix bug #1192166 - pwd test fails on touch images

7. By Paul Larson on 2013-06-16

ping something on the local network instead of an external host

=== added directory 'ifconfig'
=== added file 'ifconfig/tc_control'
--- ifconfig/tc_control	1970-01-01 00:00:00 +0000
+++ ifconfig/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,14 @@
+description: make sure wlan0 exists
+dependencies: none
+action: |
+  1. ifconfig
+  2. Make sure there is a wlan0
+expected_results: |
+  1. there is a wlan0 on the system
+type: userland
+timeout: 60
+command: ifconfig; ifconfig | grep wlan0
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'install'
=== added file 'install/tc_control'
--- install/tc_control	1970-01-01 00:00:00 +0000
+++ install/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: apt-get install works
+dependencies: openssh-server
+action: |
+  1. apt-get install -y curl
+expected_results: |
+  1. apt-get install can install a package and ends with exit code 0
+type: userland
+timeout: 120
+command: apt-get install -y curl
+run_as: root
+#build_cmd:
+#tc_setup:
+tc_cleanup: apt-get remove -y curl

=== added file 'master.run'
--- master.run	1970-01-01 00:00:00 +0000
+++ master.run	2013-08-13 12:26:05 +0000
@@ -0,0 +1,5 @@
+---
+testsuites:
+  - name: smoke_touch
+    fetch_method: bzr  # one of 'bzr', 'bzr-export', 'dev', 'git'
+    fetch_location: lp:ubuntu-test-cases/touch

=== added directory 'netstat'
=== added file 'netstat/tc_control'
--- netstat/tc_control	1970-01-01 00:00:00 +0000
+++ netstat/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: netstat -l lists all the services 
+dependencies: none
+action: |
+  1. netstat -l
+expected_results: |
+  1. netstat shows output and exists cleanly
+type: userland
+timeout: 60
+command: netstat -l
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'ping'
=== added file 'ping/tc_control'
--- ping/tc_control	1970-01-01 00:00:00 +0000
+++ ping/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: ping -c 5 10.97.0.1
+dependencies: None
+action: |
+  1. ping -c 5 10.97.0.1
+expected_results: |
+  1. ping ends without an error
+type: userland
+timeout: 60
+command: ping -c 5 10.97.0.1
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'pwd'
=== added file 'pwd/tc_control'
--- pwd/tc_control	1970-01-01 00:00:00 +0000
+++ pwd/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,10 @@
+description: pwd works
+dependencies: none
+action: |
+  1. cd to /tmp and verify which is the current directory
+expected_results: |
+  1. Current directory is /tmp
+type: userland
+timeout: 60
+command: ./test.sh
+run_as: phablet

=== added file 'pwd/test.sh'
--- pwd/test.sh	1970-01-01 00:00:00 +0000
+++ pwd/test.sh	2013-08-13 12:26:05 +0000
@@ -0,0 +1,7 @@
+#!/bin/bash
+echo pwd is: $(pwd)
+echo Changing to /tmp...
+cd /tmp
+echo Checking that cd worked...
+echo directory is now $(pwd), should be /tmp
+/usr/bin/test $(pwd) == '/tmp'

=== added directory 'route'
=== added file 'route/tc_control'
--- route/tc_control	1970-01-01 00:00:00 +0000
+++ route/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: route shows wlan0 routes
+dependencies: none
+action: |
+  1. route | grep wlan0
+expected_results: |
+  1. route shows wlan0 routes
+type: userland
+timeout: 60
+command: route | grep wlan0
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'systemsettle'
=== added file 'systemsettle/systemsettle.sh'
--- systemsettle/systemsettle.sh	1970-01-01 00:00:00 +0000
+++ systemsettle/systemsettle.sh	2013-08-13 12:26:05 +0000
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+calc () { awk "BEGIN{ print $* }" ;}
+
+cleanup () { rm -f $top_log $vmstat_log $vmstat_log.reduced; }
+
+if test -z "$1"; then
+   echo "ERROR: you need to provide the average idle value"
+   echo "Usage: systemsettle.sh <avg-idle>"
+   echo "       - e.g. systemsettle.sh 99.25"
+   exit 129
+fi
+
+# minimum average idle level required to succeed
+idle_avg_min=$1
+
+# how many total attempts to settle the system
+settle_max=10
+
+# measurement details: vmstat $vmstat_wait $vmstat_repeat
+vmstat_wait=6
+vmstat_repeat=10
+
+# how many samples to ignore
+vmstat_ignore=1
+
+# tweak cut field by arch
+if uname -m | grep -q armv7; then
+  idle_pos=16
+elif uname -m | grep -q i.86; then
+  idle_pos=15
+else
+  echo "machine \'`uname -m`\' not supported"
+  exit 128
+fi
+
+# set and calc more runtime values
+vmstat_tail=`calc $vmstat_repeat - $vmstat_ignore`
+settle_count=0
+idle_avg=0
+
+echo "System Settle run - quiesce the system"
+echo "--------------------------------------"
+echo
+echo "  + cmd: \'vmstat $vmstat_wait $vmstat_repeat\' ignoring first $vmstat_ignore (tail: $vmstat_tail)"
+echo
+
+trap cleanup EXIT INT QUIT ILL KILL SEGV TERM
+vmstat_log=`mktemp -t`
+top_log=`mktemp -t`
+
+while test `calc $idle_avg '<' $idle_avg_min` = 1 -a "$settle_count" -lt "$settle_max"; do
+  echo Starting settle run $settle_count:
+
+  # get vmstat
+  vmstat $vmstat_wait $vmstat_repeat | tee $vmstat_log
+  cat $vmstat_log | tail -n $vmstat_tail > $vmstat_log.reduced
+
+  # log top output for potential debugging
+  echo "TOP DUMP (after settle run: $settle_count)" >> $top_log
+  echo "========================" >> $top_log
+  top -n 1 -b >> $top_log
+  echo >> $top_log
+
+  # calc average of idle field for this measurement
+  sum=0
+  count=0
+  while read line; do
+     idle=`echo $line | sed -e 's/\s\s*/ /g' | cut -d ' ' -f 15`
+     sum=`calc $sum + $idle`
+     count=`calc $count + 1`
+  done < $vmstat_log.reduced
+
+  idle_avg=`calc $sum.0 / $count.0`
+  settle_count=`calc $settle_count + 1`
+
+  echo
+  echo "Measurement:"
+  echo "  + idle level: $idle_avg"
+  echo "  + idle sum: $sum / count: $count"
+  echo
+done
+
+if test `calc $idle_avg '<' $idle_avg_min` = 1; then
+  echo "System failed to settle to target idle level ($idle_avg_min)"
+  echo "   + check out the following top log taken at each retry:"
+  cat $top_log
+  echo
+  echo "system did not settle. FAILED."
+  exit 1
+else
+  echo "system settled. SUCCESS"
+  exit 0
+fi
+

=== added file 'systemsettle/tc_control'
--- systemsettle/tc_control	1970-01-01 00:00:00 +0000
+++ systemsettle/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,10 @@
+description: check if system settles to idle average > 99.25%
+dependencies: none
+action: |
+  1. run systemsettle.sh to wait for system to become idle
+expected_results: |
+  1. run systemsettle.sh succeeds
+type: userland
+timeout: 720
+command: ./systemsettle.sh 99.25
+run_as: root

=== added file 'ts_control'
--- ts_control	1970-01-01 00:00:00 +0000
+++ ts_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,3 @@
+#build_cmd: echo building
+#ts_setup: echo setting up
+#ts_cleanup: echo cleaning up

=== added file 'tslist.run'
--- tslist.run	1970-01-01 00:00:00 +0000
+++ tslist.run	2013-08-13 12:26:05 +0000
@@ -0,0 +1,11 @@
+- test: pwd
+- test: uname
+- test: vmstat
+- test: systemsettle
+- test: netstat
+- test: ifconfig
+- test: route
+- test: ping
+- test: update
+- test: install
+- test: unity8

=== added directory 'uname'
=== added file 'uname/tc_control'
--- uname/tc_control	1970-01-01 00:00:00 +0000
+++ uname/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: uname shows an ubuntu-phablet image installed
+dependencies: CHANGE_ME
+action: |
+  1. uname -a|grep ubuntu-phablet
+expected_results: |
+  1. ubuntu-phablet is installed as expected
+type: userland
+timeout: 60
+command: uname -a | grep ubuntu-phablet
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'unity8'
=== added file 'unity8/tc_control'
--- unity8/tc_control	1970-01-01 00:00:00 +0000
+++ unity8/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,10 @@
+description: check for unity8 process
+dependencies: None
+action: |
+  1. pgrep unity8
+expected_results: |
+  1. pgrep exits with 0, indicating it found a matching process
+type: userland
+timeout: 60
+command: /usr/bin/pgrep unity8
+run_as: phablet

=== added directory 'update'
=== added file 'update/tc_control'
--- update/tc_control	1970-01-01 00:00:00 +0000
+++ update/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: apt-get update works
+dependencies: apt
+action: |
+  1. apt-get update
+expected_results: |
+  1. apt-get update works
+type: userland
+timeout: 120
+command: apt-get update
+run_as: root
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'vmstat'
=== added file 'vmstat/tc_control'
--- vmstat/tc_control	1970-01-01 00:00:00 +0000
+++ vmstat/tc_control	2013-08-13 12:26:05 +0000
@@ -0,0 +1,13 @@
+description: Execute vmstat and capture output
+dependencies: none
+action: |
+  1. vmstat
+expected_results: |
+  1. vmstat finishes without an error
+type: userland
+timeout: 60
+command: vmstat
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup: