Merge lp:~asac/ubuntu-test-cases/use-top-and-always-dumb-toplog into lp:ubuntu-test-cases

[Alexander Sack]

+ use top in batchmode instead of vmstat
+ improve logging; echo cli arguments given as well as
+ dumb top_log regardless of success and failyure
+ top_log is now more useful to see what was going on as we dumb exactly what was measured

[lp:~asac/ubuntu-test-cases/use-top-and-always-dumb-toplog]

14. By Alexander Sack on 2013-08-19

fix idle_avg calc to deal better with non-natural numbers getting passed

15. By Alexander Sack on 2013-08-19

improve systemsettle idle measurement run logging now that we don't run vmstat anymore

16. By Alexander Sack on 2013-08-19

since we now use top instead of vmstat, let's use s/vmstat/top/ for all variables as well

17. By Alexander Sack on 2013-08-19

don't log the top command used in systemsettle anymore

Unmerged revisions

17. By Alexander Sack on 2013-08-19

don't log the top command used in systemsettle anymore

16. By Alexander Sack on 2013-08-19

since we now use top instead of vmstat, let's use s/vmstat/top/ for all variables as well

15. By Alexander Sack on 2013-08-19

improve systemsettle idle measurement run logging now that we don't run vmstat anymore

14. By Alexander Sack on 2013-08-19

fix idle_avg calc to deal better with non-natural numbers getting passed

13. By Alexander Sack on 2013-08-19

improve logging; echo cli args used and always echo top_log regardless of success/fail

12. By Alexander Sack on 2013-08-19

use top instead of vmstat; this gives us top measurements all time during run

11. By Paul Larson on 2013-08-16

Add options and extra configurability to systemsettle.sh

10. By Andy Doan on 2013-08-14

add systemsettle test that waits for idle

9. By Paul Larson on 2013-06-18

Add a smoke test for detecting presence of the UI on ubuntu-touch

8. By Paul Larson on 2013-06-18

Fix bug #1192166 - pwd test fails on touch images

=== added directory 'ifconfig'
=== added file 'ifconfig/tc_control'
--- ifconfig/tc_control	1970-01-01 00:00:00 +0000
+++ ifconfig/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,14 @@
+description: make sure wlan0 exists
+dependencies: none
+action: |
+  1. ifconfig
+  2. Make sure there is a wlan0
+expected_results: |
+  1. there is a wlan0 on the system
+type: userland
+timeout: 60
+command: ifconfig; ifconfig | grep wlan0
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'install'
=== added file 'install/tc_control'
--- install/tc_control	1970-01-01 00:00:00 +0000
+++ install/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: apt-get install works
+dependencies: openssh-server
+action: |
+  1. apt-get install -y curl
+expected_results: |
+  1. apt-get install can install a package and ends with exit code 0
+type: userland
+timeout: 120
+command: apt-get install -y curl
+run_as: root
+#build_cmd:
+#tc_setup:
+tc_cleanup: apt-get remove -y curl

=== added file 'master.run'
--- master.run	1970-01-01 00:00:00 +0000
+++ master.run	2013-08-19 10:31:04 +0000
@@ -0,0 +1,5 @@
+---
+testsuites:
+  - name: smoke_touch
+    fetch_method: bzr  # one of 'bzr', 'bzr-export', 'dev', 'git'
+    fetch_location: lp:ubuntu-test-cases/touch

=== added directory 'netstat'
=== added file 'netstat/tc_control'
--- netstat/tc_control	1970-01-01 00:00:00 +0000
+++ netstat/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: netstat -l lists all the services 
+dependencies: none
+action: |
+  1. netstat -l
+expected_results: |
+  1. netstat shows output and exists cleanly
+type: userland
+timeout: 60
+command: netstat -l
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'ping'
=== added file 'ping/tc_control'
--- ping/tc_control	1970-01-01 00:00:00 +0000
+++ ping/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: ping -c 5 10.97.0.1
+dependencies: None
+action: |
+  1. ping -c 5 10.97.0.1
+expected_results: |
+  1. ping ends without an error
+type: userland
+timeout: 60
+command: ping -c 5 10.97.0.1
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'pwd'
=== added file 'pwd/tc_control'
--- pwd/tc_control	1970-01-01 00:00:00 +0000
+++ pwd/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,10 @@
+description: pwd works
+dependencies: none
+action: |
+  1. cd to /tmp and verify which is the current directory
+expected_results: |
+  1. Current directory is /tmp
+type: userland
+timeout: 60
+command: ./test.sh
+run_as: phablet

=== added file 'pwd/test.sh'
--- pwd/test.sh	1970-01-01 00:00:00 +0000
+++ pwd/test.sh	2013-08-19 10:31:04 +0000
@@ -0,0 +1,7 @@
+#!/bin/bash
+echo pwd is: $(pwd)
+echo Changing to /tmp...
+cd /tmp
+echo Checking that cd worked...
+echo directory is now $(pwd), should be /tmp
+/usr/bin/test $(pwd) == '/tmp'

=== added directory 'route'
=== added file 'route/tc_control'
--- route/tc_control	1970-01-01 00:00:00 +0000
+++ route/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: route shows wlan0 routes
+dependencies: none
+action: |
+  1. route | grep wlan0
+expected_results: |
+  1. route shows wlan0 routes
+type: userland
+timeout: 60
+command: route | grep wlan0
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'systemsettle'
=== added file 'systemsettle/systemsettle.sh'
--- systemsettle/systemsettle.sh	1970-01-01 00:00:00 +0000
+++ systemsettle/systemsettle.sh	2013-08-19 10:31:04 +0000
@@ -0,0 +1,125 @@
+#!/bin/bash
+
+set -e
+
+# default exit code storage
+dump_error=1
+
+calc () { awk "BEGIN{ print $* }" ;}
+
+cleanup () {
+  if ! test "$dump_error" = 0; then
+    echo "Check out the following top log taken at each retry:"
+
+    echo
+    # dumb toplog indented
+    while read line; do
+      echo "  $line"
+    done < $top_log
+    # dont rerun this logic in case we get multiple signals
+    dump_error=0
+  fi
+  rm -f $top_log $vmstat_log $vmstat_log.reduced
+}
+
+function show_usage() {
+   echo "Usage:"
+   echo " $0 [options]"
+   echo "Options:"
+   echo " -r  run forever without exiting"
+   echo " -p  minimum idle percent to wait for (Default: 99)"
+   echo " -c  number of times to run vmstat at each iteration (Default: 10)"
+   echo " -d  seconds to delay between each vmstat iteration (Default: 6)"
+   echo " -i  vmstat measurements to ignore from each loop (Default: 1)"
+   echo " -m  maximum loops of vmstat before giving up if minimum idle"
+   echo "     percent is not reached (Default: 1)"
+   exit 129
+}
+
+while getopts "h?rp:c:d:i:m:" opt; do
+    case "$opt" in
+        h|\?) show_usage
+              ;;
+        r)    settle_prefix='-'
+              ;;
+        p)    idle_avg_min=$OPTARG
+              ;;
+        c)    vmstat_repeat=$OPTARG
+              ;;
+        d)    vmstat_wait=$OPTARG
+              ;;
+        i)    vmstat_ignore=$OPTARG
+              ;;
+        m)    settle_max=$OPTARG
+              ;;
+    esac
+done
+
+# minimum average idle level required to succeed
+idle_avg_min=${idle_avg_min:-99}
+# measurement details: vmstat $vmstat_wait $vmstat_repeat
+vmstat_repeat=${vmstat_repeat:-10}
+vmstat_wait=${vmstat_wait:-6}
+# how many samples to ignore
+vmstat_ignore=${vmstat_ignore:-1}
+# how many total attempts to settle the system
+settle_max=${settle_max:-10}
+
+# set and calc more runtime values
+vmstat_tail=`calc $vmstat_repeat - $vmstat_ignore`
+settle_count=0
+idle_avg=0
+
+echo "System Settle run - quiesce the system"
+echo "--------------------------------------"
+echo
+echo "  cmd            = 'top -b -d $vmstat_wait -n $vmstat_repeat' ignoring first $vmstat_ignore (tail: $vmstat_tail)"
+echo "  idle_avg_min   = '$idle_avg_min'"
+echo "  vmstat_repeat  = '$vmstat_repeat'"
+echo "  vmstat_wait    = '$vmstat_wait'"
+echo "  vmstat_ignore  = '$vmstat_ignore'"
+echo "  settle_max     = '$settle_max'"
+echo "  run_forever    = '$settle_prefix' (- = yes)"
+echo
+
+trap cleanup EXIT INT QUIT ILL KILL SEGV TERM
+vmstat_log=`mktemp -t`
+top_log=`mktemp -t`
+
+while test `calc $idle_avg '<' $idle_avg_min` = 1 -a "$settle_prefix$settle_count" -lt "$settle_max"; do
+  echo Starting settle run $settle_count:
+
+  # get vmstat
+  echo "TOP DUMP (after settle run: $settle_count)" >> $top_log
+  echo "========================" >> $top_log
+  top -b -d $vmstat_wait -n $vmstat_repeat >> $top_log
+  cat $top_log | grep '.Cpu.*' | tail -n $vmstat_tail > $vmstat_log.reduced
+  echo >> $top_log
+
+  # calc average of idle field for this measurement
+  sum=0
+  count=0
+  while read line; do
+     idle=`echo $line | sed -e 's/.* \([0-9\.]*\) id.*/\1/'`
+     sum=`calc $sum + $idle`
+     count=`calc $count + 1`
+  done < $vmstat_log.reduced
+
+  idle_avg=`calc $sum.0 / $count.0`
+  settle_count=`calc $settle_count + 1`
+
+  echo
+  echo "Measurement:"
+  echo "  + idle level: $idle_avg"
+  echo "  + idle sum: $sum / count: $count"
+  echo
+done
+
+if test `calc $idle_avg '<' $idle_avg_min` = 1; then
+  echo "system not settled. FAIL"
+  exit 1
+else
+  echo "system settled. SUCCESS"
+  exit 0
+fi
+

=== added file 'systemsettle/tc_control'
--- systemsettle/tc_control	1970-01-01 00:00:00 +0000
+++ systemsettle/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,10 @@
+description: check if system settles to idle average > 99.25%
+dependencies: none
+action: |
+  1. Take CPU load samples for 10 minutes and fail if average idle never goes above 99.25% percent
+expected_results: |
+  1. When doing nothing, system calms down to at least 99.25% idle level
+type: userland
+timeout: 720
+command: ./systemsettle.sh -p 99.25
+run_as: root

=== added file 'ts_control'
--- ts_control	1970-01-01 00:00:00 +0000
+++ ts_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,3 @@
+#build_cmd: echo building
+#ts_setup: echo setting up
+#ts_cleanup: echo cleaning up

=== added file 'tslist.run'
--- tslist.run	1970-01-01 00:00:00 +0000
+++ tslist.run	2013-08-19 10:31:04 +0000
@@ -0,0 +1,11 @@
+- test: pwd
+- test: uname
+- test: vmstat
+- test: systemsettle
+- test: netstat
+- test: ifconfig
+- test: route
+- test: ping
+- test: update
+- test: install
+- test: unity8

=== added directory 'uname'
=== added file 'uname/tc_control'
--- uname/tc_control	1970-01-01 00:00:00 +0000
+++ uname/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: uname shows an ubuntu-phablet image installed
+dependencies: CHANGE_ME
+action: |
+  1. uname -a|grep ubuntu-phablet
+expected_results: |
+  1. ubuntu-phablet is installed as expected
+type: userland
+timeout: 60
+command: uname -a | grep ubuntu-phablet
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'unity8'
=== added file 'unity8/tc_control'
--- unity8/tc_control	1970-01-01 00:00:00 +0000
+++ unity8/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,10 @@
+description: check for unity8 process
+dependencies: None
+action: |
+  1. pgrep unity8
+expected_results: |
+  1. pgrep exits with 0, indicating it found a matching process
+type: userland
+timeout: 60
+command: /usr/bin/pgrep unity8
+run_as: phablet

=== added directory 'update'
=== added file 'update/tc_control'
--- update/tc_control	1970-01-01 00:00:00 +0000
+++ update/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: apt-get update works
+dependencies: apt
+action: |
+  1. apt-get update
+expected_results: |
+  1. apt-get update works
+type: userland
+timeout: 120
+command: apt-get update
+run_as: root
+#build_cmd:
+#tc_setup:
+#tc_cleanup:

=== added directory 'vmstat'
=== added file 'vmstat/tc_control'
--- vmstat/tc_control	1970-01-01 00:00:00 +0000
+++ vmstat/tc_control	2013-08-19 10:31:04 +0000
@@ -0,0 +1,13 @@
+description: Execute vmstat and capture output
+dependencies: none
+action: |
+  1. vmstat
+expected_results: |
+  1. vmstat finishes without an error
+type: userland
+timeout: 60
+command: vmstat
+run_as: phablet
+#build_cmd:
+#tc_setup:
+#tc_cleanup: