Merge lp:~dpniel/ubuntu-autopilot-tests/test-runner into lp:ubuntu-autopilot-tests

[Dan Chapman ]

This MP adds a test runner to run the autopilot tests in a pristine environment. See the enclosed README for info on usage.

It work pretty much the same as the ubiquity runner and is just a modified copy of that runner

[Nicholas Skaggs (nskaggs)]

With this runner, we should be able to provide an easier workflow for running these tests in a similar environment as jenkins. This should help in reproducing issues. I suppose this could also be helpful in verifying tests work when run in a pristine environment; probably useful to verify before final merge.

[Nicholas Skaggs (nskaggs)]

I'm curious about desktop-autopilot-runner/custom-installation/iso-override/etc/apt/sources.list

deb http://archive.ubuntu.com/ubuntu/ trusty main restricted universe multiverse

We're pushing trusty in there; is there a more generic way to specify what we want? Going forward trusty isn't going to be it, and I don't want to push updates for something so trivial.

[Nicholas Skaggs (nskaggs)]

Looks good. I know jibel likes to push things into /tmp, so I suppose I'm ok with using /tmp to store the results. I personally don't like putting longterm things in there.

Overall, I approve, just have the one question above.

review: Needs Information

[Dan Chapman  (dpniel)]

It's easy enough to either re-write/update the sources.list file. I'll get that updated.

Where would you rather the artefacts ended up? as I want it to be more of a general purpose tool for running any autopilot test than the ubiquity runner currently is.

[Nicholas Skaggs (nskaggs)]

I would rather the results ended up in /home.. /home/autopilot-testrunner?

[Dan Chapman  (dpniel)]

Sounds good to me. I'll change that

[Adam Smith (adam-disc0tech)]

It would be great to be able run files from a local path instead of LP.

[lp:~dpniel/ubuntu-autopilot-tests/test-runner]

67. By Dan Chapman  on 2014-03-04

Removed apt/sources.list and just add-apt-repo universe & multiverse instead

Artifacts are now stored in $HOME/ubuntu_autopilot.tests instead of /tmp

Updated README

[Dan Chapman  (dpniel)]

Ok so changes made.

1) Removed the hardcoded sources.list and now just enable repositories using add-apt-repository
2) Artefacts now stored in $HOME/ubuntu_autopilot.tests rather than temp directory
3) Updated README

@Adam From an MP review / Jenkins point of view pulling from bzr branches is ideal, so I don't want to make any changes in this branch. But i agree the ability to use local directories for developing tests would be a nice feature. Once this is merged if you could file a bug for a new feature that would be great :-)

[lp:~dpniel/ubuntu-autopilot-tests/test-runner]

68. By Dan Chapman  on 2014-03-04

added retrieval of upstart logs for test artefacts

69. By Dan Chapman  on 2014-03-04

added partner repository

70. By Dan Chapman  on 2014-03-04

remove partner repo as it's archive.canonical.com and not archive.ubuntu.com

[Nicholas Skaggs (nskaggs)]

Here and I thought this was waiting on you. Approved on my end ;-)

review: Approve

=== added file 'README.md'
--- README.md	1970-01-01 00:00:00 +0000
+++ README.md	2014-03-04 19:06:35 +0000
@@ -0,0 +1,113 @@
+Introduction
+============
+
+This document describes how to setup and run automated autopilot tests for desktop applications.
+The test runner works for Ubuntu, Xubuntu, Lubuntu, Ubuntu-GNOME and Edubuntu. and should work
+for all other flavours of Ubuntu.
+
+Source code: `lp:ubuntu-autopilot-tests`
+
+Tests use python3 version of autopilot. So any current tests that use python 2.7 should be ported
+to be supported by python3
+
+
+Project Structure
+=================
+
+The project is structured as follow:
+
+ * *ubuntu-autopilot-tests/* Contains the tests and the test runner
+
+   * *desktop-autopilot-runner/* Runner to setup a VM from an ISO
+     and execute autopilot tests
+
+     * *config/* Contains configuration examples to override default values of
+       the runner
+
+     * *custom-installation/iso-override/* Content of this directory will
+       override the content on the ISO
+
+   * *ubuntu_autopilot_tests/* Autopilot tests for the default applications in Ubuntu
+   
+   * *xubuntu_autopilot_tests/* Autopilot tests for the default applications in Xubuntu
+
+
+Running the tests
+=================
+
+Directly on local machine
+----------------------------
+
+Install the following dependencies :
+
+    $ sudo apt-get install python3-autopilot libautopilot-gtk python3-xlib
+
+To run the tests, go to the root folder of the flavor's tests you want to run.
+
+Example:
+
+Run in Terminal (for ubuntu):
+
+    $ cd ubiquity-autopilot-tests/ubuntu_autopilot_tests
+    $ autopilot-py3 run TESTNAME (where TESTNAME is the directory name of the application you want to test)
+
+To execute the *gedit* test suite, Run in Terminal:
+
+    $ cd ubiquity-autopilot-tests/ubuntu_autopilot_tests
+    $ autopilot-py3 run gedit
+
+Other tests are available from *ubuntu_autopilot_tests/*. To get a list
+of available tests run:
+
+    $ autopilot-py3 list ubuntu_autopilot_tests or xubuntu_autopilot_tests
+
+On a local machine using the test runner (recommended before proposing a merge for new tests)
+---------------------------------------------------------------------------------------------
+
+ * Install the following dependencies:
+
+        $ sudo apt-get install bsdtar qemu-system-x86 bzr xz-utils cpio
+
+ * Download a desktop image from *http//cdimage.ubuntu.com/*
+
+ * Execute the command:
+    
+        $ ./desktop-autopilot-runner/run-autopilot-test <ISO> 
+        e.g
+        $ ./desktop-autopilot-runner/run-autopilot-test ~/iso/ubuntu/trusty-desktop-amd64.iso
+        
+ * All tests are run from the root of the flavors test suite (eg ubuntu_autopilot_tests/*)
+   So all tests need to be abe to run when be started from this point. When specifying a
+   test/testsuite to run, a dot path to a specific test can be used as long as it starts from the
+   suites root directory
+   
+   Example:
+   
+        gedit.test_gedit.GeditWindowTests.test_window_visible
+
+
+ * If you want to watch what is running use option `--sdl`
+ 
+ * If you want to select a test suite to run use option `--test` (e.g --test fileroller)
+ 
+ * If you want need to use any extra packages or select flavor test suite this can be done
+   using a config file, like the one found in ./desktop-autopilot-runner/config/testrunner.cfg.
+   And can be used with the `-T` and `--testconfig` options. Also see config for other available options
+
+ * If your system has enough memory you can run tests in memory (in /dev/shm
+   actually) with option `-s|--shm`
+ 
+ * If you require test environment to continue running after the test,
+   you can set no shutdown in the config file.
+   
+   Note: this is still restricted by the global timeout of the test runner which can be overridden
+         by setting the TIMOUT variable when executing the runner
+         
+         Example:
+         
+         $ TIMEOUT=123412341234 ./desktop-autopilot-runner/run-autopilot-test --sdl ~/iso/ubuntu/trusty-desktop-amd64.iso
+   
+   * And of course `-h|--help` for a list of available options
+
+   * At the end of the run, results are collected in your home directory`$HOME/ubuntu_autopilot.tests`
+

=== added directory 'desktop-autopilot-runner'
=== added directory 'desktop-autopilot-runner/config'
=== added file 'desktop-autopilot-runner/config/testrunner.cfg'
--- desktop-autopilot-runner/config/testrunner.cfg	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/config/testrunner.cfg	2014-03-04 19:06:35 +0000
@@ -0,0 +1,35 @@
+#
+# Example configuration file for the test runner
+# This file allows user to override some of the variables defined in the
+# testrunner
+#
+# Branch to fetch containing autopilot tests
+TSBRANCH=lp:ubuntu-autopilot-tests
+
+# Testsuite directory for flavor under test
+FLAVOR_TESTSUITE="ubuntu_autopilot_tests"
+
+# Additional artifacts to collect at the end of the test
+ARTIFACTS="/var/log/upstart/" 
+
+# Additional options for autopilot 
+# -f xml <FILE> is always appended to this option
+AP_OPTS="-v"
+
+# By default the test environment powers off at the end of the test. Set this to
+# 0 to keep it running
+SHUTDOWN=0
+
+# Extra packages to install. EXTRAPACKAGE is appended to PACKAGES defined in the
+# runner
+EXTRAPACKAGES="eatmydata"
+
+# Test timeout.
+# Note that this apply to autopilot running inside the test environment and
+# doesn't affect the global timeout of the runner set in the main script
+TIMEOUT=600
+
+# Debug Mode
+# Set to 1 to enable debug mode. This will start an xterm in the live session
+# for example.
+DEBUG=1

=== added directory 'desktop-autopilot-runner/custom-installation'
=== added directory 'desktop-autopilot-runner/custom-installation/hooks'
=== added file 'desktop-autopilot-runner/custom-installation/hooks/casper-bottom.sh'
--- desktop-autopilot-runner/custom-installation/hooks/casper-bottom.sh	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/custom-installation/hooks/casper-bottom.sh	2014-03-04 19:06:35 +0000
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+#
+#  Put here every steps that must be executed on the target system and can not
+#  be done with the iso-override facility
+#
+
+PREREQ=""
+DESCRIPTION="Running custom script..."
+
+prereqs()
+{
+       echo "$PREREQ"
+}
+
+case $1 in
+# get pre-requisites
+    prereqs)
+       prereqs
+       exit 0
+       ;;
+esac
+
+. /scripts/casper-functions
+
+log_begin_msg "$DESCRIPTION"
+
+sed -i 's/^%admin.*/%admin ALL=(ALL) NOPASSWD: ALL/' /root/etc/sudoers
+
+log_end_msg

=== added directory 'desktop-autopilot-runner/custom-installation/iso-override'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc/rsyslog.d'
=== added file 'desktop-autopilot-runner/custom-installation/iso-override/etc/rsyslog.d/50-default.conf'
--- desktop-autopilot-runner/custom-installation/iso-override/etc/rsyslog.d/50-default.conf	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/custom-installation/iso-override/etc/rsyslog.d/50-default.conf	2014-03-04 19:06:35 +0000
@@ -0,0 +1,69 @@
+#  Default rules for rsyslog.
+#
+#			For more information see rsyslog.conf(5) and /etc/rsyslog.conf
+
+#
+# First some standard log files.  Log by facility.
+#
+auth,authpriv.*			/var/log/auth.log
+*.*;auth,authpriv.none		-/dev/ttyS0
+*.*;auth,authpriv.none		-/var/log/syslog
+#cron.*				/var/log/cron.log
+#daemon.*			-/var/log/daemon.log
+kern.*				-/var/log/kern.log
+#lpr.*				-/var/log/lpr.log
+mail.*				-/var/log/mail.log
+#user.*				-/var/log/user.log
+
+#
+# Logging for the mail system.  Split it up so that
+# it is easy to write scripts to parse these files.
+#
+#mail.info			-/var/log/mail.info
+#mail.warn			-/var/log/mail.warn
+mail.err			/var/log/mail.err
+
+#
+# Logging for INN news system.
+#
+news.crit			/var/log/news/news.crit
+news.err			/var/log/news/news.err
+news.notice			-/var/log/news/news.notice
+
+#
+# Some "catch-all" log files.
+#
+#*.=debug;\
+#	auth,authpriv.none;\
+#	news.none;mail.none	-/var/log/debug
+#*.=info;*.=notice;*.=warn;\
+#	auth,authpriv.none;\
+#	cron,daemon.none;\
+#	mail,news.none		-/var/log/messages
+
+#
+# Emergencies are sent to everybody logged in.
+#
+*.emerg				*
+
+#
+# I like to have messages displayed on the console, but only on a virtual
+# console I usually leave idle.
+#
+#daemon,mail.*;\
+#	news.=crit;news.=err;news.=notice;\
+#	*.=debug;*.=info;\
+#	*.=notice;*.=warn	/dev/ttyS0
+
+# The named pipe /dev/xconsole is for the `xconsole' utility.  To use it,
+# you must invoke `xconsole' with the `-file' option:
+# 
+#    $ xconsole -file /dev/xconsole [...]
+#
+# NOTE: adjust the list below, or you'll go crazy if you have a reasonably
+#      busy site..
+#
+daemon.*;mail.*;\
+	news.err;\
+	*.=debug;*.=info;\
+	*.=notice;*.=warn	|/dev/xconsole

=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/autostart'
=== added file 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/autostart/autopilot.desktop'
--- desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/autostart/autopilot.desktop	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/autostart/autopilot.desktop	2014-03-04 19:06:35 +0000
@@ -0,0 +1,9 @@
+[Desktop Entry]
+Encoding=UTF-8
+Name=Fire-up Autopilot tests for Ubiquity
+Comment=Starts the Autopilot tests
+Exec=/usr/local/bin/run-autopilot.sh -R
+Terminal=false
+Type=Application
+Categories=
+X-GNOME-Autostart-Delay=10

=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/lxsession'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/lxsession/Lubuntu'
=== added file 'desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/lxsession/Lubuntu/autostart'
--- desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/lxsession/Lubuntu/autostart	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/custom-installation/iso-override/etc/xdg/lxsession/Lubuntu/autostart	2014-03-04 19:06:35 +0000
@@ -0,0 +1,1 @@
+@/usr/local/bin/run-autopilot.sh -R

=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/usr'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/usr/local'
=== added directory 'desktop-autopilot-runner/custom-installation/iso-override/usr/local/bin'
=== added file 'desktop-autopilot-runner/custom-installation/iso-override/usr/local/bin/run-autopilot.sh'
--- desktop-autopilot-runner/custom-installation/iso-override/usr/local/bin/run-autopilot.sh	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/custom-installation/iso-override/usr/local/bin/run-autopilot.sh	2014-03-04 19:06:35 +0000
@@ -0,0 +1,332 @@
+#!/bin/sh
+
+#
+# This script runs autopilot
+
+# Copyright (C) 2013, Canonical Ltd (http://www.canonical.com/)
+#
+# Author: Jean-Baptiste Lallement <[e-mail]>
+#         Dan Chapman <[e-mail]> (Modified from original ubiquity
+#                                           autopilot test runner)
+#
+# This software is free software: you can redistribute it 
+# and/or modify it under the terms of the GNU General Public License 
+# as published by the Free Software Foundation, either version 3 of 
+# the License, or (at your option) any later version.
+# 
+# This software is distributed in the hope that it will 
+# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty 
+# of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this software.  If not, see <http://www.gnu.org/licenses/>.
+#
+set -eu
+
+# Lock management to prevent this script of running twice
+LOCKFILE=/tmp/$(basename $0).lock
+if [ -e ${LOCKFILE} ] && kill -0 $(cat ${LOCKFILE}); then
+    echo "W: $(basename $0) already running. Exiting!"
+    exit
+fi
+echo $$>${LOCKFILE}
+
+# The following variables can be overridden with a configuration file
+TSBRANCH=lp:ubuntu-autopilot-tests/production
+FLAVOR_TESTSUITE="ubuntu_autopilot_tests"
+EXTRAPACKAGES=""
+ARTIFACTS=""
+AP_OPTS="-vv"
+SHUTDOWN=1
+TIMEOUT=1200  # 20min timeout
+DEBUG=0
+
+# Custom configuration
+# Do not use the variable TESTBASE because we don't want it to be overridden
+[ -f /var/local/autopilot/config ] && . /var/local/autopilot/config
+
+TESTBASE=/var/local/autopilot/
+AP_ARTIFACTS=$TESTBASE/videos/
+AP_RESULTS=$TESTBASE/junit/
+AP_LOGS=$TESTBASE/logs/
+AP_TESTSUITES=$TESTBASE/testsuites
+AP_LOGFILE=$AP_LOGS/autopilot.log
+AP_SUMMARY=$AP_LOGS/summary.log
+AP_INFO=$AP_LOGS/build_info.txt
+RMD_OPTS="-r -rd $AP_ARTIFACTS --record-options=--fps=6,--no-wm-check"
+SPOOLDIR=$TESTBASE/spool
+TSEXPORT=$HOME/ubuntu_autopilot-autopilot
+SESSION_LOG=""
+# Append mandatory artifacts
+ARTIFACTS="$TESTBASE /var/log/syslog /var/crash $HOME/.cache/upstart $ARTIFACTS"
+
+
+# Specific configurations for various DE
+case $SESSION in
+    ubuntu)    # Covers Ubuntu and Edubuntu
+        SESSION_LOG=$HOME/.cache/upstart/gnome-session.log
+        ;;
+    xubuntu)
+        SESSION_LOG=$HOME/.cache/upstart/startxfce4.log
+        ;;
+    Lubuntu)
+        SESSION_LOG=$HOME/.cache/lxsession/Lubuntu/run.log
+        ARTIFACTS="$ARTIFACTS $HOME/.cache/lxsession"
+        ;;
+    gnome)     # ubuntu-gnome
+        SESSION_LOG=$HOME/.cache/upstart/gnome-session.log
+esac
+
+PACKAGES="bzr ssh python3-autopilot libautopilot-gtk python3-xlib \
+    recordmydesktop"
+
+export DEBIAN_FRONTEND=noninteractive
+
+on_exit() {
+    # Exit handler
+    echo "I: Archiving artifacts"
+    archive=/tmp/artifacts
+
+    
+
+    for artifact in $ARTIFACTS; do
+        [ -e "$artifact" ] && sudo tar rf ${archive}.tar $artifact || true
+    done
+
+    # Find a better way. ttys are a bit limited and sometimes output is
+    # truncated or messages are skipped by the kernel if it goes too fast.
+    if [ -f ${archive}.tar ]; then
+        sudo stty -F /dev/ttyS1 raw speed 115200
+        gzip -9 -c ${archive}.tar > ${archive}.tgz
+        sudo sh -c "cat ${archive}.tgz>/dev/ttyS1"
+    fi
+
+    rm -f ${LOCKFILE}
+    shutdown_host
+}
+trap on_exit EXIT INT QUIT ABRT PIPE TERM
+
+usage() {
+    # Display usage and exit
+    cat<<EOF
+Usage: $(basename $0) [OPTIONS...]
+Run autopilot tests in $SPOOLDIR
+
+Options:
+  -h, --help      This help
+  -d, --debug     Enable debug mode
+  -N, --new       Restart all the tests in $AP_TESTSUITES otherwise
+                  only the remaining tests in $SPOOLDIR are run
+  -R, --norecord  Do not use recordmydesktop.
+  -S, --noshutdown
+                  Do not shutdown the system after the tests
+
+EOF
+    exit 1
+}
+
+# retry command a certain number of time
+# $1 is the number of retry
+# $2 is the delay in second between retries
+# the command is then passed
+retry_cmd() {
+    # Tries to execute $@ $loop times with a delay of $delay between retries
+    # before aborting
+    loop=$1
+    loopcnt=$1  # Just used to print the status on failure
+    delay=$2
+    shift
+    shift
+    while [ $loop -gt 0 ]; do
+        rc=0
+        $@ || rc=$?
+        [ $rc -eq 0 ] && return
+        loop=$((loop - 1))
+        sleep $delay
+    done
+
+    echo "E: Command failed after $loopcnt tries: $@"
+    echo "E: Aborting!"
+    exit 1
+}
+
+setup_tests() {
+    # Prepares the environment for the tests
+    flag=$HOME/.ap_setup_done
+
+    [ -e "$flag" ] && return 0
+
+
+    if [ $DEBUG -ne 0 ]; then
+        # Put here everything you want to run in debug mode
+        xterm &  # Easier to debug from a live session, and rarely broken
+    fi
+
+    sudo stty -F /dev/ttyS0 raw speed 115200
+    
+    tail_logs $SESSION_LOG /var/log/syslog
+    # Disable notifications and screensaver
+    if which gsettings >/dev/null 2>&1; then 
+        echo "I: Disabling crash notifications"
+        gsettings set com.ubuntu.update-notifier show-apport-crashes false
+        echo "I: Disabling screensaver"
+        gsettings set org.gnome.desktop.screensaver idle-activation-enabled false
+    fi
+
+    # Loads the list of test and queue them in test spool
+    sudo mkdir -p $SPOOLDIR $AP_ARTIFACTS $AP_RESULTS $AP_LOGS
+    sudo chown -R $USER:$USER $TESTBASE $SPOOLDIR $AP_ARTIFACTS $AP_RESULTS $AP_LOGS
+    
+    echo "I: enabling universe and multiverse repositories"
+    retry_cmd 3 30 sudo add-apt-repository universe -y
+    retry_cmd 3 30 sudo add-apt-repository multiverse -y
+    # add partner repository aswell
+    #retry_cmd 3 30 sudo add-apt-repository "deb http://archive.canonical.com/ubuntu $(lsb_release -sc) partner" -y
+
+    echo "I: Installating additional packages"
+    retry_cmd 3 30 sudo apt-get update
+    retry_cmd 3 30 sudo apt-get install -yq $PACKAGES $EXTRAPACKAGES
+    
+
+    echo "I: Branch $TSBRANCH"
+    bzr export $TSEXPORT $TSBRANCH
+
+    if [ -e "$AP_TESTSUITES" ]; then
+        (cd $SPOOLDIR; touch $(cat $AP_TESTSUITES))
+    fi
+
+    cat>$AP_INFO<<EOF
+Image Id:      $(cat /cdrom/.disk/info)
+Test branch:   ${TSBRANCH}
+Test revno:    $(bzr revno $TSBRANCH)
+EOF
+    
+    cat<<EOF
+
+============================== Test Info ============================== 
+$(cat $AP_INFO)
+======================================================================= 
+
+EOF
+    touch $flag
+
+    dpkg -l > $AP_LOGS/packages.list
+}
+
+shutdown_host() {
+    # Shutdown host
+    sleep 10
+    if [ $SHUTDOWN -eq 1 ]; then
+        echo "I: Shutting down test environment"
+        sudo shutdown -h now
+    else
+        echo "I: Shutdown disabled, host will keep running"
+    fi
+}
+
+tail_logs() {
+    # Tail log files in -F mode in background
+    #   
+    # $@ List of log files
+    for log in $@; do
+        if [ -f "$log" ]; then
+            sudo sh -c "/bin/busybox tail -n0 -f $log | mawk -Winteractive -v logfile=\"$log\" '{print logfile\":\",\$0}' > /dev/ttyS0" &
+        fi
+    done
+    }
+
+
+run_tests() {
+    # Runs all the tests in spooldir
+    #
+    # $1: Spool directory
+    spooldir=$1
+    if [ ! -d $spooldir ]; then
+        echo "E: '$spooldir is not a directory. Exiting!"
+        exit 1
+    fi
+
+    if ! which autopilot-py3 >/dev/null 2>&1; then
+        echo "E: autopilot is required to run autopilot tests"
+        echo "autopilot_installed (see autopilot.log for details): ERROR" >> $AP_SUMMARY
+        shutdown_host
+        exit 1
+    fi
+    echo "autopilot_installed: PASS" >> $AP_SUMMARY
+
+    exec >>$AP_LOGFILE
+    exec 2>&1
+    touch $AP_LOGFILE
+    tail_logs $AP_LOGFILE
+
+    echo "Running autopilot tests"
+    cd $TSEXPORT/$FLAVOR_TESTSUITE/
+    
+    tail_logs /var/log/installer/debug
+    for testfile in $(ls -d $spooldir/* 2>/dev/null); do
+        testname=$(basename $testfile)
+        echo "I: Running autopilot run $testname $AP_OPTS -o $AP_RESULTS/$testname.xml"
+        aprc=0
+        timeout -s 9 -k 30 $TIMEOUT autopilot-py3 run $testname $AP_OPTS -f xml -o $AP_RESULTS/${testname}.xml||aprc=$?
+        if [ $aprc -gt 0 ]; then
+            echo "${testname}: FAIL" >> $AP_SUMMARY
+        else
+            echo "${testname}: DONE" >> $AP_SUMMARY
+        fi
+        sudo rm -f $testfile
+    done
+}
+
+reset_test() {
+    # Reset the tests for a new run
+    rm -f $HOME/.ap_setup_done $AP_SUMMARY $AP_LOGFILE $SPOOLDIR/* $AP_ARTIFACTS/* $AP_RESULTS/*
+}
+SHORTOPTS="hdNRS"
+LONGOPTS="help,debug,new,norecord,noshutdown"
+
+TEMP=$(getopt -o $SHORTOPTS --long $LONGOPTS -- "$@")
+eval set -- "$TEMP"
+
+while true ; do
+    case "$1" in
+    -h|--help)
+        usage;;
+    -d|--debug)
+        set -x
+        shift;;
+    -N|--new)
+        reset_test
+        shift;;
+    -R|--norecord)
+        RMD_OPTS=""
+        shift;;
+    -S|--noshutdown)
+        SHUTDOWN=0
+        shift;;
+    --) shift;
+        break;;
+    *) usage;;
+    esac
+done
+
+setup_tests
+if [ -f "/usr/lib/libeatmydata/libeatmydata.so" ]; then
+    echo "I: Enabling eatmydata"
+    export LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}/usr/lib/libeatmydata/libeatmydata.so"
+fi
+# Specific option for recordmydesktop for unity tests
+# It is suspected to caused memory fragmentation and make the test crash
+if [ -e "$AP_TESTSUITES" ]; then
+    if grep -qw "unity" "$AP_TESTSUITES" 2>/dev/null; then
+        RMD_OPTS="$RMD_OPTS --record-options=--fps=6,--no-wm-check"
+    fi
+fi
+
+if which recordmydesktop >/dev/null 2>&1; then
+    AP_OPTS="$AP_OPTS $RMD_OPTS"
+fi
+
+run_tests $SPOOLDIR
+
+exit 0

=== added directory 'desktop-autopilot-runner/custom-installation/packages'
=== added file 'desktop-autopilot-runner/run-autopilot-test'
--- desktop-autopilot-runner/run-autopilot-test	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/run-autopilot-test	2014-03-04 19:06:35 +0000
@@ -0,0 +1,392 @@
+#!/bin/bash
+
+#
+# Run Ubuntu Desktop Autopilot Tests in KVM
+#
+# This script prepares a test environment to run ubuntu autopilot test, run
+# the tests and collect test artifacts
+#
+
+# Copyright (C) 2013, Canonical Ltd (http://www.canonical.com/)
+#
+# Author: Jean-Baptiste Lallement <[e-mail]>
+#         Dan Chapman <[e-mail]> (Modified from original ubiquity
+#                                           autopilot test runner)
+#
+# This software is free software: you can redistribute it 
+# and/or modify it under the terms of the GNU General Public License 
+# as published by the Free Software Foundation, either version 3 of 
+# the License, or (at your option) any later version.
+# 
+# This software is distributed in the hope that it will 
+# be useful, but WITHOUT ANY WARRANTY; without even the implied warranty 
+# of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this software.  If not, see <http://www.gnu.org/licenses/>.
+#
+set -u
+
+RC=0
+BINDIR=$(dirname $(readlink -f $0))
+
+DISKSIZE=12G
+RAMSIZE=2048
+VCPU=2
+REQUIREDPKGS="bsdtar qemu-system-x86 bzr xz-utils cpio"
+OVERRIDEDIR=""
+WORKSPACE="${WORKSPACE:-$HOME/ubuntu_autopilot.tests}"
+TESTNAME="${TESTNAME:-gedit}"
+TESTCONFIG=""
+TIMEOUT=${TIMEOUT:-3600}
+DELAY=5
+SSHPORT=$(shuf -i 2000-65000 -n 1 -z)
+VNCPORT=$(shuf -i 1-2000 -n 1 -z)
+VNCOPTS="-vnc localhost:$VNCPORT"
+
+QEMUOPTS="-m $RAMSIZE -smp $VCPU \
+    -localtime -no-reboot \
+    -net nic,model=virtio -net user -redir tcp:$SSHPORT::22 \
+    -daemonize
+    "
+
+STARTAT=$(date)
+ENDAT=""
+
+on_exit() {
+    # Exit Handler
+    #
+    # Exit program with status in $RC
+    [ -n "$ENDAT" ] && exit $RC
+    ENDAT=$(date)
+    if [ -f "$WORKDIR/qemu.pid" ]; then
+        echo "I: Terminating qemu"
+        kill -9 $(cat $WORKDIR/qemu.pid) 2>/dev/null
+    fi
+    echo "I: Cleaning working directory"
+    [ -f "$DISKIMG" ] && rm -f "$DISKIMG"
+    [ -d "$WORKDIR" ] && rm -Rf "$WORKDIR"
+
+    hrule
+    echo -en "Run ended on $ENDAT\n"
+    hrule
+    exit $RC
+}
+
+trap on_exit EXIT INT QUIT ABRT PIPE TERM
+WORKDIR=$(mktemp -d /tmp/$(basename $0).XXXXXX)
+DISKDIR=$WORKDIR
+DISKIMG=""
+
+usage() {
+    # Display script usage
+    cat<<EOF
+Usage: $(basename $0) [OPTIONS...] <iso file>
+    Prepares a system from an ISO and run autopilot tests for ubuntu_autopilot
+
+Arguments:
+    <iso file>      Path to a valid Ubuntu Desktop ISO 
+Options:
+    -h, --help      This help
+    -d, --debug     Enable debug mode
+    -o, --override PATH
+                    Directory to copy into the target filesystem
+    -S, --disksize DISKSIZE
+                    Set disk size (default: $DISKSIZE)
+    -s, --shm       Write disk image into /dev/shm
+        --sdl       Uses SDL instead of VNC by default
+    -t, --test TESTNAME
+                    Name of the test to run (default: $TESTNAME)
+    -T, --testconfig FILE
+                    Load test configuration from FILE
+
+EOF
+    RC=1 && exit
+}
+
+hrule() { seq -s_ 78|tr -d [[:digit:]]; echo; }
+
+check_prerequisites() {
+    # Check and set requirements to run this test
+    #
+    # Check and set the requirements to run this test. If any of the
+    # requirement is missing the programs exit with error
+    #
+    # Args:
+    #   $@: List of required packages
+    #
+    # Returns
+    #   Exit program is a requirement is not met
+    echo "I: Checking system requirements"
+    for pkg in $@; do
+        if ! dpkg-query -W -f'${Status}' $pkg|grep -q "install ok installed" 2>/dev/null; then
+            echo "E: $pkg is required and not installed on this system. Exiting!"
+            RC=1
+            exit
+        fi
+    done
+
+    QEMUCMD=$(which qemu-system-x86_64||true) 
+    if [ -z "$QEMUCMD" ]; then
+        echo "E: qemu is required and not installed on this system (run sudo apt-get install qemu). Exiting!"
+        RC=1
+        exit
+    fi
+    QEMUCMD="$(which eatmydata 2>/dev/null||true) $QEMUCMD"
+    QEMUOPTS="$QEMUOPTS -enable-kvm $VNCOPTS"
+
+}
+
+extract_kernel() {
+    # Extract kernel and initrd 
+    #
+    # Extract kernel and initrd from an ISO image
+    #
+    # Args:
+    #   $1: Path to a valid ISO
+    #   $2: Destination directory
+    src=$1
+    dst=$2
+    echo "I: Extracting kernel and initrd from iso"
+
+    if [ ! -f "$src" ]; then
+        echo "E: The file '$src' does not exists. Exiting!"
+        RC=2
+        exit
+    fi
+
+    if [ ! -d "$dst" ]; then
+        echo "E: The directory '$dst' does not exists or is not a directory. Exiting!"
+        RC=2
+        exit
+    fi
+
+    files="casper/initrd.lz casper/$(basename $KERNEL)"
+    echo "I: extracting files from $src: $files"
+    bsdtar xf $src -C $dst $files
+    ret=$?
+    if [ $ret -gt 0 ]; then
+        echo "E: bsdtar failed. Aborting!"
+        RC=$ret
+        exit
+    fi
+}
+
+create_disk_image() {
+    # Create a disk image 
+    #
+    # Args:
+    #   $1: Path of the disk image to create
+    #   $2: Size of the disk
+    diskpath=$1
+    disksize=$2
+    echo "I: Creating disk image"
+    qemu-img create -f qcow2 $diskpath $disksize
+    ret=$?
+    if [ $ret -gt 0 ]; then
+        echo "E: qemu-img create failed. Aborting!"
+        RC=$ret
+        exit
+    fi
+}
+prepare_initrd() {
+    # Put test payload into initrd
+    #
+    # Prepares the initrd and loads a payload that'll botstrap the test
+    #
+    # Args:
+    #   $1: working directory
+    #   $2: Path to initrd
+    #   $3: Path to test directory that contains the override of the target
+    #       file system
+    # Returns:
+    #   Nothing
+    workdir=$1/initrd
+    initrd=$2
+    initrdlz=${initrd}.lz
+    usercustomdir=$3
+    customdir=${BINDIR}/custom-installation
+
+    echo "I: Preparing initrd"
+    if [ ! -f "$initrdlz" ]; then
+        echo "E: file '$(basename $initrdlz)' not found. Aborting!"
+        RC=1
+        exit 
+    fi
+    mkdir -p $workdir
+
+    (
+        cd $workdir
+        xzcat $initrdlz|cpio --quiet -ivd 2>/dev/null
+        echo "I: Copying overrides"
+        cp -af $customdir $workdir/
+        [ -d "$usercustomdir" ] && cp -af "$usercustomdir" $workdir/
+        mkdir -p $workdir/custom-installation/iso-override/var/local/autopilot
+        echo "$TESTNAME" > $workdir/custom-installation/iso-override/var/local/autopilot/testsuites
+        [ -f "$TESTCONFIG" ] && cp $TESTCONFIG $workdir/custom-installation/iso-override/var/local/autopilot/config
+
+        echo "I: Repacking initrd"
+        [ -f "$initrd" ] && rm -f $initrd
+        find .|cpio --quiet -o -H newc > $initrd
+    )
+}
+
+boot_image() {
+    # Boot an image with a iso, a kernel, an initrd and a kernel cmdline
+    #
+    # Args:
+    #   $1: Working directory
+    #   $1: Path to ISO file
+    #   $2: Path to disk image
+    #   $3: Path to initrd file
+    #   $4: Path to kernel file
+    #   $5: Kernel command line
+    workdir=$1
+    isofile=$2
+    diskimg=$3
+    initrd=$4
+    kernel=$5
+    cmdline="$6"
+
+    for f in $2 $3 $4 $5; do
+        if [ ! -f "$f" ]; then
+            echo "E: Required file not found. Aborting!"
+            RC=1
+            exit
+        fi
+    done
+    echo "I: Booting image"
+
+    $QEMUCMD $QEMUOPTS -drive file=$diskimg,if=virtio -cdrom $isofile -boot d \
+        -initrd $initrd -kernel $kernel -append "$cmdline" \
+        -serial file:$workdir/ubuntu_autopilot.ttyS0 \
+        -serial file:$workdir/ubuntu_autopilot.ttyS1 \
+        -pidfile $workdir/qemu.pid
+}
+
+collect_artifacts() {
+    echo "I: Collecting test artifacts"
+    [ $# -eq 0 ] && return
+    [ -z "$WORKSPACE" ] && return
+
+    resultdir=$WORKSPACE/results/
+    mkdir -p $resultdir
+
+    rsync -avH $@ $resultdir
+
+    if [ -f "$resultdir/artifacts.tgz" ]; then
+        echo "I: Extracting artifacts to result directory"
+        tar xzf $resultdir/artifacts.tgz -C $resultdir/
+        rm -f "$resultdir/artifacts.tgz"
+    fi
+    find $resultdir | xargs touch
+    echo "I: Artifacts stored in $resultdir"
+}
+
+tail_logs() {
+    # Tail log files in -F mode in background
+    #   
+    # $@ List of log files
+    for log in $@; do
+        #tail -n0 -F --pid=$$ $log | mawk -Winteractive -v logfile="$log" '{print logfile":",$0}' &
+        tail -n0 -F --pid=$$ $log &
+    done
+}
+
+SHORTOPTS="hdo:S:st:T:"
+LONGOPTS="help,debug,override:,disksize:,sdl,shm,test:,testconfig:"
+
+TEMP=$(getopt -o $SHORTOPTS --long $LONGOPTS -- "$@")
+eval set -- "$TEMP"
+
+exec 2>&1
+
+while true ; do
+    case "$1" in
+        -h|--help)
+            usage;;
+        -d|--debug)
+            set -x
+            shift;;
+        -o|--override)
+            OVERRIDEDIR=$2
+            shift 2;;
+        -S|--disksize)
+            DISKSIZE=$2
+            shift 2;;
+        -s|--shm)
+            DISKDIR=/dev/shm
+            shift;;
+        --sdl)
+            VNCOPTS=""
+            shift;;
+        -t|--test)
+            TESTNAME=$2
+            shift 2;;
+        -T|--testconfig)
+            TESTCONFIG=$(readlink -f $2)
+            if [ ! -f "$TESTCONFIG" ]; then
+                echo "E: Configuration file not found :'$TESTCONFIG'. Aborting!"
+                exit 1
+            fi
+            shift 2;;
+        --)
+            shift;
+            break;;
+        *)
+            usage;;
+    esac
+done
+
+[ $# -ne 1 ] && usage
+ISO="$1"
+hrule
+cat<<EOF
+
+╦ ╦┌┐ ┬ ┬┌┐┌┌┬┐┬ ┬   ╔═╗┬ ┬┌┬┐┌─┐┌─┐┬┬  ┌─┐┌┬┐ ╔╦╗┌─┐┌─┐┌┬┐┌─┐
+║ ║├┴┐│ ││││ │ │ │───╠═╣│ │ │ │ │├─┘││  │ │ │───║ ├┤ └─┐ │ └─┐
+╚═╝└─┘└─┘┘└┘ ┴ └─┘   ╩ ╩└─┘ ┴ └─┘┴  ┴┴─┘└─┘ ┴   ╩ └─┘└─┘ ┴ └─┘
+EOF
+hrule
+echo -en "Starting run on $STARTAT\n"
+hrule
+
+DISKIMG=$DISKDIR/disk.img
+INITRD=$WORKDIR/casper/initrd
+KERNEL=$WORKDIR/casper/vmlinuz
+if bsdtar tf $ISO casper/vmlinuz.efi >/dev/null 2>&1; then
+    KERNEL=${KERNEL}.efi
+fi
+KERNELOPTS="boot=casper DEBCONF_DEBUG=developer -- debconf/priority=critical locale=en_US console-setup/ask_detect=false console-setup/layoutcode=us automatic-ubuntu_autopilot"
+KERNELOPTS="boot=casper DEBCONF_DEBUG=developer -- debconf/priority=critical locale=en_US console-setup/ask_detect=false console-setup/layoutcode=us noprompt console=ttyS0,115200"
+check_prerequisites $REQUIREDPKGS
+extract_kernel $ISO $WORKDIR
+create_disk_image $DISKIMG $DISKSIZE
+prepare_initrd $WORKDIR ${INITRD} "$OVERRIDEDIR"
+boot_image $WORKDIR $ISO $DISKIMG ${INITRD} $KERNEL "$KERNELOPTS"
+
+tail_logs $WORKDIR/ubuntu_autopilot.ttyS0 
+
+PIDFILE=$WORKDIR/qemu.pid
+while sleep $DELAY; do
+    if [ ! -f "$PIDFILE" ]; then
+        echo "I: PID file not found: $PIDFILE"
+        break
+    fi
+    if ! kill -0 $(cat $WORKDIR/qemu.pid) 2>/dev/null; then
+        echo "I: Qemu process terminated."
+        break
+    fi
+    TIMEOUT=$((TIMEOUT - DELAY))
+    if [ $TIMEOUT -le 0 ]; then
+        echo "E: Test Timed out!"
+        break
+    fi
+done
+
+mkdir -p $WORKDIR/logs
+cp $WORKDIR/ubuntu_autopilot.ttyS0 $WORKDIR/logs/console
+cp $WORKDIR/ubuntu_autopilot.ttyS1 $WORKDIR/logs/artifacts.tgz
+
+collect_artifacts $WORKDIR/logs/

=== added file 'desktop-autopilot-runner/splitlogs.awk'
--- desktop-autopilot-runner/splitlogs.awk	1970-01-01 00:00:00 +0000
+++ desktop-autopilot-runner/splitlogs.awk	2014-03-04 19:06:35 +0000
@@ -0,0 +1,14 @@
+{
+    split($0, line, ":")
+    filename=line[1]
+    # Get basename and trim line
+    sub(/^.*\//, "", filename)
+    sub(/[ \t\r\n]+/, "", filename)
+    # Remove filename from line
+    logline=$0
+    sub(/^[^:]*: /, "", logline)
+
+    if (filename) {
+        print logline>filename;
+    }
+}