Merge lp:~canonical-ci-engineering/ubuntu-test-cases/boottest into lp:ubuntu-test-cases/touch

[Paul Larson]

As discussed in the standup today, let's merge this into the same branch we normally use for smoke testing.

[Paul Larson (pwlars)]

See also: https://code.launchpad.net/~pwlars/auto-package-testing/use-touch-branch/+merge/249426 which makes use of this in the jenkins template

[Francis Ginther (fginther)]

There is some cleanup that could be done, but it's not urgent if this needs to land right away. I've created an MP with the cleanup here:
https://code.launchpad.net/~fginther/ubuntu-test-cases/boottest-cleanup/+merge/249445

I'll approve as the cleanup can just as easily be done after the merge.

review: Approve

[lp:~canonical-ci-engineering/ubuntu-test-cases/boottest]

407. By Paul Larson on 2015-02-12

Remove defunct boottest.py and its tests and minor README cleanup.

=== added directory 'config'
=== added file 'config/boottest.rc.example'
--- config/boottest.rc.example	1970-01-01 00:00:00 +0000
+++ config/boottest.rc.example	2015-02-12 15:55:25 +0000
@@ -0,0 +1,9 @@
+# This file is sourced by boottest.sh
+
+# Example Config Follows - These are the current defaults
+#
+# REVISION=81
+# ARCH=krillin
+# PHABLET_PASSWORD=0000
+# ADT_TIMEOUT=600
+# VERSION=1.2.51-0ubuntu3

=== added file 'scripts/boottest.sh'
--- scripts/boottest.sh	1970-01-01 00:00:00 +0000
+++ scripts/boottest.sh	2015-02-12 15:55:25 +0000
@@ -0,0 +1,154 @@
+#!/bin/bash
+set -ex
+
+# Where am I ?
+BASEDIR=$(dirname $(readlink -f $0))/..
+
+# The release to test the package on.
+export RELEASE=${1:-vivid}
+# The source package under test.
+export SRC_PKG_NAME=${2:-libpng}
+# The phone name.
+export NODE_NAME=$3
+# Default adt-run timeout
+export ADT_TIMEOUT=${ADT_TIMEOUT:-600}
+
+# XXX psivaa 20150130: This is to use /var/local/boottest
+# directory in tachash for rsyncing the results back.
+# This should be revisited and fixed when the actual directory
+# is decided for final
+# May need tweaking/ removing the boottest section of /etc/rsyncd.conf
+# in tachash
+export RSYNC_DEST=${RSYNC_DEST:-rsync://tachash.ubuntu-ci/boottest/}
+
+# If the NODE_NAME is unset, we're running locally, the commands that
+# requires a phone are prefixed with "[ -z ${NODE_NAME} ] ||"
+# If you have a phone available locally, set ANDROID_SERIAL and NODE_NAME=yes
+
+# These can be set by jenkins when running in that context
+
+# ANDROID_SERIAL: The phone ID.
+[ -z ${NODE_NAME} ] || export ANDROID_SERIAL=${ANDROID_SERIAL:-$(${BASEDIR}/scripts/get-adb-id ${NODE_NAME})}
+# The package version to test
+export VERSION=${VERSION:-1.2.51-0ubuntu3}
+
+BOOTTESTRC=${HOME}/.ubuntu-ci/boottest.rc
+if [ -f $BOOTTESTRC ]; then
+	source $BOOTTESTRC
+fi
+
+# FIXME: Should be provided -- vila 2015-01-26
+ARCH="${ARCH:-krillin}"
+
+PHABLET_PASSWORD="${PHABLET_PASSWORD-0000}"
+
+# The provision.sh and run-smoke scripts can install extra packages to meet
+# the needs of image testing. Since we are installing specific packages from
+# the archive, this needs to be disabled.
+export SKIP_CLICK=1
+export SKIP_TESTCONFIG=1
+
+# Ensures we start with a usable phone
+[ -z ${NODE_NAME} ] || ${BASEDIR}/scripts/recover.py ${NODE_NAME}
+
+TESTS=${BASEDIR}/tests
+
+# Provision the device
+# Allow the image revision to be overridden if the latest is unusable
+REVISION="${REVISION:-0}"
+PROV_CMD="${BASEDIR}/scripts/provision.sh \
+    -r $REVISION \
+    -n ${HOME}/.ubuntu-ci/wifi.conf -w"
+[ -z ${NODE_NAME} ] || ${PROV_CMD} -s ${ANDROID_SERIAL}
+
+# Generate the adt-run setup-command
+rm -f adt-commands || true
+# apt-get update like adt-run does it
+echo "(apt-get update || (sleep 15; apt-get update))" >> adt-commands
+
+
+# --no-built-binaries should come first
+# --debug helps while debugging, can be removed otherwise
+ADT_CMD="timeout ${ADT_TIMEOUT} adt-run --debug --no-built-binaries"
+# ADT_VIRT can be overridden for local tests, 
+# it defaults to ${ANDROID_SERIAL} phone via the adb/ssh combo
+ADT_VIRT=${ADT_VIRT:-adt-virt-ssh -s /usr/share/autopkgtest/ssh-setup/adb \
+    -- -s "${ANDROID_SERIAL}"}
+# - setting up -proposed and doing apt-get update
+# - via adt-virt-ssh with a setup from adb
+# - using --apt-upgrade to ensure we only deal with packages already on the
+#   phone (see above)
+ADT_OPTS="--apt-pocket=proposed\
+    --setup-commands=adt-commands \
+    --- ${ADT_VIRT}"
+
+
+# Inject the package name into getpkgsrc DEP8 test
+FROM=${TESTS}/getpkgsrc/debian/tests/getpkgsrc.template
+TARGET=${TESTS}/getpkgsrc/debian/tests/getpkgsrc
+sed -e "s/{{ source_package }}/${SRC_PKG_NAME}/" ${FROM} > ${TARGET}
+
+# Execute a first test to get the package source tree from the testbed.
+PKG_SRC_DIR=pkgsrc
+rm -fr ${PKG_SRC_DIR} || true
+set +e
+${ADT_CMD} --unbuilt-tree ${TESTS}/getpkgsrc -o ${PKG_SRC_DIR} ${ADT_OPTS}
+RET=$?
+set -e
+if [ $RET -ne 0 ]; then
+    # Something went wrong with the testbed and/or adt-run itself
+    errfile=${PKG_SRC_DIR}/${RELEASE}_${ARCH}_${SRC_PKG_NAME}_$(date +%Y%m%d-%H%M%S).error
+    echo "$RELEASE $ARCH $SRC_PKG_NAME" > $errfile
+    [ -f "$errfile" ] && rsync -a $errfile $RSYNC_DEST/${RELEASE}/tmp/ || true
+    # Ensure we leave a usable phone
+    [ -z ${NODE_NAME} ] || test-runner/scripts/recover.py ${NODE_NAME}
+
+    exit $RET
+fi
+
+if [ -n "${FORCE_FAILURE}" ]; then
+	# Force a boottest failure by running an alternate DEP8 test
+	set +e
+	${ADT_CMD} --unbuilt-tree ${TESTS}/bootfail -o results ${ADT_OPTS}
+	RET=$?
+	set -e
+else
+	# Inject the boot DEP8 test into the package source tree
+	SOURCE_DIR=$(ls -d ${PKG_SRC_DIR}/artifacts/${SRC_PKG_NAME}-*)
+	FROM=${TESTS}/boottest/debian/tests
+	TARGET="${SOURCE_DIR}/debian/tests"
+	mkdir -p ${TARGET} # For packages that don't define DEP8 tests
+        # Inject the binary packages built previously
+        BIN_PACKAGES=$(tr '\n' ',' < ${PKG_SRC_DIR}/artifacts/needs_install.packages | sed -e s/,$//)
+        sed -e "s/{{ bin_packages }}/${BIN_PACKAGES}/" \
+            ${FROM}/control.template > ${TARGET}/control
+	cp ${FROM}/boottest ${TARGET}
+
+	# Now execute the boot test from inside the pkg source tree
+	set +e
+	${ADT_CMD} --unbuilt-tree ${SOURCE_DIR} -o results ${ADT_OPTS}
+	RET=$?
+	set -e
+fi
+
+# Return Skipped as Passed
+[ $RET -eq 2 ] && RET=0
+
+if [ -e "results/testpkg-version" -a -e "results/testbed-packages" ]; then
+    result='PASS'
+    resultfile=results/${RELEASE}_${ARCH}_${SRC_PKG_NAME}_$(date +%Y%m%d-%H%M%S).result
+    [ $RET -gt 0 ] && result="FAIL"
+    set +x  # quiet mode as it pollutes output
+    echo "$RELEASE $ARCH $(cat results/testpkg-version) $result $(sort -u results/*-packages|tr -s '[\n\t]' ' ')" > $resultfile
+    [ -f "$resultfile" ] && rsync -a $resultfile $RSYNC_DEST/${RELEASE}/tmp/ || true
+else
+    # Something went wrong with the testbed
+    errfile=results/${RELEASE}_${ARCH}_${SRC_PKG_NAME}_$(date +%Y%m%d-%H%M%S).error
+    echo "$RELEASE $ARCH $SRC_PKG_NAME" > $errfile
+    [ -f "$errfile" ] && rsync -a $errfile $RSYNC_DEST/${RELEASE}/tmp/ || true
+fi
+
+# Ensure we leave a usable phone
+[ -z ${NODE_NAME} ] || test-runner/scripts/recover.py ${NODE_NAME}
+
+exit $RET

=== modified file 'scripts/device_info.py'
--- scripts/device_info.py	2015-01-29 15:30:26 +0000
+++ scripts/device_info.py	2015-02-12 15:55:25 +0000
@@ -263,6 +263,7 @@
     "mako-20": TouchDevice("mako", "05083705e0d29402",
                            relay_url="http://decatur.ubuntu-ci",
                            bank=1, power_pin=2, volume_down_pin=3),
+    "mako-fginther": TouchDevice("mako", "04c3c2be1d5248b3"),
     "ps-manta-01": TouchDevice("manta", "R32D203DDZR"),
     "manta-01": TouchDevice("manta", "R32D102RPZL"),
     "manta-02": TouchDevice("manta", "R32D102RPPK"),

=== added directory 'tests/bootfail'
=== added file 'tests/bootfail/README'
--- tests/bootfail/README	1970-01-01 00:00:00 +0000
+++ tests/bootfail/README	2015-02-12 15:55:25 +0000
@@ -0,0 +1,15 @@
+!!CAUTION!!
+Do not execute this test on a device without manual control.
+The test will render the device unbootable and disable adb.
+Manual control is required to return the device to fastboot.
+
+To run the tests:
+
+$ export ANDROID_SERIAL=[adb device id]
+$ phablet-config writable-image
+
+$ adt-run --no-built-binaries --unbuilt-tree bootfail -o log-dir --- \
+      ssh -s /usr/share/autopkgtest/ssh-setup/adb -- -s $ANDROID_SERIAL
+
+Exit code is '0' if it passed.
+Logs are stored under log-dir as specified with the '-o' option.

=== added directory 'tests/bootfail/debian'
=== added file 'tests/bootfail/debian/changelog'
--- tests/bootfail/debian/changelog	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/changelog	2015-02-12 15:55:25 +0000
@@ -0,0 +1,5 @@
+bootfail (0.1) vivid; urgency=medium
+
+  * Initial release.
+
+ -- Francis Ginther <[e-mail]>  Mon, 26 Jan 2015 21:34:20 -0600

=== added file 'tests/bootfail/debian/compat'
--- tests/bootfail/debian/compat	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/compat	2015-02-12 15:55:25 +0000
@@ -0,0 +1,1 @@
+9

=== added file 'tests/bootfail/debian/control'
--- tests/bootfail/debian/control	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/control	2015-02-12 15:55:25 +0000
@@ -0,0 +1,10 @@
+Source: bootfail
+Section: misc
+Maintainer: Canonical CI Engineering <[e-mail]>
+Build-Depends: debhelper (>= 9),
+Standards-Version: 3.9.4
+
+Package: bootfail
+Architecture: all
+Description: A boottest dep8 test package designed to fail
+ A boottest dep8 test package designed to fail.

=== added file 'tests/bootfail/debian/rules'
--- tests/bootfail/debian/rules	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/rules	2015-02-12 15:55:25 +0000
@@ -0,0 +1,4 @@
+#!/usr/bin/make -f
+
+%:
+	dh $@

=== added directory 'tests/bootfail/debian/tests'
=== added file 'tests/bootfail/debian/tests/bootfail'
--- tests/bootfail/debian/tests/bootfail	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/tests/bootfail	2015-02-12 15:55:25 +0000
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+rc=0
+
+if [ "${ADT_REBOOT_MARK}" == "reboot_mark" ]; then
+    ls > "${ADT_ARTIFACTS}"/ls.txt
+    rc=$?
+else 
+    # CAUITON! The following will render a phone device unbootable
+    # It also requires manual control to reset the device into fastboot
+    sudo apt-get remove -y --force-yes lxc
+    /tmp/autopkgtest-reboot reboot_mark
+    echo "Are we having fun yet?"
+fi
+exit $rc

=== added file 'tests/bootfail/debian/tests/control'
--- tests/bootfail/debian/tests/control	1970-01-01 00:00:00 +0000
+++ tests/bootfail/debian/tests/control	2015-02-12 15:55:25 +0000
@@ -0,0 +1,3 @@
+Tests: bootfail
+Depends:
+Restrictions: needs-root

=== added directory 'tests/boottest'
=== added file 'tests/boottest/README'
--- tests/boottest/README	1970-01-01 00:00:00 +0000
+++ tests/boottest/README	2015-02-12 15:55:25 +0000
@@ -0,0 +1,10 @@
+To run the tests:
+
+$ export ANDROID_SERIAL=[adb device id]
+$ phablet-config writable-image
+
+$ adt-run --unbuilt-tree --no-built-binaries boottest -o log-dir --- \
+      ssh -s /usr/share/autopkgtest/ssh-setup/adb -- -s $ANDROID_SERIAL
+
+Exit code is '0' if it passed.
+Logs are stored under log-dir as specified with the '-o' option.

=== added directory 'tests/boottest/debian'
=== added file 'tests/boottest/debian/changelog'
--- tests/boottest/debian/changelog	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/changelog	2015-02-12 15:55:25 +0000
@@ -0,0 +1,5 @@
+boottest (0.1) vivid; urgency=medium
+
+  * Initial release.
+
+ -- Francis Ginther <[e-mail]>  Fri, 16 Jan 2015 14:34:34 -0600

=== added file 'tests/boottest/debian/compat'
--- tests/boottest/debian/compat	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/compat	2015-02-12 15:55:25 +0000
@@ -0,0 +1,1 @@
+9

=== added file 'tests/boottest/debian/control'
--- tests/boottest/debian/control	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/control	2015-02-12 15:55:25 +0000
@@ -0,0 +1,10 @@
+Source: boottest
+Section: misc
+Maintainer: Canonical CI Engineering <[e-mail]>
+Build-Depends: debhelper (>= 9),
+Standards-Version: 3.9.4
+
+Package: boottest
+Architecture: all
+Description: A boottest dep8 test package
+ A boottest dep8 test package.

=== added file 'tests/boottest/debian/rules'
--- tests/boottest/debian/rules	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/rules	2015-02-12 15:55:25 +0000
@@ -0,0 +1,4 @@
+#!/usr/bin/make -f
+
+%:
+	dh $@

=== added directory 'tests/boottest/debian/tests'
=== added file 'tests/boottest/debian/tests/boottest'
--- tests/boottest/debian/tests/boottest	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/tests/boottest	2015-02-12 15:55:25 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e
+
+rc=0
+
+if [ "${ADT_REBOOT_MARK}" == "reboot_mark" ]; then
+    ls > "${ADT_ARTIFACTS}"/ls.txt
+    rc=$?
+else 
+    /tmp/autopkgtest-reboot reboot_mark
+    echo "Are we having fun yet?"
+fi
+exit $rc

=== added file 'tests/boottest/debian/tests/control.template'
--- tests/boottest/debian/tests/control.template	1970-01-01 00:00:00 +0000
+++ tests/boottest/debian/tests/control.template	2015-02-12 15:55:25 +0000
@@ -0,0 +1,4 @@
+Tests: boottest
+# Make adt-run install the binary packages by declaring a dep
+Depends: {{ bin_packages }}
+Restrictions: needs-root

=== added directory 'tests/getpkgsrc'
=== added directory 'tests/getpkgsrc/debian'
=== added file 'tests/getpkgsrc/debian/changelog'
--- tests/getpkgsrc/debian/changelog	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/changelog	2015-02-12 15:55:25 +0000
@@ -0,0 +1,5 @@
+getpkgsrc (0.1) vivid; urgency=medium
+
+  * Initial release.
+
+ -- Vincent Ladeuil <[e-mail]>  Sat, 24 Jan 2015 16:07:34 +0100

=== added file 'tests/getpkgsrc/debian/compat'
--- tests/getpkgsrc/debian/compat	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/compat	2015-02-12 15:55:25 +0000
@@ -0,0 +1,1 @@
+9

=== added file 'tests/getpkgsrc/debian/control'
--- tests/getpkgsrc/debian/control	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/control	2015-02-12 15:55:25 +0000
@@ -0,0 +1,9 @@
+Source: getpkgsrc
+Section: misc
+Maintainer: Canonical CI Engineering <[e-mail]>
+Build-Depends: debhelper (>= 9),
+Standards-Version: 3.9.4
+
+Package: getpkgsrc
+Architecture: all
+Description: A dep8 test providing a package source from inside the testbed.

=== added file 'tests/getpkgsrc/debian/rules'
--- tests/getpkgsrc/debian/rules	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/rules	2015-02-12 15:55:25 +0000
@@ -0,0 +1,4 @@
+#!/usr/bin/make -f
+
+%:
+	dh $@

=== added directory 'tests/getpkgsrc/debian/tests'
=== added file 'tests/getpkgsrc/debian/tests/control'
--- tests/getpkgsrc/debian/tests/control	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/tests/control	2015-02-12 15:55:25 +0000
@@ -0,0 +1,2 @@
+Tests: getpkgsrc
+Depends: dpkg-dev

=== added file 'tests/getpkgsrc/debian/tests/getpkgsrc.template'
--- tests/getpkgsrc/debian/tests/getpkgsrc.template	1970-01-01 00:00:00 +0000
+++ tests/getpkgsrc/debian/tests/getpkgsrc.template	2015-02-12 15:55:25 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -e
+
+cd ${ADT_ARTIFACTS}
+# List installed packages, removing the ':{arch}' suffix
+dpkg-query -f '${binary:Package}\n' -W | sed -e 's/:.*$//' > installed.packages
+# adt-run complains and fails the test if sdterr is not empty
+# redirect stderr to preserve the output for reference
+apt-get source {{ source_package }} 2> apt-get-source.stderr
+# Get the defined binary packages
+grep ^Package: {{ source_package }}*/debian/control | sed -e 's/^Package: //' | sort > binary.packages
+# Get the binary packages already installed
+comm  -1 -2 binary.packages installed.packages > needs_install.packages