Merge lp:~mvo/snappy/snappy-merge-integration-tests into lp:~snappy-dev/snappy/snappy-moved-to-github

[Michael Vogt]

Integrate lp:snappy/selftest branch into snappy itself

This branch integrates the lp:snappy/selftest branch into snappy itself and provides support for running the integration tests as part of the run-checks.sh command.

The way this works is essentially:
- build a deb package from the current tree
- build a fresh ubuntu core test image
- copy/install the ubuntu-snappy deb package into the new image
- run the integration-tests inside this new image via "autopkgtest"

This should allow us to run the full testsuite for each MP via tarmac.

Some open questions:
- can we avoid using sudo for the ubuntu-device-flash generation of the image somehow?
- what does need to change on the CI side so that the new way of running the tests is supported?
- can we add that to our tarmac infrastructure somehow?

[John Lenton (chipaca)]

drooling at running this in tarmac, here. When can we drop kpartx? (aiui that's where the need for sudo comes from)

review: Approve

[Sergio Schvezov (sergiusens)]

hey this is a great initiative!

review: Needs Information

[Sergio Schvezov (sergiusens)]

> drooling at running this in tarmac, here. When can we drop kpartx? (aiui
> that's where the need for sudo comes from)

Can you think of anything that doesn't require something similar to kpartx?

I can think of having an option for split image files, but you would still need sudo to install grub, unless more people get on board the idea of using a grub img file (dd'ed ala u-boot) and all the other stuff that goes on.

[Sergio Schvezov (sergiusens)]

hmm, we also want to disable the autopilot; one way of doing this is to create an oem package with a config for it to be disabled by default.

[Leo Arias (elopio)]

All good, what could possibly go wrong

\o/

The tests pass. That's an awesome start for me.
I would like to see the tests running from debian/control/tests instead of

16 +++ integration-tests/run-in-image/debian/tests/control

Do you think it is not a good idea to get them running during proposed-migration?

[Sergio Schvezov (sergiusens)]

On Thu, May 21, 2015 at 09:30:57PM -0000, Leo Arias wrote:
> All good, what could possibly go wrong
>
> \o/
>
> The tests pass. That's an awesome start for me.
> I would like to see the tests running from debian/control/tests instead of
>
> 16 +++ integration-tests/run-in-image/debian/tests/control

Won't this be tricky since it requires an internet connection?

> Do you think it is not a good idea to get them running during proposed-migration?

That's being worked on by the ci team already.

[lp:~mvo/snappy/snappy-merge-integration-tests]

466. By Michael Vogt on 2015-05-22

sync with lp:snappy/selftest

[Leo Arias (elopio)]

> On Thu, May 21, 2015 at 09:30:57PM -0000, Leo Arias wrote:
> > All good, what could possibly go wrong
> >
> > \o/
> >
> > The tests pass. That's an awesome start for me.
> > I would like to see the tests running from debian/control/tests instead of
> >
> > 16 +++ integration-tests/run-in-image/debian/tests/control
>
> Won't this be tricky since it requires an internet connection?

Yes, we would need to deploy a local fake server that replies with the right messages. Then the tests will be able to run with the fake when there's no internet, or with the real server when there is. If CI is going to run the suite against the real server often enough, probably there's no need for the fake.

[Federico Gimenez (fgimenez)]

Looks great, some inline comments.

Thanks!

[lp:~mvo/snappy/snappy-merge-integration-tests]

467. By Michael Vogt on 2015-06-09

merged lp:snappy

468. By Michael Vogt on 2015-06-09

do not try to sign the deb

[Leo Arias (elopio)]

Successful run: http://paste.ubuntu.com/11676223/

Thanks mvo.

pitti is right, of course. We need to figure out how to generate an image that contains snappy from the code of the branch.

So lets land this and iterate.

review: Approve

[Ricardo Salveti (rsalveti)]

From IRC:

<rsalveti> elopio: so, ideally the way to change the image is basically as pitti said, which is creating another clean image with livecd-rootfs
<rsalveti> elopio: the proposed-migration work that the ci team is doing basically that, but using launchpad to build the new image
<rsalveti> we should just have a script or something that can easily do that locally (something like rootstock, ogra_ ^^)
<rsalveti> elopio: we definitely don't want to change the image by installing debs in runtime, as that's not the same path used when we produce the final images
<ogra_> i can look into making rootstock work for core

=== added directory 'integration-tests'
=== added directory 'integration-tests/debs'
=== added directory 'integration-tests/image'
=== added directory 'integration-tests/run-in-image'
=== added file 'integration-tests/run-in-image/README'
--- integration-tests/run-in-image/README	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/README	2015-06-09 15:41:05 +0000
@@ -0,0 +1,16 @@
+# Automatic testing for snappy
+
+
+## qemu based x86 testing
+
+Run locally with:
+$ adt-run $(pwd)/ --- ssh -s snappy -- -i /path/to/snappy.img
+
+
+## BeagleBoneBlack real HW testing
+
+With autopkgtest 3.14 it can also drive a BBB directly, this requires
+that the bbb contains a clean image. Then run:
+
+$ adt-run $(pwd)/ --- ssh --reboot -l ubuntu -P ubuntu --reboot -H webdm.local
+

=== added directory 'integration-tests/run-in-image/debian'
=== added directory 'integration-tests/run-in-image/debian/tests'
=== added file 'integration-tests/run-in-image/debian/tests/control'
--- integration-tests/run-in-image/debian/tests/control	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/debian/tests/control	2015-06-09 15:41:05 +0000
@@ -0,0 +1,2 @@
+Test-Command: ./snappy-selftest --yes-really
+Depends:

=== added file 'integration-tests/run-in-image/snappy-selftest'
--- integration-tests/run-in-image/snappy-selftest	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/snappy-selftest	2015-06-09 15:41:05 +0000
@@ -0,0 +1,51 @@
+#!/bin/bash
+
+set -e
+
+. tests/framework
+
+SNAPPY=snappy
+
+MYDIR=$(dirname $0)
+
+if [ "$1" != "--yes-really" ]; then
+    echo "The selftest may be destructive, please run with:"
+    echo "   --yes-really"
+    echo "if you know what you are doing"
+    exit 1
+fi
+
+. tests/settings
+
+. tests/common.sh
+
+# prepare the environment
+if [ -z "$ADT_REBOOT_MARK" ]; then
+    sudo snappy remove "$framework" 2>&1 || true
+    sudo snappy remove hello-world 2>&1 || true
+    sudo snappy remove xkcd-webserver 2>&1 || true
+fi
+
+for test in $MYDIR/tests/*_test_*[!~#]; do
+    CURRENT_TEST=$(basename $test)
+
+    # after a reboot, skip ahead to the test that triggered it
+    if [ -n "$ADT_REBOOT_MARK" ]; then
+        if [ "$ADT_REBOOT_MARK" == "$CURRENT_TEST" ]; then
+            echo "****** Resuming $test after reboot"
+        else
+            continue
+        fi
+    else
+        echo "****** Running $test"
+    fi
+
+    . "$test"
+    if ! (test 2>&1); then
+        echo "FAILED: $test"
+        exit 1
+    fi
+
+    # clear reboot marker
+    unset ADT_REBOOT_MARK
+done

=== added directory 'integration-tests/run-in-image/tests'
=== added file 'integration-tests/run-in-image/tests/01_test_info'
--- integration-tests/run-in-image/tests/01_test_info	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/01_test_info	2015-06-09 15:41:05 +0000
@@ -0,0 +1,4 @@
+test() {
+    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
+    test_regexp "^frameworks: ($framework|)*$" $SNAPPY info
+}

=== added file 'integration-tests/run-in-image/tests/02_test_versions'
--- integration-tests/run-in-image/tests/02_test_versions	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/02_test_versions	2015-06-09 15:41:05 +0000
@@ -0,0 +1,7 @@
+test() {
+    T="ubuntu-core.*201?"
+    test_regexp "$T" $SNAPPY list
+
+    T="Name.*Date.*Version.*Developer"
+    test_regexp "$T" $SNAPPY list
+}

=== added file 'integration-tests/run-in-image/tests/03_test_apt'
--- integration-tests/run-in-image/tests/03_test_apt	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/03_test_apt	2015-06-09 15:41:05 +0000
@@ -0,0 +1,4 @@
+test() {
+       T="Ubuntu Core does not use apt-get, see 'snappy --help'!"
+       test_equal "$T" apt-get update
+}
\ No newline at end of file

=== added file 'integration-tests/run-in-image/tests/04_test_install_hello'
--- integration-tests/run-in-image/tests/04_test_install_hello	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/04_test_install_hello	2015-06-09 15:41:05 +0000
@@ -0,0 +1,7 @@
+test() {
+       T="hello-world[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+(\.[0-9]+)+[[:space:]]+canonical"
+       test_regexp "$T" sudo $SNAPPY install hello-world
+       
+       T="Hello World!"
+       test_equal "$T" hello-world.echo
+}

=== added file 'integration-tests/run-in-image/tests/05_test_install_nonexistant'
--- integration-tests/run-in-image/tests/05_test_install_nonexistant	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/05_test_install_nonexistant	2015-06-09 15:41:05 +0000
@@ -0,0 +1,5 @@
+test() {
+        pkg=fizzler
+        T="^${pkg} failed to install: snappy package not found"
+        test_regexp "$T" sudo $SNAPPY install $pkg 
+}

=== added file 'integration-tests/run-in-image/tests/06_test_search_framework'
--- integration-tests/run-in-image/tests/06_test_search_framework	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/06_test_search_framework	2015-06-09 15:41:05 +0000
@@ -0,0 +1,4 @@
+test() {
+       T="$framework[[:space:]]+[0-9.]+[[:space:]]+.*"
+       test_regexp "$T" $SNAPPY search "$framework" 
+}

=== added file 'integration-tests/run-in-image/tests/07_test_install_framework'
--- integration-tests/run-in-image/tests/07_test_install_framework	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/07_test_install_framework	2015-06-09 15:41:05 +0000
@@ -0,0 +1,33 @@
+test() {
+    if after_reboot; then
+        # ensure the service comes up after a reboot, give it time
+        # to start (i.e. avoid race between framework and ssh)
+        TIMEOUT=60
+        for i in $(seq $TIMEOUT); do
+            services=$(systemctl | grep "${framework}_.*.service")
+            if [ -n "$services" ]; then
+                echo "Service $framework ready"
+                break
+            fi
+            echo "Waiting for $framework to come up ($i)"
+            sleep 1
+        done
+        if [ $i = $TIMEOUT ]; then
+            fail "$framework provided no services"
+        fi
+        return
+    fi
+
+    # we test that the final line contains the framework command
+    # FIXME: test progress here too
+    T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
+    test_regexp "$T" sudo $SNAPPY install "$framework_full"
+
+    # ensure its there
+    services=$(systemctl | grep "${framework}_.*.service")
+    [ -n "$services" ] || fail "$framework provided no services"
+
+    if can_reboot; then
+        reboot
+    fi
+}

=== added file 'integration-tests/run-in-image/tests/08_test_versions_has_framework'
--- integration-tests/run-in-image/tests/08_test_versions_has_framework	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/08_test_versions_has_framework	2015-06-09 15:41:05 +0000
@@ -0,0 +1,4 @@
+test() {
+    T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
+    test_regexp "$T" $SNAPPY list
+}

=== added file 'integration-tests/run-in-image/tests/10_test_info_has_stuff'
--- integration-tests/run-in-image/tests/10_test_info_has_stuff	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/10_test_info_has_stuff	2015-06-09 15:41:05 +0000
@@ -0,0 +1,5 @@
+test() {
+    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
+    test_regexp "^frameworks: $framework" $SNAPPY info
+    test_regexp "^apps:.*\<hello-world\>" $SNAPPY info
+}

=== added file 'integration-tests/run-in-image/tests/11_test_xkcd_listens'
--- integration-tests/run-in-image/tests/11_test_xkcd_listens	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/11_test_xkcd_listens	2015-06-09 15:41:05 +0000
@@ -0,0 +1,15 @@
+test() {
+       # FIXME: add $SNAPPY search webserver
+       
+       sudo $SNAPPY install xkcd-webserver.canonical
+
+       # FIXME: sucks, needed because "systemctl start" does not
+       #        wait until the service is really started
+       sleep 1
+
+       T="HTTP/1.0 200 OK"
+       TEMPF=$(tempfile)
+       printf "GET / HTTP/1.0\n\n"|nc localhost 80 > $TEMPF
+       test_regexp "$T" cat $TEMPF
+       rm -f $TEMPF
+}

=== added file 'integration-tests/run-in-image/tests/80_test_failover'
--- integration-tests/run-in-image/tests/80_test_failover	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/80_test_failover	2015-06-09 15:41:05 +0000
@@ -0,0 +1,48 @@
+# Test that we rollback on failed upgrades
+# (needs autopkgtest with a supporting runner).
+
+test() {
+    can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
+
+    # debug
+    version_info
+    boot_info
+
+    if after_reboot; then
+        # reboot: ensure we booted back into original
+        orig_current=$(cat "${ADT_ARTIFACTS}/current")
+        if [ "$orig_current" != "$current" ]; then
+            fail "did not failover to good version (\"$orig_current\" != \"$current\")"
+        fi
+
+        # FIXME: grep log(other partition) for the crash to ensure it
+        #        really worked
+        # FIXME2: reboot again as regression test for #1449904
+
+        # we booted, cleanup the panic for the next tests to work
+        sudo mount -o remount,rw /writable/cache/system
+        sudo rm /writable/cache/system/etc/rc.local
+        return
+    fi
+
+    # fake new available version by doing a current--
+    switch_channel "s/build_number: $current/build_number: $((current-1))/" /etc/system-image/channel.ini
+
+    # we should have something now :)
+    version_info
+
+    [ $avail -gt $current ] || fail "$avail is not newer than $current"
+
+    # save version for post-upgrade test
+    save_version_info $avail $current
+
+    echo "upgrading..."
+    sudo snappy update
+
+    # break the upgrade by inserting a broken rc.local file onto the other
+    # partition
+    sudo mount -o remount,rw /writable/cache/system/
+    printf "#!/bin/sh\nprintf c > /proc/sysrq-trigger" | sudo tee /writable/cache/system/etc/rc.local
+    
+    reboot
+}

=== added file 'integration-tests/run-in-image/tests/90_test_upgrade'
--- integration-tests/run-in-image/tests/90_test_upgrade	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/90_test_upgrade	2015-06-09 15:41:05 +0000
@@ -0,0 +1,45 @@
+# Test that we can upgrade to the latest image version and roll back to the
+# original version. Test will be skipped the testbed does not offer reboot
+# (needs autopkgtest with a supporting runner).
+
+test() {
+    can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
+
+    # debug
+    version_info    
+    boot_info
+
+    if after_reboot; then
+        # second reboot: after rollback
+        if [ -e "${ADT_ARTIFACTS}/rolled_back" ]; then
+            orig_current=$(cat "${ADT_ARTIFACTS}/current")
+            [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current ($all_versions)"
+            return
+        fi
+
+        # first reboot: after upgrade
+        orig_avail=$(cat "${ADT_ARTIFACTS}/avail")
+        [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version $orig_avail != $current ($all_versions)"
+
+        echo "rolling back..."
+        sudo snappy rollback ubuntu-core
+        touch "${ADT_ARTIFACTS}/rolled_back"
+        reboot
+        return
+    fi
+
+    # fake new available version by doing a current--
+    switch_channel "s/build_number: $current/build_number: $((current-1))/"
+
+    # we should have something now :)
+    version_info
+    
+    [ $avail -gt $current ] || fail "$avail is not newer than $current"
+
+    # save version for post-upgrade test
+    save_version_info $avail $current    
+
+    echo "upgrading..."
+    sudo snappy update
+    reboot
+}

=== added file 'integration-tests/run-in-image/tests/91_test_upgrade_from_stable_image'
--- integration-tests/run-in-image/tests/91_test_upgrade_from_stable_image	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/91_test_upgrade_from_stable_image	2015-06-09 15:41:05 +0000
@@ -0,0 +1,51 @@
+# Test that we can upgrade from the latest *stable* image version to the
+# current edge image version and rollback original version.
+#
+# Make image for this with:
+#
+# $ sudo ubuntu-device-flash core  --channel=stable 15.04 -o snappy-stable.img --developer-mode --enable-ssh
+#
+# Run with:
+# adt-run upgrade-test/ --- ssh -s snappy -- -i /path/to/snappy-stable.img
+
+test() {
+    can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
+
+    # debug
+    version_info    
+    boot_info
+
+    if after_reboot; then
+        # second reboot: after rollback
+        if [ -e "${ADT_ARTIFACTS}/rolled_back-stable" ]; then
+            orig_current=$(cat "${ADT_ARTIFACTS}/current")
+            [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current ($all_versions)"
+            return
+        fi
+
+        # first reboot: after upgrade
+        orig_avail=$(cat "${ADT_ARTIFACTS}/avail")
+        [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version ($all_versions)"
+
+        echo "rolling back..."
+        sudo snappy rollback ubuntu-core
+        touch "${ADT_ARTIFACTS}/rolled_back-stable"
+        reboot
+        return
+    fi
+
+    # switch channel
+    switch_channel 's#channel: ubuntu-core/\(.*\)/stable#channel: ubuntu-core/\1/edge#'
+    
+    # now upgrade to latest edge from stable
+    version_info
+    
+    [ $avail -gt $current ] || fail "$avail is not newer than $current"
+
+    # save version for post-upgrade test
+    save_version_info $avail $current    
+
+    echo "upgrading..."
+    sudo snappy update
+    reboot
+}

=== added file 'integration-tests/run-in-image/tests/common.sh'
--- integration-tests/run-in-image/tests/common.sh	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/common.sh	2015-06-09 15:41:05 +0000
@@ -0,0 +1,59 @@
+#!/bin/sh
+#
+# (not really executable, just to make editors happy)
+
+version_info() {
+    versions=$(snappy list)
+    current=$(echo "$versions" | awk '/^ubuntu-core/ {print $3}')
+    echo "current version: $current"
+    all_version="$(snappy list -v)"
+
+    # something went wrong here with the extraction
+    [ "$current" != "${current#[0-9]}" ] || fail "\"$current\" is not numeric"
+
+    versions=$(snappy list -u)
+    avail=$(echo "$versions" | awk '/^ubuntu-core/ {print $3}')
+    echo "available version: $avail"
+    [ "$avail" != "${avail#[0-9]}" ] || fail  "\"$avail\" is not numeric"
+}
+
+# debug info really
+boot_info() {
+    echo "Boot order"
+
+    if [ -e /boot/grub/grub.cfg ]; then
+        # FIXME: why do we have two places that define what root to boot?
+        
+        # check grub.cfg config
+        grep root=LABEL /boot/grub/grub.cfg|head -n1
+        # show /boot/grub/grubenv
+        grub-editenv - list
+    elif [ -e /boot/uboot/snappy-system.txt ]; then
+        # show uboot config
+        egrep '(snappy_mode|snappy_ab)' /boot/uboot/snappy-system.txt
+    else
+        fail "unknown bootloader"
+    fi
+    echo "---"
+}
+
+switch_channel() {
+    SED_EXPR="$1"
+    sudo mount -o remount,rw /
+    sudo sed -i "${SED_EXPR}" /etc/system-image/channel.ini
+    sudo mount -o remount,ro /
+
+    if [ -e /writable/cache/system/etc/system-image/channel.ini ]; then
+        sudo mount -o remount,rw /writable/cache/system/
+        sudo sed -i "${SED_EXPR}" /writable/cache/system/etc/system-image/channel.ini
+        sudo mount -o remount,ro /writable/cache/system/
+    fi
+}
+
+save_version_info() {
+    avail=$1
+    current=$2
+    
+    echo "$avail" > "${ADT_ARTIFACTS}/avail"
+    echo "$current" > "${ADT_ARTIFACTS}/current"
+}

=== added file 'integration-tests/run-in-image/tests/framework'
--- integration-tests/run-in-image/tests/framework	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/framework	2015-06-09 15:41:05 +0000
@@ -0,0 +1,46 @@
+fail() {
+    echo "ERROR" $@
+    exit 1
+}
+
+test_equal() {
+    local expected="$1"
+    shift
+    "$@" 2>&1 | diff -u <(echo "$expected") -
+}
+
+test_regexp() {
+    local REGEXP="$1"
+    shift
+    local ACTUAL=$("$@" 2>&1)
+    if ! echo "$ACTUAL" | grep -E -q "$REGEXP"; then
+        echo "Can not find '$REGEXP' in '$ACTUAL'"
+        fail
+    fi
+}
+
+# check if reboot is supported; this needs running with autopkgtest on a
+# supported testbed (QEMU or ssh with a setup script that announces the
+# "reboot" capability)
+can_reboot() {
+    [ -x /tmp/autopkgtest-reboot ]
+}
+
+# reboot the testbed; the same test will be started again, check after_reboot()
+# to see in which state you are in
+reboot() {
+    if [ -z "$CURRENT_TEST" ]; then
+        echo 'ERROR: $CURRENT_TEST not defined' >&2
+        exit 1
+    fi
+    echo "Rebooting testbed..."
+    sudo /tmp/autopkgtest-reboot "$CURRENT_TEST"
+}
+
+# true if the current test reboot()ed and is now restarted after boot; false if
+# it's the first time the test runs (or reboot is not supported)
+after_reboot() {
+    [ "$ADT_REBOOT_MARK" = "$CURRENT_TEST" ]
+}
+
+# vim: filetype=sh shiftwidth=4

=== added file 'integration-tests/run-in-image/tests/settings'
--- integration-tests/run-in-image/tests/settings	1970-01-01 00:00:00 +0000
+++ integration-tests/run-in-image/tests/settings	2015-06-09 15:41:05 +0000
@@ -0,0 +1,3 @@
+# name of framework that will be installed for testing
+framework=docker
+framework_full=docker

=== added file 'integration-tests/selftest'
--- integration-tests/selftest	1970-01-01 00:00:00 +0000
+++ integration-tests/selftest	2015-06-09 15:41:05 +0000
@@ -0,0 +1,68 @@
+#!/bin/sh
+
+set -ex
+
+HERE="$(dirname $0)"
+
+# build the updated deb packages for the image
+rm -f "$HERE/debs/*"
+# the "-us -uc " flags tell it not to sign it
+bzr-buildpackage --result-dir="$HERE/debs" "$HERE/.." -- -us -uc
+
+# build a base image
+IMAGE="$HERE/image/snappy.img"
+rm -f "$IMAGE/*"
+# FIXME: hrm, hrm, needs sudo to work on testbed
+#
+# FIXME2: use wget here instead and just get the latest image from
+#         cdimage
+sudo ubuntu-device-flash core -o "$IMAGE" rolling --channel edge --enable-ssh --developer-mode
+
+# base cmd
+QEMU_CMD="qemu-system-$(uname -m) -enable-kvm -m 768 -localtime -nographic -net user -net nic,model=virtio"
+
+# fire it up
+port=11022
+SERIAL=stdio
+$QEMU_CMD -drive file="$IMAGE",if=virtio -redir tcp:$port::22 -monitor none -serial $SERIAL &
+QEMU_PID=$!
+trap "kill $QEMU_PID" INT QUIT
+
+# wait until the image is ready
+SSH="ssh -oStrictHostKeyChecking=no -o UserKnownHostsFile=\"$HERE/image/known_hosts\" -p $port ubuntu@localhost"
+SCP="scp -oStrictHostKeyChecking=no -o UserKnownHostsFile=\"$HERE/image/known_hosts\" -P $port"
+for i in $(seq 100); do
+    if $SSH true; then
+        break
+    fi
+    sleep 1
+done
+if [ $i = 100 ]; then
+    echo "Failed to setup qemu"
+    exit 1
+fi
+
+# install debs
+$SSH rm -rf /tmp/debs
+$SSH mkdir /tmp/debs
+$SCP "$HERE"/debs/*.deb ubuntu@localhost:/tmp/debs
+$SSH sudo mount -o remount,rw /
+$SSH sudo dpkg -i /tmp/debs/*.deb
+
+# shut it down
+$SSH sudo halt --poweroff || true
+
+# wait for qemu pid
+for i in $(seq 100); do
+    if [ ! -e /proc/$QEMU_PID/exe ]; then
+        break
+    fi
+    i=$((i+1))
+    sleep 1
+done
+
+# now run the tests against the image
+(cd "$HERE";
+ adt-run run-in-image/ --- ssh -s snappy -- -i image/snappy.img;
+)
+

=== modified file 'run-checks'
--- run-checks	2015-05-19 11:49:41 +0000
+++ run-checks	2015-06-09 15:41:05 +0000
@@ -55,4 +55,10 @@
     exit 1
 fi
 
+# integration tests
+if which adt-run >/dev/null 2>&1; then
+    echo "Running integration tests"
+    ./integration-tests/selftest
+fi
+
 echo "All good, what could possibly go wrong"