Merge lp:~mvo/snappy/selftest-upgrade-rollback into lp:~snappy-dev/snappy/snappy-moved-to-github

[Michael Vogt]

Always do a basic upgrade/rollback test

[Sergio Schvezov (sergiusens)]

Hey, nice to get this all in adt!

I added a few random comments one can lead to false negatives, the other one is just terminology.

Thanks!

[lp:~mvo/snappy/selftest-upgrade-rollback]

43. By Michael Vogt on 2015-05-08

add upgrade test for stable->edge

Unmerged revisions

43. By Michael Vogt on 2015-05-08

add upgrade test for stable->edge

42. By Michael Vogt on 2015-05-08

update to always include upgrade/rollback test

41. By James Hunt on 2015-04-22

* snappy-selftest: Don't use after_reboot() as it assumes it's being
  called from within the test loop.
* tests/framework: after_reboot(): Revert to previous logic.

40. By James Hunt on 2015-04-22

* snappy-selftest:
  - Guard all removals and stop snappy writing to stderr
    (which would trigger a test failure).
* tests/01_test_info: Update for edge channel.
* tests/10_test_info_has_stuff: Update for edge channel.
* tests/framework: after_reboot(): Fix test.

39. By James Hunt on 2015-04-22

* Only remove the framework before the tests first run - it needs to
  still be there after a reboot.

38. By James Hunt on 2015-04-21

* tests/01_test_info: selftest is not a snap.
* tests/10_test_info_has_stuff: Correct regex for hello-world (cope if
  other apps are also installed).

37. By James Hunt on 2015-04-20

* Replaced webdm tests with docker by setting the framework in
  tests/settings.

36. By James Hunt on 2015-04-20

* tests/05_test_install_nonexistant: Snappy output has changed again.

35. By James Hunt on 2015-04-17

* snappy-selftest: Drop developer names from packages.
* tests/04_test_install_hello:
  - Drop '.canonical' developer name.
  - Correct name of called script.
* tests/11_test_xkcd_listens: Drop '.canonical' developer name.

34. By James Hunt on 2015-04-17

* debian/tests/control: Pass '--yes-really' to enable DEP-8 tests.

=== added file 'README'
--- README	1970-01-01 00:00:00 +0000
+++ README	2015-05-08 13:39:06 +0000
@@ -0,0 +1,2 @@
+Run with:
+$ adt-run upgrade-test/ --- ssh -s snappy -- -i /path/to/snappy.img

=== added directory 'debian'
=== renamed directory 'debian' => 'debian.moved'
=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control	1970-01-01 00:00:00 +0000
+++ debian/tests/control	2015-05-08 13:39:06 +0000
@@ -0,0 +1,2 @@
+Test-Command: ./snappy-selftest --yes-really
+Depends:

=== added directory 'meta'
=== added file 'meta/package.yaml'
--- meta/package.yaml	1970-01-01 00:00:00 +0000
+++ meta/package.yaml	2015-05-08 13:39:06 +0000
@@ -0,0 +1,8 @@
+name: snappy-selftest
+version: 0.1
+vendor: Michael Vogt <[e-mail]>
+binaries:
+ - name: snappy-selftest
+   security-policy:
+      apparmor: meta/snappy-selftest.profile
+      seccomp: meta/snappy-selftest.seccomp

=== added file 'meta/readme.md'
--- meta/readme.md	1970-01-01 00:00:00 +0000
+++ meta/readme.md	2015-05-08 13:39:06 +0000
@@ -0,0 +1,3 @@
+snappy self test
+
+run selftest via /apps/snappy-selftest/current/snappy-selftest

=== added file 'meta/snappy-selftest.profile'
--- meta/snappy-selftest.profile	1970-01-01 00:00:00 +0000
+++ meta/snappy-selftest.profile	2015-05-08 13:39:06 +0000
@@ -0,0 +1,34 @@
+#include <tunables/global>
+
+# This profile is copied from the unconfined example profile
+#
+# Its even more permissive as it allows loading other profiles
+# too
+
+# Define vars with unconfined since autopilot rules may reference them
+###VAR###
+
+# TODO: when v3 userspace lands, use:
+# ###PROFILEATTACH### (unconfined) {}
+
+# v2 compatible wildly permissive profile
+###PROFILEATTACH### (attach_disconnected) {
+  capability,
+  network,
+  / rwkl,
+  /** rwlkm,
+  /** pix,
+
+  mount,
+  remount,
+  umount,
+  dbus,
+  signal,
+  ptrace,
+  unix,
+
+  # this is added on top of the unconfined profile and allows
+  # the snappy selftest to run apps like "hello-world" that will
+  # switch to a new aa-profile
+  change_profile -> *,
+}

=== added file 'meta/snappy-selftest.seccomp'
--- meta/snappy-selftest.seccomp	1970-01-01 00:00:00 +0000
+++ meta/snappy-selftest.seccomp	2015-05-08 13:39:06 +0000
@@ -0,0 +1,1 @@
+@unrestricted

=== added file 'snappy-selftest'
--- snappy-selftest	1970-01-01 00:00:00 +0000
+++ snappy-selftest	2015-05-08 13:39:06 +0000
@@ -0,0 +1,49 @@
+#!/bin/bash
+
+set -e
+
+. tests/framework
+
+SNAPPY=snappy
+
+MYDIR=$(dirname $0)
+
+if [ "$1" != "--yes-really" ]; then
+    echo "The selftest may be destructive, please run with:"
+    echo "   --yes-really"
+    echo "if you know what you are doing"
+    exit 1
+fi
+
+. tests/settings
+
+# prepare the environment
+if [ -z "$ADT_REBOOT_MARK" ]; then
+    sudo snappy remove "$framework" 2>&1 || true
+    sudo snappy remove hello-world 2>&1 || true
+    sudo snappy remove xkcd-webserver 2>&1 || true
+fi
+
+for test in $MYDIR/tests/*_test_*; do
+    CURRENT_TEST=$(basename $test)
+
+    # after a reboot, skip ahead to the test that triggered it
+    if [ -n "$ADT_REBOOT_MARK" ]; then
+        if [ "$ADT_REBOOT_MARK" == "$CURRENT_TEST" ]; then
+            echo "Resuming $test after reboot"
+        else
+            continue
+        fi
+    else
+        echo "Running $test"
+    fi
+
+    . "$test"
+    if ! (test 2>&1); then
+        echo "FAILED: $test"
+        exit 1
+    fi
+
+    # clear reboot marker
+    unset ADT_REBOOT_MARK
+done

=== added directory 'tests'
=== added file 'tests/01_test_info'
--- tests/01_test_info	1970-01-01 00:00:00 +0000
+++ tests/01_test_info	2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
+test() {
+    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
+    test_regexp "^frameworks: ($framework|)*$" $SNAPPY info
+}

=== added file 'tests/02_test_versions'
--- tests/02_test_versions	1970-01-01 00:00:00 +0000
+++ tests/02_test_versions	2015-05-08 13:39:06 +0000
@@ -0,0 +1,7 @@
+test() {
+    T="ubuntu-core.*201?"
+    test_regexp "$T" $SNAPPY list
+
+    T="Name.*Date.*Version.*Developer"
+    test_regexp "$T" $SNAPPY list
+}

=== added file 'tests/03_test_apt'
--- tests/03_test_apt	1970-01-01 00:00:00 +0000
+++ tests/03_test_apt	2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
+test() {
+       T="Ubuntu Core does not use apt-get, see 'snappy --help'!"
+       test_equal "$T" apt-get update
+}
\ No newline at end of file

=== added file 'tests/04_test_install_hello'
--- tests/04_test_install_hello	1970-01-01 00:00:00 +0000
+++ tests/04_test_install_hello	2015-05-08 13:39:06 +0000
@@ -0,0 +1,7 @@
+test() {
+       T="hello-world[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+(\.[0-9]+)+[[:space:]]+canonical"
+       test_regexp "$T" sudo $SNAPPY install hello-world
+       
+       T="Hello World!"
+       test_equal "$T" hello-world.echo
+}

=== added file 'tests/05_test_install_nonexistant'
--- tests/05_test_install_nonexistant	1970-01-01 00:00:00 +0000
+++ tests/05_test_install_nonexistant	2015-05-08 13:39:06 +0000
@@ -0,0 +1,5 @@
+test() {
+        pkg=fizzler
+        T="^${pkg} failed to install: snappy package not found"
+        test_regexp "$T" sudo $SNAPPY install $pkg 
+}

=== added file 'tests/06_test_search_framework'
--- tests/06_test_search_framework	1970-01-01 00:00:00 +0000
+++ tests/06_test_search_framework	2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
+test() {
+       T="$framework[[:space:]]+[0-9.]+[[:space:]]+.*"
+       test_regexp "$T" $SNAPPY search "$framework" 
+}

=== added file 'tests/07_test_install_framework'
--- tests/07_test_install_framework	1970-01-01 00:00:00 +0000
+++ tests/07_test_install_framework	2015-05-08 13:39:06 +0000
@@ -0,0 +1,21 @@
+test() {
+    if after_reboot; then
+        # ensure the service comes up after a reboot
+        services=$(systemctl | grep "${framework}_.*.service")
+        [ -n "$services" ] || fail "$framework provided no services"
+        return
+    fi
+
+    # we test that the final line contains the framework command
+    # FIXME: test progress here too
+    T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
+    test_regexp "$T" sudo $SNAPPY install "$framework_full"
+
+    # ensure its there
+    services=$(systemctl | grep "${framework}_.*.service")
+    [ -n "$services" ] || fail "$framework provided no services"
+
+    if can_reboot; then
+        reboot
+    fi
+}

=== added file 'tests/08_test_versions_has_framework'
--- tests/08_test_versions_has_framework	1970-01-01 00:00:00 +0000
+++ tests/08_test_versions_has_framework	2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
+test() {
+    T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
+    test_regexp "$T" $SNAPPY list
+}

=== added file 'tests/10_test_info_has_stuff'
--- tests/10_test_info_has_stuff	1970-01-01 00:00:00 +0000
+++ tests/10_test_info_has_stuff	2015-05-08 13:39:06 +0000
@@ -0,0 +1,5 @@
+test() {
+    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
+    test_regexp "^frameworks: $framework" $SNAPPY info
+    test_regexp "^apps:.*\<hello-world\>" $SNAPPY info
+}

=== added file 'tests/11_test_xkcd_listens'
--- tests/11_test_xkcd_listens	1970-01-01 00:00:00 +0000
+++ tests/11_test_xkcd_listens	2015-05-08 13:39:06 +0000
@@ -0,0 +1,15 @@
+test() {
+       # FIXME: add $SNAPPY search webserver
+       
+       sudo $SNAPPY install xkcd-webserver.canonical
+
+       # FIXME: sucks, needed because "systemctl start" does not
+       #        wait until the service is really started
+       sleep 1
+
+       T="HTTP/1.0 200 OK"
+       TEMPF=$(tempfile)
+       printf "GET / HTTP/1.0\n\n"|nc localhost 80 > $TEMPF
+       test_regexp "$T" cat $TEMPF
+       rm -f $TEMPF
+}

=== added file 'tests/90_test_upgrade'
--- tests/90_test_upgrade	1970-01-01 00:00:00 +0000
+++ tests/90_test_upgrade	2015-05-08 13:39:06 +0000
@@ -0,0 +1,67 @@
+# Test that we can upgrade to the latest image version and roll back to the
+# original version. Test will be skipped the testbed does not offer reboot
+# (needs autopkgtest with a supporting runner).
+
+test() {
+    can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
+
+    versions=$(snappy list)
+    current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
+    echo "original current snappy version: $current"
+
+    # debug
+    echo "Boot order"
+    grep root=LABEL /boot/grub/grub.cfg
+
+    if after_reboot; then
+        # second reboot: after rollback
+        if [ -e $ADT_ARTIFACTS/rolled_back ]; then
+            orig_current=$(cat $ADT_ARTIFACTS/current)
+            [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current"
+            return
+        fi
+
+        # first reboot: after upgrade
+        orig_avail=$(cat $ADT_ARTIFACTS/avail)
+        [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version"
+
+        echo "rolling back..."
+        sudo snappy rollback ubuntu-core
+        touch $ADT_ARTIFACTS/rolled_back
+        reboot
+        return
+    fi
+
+    # fake new available version by doing a current--
+    sudo mount -o remount,rw /
+    sudo sed -i "s/build_number: $current/build_number: $((current-1))/" /etc/system-image/channel.ini
+    sudo mount -o remount,ro /
+
+    if [ -e /writable/cache/system/etc/system-image/channel.ini ]; then
+        sudo mount -o remount,rw /writable/cache/system/
+        sudo sed -i 's/build_number: $current/build_number: $((current-1))/' /writable/cache/system/etc/system-image/channel.ini
+        sudo mount -o remount,ro /writable/cache/system/
+    fi
+
+    # we should have something now :)
+    versions=$(snappy list)
+    current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
+    echo "faked current snappy version: $current"
+
+    versions=$(snappy list -u)
+    avail=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
+    echo "available snappy version to upgrade: $avail"
+    
+    # skip if not numeric, e. g. "-"
+    [ "$avail" != "${avail#[0-9]}" ] || { echo "SKIP: no image to upgrade to"; return; }
+
+    [ $avail -gt $current ] || fail "$avail is not newer than $current"
+
+    # save version for post-upgrade test
+    echo "$avail" > $ADT_ARTIFACTS/avail
+    echo "$current" > $ADT_ARTIFACTS/current
+
+    echo "upgrading..."
+    sudo snappy update
+    reboot
+}

=== added file 'tests/91_test_upgrade_from_stable_image'
--- tests/91_test_upgrade_from_stable_image	1970-01-01 00:00:00 +0000
+++ tests/91_test_upgrade_from_stable_image	2015-05-08 13:39:06 +0000
@@ -0,0 +1,69 @@
+# Test that we can upgrade from the latest *stable* image version to the
+# current edge image version and rollback original version.
+#
+# Make image for this with:
+#
+# $ sudo ubuntu-device-flash core  --channel=stable 15.04 -o snappy-stable.img --developer-mode --enable-ssh
+#
+# Run with:
+# adt-run upgrade-test/ --- ssh -s snappy -- -i /path/to/snappy-stable.img
+
+test() {
+    can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
+
+    versions=$(snappy list)
+    current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
+    echo "original current snappy version: $current"
+
+    # debug
+    echo "Boot order"
+    grep root=LABEL /boot/grub/grub.cfg
+
+    if after_reboot; then
+        # second reboot: after rollback
+        if [ -e $ADT_ARTIFACTS/rolled_back-stable ]; then
+            orig_current=$(cat $ADT_ARTIFACTS/current)
+            [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current"
+            return
+        fi
+
+        # first reboot: after upgrade
+        orig_avail=$(cat $ADT_ARTIFACTS/avail)
+        [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version"
+
+        echo "rolling back..."
+        sudo snappy rollback ubuntu-core
+        touch $ADT_ARTIFACTS/rolled_back-stable
+        reboot
+        return
+    fi
+
+    # switch channel
+    sudo mount -o remount,rw /
+    sudo sed -i 's#channel: ubuntu-core/\(.*\)/stable#channel: ubuntu-core/\1/edge#' /etc/system-image/channel.ini
+    sudo mount -o remount,ro /
+
+    if [ -e /writable/cache/system/etc/system-image/channel.ini ]; then
+        sudo mount -o remount,rw /writable/cache/system/
+        sudo sed -i 's#channel: ubuntu-core/\(.*\)/stable#channel: ubuntu-core/\1/edge#' /writable/cache/system/etc/system-image/channel.ini
+        sudo mount -o remount,ro /writable/cache/system/
+    fi
+
+    # now upgrade to latest edge from stable
+    versions=$(snappy list -u)
+    avail=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
+    echo "available snappy version to upgrade: $avail"
+    
+    # skip if not numeric, e. g. "-"
+    [ "$avail" != "${avail#[0-9]}" ] || { echo "SKIP: no image to upgrade to"; return; }
+
+    [ $avail -gt $current ] || fail "$avail is not newer than $current"
+
+    # save version for post-upgrade test
+    echo "$avail" > $ADT_ARTIFACTS/avail
+    echo "$current" > $ADT_ARTIFACTS/current
+
+    echo "upgrading..."
+    sudo snappy update
+    reboot
+}

=== added file 'tests/framework'
--- tests/framework	1970-01-01 00:00:00 +0000
+++ tests/framework	2015-05-08 13:39:06 +0000
@@ -0,0 +1,46 @@
+fail() {
+    echo "ERROR" $@
+    exit 1
+}
+
+test_equal() {
+    local expected="$1"
+    shift
+    "$@" 2>&1 | diff -u <(echo "$expected") -
+}
+
+test_regexp() {
+    local REGEXP="$1"
+    shift
+    local ACTUAL=$("$@" 2>&1)
+    if ! echo "$ACTUAL" | grep -E -q "$REGEXP"; then
+        echo "Can not find '$REGEXP' in '$ACTUAL'"
+        fail
+    fi
+}
+
+# check if reboot is supported; this needs running with autopkgtest on a
+# supported testbed (QEMU or ssh with a setup script that announces the
+# "reboot" capability)
+can_reboot() {
+    [ -x /tmp/autopkgtest-reboot ]
+}
+
+# reboot the testbed; the same test will be started again, check after_reboot()
+# to see in which state you are in
+reboot() {
+    if [ -z "$CURRENT_TEST" ]; then
+        echo 'ERROR: $CURRENT_TEST not defined' >&2
+        exit 1
+    fi
+    echo "Rebooting testbed..."
+    sudo /tmp/autopkgtest-reboot "$CURRENT_TEST"
+}
+
+# true if the current test reboot()ed and is now restarted after boot; false if
+# it's the first time the test runs (or reboot is not supported)
+after_reboot() {
+    [ "$ADT_REBOOT_MARK" = "$CURRENT_TEST" ]
+}
+
+# vim: filetype=sh shiftwidth=4

=== added file 'tests/settings'
--- tests/settings	1970-01-01 00:00:00 +0000
+++ tests/settings	2015-05-08 13:39:06 +0000
@@ -0,0 +1,3 @@
+# name of framework that will be installed for testing
+framework=docker
+framework_full=docker