Merge lp:~mvo/snappy/selftest-upgrade-rollback into lp:~snappy-dev/snappy/snappy-moved-to-github

Proposed by Michael Vogt
Status: Superseded
Proposed branch: lp:~mvo/snappy/selftest-upgrade-rollback
Merge into: lp:~snappy-dev/snappy/snappy-moved-to-github
Diff against target: 470 lines (+360/-0) (has conflicts)
21 files modified
README (+2/-0)
debian/tests/control (+2/-0)
meta/package.yaml (+8/-0)
meta/readme.md (+3/-0)
meta/snappy-selftest.profile (+34/-0)
meta/snappy-selftest.seccomp (+1/-0)
snappy-selftest (+49/-0)
tests/01_test_info (+4/-0)
tests/02_test_versions (+7/-0)
tests/03_test_apt (+4/-0)
tests/04_test_install_hello (+7/-0)
tests/05_test_install_nonexistant (+5/-0)
tests/06_test_search_framework (+4/-0)
tests/07_test_install_framework (+21/-0)
tests/08_test_versions_has_framework (+4/-0)
tests/10_test_info_has_stuff (+5/-0)
tests/11_test_xkcd_listens (+15/-0)
tests/90_test_upgrade (+67/-0)
tests/91_test_upgrade_from_stable_image (+69/-0)
tests/framework (+46/-0)
tests/settings (+3/-0)
Conflict adding file debian.  Moved existing file to debian.moved.
To merge this branch: bzr merge lp:~mvo/snappy/selftest-upgrade-rollback
Reviewer Review Type Date Requested Status
Snappy Developers Pending
Review via email: mp+258605@code.launchpad.net

This proposal has been superseded by a proposal from 2015-05-08.

Description of the change

Always do a basic upgrade/rollback test

To post a comment you must log in.
Revision history for this message
Sergio Schvezov (sergiusens) wrote :

Hey, nice to get this all in adt!

I added a few random comments one can lead to false negatives, the other one is just terminology.

Thanks!

43. By Michael Vogt

add upgrade test for stable->edge

Unmerged revisions

43. By Michael Vogt

add upgrade test for stable->edge

42. By Michael Vogt

update to always include upgrade/rollback test

41. By James Hunt

* snappy-selftest: Don't use after_reboot() as it assumes it's being
  called from within the test loop.
* tests/framework: after_reboot(): Revert to previous logic.

40. By James Hunt

* snappy-selftest:
  - Guard all removals and stop snappy writing to stderr
    (which would trigger a test failure).
* tests/01_test_info: Update for edge channel.
* tests/10_test_info_has_stuff: Update for edge channel.
* tests/framework: after_reboot(): Fix test.

39. By James Hunt

* Only remove the framework before the tests first run - it needs to
  still be there after a reboot.

38. By James Hunt

* tests/01_test_info: selftest is not a snap.
* tests/10_test_info_has_stuff: Correct regex for hello-world (cope if
  other apps are also installed).

37. By James Hunt

* Replaced webdm tests with docker by setting the framework in
  tests/settings.

36. By James Hunt

* tests/05_test_install_nonexistant: Snappy output has changed again.

35. By James Hunt

* snappy-selftest: Drop developer names from packages.
* tests/04_test_install_hello:
  - Drop '.canonical' developer name.
  - Correct name of called script.
* tests/11_test_xkcd_listens: Drop '.canonical' developer name.

34. By James Hunt

* debian/tests/control: Pass '--yes-really' to enable DEP-8 tests.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added file 'README'
--- README 1970-01-01 00:00:00 +0000
+++ README 2015-05-08 13:39:06 +0000
@@ -0,0 +1,2 @@
1Run with:
2$ adt-run upgrade-test/ --- ssh -s snappy -- -i /path/to/snappy.img
03
=== added directory 'debian'
=== renamed directory 'debian' => 'debian.moved'
=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control 1970-01-01 00:00:00 +0000
+++ debian/tests/control 2015-05-08 13:39:06 +0000
@@ -0,0 +1,2 @@
1Test-Command: ./snappy-selftest --yes-really
2Depends:
03
=== added directory 'meta'
=== added file 'meta/package.yaml'
--- meta/package.yaml 1970-01-01 00:00:00 +0000
+++ meta/package.yaml 2015-05-08 13:39:06 +0000
@@ -0,0 +1,8 @@
1name: snappy-selftest
2version: 0.1
3vendor: Michael Vogt <mvo@ubuntu.com>
4binaries:
5 - name: snappy-selftest
6 security-policy:
7 apparmor: meta/snappy-selftest.profile
8 seccomp: meta/snappy-selftest.seccomp
09
=== added file 'meta/readme.md'
--- meta/readme.md 1970-01-01 00:00:00 +0000
+++ meta/readme.md 2015-05-08 13:39:06 +0000
@@ -0,0 +1,3 @@
1snappy self test
2
3run selftest via /apps/snappy-selftest/current/snappy-selftest
04
=== added file 'meta/snappy-selftest.profile'
--- meta/snappy-selftest.profile 1970-01-01 00:00:00 +0000
+++ meta/snappy-selftest.profile 2015-05-08 13:39:06 +0000
@@ -0,0 +1,34 @@
1#include <tunables/global>
2
3# This profile is copied from the unconfined example profile
4#
5# Its even more permissive as it allows loading other profiles
6# too
7
8# Define vars with unconfined since autopilot rules may reference them
9###VAR###
10
11# TODO: when v3 userspace lands, use:
12# ###PROFILEATTACH### (unconfined) {}
13
14# v2 compatible wildly permissive profile
15###PROFILEATTACH### (attach_disconnected) {
16 capability,
17 network,
18 / rwkl,
19 /** rwlkm,
20 /** pix,
21
22 mount,
23 remount,
24 umount,
25 dbus,
26 signal,
27 ptrace,
28 unix,
29
30 # this is added on top of the unconfined profile and allows
31 # the snappy selftest to run apps like "hello-world" that will
32 # switch to a new aa-profile
33 change_profile -> *,
34}
035
=== added file 'meta/snappy-selftest.seccomp'
--- meta/snappy-selftest.seccomp 1970-01-01 00:00:00 +0000
+++ meta/snappy-selftest.seccomp 2015-05-08 13:39:06 +0000
@@ -0,0 +1,1 @@
1@unrestricted
02
=== added file 'snappy-selftest'
--- snappy-selftest 1970-01-01 00:00:00 +0000
+++ snappy-selftest 2015-05-08 13:39:06 +0000
@@ -0,0 +1,49 @@
1#!/bin/bash
2
3set -e
4
5. tests/framework
6
7SNAPPY=snappy
8
9MYDIR=$(dirname $0)
10
11if [ "$1" != "--yes-really" ]; then
12 echo "The selftest may be destructive, please run with:"
13 echo " --yes-really"
14 echo "if you know what you are doing"
15 exit 1
16fi
17
18. tests/settings
19
20# prepare the environment
21if [ -z "$ADT_REBOOT_MARK" ]; then
22 sudo snappy remove "$framework" 2>&1 || true
23 sudo snappy remove hello-world 2>&1 || true
24 sudo snappy remove xkcd-webserver 2>&1 || true
25fi
26
27for test in $MYDIR/tests/*_test_*; do
28 CURRENT_TEST=$(basename $test)
29
30 # after a reboot, skip ahead to the test that triggered it
31 if [ -n "$ADT_REBOOT_MARK" ]; then
32 if [ "$ADT_REBOOT_MARK" == "$CURRENT_TEST" ]; then
33 echo "Resuming $test after reboot"
34 else
35 continue
36 fi
37 else
38 echo "Running $test"
39 fi
40
41 . "$test"
42 if ! (test 2>&1); then
43 echo "FAILED: $test"
44 exit 1
45 fi
46
47 # clear reboot marker
48 unset ADT_REBOOT_MARK
49done
050
=== added directory 'tests'
=== added file 'tests/01_test_info'
--- tests/01_test_info 1970-01-01 00:00:00 +0000
+++ tests/01_test_info 2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
1test() {
2 test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
3 test_regexp "^frameworks: ($framework|)*$" $SNAPPY info
4}
05
=== added file 'tests/02_test_versions'
--- tests/02_test_versions 1970-01-01 00:00:00 +0000
+++ tests/02_test_versions 2015-05-08 13:39:06 +0000
@@ -0,0 +1,7 @@
1test() {
2 T="ubuntu-core.*201?"
3 test_regexp "$T" $SNAPPY list
4
5 T="Name.*Date.*Version.*Developer"
6 test_regexp "$T" $SNAPPY list
7}
08
=== added file 'tests/03_test_apt'
--- tests/03_test_apt 1970-01-01 00:00:00 +0000
+++ tests/03_test_apt 2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
1test() {
2 T="Ubuntu Core does not use apt-get, see 'snappy --help'!"
3 test_equal "$T" apt-get update
4}
0\ No newline at end of file5\ No newline at end of file
16
=== added file 'tests/04_test_install_hello'
--- tests/04_test_install_hello 1970-01-01 00:00:00 +0000
+++ tests/04_test_install_hello 2015-05-08 13:39:06 +0000
@@ -0,0 +1,7 @@
1test() {
2 T="hello-world[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+(\.[0-9]+)+[[:space:]]+canonical"
3 test_regexp "$T" sudo $SNAPPY install hello-world
4
5 T="Hello World!"
6 test_equal "$T" hello-world.echo
7}
08
=== added file 'tests/05_test_install_nonexistant'
--- tests/05_test_install_nonexistant 1970-01-01 00:00:00 +0000
+++ tests/05_test_install_nonexistant 2015-05-08 13:39:06 +0000
@@ -0,0 +1,5 @@
1test() {
2 pkg=fizzler
3 T="^${pkg} failed to install: snappy package not found"
4 test_regexp "$T" sudo $SNAPPY install $pkg
5}
06
=== added file 'tests/06_test_search_framework'
--- tests/06_test_search_framework 1970-01-01 00:00:00 +0000
+++ tests/06_test_search_framework 2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
1test() {
2 T="$framework[[:space:]]+[0-9.]+[[:space:]]+.*"
3 test_regexp "$T" $SNAPPY search "$framework"
4}
05
=== added file 'tests/07_test_install_framework'
--- tests/07_test_install_framework 1970-01-01 00:00:00 +0000
+++ tests/07_test_install_framework 2015-05-08 13:39:06 +0000
@@ -0,0 +1,21 @@
1test() {
2 if after_reboot; then
3 # ensure the service comes up after a reboot
4 services=$(systemctl | grep "${framework}_.*.service")
5 [ -n "$services" ] || fail "$framework provided no services"
6 return
7 fi
8
9 # we test that the final line contains the framework command
10 # FIXME: test progress here too
11 T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
12 test_regexp "$T" sudo $SNAPPY install "$framework_full"
13
14 # ensure its there
15 services=$(systemctl | grep "${framework}_.*.service")
16 [ -n "$services" ] || fail "$framework provided no services"
17
18 if can_reboot; then
19 reboot
20 fi
21}
022
=== added file 'tests/08_test_versions_has_framework'
--- tests/08_test_versions_has_framework 1970-01-01 00:00:00 +0000
+++ tests/08_test_versions_has_framework 2015-05-08 13:39:06 +0000
@@ -0,0 +1,4 @@
1test() {
2 T="$framework[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+"
3 test_regexp "$T" $SNAPPY list
4}
05
=== added file 'tests/10_test_info_has_stuff'
--- tests/10_test_info_has_stuff 1970-01-01 00:00:00 +0000
+++ tests/10_test_info_has_stuff 2015-05-08 13:39:06 +0000
@@ -0,0 +1,5 @@
1test() {
2 test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
3 test_regexp "^frameworks: $framework" $SNAPPY info
4 test_regexp "^apps:.*\<hello-world\>" $SNAPPY info
5}
06
=== added file 'tests/11_test_xkcd_listens'
--- tests/11_test_xkcd_listens 1970-01-01 00:00:00 +0000
+++ tests/11_test_xkcd_listens 2015-05-08 13:39:06 +0000
@@ -0,0 +1,15 @@
1test() {
2 # FIXME: add $SNAPPY search webserver
3
4 sudo $SNAPPY install xkcd-webserver.canonical
5
6 # FIXME: sucks, needed because "systemctl start" does not
7 # wait until the service is really started
8 sleep 1
9
10 T="HTTP/1.0 200 OK"
11 TEMPF=$(tempfile)
12 printf "GET / HTTP/1.0\n\n"|nc localhost 80 > $TEMPF
13 test_regexp "$T" cat $TEMPF
14 rm -f $TEMPF
15}
016
=== added file 'tests/90_test_upgrade'
--- tests/90_test_upgrade 1970-01-01 00:00:00 +0000
+++ tests/90_test_upgrade 2015-05-08 13:39:06 +0000
@@ -0,0 +1,67 @@
1# Test that we can upgrade to the latest image version and roll back to the
2# original version. Test will be skipped the testbed does not offer reboot
3# (needs autopkgtest with a supporting runner).
4
5test() {
6 can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
7
8 versions=$(snappy list)
9 current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
10 echo "original current snappy version: $current"
11
12 # debug
13 echo "Boot order"
14 grep root=LABEL /boot/grub/grub.cfg
15
16 if after_reboot; then
17 # second reboot: after rollback
18 if [ -e $ADT_ARTIFACTS/rolled_back ]; then
19 orig_current=$(cat $ADT_ARTIFACTS/current)
20 [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current"
21 return
22 fi
23
24 # first reboot: after upgrade
25 orig_avail=$(cat $ADT_ARTIFACTS/avail)
26 [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version"
27
28 echo "rolling back..."
29 sudo snappy rollback ubuntu-core
30 touch $ADT_ARTIFACTS/rolled_back
31 reboot
32 return
33 fi
34
35 # fake new available version by doing a current--
36 sudo mount -o remount,rw /
37 sudo sed -i "s/build_number: $current/build_number: $((current-1))/" /etc/system-image/channel.ini
38 sudo mount -o remount,ro /
39
40 if [ -e /writable/cache/system/etc/system-image/channel.ini ]; then
41 sudo mount -o remount,rw /writable/cache/system/
42 sudo sed -i 's/build_number: $current/build_number: $((current-1))/' /writable/cache/system/etc/system-image/channel.ini
43 sudo mount -o remount,ro /writable/cache/system/
44 fi
45
46 # we should have something now :)
47 versions=$(snappy list)
48 current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
49 echo "faked current snappy version: $current"
50
51 versions=$(snappy list -u)
52 avail=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
53 echo "available snappy version to upgrade: $avail"
54
55 # skip if not numeric, e. g. "-"
56 [ "$avail" != "${avail#[0-9]}" ] || { echo "SKIP: no image to upgrade to"; return; }
57
58 [ $avail -gt $current ] || fail "$avail is not newer than $current"
59
60 # save version for post-upgrade test
61 echo "$avail" > $ADT_ARTIFACTS/avail
62 echo "$current" > $ADT_ARTIFACTS/current
63
64 echo "upgrading..."
65 sudo snappy update
66 reboot
67}
068
=== added file 'tests/91_test_upgrade_from_stable_image'
--- tests/91_test_upgrade_from_stable_image 1970-01-01 00:00:00 +0000
+++ tests/91_test_upgrade_from_stable_image 2015-05-08 13:39:06 +0000
@@ -0,0 +1,69 @@
1# Test that we can upgrade from the latest *stable* image version to the
2# current edge image version and rollback original version.
3#
4# Make image for this with:
5#
6# $ sudo ubuntu-device-flash core --channel=stable 15.04 -o snappy-stable.img --developer-mode --enable-ssh
7#
8# Run with:
9# adt-run upgrade-test/ --- ssh -s snappy -- -i /path/to/snappy-stable.img
10
11test() {
12 can_reboot || { echo "SKIP: cannot reboot testbed"; return; }
13
14 versions=$(snappy list)
15 current=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
16 echo "original current snappy version: $current"
17
18 # debug
19 echo "Boot order"
20 grep root=LABEL /boot/grub/grub.cfg
21
22 if after_reboot; then
23 # second reboot: after rollback
24 if [ -e $ADT_ARTIFACTS/rolled_back-stable ]; then
25 orig_current=$(cat $ADT_ARTIFACTS/current)
26 [ "$orig_current" = "$current" ] || fail "did not roll back to version $orig_current"
27 return
28 fi
29
30 # first reboot: after upgrade
31 orig_avail=$(cat $ADT_ARTIFACTS/avail)
32 [ "$orig_avail" = "$current" ] || fail "did not upgrade to current version"
33
34 echo "rolling back..."
35 sudo snappy rollback ubuntu-core
36 touch $ADT_ARTIFACTS/rolled_back-stable
37 reboot
38 return
39 fi
40
41 # switch channel
42 sudo mount -o remount,rw /
43 sudo sed -i 's#channel: ubuntu-core/\(.*\)/stable#channel: ubuntu-core/\1/edge#' /etc/system-image/channel.ini
44 sudo mount -o remount,ro /
45
46 if [ -e /writable/cache/system/etc/system-image/channel.ini ]; then
47 sudo mount -o remount,rw /writable/cache/system/
48 sudo sed -i 's#channel: ubuntu-core/\(.*\)/stable#channel: ubuntu-core/\1/edge#' /writable/cache/system/etc/system-image/channel.ini
49 sudo mount -o remount,ro /writable/cache/system/
50 fi
51
52 # now upgrade to latest edge from stable
53 versions=$(snappy list -u)
54 avail=$(echo "$versions" | awk '/ubuntu-core/ {print $3}')
55 echo "available snappy version to upgrade: $avail"
56
57 # skip if not numeric, e. g. "-"
58 [ "$avail" != "${avail#[0-9]}" ] || { echo "SKIP: no image to upgrade to"; return; }
59
60 [ $avail -gt $current ] || fail "$avail is not newer than $current"
61
62 # save version for post-upgrade test
63 echo "$avail" > $ADT_ARTIFACTS/avail
64 echo "$current" > $ADT_ARTIFACTS/current
65
66 echo "upgrading..."
67 sudo snappy update
68 reboot
69}
070
=== added file 'tests/framework'
--- tests/framework 1970-01-01 00:00:00 +0000
+++ tests/framework 2015-05-08 13:39:06 +0000
@@ -0,0 +1,46 @@
1fail() {
2 echo "ERROR" $@
3 exit 1
4}
5
6test_equal() {
7 local expected="$1"
8 shift
9 "$@" 2>&1 | diff -u <(echo "$expected") -
10}
11
12test_regexp() {
13 local REGEXP="$1"
14 shift
15 local ACTUAL=$("$@" 2>&1)
16 if ! echo "$ACTUAL" | grep -E -q "$REGEXP"; then
17 echo "Can not find '$REGEXP' in '$ACTUAL'"
18 fail
19 fi
20}
21
22# check if reboot is supported; this needs running with autopkgtest on a
23# supported testbed (QEMU or ssh with a setup script that announces the
24# "reboot" capability)
25can_reboot() {
26 [ -x /tmp/autopkgtest-reboot ]
27}
28
29# reboot the testbed; the same test will be started again, check after_reboot()
30# to see in which state you are in
31reboot() {
32 if [ -z "$CURRENT_TEST" ]; then
33 echo 'ERROR: $CURRENT_TEST not defined' >&2
34 exit 1
35 fi
36 echo "Rebooting testbed..."
37 sudo /tmp/autopkgtest-reboot "$CURRENT_TEST"
38}
39
40# true if the current test reboot()ed and is now restarted after boot; false if
41# it's the first time the test runs (or reboot is not supported)
42after_reboot() {
43 [ "$ADT_REBOOT_MARK" = "$CURRENT_TEST" ]
44}
45
46# vim: filetype=sh shiftwidth=4
047
=== added file 'tests/settings'
--- tests/settings 1970-01-01 00:00:00 +0000
+++ tests/settings 2015-05-08 13:39:06 +0000
@@ -0,0 +1,3 @@
1# name of framework that will be installed for testing
2framework=docker
3framework_full=docker

Subscribers

People subscribed via source and target branches