Merge ~kissiel/hwcert-tools:add-beta-snap-build-step into hwcert-tools:mini-ci-legacy

[Maciej Kisielewski]

Snapcraft Testing: bring testing of beta branch and tidy up the scripts

[Jonathan Cave (jocave)]

Looks good to me.

Thanks for adding the --devmode and also the testing of beta. As both edge and beta are essentially rolling builds I think it is good to test these.

review: Approve

[Taihsiang Ho (tai271828)]

We need this story https://trello.com/c/zzQ1cetn/552-when-i-work-on-tools-scripts-for-the-tpe-lab-i-want-them-to-be-organized-under-a-sub-project-like-checkbox-project-so-i-can-know
to avoid using branches as repos ; )

[Taihsiang Ho (tai271828)]

Dry-run did not pass on real mini-ci. I did not get the email notification. I tried to manually reproduce again but the flow interrupted. I will check this is caused by this change or the mini-ci framework itself.

I rolled back to another version with --devimode appended only for the current production mini-ci.

[Jonathan Cave (jocave)]

I think a true negative is better than a false positive

[Taihsiang Ho (tai271828)]

Agree.

My previous comment meant that dry-run did not run through the test because something crashed. That is, mini-ci for checkbox-snappy build died. All the emails sent today were not from this MR. I am checking the root cause.

[Taihsiang Ho (tai271828)]

It seems that smoke-test has warm reboot test now to cause the weird behavior of mini-ci. mini-ci does not expect reboot during the test unit. Reboot confuses mini-ci. I will fix mini-ci to fit this checkbox-snappy test by checking if the systemd file is there for the reboot test job.

I will merge mini-ci fix first and then this test script.

[Taihsiang Ho (tai271828)]

I realized the reboot test of smoke-test[1] could not trace the this test script as well and not only make the mini-ci itself confused. Please see my line comments.

The first thing came to my mind is to prepare another specific test plan for mini-ci rather than an updated test script and changing the flow of the current mini-ci, because it is easier and won't affect the other components in the mini-ci flow, which is not designated to be able to trace reboot status.

[1]

------------[ Running job 471 / 494. Estimated time left: unknown ]-------------
---------------------------[ Perform warm reboot 1 ]----------------------------
ID: 2013.com.canonical.certification::warm-boot-loop-reboot1
Category: 2013.com.canonical.certification::stress-tests/warm-boot
Waiting for the system to shut down or reboot...
Created symlink from /etc/systemd/system/multi-user.target.wants/plainbox-autostart.service to /etc/systemd/system/plainbox-autostart.service.
... 8< -------------------------------------------------------------------------
Connection to 10.101.48.116 closed by remote host.
Connection to 10.101.48.116 closed.

review: Needs Fixing

Unmerged commits

7391692... by Maciej Kisielewski on 2017-03-22

add checking of the beta branch of checkbox-snappy

Also:
  use lp:checkbox-snappy instead of the packaging branch from ~checkbox-dev
  make failing one branch not stop the whole script
  invoke checkbox-snappy from ~/ to avoid problems with calling from tmps

Signed-off-by: Maciej Kisielewski <email address hidden>

diff --git a/src/www/snapcraft/run-snapcraft-test.sh b/src/www/snapcraft/run-snapcraft-test.sh
index 0c192ad..c54522c 100755
--- a/src/www/snapcraft/run-snapcraft-test.sh
+++ b/src/www/snapcraft/run-snapcraft-test.sh
@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 # This file is part of Checkbox.
 #
-# Copyright 2016 Canonical Ltd.
+# Copyright 2016-2017 Canonical Ltd.
 # Written by:
 #   Maciej Kisielewski <[e-mail]>
 #
@@ -50,18 +50,50 @@ function run_step {
     echo -n elapsed: ; cat $LOG_DIR/timing.dat
     if [ $outcome -ne 0 ]; then
         pastebin_logs
-        clean_up
-        echo "STOPPING EXECUTION"
-        exit 1
+        return 1
     fi
 }
 
-run_step "git-clone" "git clone -b master https://git.launchpad.net/~checkbox-dev/plainbox-provider-snappy/+git/packaging"
-run_step "update-snapcraft" "snapcraft update"
-(cd packaging && run_step "run-snapcraft" "snapcraft snap -o ../checkbox-snappy.snap")
-run_step "verify-build" "ls -al checkbox-snappy.snap"
-run_step "install-checkbox-snappy" "sudo snap install ./checkbox-snappy.snap"
-run_step "run smoke-test" "checkbox-snappy.smoke-test"
+# master branch test
+function test_master {
+    run_step "git-clone" "git clone -b master https://git.launchpad.net/checkbox-snappy packaging" || return 1
+    run_step "update-snapcraft" "snapcraft update" || return 1
+    (cd packaging && run_step "run-snapcraft" "snapcraft snap -o ../checkbox-snappy.snap") || return 1
+    run_step "verify-build" "ls -al checkbox-snappy.snap" || return 1
+    run_step "install-checkbox-snappy" "sudo snap install --devmode ./checkbox-snappy.snap" || return 1
+    (cd && run_step "run smoke-test" "checkbox-snappy.smoke-test") || return 1
+    return 0
+}
+test_master
+master_result=$?
+if [[ $master_result -eq 1 ]]; then
+    echo "Testing Checkbox snap for branch master failed!"
+fi
 pastebin_logs
+
+# beta branch test
+function test_beta {
+    run_step "git-clone" "git clone -b beta https://git.launchpad.net/checkbox-snappy packaging-beta" || return 1
+    run_step "update-snapcraft" "snapcraft update" || return 1
+    (cd packaging-beta && run_step "run-snapcraft" "snapcraft snap -o ../checkbox-snappy-beta.snap") || return 1
+    run_step "verify-build" "ls -al checkbox-snappy-beta.snap" || return 1
+    run_step "install-checkbox-snappy" "sudo snap install --devmode ./checkbox-snappy-beta.snap" || return 1
+    (cd && run_step "run smoke-test" "checkbox-snappy.smoke-test") || return 1
+    return 0
+}
+test_beta
+beta_result=$?
+if [[ $beta_result -eq 1 ]]; then
+    echo "Testing Checkbox snap for branch beta failed!"
+fi
+pastebin_logs
+
 clean_up
-echo "ALL STEPS PASSED"
+echo "$master_result"
+echo "$beta_result"
+if [[ $master_result -eq 0 && $beta_result -eq 0 ]]; then
+    echo "ALL STEPS PASSED"
+    exit 0
+else
+    exit 1
+fi