Merge lp:~elopio/snappy/bbb_integration into lp:~snappy-dev/snappy/snappy-moved-to-github

[Leo Arias]

Integration tests in the latest package working on beaglebone black.
Integration tests README updated.

following branches with shell, update and failover.

[Leo Arias (elopio)]

There seems to be a problem when running the update twice on beagle. The second time, after the reboot, this happens: http://paste.ubuntu.com/11839856/

[Federico Gimenez (fgimenez)]

Same here after the second update (both via fake), working fine on kvm.

Otherwise, the changes look good and don't seem to be the cause of this.

review: Approve

[Leo Arias (elopio)]

Let me remove the update for now. Then this would be progress, as we will have the latest tests passing on beagle.

[lp:~elopio/snappy/bbb_integration]

537. By Leo Arias on 2015-07-08

Removed the update for now.

538. By Leo Arias on 2015-07-08

Updated the comment.

=== renamed file '_integration-tests/README' => '_integration-tests/README.md'
--- _integration-tests/README	2015-06-25 18:44:08 +0000
+++ _integration-tests/README.md	2015-07-08 16:08:32 +0000
@@ -1,21 +1,69 @@
-# Automatic testing for snappy
-
-## qemu based x86 testing
-
-Run locally with:
-$ adt-run $(pwd)/ --- ssh -s snappy -- -i /path/to/snappy.img
-
-
-## BeagleBoneBlack real HW testing
-
-With autopkgtest 3.14 it can also drive a BBB directly, this requires
-that the bbb contains a clean image. Then run:
-
-$ adt-run $(pwd)/ --- ssh --reboot -l ubuntu -P ubuntu --reboot -H webdm.local
-
-## Native Go tests
-
-Having golang installed, you can execute the full suite (including both go and
-shell tests) with:
-
-$ go run _integration-test/main.go
+# Integration testing for snappy
+
+## Requirements
+
+ *  autopkgtest (>= 3.15.1)
+
+    Get the latest autopkgtest deb from
+    https://packages.debian.org/sid/all/autopkgtest/download
+
+ *  Internet access in the test bed.
+
+## Testing a virtual machine
+
+You can execute the full integration suite in a local virtual machine with:
+
+    go run _integration-test/main.go
+
+The test runner will create the snappy images with `ubuntu-device-flash`, so it
+will ask for your password to run this command with `sudo`.
+
+## Testing snappy from a branch
+
+With the --snappy-from-branch flag, the snappy CLI command will be compiled
+from the current branch, copied to the test bed and used during the integration
+tests:
+
+    go run _integration-tests/main.go --snappy-from-branch
+
+You can use this flag to test in a remote machine too.
+
+## Filtering the tests to run
+
+With the --filter flag you can select the tests to run. For instance you can
+pass MyTestSuite, MyTestSuite.FirstCustomTest or MyTestSuite.*CustomTest:
+
+    go run _integration-tests/main.go --filter MyTestSuite.FirstCustomTest
+
+## Testing a remote machine
+
+You can execute the integration suite in a remote snappy machine with:
+
+    go run _integration-test/main.go --ip {testbed-ip} --port {testbed-port} \
+    --arch {testbed-arch}
+
+The test runner will use `ssh-copy-id` to send your identity file to the
+testbed, so it will ask for the password of the ubuntu user in the test bed.
+
+When running in a remote machine, the test runner assumes the test bed is in
+the latest rolling edge version, and it will skip all the tests that
+require a different version. See the following section for instructions for
+setting up a BeagleBone Black as the test bed.
+
+## Testing a BeagleBone Black
+
+First flash the latest rolling edge version into the sd card
+(replacing /dev/sdX with the path to your card):
+
+    sudo ubuntu-device-flash core rolling --channel edge --oem beagleblack \
+    --developer-mode --enable-ssh -o ubuntu-rolling-edge-armhf-bbb.img
+
+    sudo dd if=ubuntu-rolling-edge-armhf-bbb.img of=/dev/sdX bs=32M
+    sync
+
+Then boot the board with the sd card, make sure that it is connected to the
+same network as the test runner host, and find the {beaglebone-ip}.
+
+Run the tests with:
+
+    go run _integration-tests/main.go --ip {beaglebone-ip} --arch arm

=== modified file '_integration-tests/main.go'
--- _integration-tests/main.go	2015-07-07 04:20:01 +0000
+++ _integration-tests/main.go	2015-07-08 16:08:32 +0000
@@ -80,8 +80,9 @@
 	} else {
 		execCommand("ssh-copy-id", "-p", strconv.Itoa(testbedPort),
 			"ubuntu@"+testbedIP)
-		// Run the shell tests. TODO: Also run the other tests.
-		adtRun(rootPath, "", []string{}, remoteTestbedSSHOptions(testbedIP, testbedPort), true)
+		// TODO: Also run the other tests.
+		adtRun(rootPath, testFilter, []string{"latest"},
+			remoteTestbedSSHOptions(testbedIP, testbedPort), false)
 	}
 }
 

=== modified file '_integration-tests/tests/latest/build_test.go'
--- _integration-tests/tests/latest/build_test.go	2015-07-04 05:49:13 +0000
+++ _integration-tests/tests/latest/build_test.go	2015-07-08 16:08:32 +0000
@@ -61,9 +61,9 @@
 	expected = "" +
 		"Installing " + snapName + "\n" +
 		".*Signature check failed, but installing anyway as requested\n" +
-		"Name          Date       Version Developer \n" +
+		"Name +Date +Version +Developer \n" +
 		".*\n" +
-		basicSnapName + "   .* .*  sideload  \n" +
+		basicSnapName + " +.* +.* +sideload  \n" +
 		".*\n"
 
 	c.Check(installOutput, Matches, expected)

=== modified file '_integration-tests/tests/latest/install_test.go'
--- _integration-tests/tests/latest/install_test.go	2015-07-08 08:49:22 +0000
+++ _integration-tests/tests/latest/install_test.go	2015-07-08 16:08:32 +0000
@@ -42,12 +42,12 @@
 func (s *installSuite) TestInstallSnapMustPrintPackageInformation(c *C) {
 	installOutput := InstallSnap(c, "hello-world")
 
-	expected := "" +
+	expected := "(?ms)" +
 		"Installing hello-world\n" +
-		"Name          Date       Version Developer \n" +
-		".*\n" +
-		"hello-world   .* .*  canonical \n" +
-		".*\n"
+		"Name +Date +Version +Developer \n" +
+		".*" +
+		"^hello-world +.* +.* +canonical \n" +
+		".*"
 
 	c.Assert(installOutput, Matches, expected)
 }

=== modified file '_integration-tests/tests/update/update_test.go'
--- _integration-tests/tests/update/update_test.go	2015-07-06 22:27:42 +0000
+++ _integration-tests/tests/update/update_test.go	2015-07-08 16:08:32 +0000
@@ -44,6 +44,7 @@
 		CallUpdate(c)
 		Reboot(c)
 	} else if AfterReboot(c) {
+		RemoveRebootMark(c)
 		c.Assert(GetCurrentVersion(c) > GetSavedVersion(c), Equals, true)
 	}
 }