Merge lp:~fgimenez/snappy/go-functional-tests into lp:~snappy-dev/snappy/snappy-moved-to-github

[Federico Gimenez]

Wrapper for go-based dep8 tests and first functional test.

Wrapper for go-based dep8 tests and first functional test

[Federico Gimenez (fgimenez)]

@Sergio thanks a lot for your review!

Ok, now it doesn't check the install output, just the error. You mentioned checking an error output, how could this be done?

Now it's still checking the output from the binary, maybe this can be removed for the same reason, as we are checking for the error response here too?

Cheers!

[Leo Arias (elopio)]

> Ok, now it doesn't check the install output, just the error. You mentioned
> checking an error output, how could this be done?

I think what Sergio meant was that we should use a non-fatal assertion, so the tests continues even if the output of the install command is not what we expect. We can do this by just changing c.Assert by c.Check. I'll propose this in a new branch to be merged with yours.

[Leo Arias (elopio)]

After thinking it twice, instead of using a non-fatal assertion, I prefer to split the test in two. I'm testing the branch.

[Leo Arias (elopio)]

Leaving my approval here for the record. Somebody else should +1. Adding mvo as reviewer, because he did the original selftest. We also might need a +1 from an ubuntu core dev; luckily we have plenty on the team :)

review: Approve

[John Lenton (chipaca)]

several nits / stylistic questions. Only thing really needing fixing is the quoting of MYDIR.

review: Needs Fixing

[Federico Gimenez (fgimenez)]

@Chipaca thanks :) it should be fixed now, i've taken the opportunity to quote other occurrence of $MYDIR in the shell script.

Cheers!

[John Lenton (chipaca)]

review: Approve

[Leo Arias (elopio)]

All good, what could possibly go wrong

:D

review: Approve

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~fgimenez/snappy/go-functional-tests into lp:snappy failed. Below is the output from the failed tests.

Checking docs
Checking formatting
Installing godeps
Install golint
Obtaining dependencies
update code.google.com/p/go.crypto failed; trying to fetch newer version
update github.com/blakesmith/ar failed; trying to fetch newer version
code.google.com/p/go.crypto now at 69e2a90ed92d03812364aeb947b7068dc42e561e
update github.com/cheggaaa/pb failed; trying to fetch newer version
github.com/blakesmith/ar now at c9a977dd0cc1392b023382c7bfa5a22af8d3b730
update github.com/jessevdk/go-flags failed; trying to fetch newer version
github.com/cheggaaa/pb now at e8c7cc515bfde3e267957a3b110080ceed51354e
update github.com/mvo5/goconfigparser failed; trying to fetch newer version
github.com/jessevdk/go-flags now at 15347ef417a300349807983f15af9e65cd2e1b3a
update gopkg.in/check.v1 failed; trying to fetch newer version
github.com/mvo5/goconfigparser now at 26426272dda20cc76aa1fa44286dc743d2972fe8
update gopkg.in/yaml.v2 failed; trying to fetch newer version
gopkg.in/check.v1 now at 64131543e7896d5bcc6bd5a76287eb75ea96c673
gopkg.in/yaml.v2 now at 49c95bdc21843256fb6c4e0d370a05f24a0bf213
Building
Running tests from /home/tarmac/tmp/tmp.Jzd2H0EvYX/src/launchpad.net/snappy
=== RUN Test
OK: 16 passed
--- PASS: Test (0.39 seconds)
PASS
coverage: 74.3% of statements
ok launchpad.net/snappy/clickdeb 0.398s coverage: 74.3% of statements
=== RUN Test
OK: 6 passed
--- PASS: Test (0.01 seconds)
PASS
coverage: 12.9% of statements
ok launchpad.net/snappy/cmd/snappy 0.030s coverage: 12.9% of statements
=== RUN Test
OK: 24 passed
--- PASS: Test (0.10 seconds)
PASS
coverage: 100.0% of statements
ok launchpad.net/snappy/coreconfig 0.103s coverage: 100.0% of statements
=== RUN Test
OK: 50 passed
--- PASS: Test (0.49 seconds)
PASS
coverage: 80.9% of statements
ok launchpad.net/snappy/helpers 0.497s coverage: 80.9% of statements
=== RUN Test
OK: 6 passed
--- PASS: Test (0.00 seconds)
PASS
coverage: 93.5% of statements
ok launchpad.net/snappy/logger 0.005s coverage: 93.5% of statements
=== RUN Test
OK: 4 passed
--- PASS: Test (0.00 seconds)
PASS
coverage: 100.0% of statements
ok launchpad.net/snappy/oauth 0.005s coverage: 100.0% of statements
=== RUN Test
OK: 36 passed
--- PASS: Test (0.19 seconds)
PASS
coverage: 81.5% of statements
ok launchpad.net/snappy/partition 0.193s coverage: 81.5% of statements
=== RUN Test
OK: 3 passed
--- PASS: Test (0.00 seconds)
PASS
coverage: 100.0% of statements
ok launchpad.net/snappy/pkg 0.004s coverage: 100.0% of statements
=== RUN Test
OK: 12 passed
--- PASS: Test (0.11 seconds)
PASS
coverage: 94.3% of statements
ok launchpad.net/snappy/policy 0.110s coverage: 94.3% of statements
=== RUN Test
OK: 5 passed
--- PASS: Test (0.20 seconds)
PASS
coverage: 91.1% of statements
ok launchpad.net/snappy/priv 0.207s coverage: 91.1% of statements
=== RUN Test
OK: 4 passed
--- PASS: Test (0.00 seconds)
PASS
coverage: 50.0% of statements
ok launchpad.net/snappy/progress 0.017s coverage: 50.0% of statements
=== RUN Test
OK: 9 passed
--- PASS: Test (0.01 seconds)
PASS
coverage: 96.4% of statements
ok launchpad.net/sn...

=== renamed directory 'integration-tests' => '_integration-tests'
=== renamed file 'integration-tests/run-in-image/README' => '_integration-tests/README'
--- integration-tests/run-in-image/README	2015-05-22 12:44:41 +0000
+++ _integration-tests/README	2015-06-25 16:10:43 +0000
@@ -14,3 +14,10 @@
 
 $ adt-run $(pwd)/ --- ssh --reboot -l ubuntu -P ubuntu --reboot -H webdm.local
 
+## Native Go tests
+
+Having golang installed, you can execute the full suite (including both go and
+shell tests) with:
+
+$ go run _integration-test/main.go
+

=== added file '_integration-tests/main.go'
--- _integration-tests/main.go	1970-01-01 00:00:00 +0000
+++ _integration-tests/main.go	2015-06-25 16:10:43 +0000
@@ -0,0 +1,113 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+
+/*
+ * Copyright (C) 2015 Canonical Ltd
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+package main
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"path/filepath"
+)
+
+const (
+	baseDir         = "/tmp/snappy-test"
+	debsTestBedPath = "/tmp/snappy-debs"
+	defaultRelease  = "rolling"
+	defaultChannel  = "edge"
+)
+
+var (
+	debsDir     = filepath.Join(baseDir, "debs")
+	testsDir    = filepath.Join(baseDir, "tests")
+	imageDir    = filepath.Join(baseDir, "image")
+	outputDir   = filepath.Join(baseDir, "output")
+	imageTarget = filepath.Join(imageDir, "snappy.img")
+	testFile    = filepath.Join(testsDir, "snappy.tests")
+)
+
+func execCommand(cmds ...string) {
+	cmd := exec.Command(cmds[0], cmds[1:]...)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	if err := cmd.Run(); err != nil {
+		log.Fatalf("Error while running %s: %s\n", cmd.Args, err)
+	}
+}
+
+func buildTests() {
+	fmt.Println("Building tests")
+	prepareTargetDir(testsDir)
+	execCommand("go", "test", "-c", "./_integration-tests/tests")
+	execCommand("mv", "tests.test", testFile)
+}
+
+func createImage(release, channel string) {
+	fmt.Println("Creating image...")
+	prepareTargetDir(imageDir)
+	execCommand(
+		"sudo", "ubuntu-device-flash", "--verbose",
+		"core", release,
+		"-o", imageTarget,
+		"--channel", channel,
+		"--developer-mode")
+}
+
+func adtRun(rootPath string) {
+	fmt.Println("Calling adt-run...")
+	prepareTargetDir(outputDir)
+	execCommand(
+		"adt-run",
+		"-B",
+		"--setup-commands", "touch /run/autopkgtest_no_reboot.stamp",
+		"--override-control", "debian/integration-tests/control",
+		"--built-tree", rootPath,
+		"--output-dir", outputDir,
+		fmt.Sprintf("--copy=%s:%s", testsDir, testsDir),
+		"---",
+		"ssh", "-s", "/usr/share/autopkgtest/ssh-setup/snappy",
+		"--", "-i", imageTarget)
+}
+
+func prepareTargetDir(targetDir string) {
+	if _, err := os.Stat(targetDir); err == nil {
+		// dir exists, remove it
+		os.RemoveAll(targetDir)
+	}
+	os.MkdirAll(targetDir, 0777)
+}
+
+func getRootPath() string {
+	dir, err := os.Getwd()
+	if err != nil {
+		log.Fatal(err)
+	}
+	return dir
+}
+
+func main() {
+	rootPath := getRootPath()
+
+	buildTests()
+
+	createImage(defaultRelease, defaultChannel)
+
+	adtRun(rootPath)
+}

=== renamed file 'integration-tests/run-in-image/snappy-selftest' => '_integration-tests/snappy-selftest'
--- integration-tests/run-in-image/snappy-selftest	2015-05-22 12:44:41 +0000
+++ _integration-tests/snappy-selftest	2015-06-25 16:10:43 +0000
@@ -2,12 +2,12 @@
 
 set -e
 
-. tests/framework
+MYDIR=$(dirname $0)
+
+. "$MYDIR"/tests/framework
 
 SNAPPY=snappy
 
-MYDIR=$(dirname $0)
-
 if [ "$1" != "--yes-really" ]; then
     echo "The selftest may be destructive, please run with:"
     echo "   --yes-really"
@@ -15,18 +15,19 @@
     exit 1
 fi
 
-. tests/settings
+. "$MYDIR"/tests/settings
 
-. tests/common.sh
+. "$MYDIR"/tests/common.sh
 
 # prepare the environment
 if [ -z "$ADT_REBOOT_MARK" ]; then
+    sudo systemctl stop snappy-autopilot.timer
     sudo snappy remove "$framework" 2>&1 || true
     sudo snappy remove hello-world 2>&1 || true
     sudo snappy remove xkcd-webserver 2>&1 || true
 fi
 
-for test in $MYDIR/tests/*_test_*[!~#]; do
+for test in "$MYDIR"/tests/*_test_*[!~#]; do
     CURRENT_TEST=$(basename $test)
 
     # after a reboot, skip ahead to the test that triggered it

=== added directory '_integration-tests/tests'
=== renamed file 'integration-tests/run-in-image/tests/01_test_info' => '_integration-tests/tests/01_test_info'
=== renamed file 'integration-tests/run-in-image/tests/02_test_versions' => '_integration-tests/tests/02_test_versions'
=== renamed file 'integration-tests/run-in-image/tests/03_test_apt' => '_integration-tests/tests/03_test_apt'
=== renamed file 'integration-tests/run-in-image/tests/05_test_install_nonexistant' => '_integration-tests/tests/05_test_install_nonexistant'
=== renamed file 'integration-tests/run-in-image/tests/06_test_search_framework' => '_integration-tests/tests/06_test_search_framework'
=== renamed file 'integration-tests/run-in-image/tests/07_test_install_framework' => '_integration-tests/tests/07_test_install_framework'
=== renamed file 'integration-tests/run-in-image/tests/08_test_versions_has_framework' => '_integration-tests/tests/08_test_versions_has_framework'
=== renamed file 'integration-tests/run-in-image/tests/10_test_info_has_stuff' => '_integration-tests/tests/10_test_info_has_stuff'
--- integration-tests/run-in-image/tests/10_test_info_has_stuff	2015-05-20 10:23:39 +0000
+++ _integration-tests/tests/10_test_info_has_stuff	2015-06-25 16:10:43 +0000
@@ -1,5 +1,4 @@
 test() {
     test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
     test_regexp "^frameworks: $framework" $SNAPPY info
-    test_regexp "^apps:.*\<hello-world\>" $SNAPPY info
 }

=== renamed file 'integration-tests/run-in-image/tests/11_test_xkcd_listens' => '_integration-tests/tests/11_test_xkcd_listens'
=== renamed file 'integration-tests/run-in-image/tests/80_test_failover' => '_integration-tests/tests/80_test_failover'
=== renamed file 'integration-tests/run-in-image/tests/90_test_upgrade' => '_integration-tests/tests/90_test_upgrade'
=== renamed file 'integration-tests/run-in-image/tests/91_test_upgrade_from_stable_image' => '_integration-tests/tests/91_test_upgrade_from_stable_image'
=== renamed file 'integration-tests/run-in-image/tests/common.sh' => '_integration-tests/tests/common.sh'
=== renamed file 'integration-tests/run-in-image/tests/framework' => '_integration-tests/tests/framework'
=== renamed file 'integration-tests/run-in-image/tests/settings' => '_integration-tests/tests/settings'
=== added file '_integration-tests/tests/snappy_test.go'
--- _integration-tests/tests/snappy_test.go	1970-01-01 00:00:00 +0000
+++ _integration-tests/tests/snappy_test.go	2015-06-25 16:10:43 +0000
@@ -0,0 +1,83 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+
+/*
+ * Copyright (C) 2015 Canonical Ltd
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+package tests
+
+import (
+	"os/exec"
+	"testing"
+
+	. "gopkg.in/check.v1"
+)
+
+// Hook up gocheck into the "go test" runner
+func Test(t *testing.T) { TestingT(t) }
+
+var _ = Suite(&InstallSuite{})
+
+type InstallSuite struct{}
+
+func (s *InstallSuite) installSnap(c *C, packageName string) string {
+	return s.execCommand(c, "sudo", "snappy", "install", packageName)
+}
+
+func (s *InstallSuite) execCommand(c *C, cmds ...string) string {
+	cmd := exec.Command(cmds[0], cmds[1:]...)
+	output, err := cmd.CombinedOutput()
+	stringOutput := string(output)
+	c.Assert(err, IsNil, Commentf("Error: %v", stringOutput))
+	return stringOutput
+}
+
+func (s *InstallSuite) SetUpSuite(c *C) {
+	s.execCommand(c, "sudo", "systemctl", "stop", "snappy-autopilot.timer")
+}
+
+func (s *InstallSuite) TearDownTest(c *C) {
+	s.execCommand(c, "sudo", "snappy", "remove", "hello-world")
+}
+
+func (s *InstallSuite) TestInstallSnapMustPrintPackageInformation(c *C) {
+	installOutput := s.installSnap(c, "hello-world")
+
+	expected := "" +
+		"Installing hello-world\n" +
+		"Name          Date       Version Developer \n" +
+		".*\n" +
+		"hello-world   .* .*  canonical \n" +
+		".*\n"
+	c.Assert(installOutput, Matches, expected)
+}
+
+func (s *InstallSuite) TestCallBinaryFromInstalledSnap(c *C) {
+	s.installSnap(c, "hello-world")
+
+	echoOutput := s.execCommand(c, "hello-world.echo")
+
+	c.Assert(echoOutput, Equals, "Hello World!\n")
+}
+
+func (s *InstallSuite) TestInfoMustPrintInstalledPackageInformation(c *C) {
+	s.installSnap(c, "hello-world")
+
+	infoOutput := s.execCommand(c, "snappy", "info")
+
+	expected := "(?ms).*^apps: hello-world\n"
+	c.Assert(infoOutput, Matches, expected)
+}

=== added directory 'debian/integration-tests'
=== added file 'debian/integration-tests/control'
--- debian/integration-tests/control	1970-01-01 00:00:00 +0000
+++ debian/integration-tests/control	2015-06-25 16:10:43 +0000
@@ -0,0 +1,6 @@
+Test-Command: /tmp/snappy-test/tests/snappy.tests -gocheck.vv -test.outputdir=$ADT_ARTIFACTS
+Restrictions: allow-stderr
+Depends: ubuntu-snappy-tests
+
+Test-Command: ./_integration-tests/snappy-selftest --yes-really
+Depends:

=== removed directory 'integration-tests/debs'
=== removed directory 'integration-tests/image'
=== removed directory 'integration-tests/run-in-image'
=== removed directory 'integration-tests/run-in-image/debian'
=== removed directory 'integration-tests/run-in-image/debian/tests'
=== removed file 'integration-tests/run-in-image/debian/tests/control'
--- integration-tests/run-in-image/debian/tests/control	2015-05-20 10:23:39 +0000
+++ integration-tests/run-in-image/debian/tests/control	1970-01-01 00:00:00 +0000
@@ -1,2 +0,0 @@
-Test-Command: ./snappy-selftest --yes-really
-Depends:

=== removed directory 'integration-tests/run-in-image/tests'
=== removed file 'integration-tests/run-in-image/tests/04_test_install_hello'
--- integration-tests/run-in-image/tests/04_test_install_hello	2015-05-20 10:23:39 +0000
+++ integration-tests/run-in-image/tests/04_test_install_hello	1970-01-01 00:00:00 +0000
@@ -1,7 +0,0 @@
-test() {
-       T="hello-world[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+(\.[0-9]+)+[[:space:]]+canonical"
-       test_regexp "$T" sudo $SNAPPY install hello-world
-       
-       T="Hello World!"
-       test_equal "$T" hello-world.echo
-}

=== removed file 'integration-tests/selftest'
--- integration-tests/selftest	2015-06-09 15:41:02 +0000
+++ integration-tests/selftest	1970-01-01 00:00:00 +0000
@@ -1,68 +0,0 @@
-#!/bin/sh
-
-set -ex
-
-HERE="$(dirname $0)"
-
-# build the updated deb packages for the image
-rm -f "$HERE/debs/*"
-# the "-us -uc " flags tell it not to sign it
-bzr-buildpackage --result-dir="$HERE/debs" "$HERE/.." -- -us -uc
-
-# build a base image
-IMAGE="$HERE/image/snappy.img"
-rm -f "$IMAGE/*"
-# FIXME: hrm, hrm, needs sudo to work on testbed
-#
-# FIXME2: use wget here instead and just get the latest image from
-#         cdimage
-sudo ubuntu-device-flash core -o "$IMAGE" rolling --channel edge --enable-ssh --developer-mode
-
-# base cmd
-QEMU_CMD="qemu-system-$(uname -m) -enable-kvm -m 768 -localtime -nographic -net user -net nic,model=virtio"
-
-# fire it up
-port=11022
-SERIAL=stdio
-$QEMU_CMD -drive file="$IMAGE",if=virtio -redir tcp:$port::22 -monitor none -serial $SERIAL &
-QEMU_PID=$!
-trap "kill $QEMU_PID" INT QUIT
-
-# wait until the image is ready
-SSH="ssh -oStrictHostKeyChecking=no -o UserKnownHostsFile=\"$HERE/image/known_hosts\" -p $port ubuntu@localhost"
-SCP="scp -oStrictHostKeyChecking=no -o UserKnownHostsFile=\"$HERE/image/known_hosts\" -P $port"
-for i in $(seq 100); do
-    if $SSH true; then
-        break
-    fi
-    sleep 1
-done
-if [ $i = 100 ]; then
-    echo "Failed to setup qemu"
-    exit 1
-fi
-
-# install debs
-$SSH rm -rf /tmp/debs
-$SSH mkdir /tmp/debs
-$SCP "$HERE"/debs/*.deb ubuntu@localhost:/tmp/debs
-$SSH sudo mount -o remount,rw /
-$SSH sudo dpkg -i /tmp/debs/*.deb
-
-# shut it down
-$SSH sudo halt --poweroff || true
-
-# wait for qemu pid
-for i in $(seq 100); do
-    if [ ! -e /proc/$QEMU_PID/exe ]; then
-        break
-    fi
-    i=$((i+1))
-    sleep 1
-done
-
-# now run the tests against the image
-(cd "$HERE";
- adt-run run-in-image/ --- ssh -s snappy -- -i image/snappy.img;
-)
-

=== modified file 'run-checks'
--- run-checks	2015-06-09 15:35:35 +0000
+++ run-checks	2015-06-25 16:10:43 +0000
@@ -58,7 +58,7 @@
 # integration tests
 if which adt-run >/dev/null 2>&1; then
     echo "Running integration tests"
-    ./integration-tests/selftest
+    go run _integration-tests/main.go
 fi
 
 echo "All good, what could possibly go wrong"