Merge lp:~elopio/snappy/test_go_info into lp:~snappy-dev/snappy/snappy-moved-to-github

[Leo Arias]

Translated the info integration test to go.

[Federico Gimenez (fgimenez)]

Looks very good, great idea checking the actual release and channel. I'm not sure I like the way that info is passed to the tests, do we have any alternative?

In any case, that file should be put under the _integration-tests/data/output directory IMO, and perhaps be cleaned up at the end of the execution (we should talk about reusing the AddCleanup in the files outside the tests too).

Otherwise +1

review: Needs Information

[Leo Arias (elopio)]

As an alternative, I thought about reading the info from channel.ini in the testbed; but then the test will be self-fulfilling because snappy reads from that file to print the info. And it would be duplicated, as we have unit tests that check the attributes of the release.

So, what I want is to make sure that we are running the tests in the image that we specified in the host. As how to pass that info from the host to the test bed, there are plenty of options. Different file formats, env vars, with the --setup-command... I just copied what we did on the sanity suite.

I'll change the file location.

[Leo Arias (elopio)]

Please give it another review. I'm not sure about removing the data files, as they can serve to debug an execution. And on prepare target we make sure that the dirs are wiped out before running a new test.

I think that the files will be collected as part of the artifacts, so maybe it's ok to remove them. But I don't know, lets talk about it and later we can make a branch that removes all the generated stuff.

[Sergio Schvezov (sergiusens)]

Just a small proposal for an alternate way to validate the output.

[Federico Gimenez (fgimenez)]

Great, we can take care of the generated files in [1]

Cheers!

[1] https://trello.com/c/1QOXRIqx/164-automation-remove-leftover-files

review: Approve

[Sergio Schvezov (sergiusens)]

A side effect of using a scanner is that you can c.Check every line for formatting (useful in snappy list where we have formatting guidelines)

[Federico Gimenez (fgimenez)]

@Sergio it looks very nice, thanks a lot :)

We could replace pattern matching with it in places where regex feels a little cumbersome, perhaps write a checker with this logic inside? (I feel very pro-custom-checkers today :D)

[Leo Arias (elopio)]

<elopio> sergiusens: I like your proposal for map + Scan, but for the format compliance tests.
<elopio> what we are doing in the integration tests is closer to pattern matching: a user will ignore big chunks of text and care only about some lines. I find regexp and .* to express that nicely.
<elopio> We have a card for the format tests, where we will want to check every single line. I'll copy your comment in there.
<elopio> what do you think?

[Leo Arias (elopio)]

Back to work in progress because this will now conflict with the branch that splits main.

[Leo Arias (elopio)]

<sergiusens> elopio: it's fine; scanning is also good to see how long a specific message takes to get on screen; but is leagues away

[Federico Gimenez (fgimenez)]

infoSuite is passing but I get this in one of the remaining shell tests:

****** Running ./_integration-tests/tests/08_test_versions_has_framework
Can not find 'docker[[:space:]]+[0-9]{4}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]+\.[0-9]+' in 'Name Date Version Developer
ubuntu-core 2015-07-15 108 ubuntu
generic-amd64 2015-07-15 1.4 canonical '
ERROR
FAILED: ./_integration-tests/tests/08_test_versions_has_framework

AFAIU this will be removed in one of the branches, right?

Thanks,

[Leo Arias (elopio)]

Thanks for the review, and yes, removed in here:
https://code.launchpad.net/~elopio/snappy/test_go_list/+merge/264794

Should land anytime now.

=== modified file '_integration-tests/main.go'
--- _integration-tests/main.go	2015-07-15 14:22:11 +0000
+++ _integration-tests/main.go	2015-07-15 15:54:02 +0000
@@ -20,8 +20,10 @@
 package main
 
 import (
+	"encoding/json"
 	"flag"
 	"fmt"
+	"io/ioutil"
 	"log"
 	"os"
 	"path/filepath"
@@ -40,13 +42,14 @@
 	defaultChannel = "edge"
 	defaultSSHPort = 22
 	defaultGoArm   = "7"
-	tplOutputDir   = "_integration-tests/data/output/"
+	dataOutputDir  = "_integration-tests/data/output/"
 	controlTpl     = "_integration-tests/data/tpl/control"
 )
 
 var (
 	commonSSHOptions   = []string{"---", "ssh"}
-	controlFile        = filepath.Join(tplOutputDir, "control")
+	configFileName     = filepath.Join(dataOutputDir, "testconfig.json")
+	controlFile        = filepath.Join(dataOutputDir, "control")
 	testPackagesLatest = []string{"latest", "failover"}
 	testPackageUpdate  = []string{"update"}
 	testPackages       = append(testPackagesLatest, testPackageUpdate...)
@@ -63,6 +66,20 @@
 	buildTests(arch)
 }
 
+func writeTestConfig(release, channel string) {
+	fmt.Println("Writing test config...")
+	testConfig := map[string]string{
+		"release": release,
+		"channel": channel,
+	}
+	fmt.Println(testConfig)
+	encoded, err := json.Marshal(testConfig)
+	if err != nil {
+		log.Fatalf("Error encoding the test config: %v", testConfig)
+	}
+	ioutil.WriteFile(configFileName, encoded, 0644)
+}
+
 func setupAndRunLocalTests(rootPath, testFilter string, img image.Image) {
 	var includeShell bool
 	if testFilter == "" {
@@ -161,7 +178,6 @@
 		log.Fatalf("Error reading adt-run control template %s", controlTpl)
 	}
 
-	utils.PrepareTargetDir(tplOutputDir)
 	outputFile, err := os.Create(controlFile)
 	if err != nil {
 		log.Fatalf("Error creating control file %s", controlFile)
@@ -224,6 +240,14 @@
 
 	buildAssets(*useSnappyFromBranch, *arch)
 
+	// TODO: generate the files out of the source tree. --elopio - 2015-07-15
+	utils.PrepareTargetDir(dataOutputDir)
+	defer os.RemoveAll(dataOutputDir)
+
+	// TODO: pass the config as arguments to the test binaries.
+	// --elopio - 2015-07-15
+	writeTestConfig(*imgRelease, *imgChannel)
+
 	rootPath := getRootPath()
 
 	if *testbedIP == "" {

=== removed file '_integration-tests/tests/01_test_info'
--- _integration-tests/tests/01_test_info	2015-06-15 15:36:29 +0000
+++ _integration-tests/tests/01_test_info	1970-01-01 00:00:00 +0000
@@ -1,4 +0,0 @@
-test() {
-    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
-    test_regexp "^frameworks: ($framework|)*$" $SNAPPY info
-}

=== removed file '_integration-tests/tests/10_test_info_has_stuff'
--- _integration-tests/tests/10_test_info_has_stuff	2015-06-16 06:05:14 +0000
+++ _integration-tests/tests/10_test_info_has_stuff	1970-01-01 00:00:00 +0000
@@ -1,4 +0,0 @@
-test() {
-    test_regexp "^release: ubuntu-core/.*/(edge|stable)" $SNAPPY info
-    test_regexp "^frameworks: $framework" $SNAPPY info
-}

=== added file '_integration-tests/tests/latest/info_test.go'
--- _integration-tests/tests/latest/info_test.go	1970-01-01 00:00:00 +0000
+++ _integration-tests/tests/latest/info_test.go	2015-07-15 15:54:02 +0000
@@ -0,0 +1,86 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+
+/*
+ * Copyright (C) 2015 Canonical Ltd
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+package latest
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+
+	. "../common"
+
+	check "gopkg.in/check.v1"
+)
+
+var _ = check.Suite(&infoSuite{})
+
+type infoSuite struct {
+	SnappySuite
+}
+
+func readConfig(c *check.C) map[string]string {
+	b, err := ioutil.ReadFile("_integration-tests/data/output/testconfig.json")
+	c.Assert(
+		err, check.IsNil, check.Commentf("Failed to read test config: %v", err))
+
+	var decoded map[string]string
+	err = json.Unmarshal(b, &decoded)
+	c.Assert(
+		err, check.IsNil, check.Commentf("Failed to decode test config: %v", err))
+	return decoded
+}
+
+func (s *infoSuite) TestInfoMustPrintReleaseAndChannel(c *check.C) {
+	infoOutput := ExecCommand(c, "snappy", "info")
+
+	config := readConfig(c)
+	expected := "(?ms)" +
+		fmt.Sprintf("^release: ubuntu-core/%s/%s\n", config["release"], config["channel"]) +
+		".*"
+
+	c.Assert(infoOutput, check.Matches, expected)
+}
+
+func (s *infoSuite) TestInfoMustPrintInstalledApps(c *check.C) {
+	InstallSnap(c, "hello-world")
+	s.AddCleanup(func() {
+		RemoveSnap(c, "hello-world")
+	})
+	infoOutput := ExecCommand(c, "snappy", "info")
+
+	expected := "(?ms)" +
+		".*" +
+		"^apps: .*hello-world.*\n"
+	c.Assert(infoOutput, check.Matches, expected)
+}
+
+func (s *infoSuite) TestInfoMustPrintInstalledFrameworks(c *check.C) {
+	InstallSnap(c, "hello-dbus-fwk.canonical")
+	s.AddCleanup(func() {
+		RemoveSnap(c, "hello-dbus-fwk.canonical")
+	})
+	infoOutput := ExecCommand(c, "snappy", "info")
+
+	expected := "(?ms)" +
+		".*" +
+		"^frameworks: .*hello-dbus-fwk.*\n" +
+		".*"
+	c.Assert(infoOutput, check.Matches, expected)
+}