Merge lp:~fgimenez/snapcraft/build-examples-test into lp:~snappy-dev/snapcraft/core

[Federico Gimenez]

plainbox test for building the examples

plainbox test for building the examples

[Sergio Schvezov (sergiusens)]

Thanks, this is a good start, looping over everything in examples will do the initial trick. In the future I will split this out a bit and use something a bit more custom for each checking if stage and snap have the right set of files for example.

The reason I set to needs fixing though is because of 'run_examples_tests.sh', it seems there is too much duplicate code here and in runtests.sh, it also introduces one more driver for them.

I'd like to see this at least driven by an envvar or command arg

./runtests.sh all (default)
./runtests.sh unit
./runtest.sh plainbox
./runtest.sh plainbox examples|normal

Thoughts?

review: Needs Fixing

[Leo Arias (elopio)]

I agree with what sergio says. I'd also like to see snapcraft run on each example, and check that it does what it's supposed to. That's way more complex and I have no idea yet how to do it, so +1 to this branch that gets all started.

fwiw, I did a little refactor in here to support dep8:
https://code.launchpad.net/~elopio/snapcraft/dep8/+merge/266998

You could put the script inside examples dir.

Thanks Federico.

[Federico Gimenez (fgimenez)]

Thanks a lot Sergio and Leo for your comments.

With the latest changes it can be invoked with the options suggested by Sergio, by default it only runs the normal test plan in plainbox, to run the examples test plan you have to be explicit:

./runtests.sh plainbox examples

Let me know what do you think. Leo, your dep8 work is already merged, right?

[Daniel Holbach (dholbach)]

I think the following changes would be good: http://pastebin.ubuntu.com/12448355/

Unfortunately does the runexamples test still explode: http://pastebin.ubuntu.com/12448397/

[Daniel Holbach (dholbach)]

And yep, the dep8 branch is in trunk.

[Federico Gimenez (fgimenez)]

Back to wip, autopkgtest not yet working

[Federico Gimenez (fgimenez)]

It should be fixed now, the examples folder was not being added to the deb and because of that was not available in the testbed, let me know if it works for you.

[lp:~fgimenez/snapcraft/build-examples-test]

172. By Federico Gimenez on 2015-09-21

added missing $

[Daniel Holbach (dholbach)]

Great work.

review: Approve

[Sergio Schvezov (sergiusens)]

As discussed during the standup, can we have a snapcraft-examples binary package that depends on the snapcraft binary package?

[Leo Arias (elopio)]

+Tests: runtests
+ runexamples

I would call them integration-tests and example-tests, or something like that.

I ran the tests, and they work. Got one failure, which is awesome. Thanks Federico!

Waiting for the examples package to approve.

[Daniel Holbach (dholbach)]

https://code.launchpad.net/~dholbach/snapcraft/snapcraft-examples/+merge/271838

[lp:~fgimenez/snapcraft/build-examples-test]

173. By Federico Gimenez on 2015-09-21

merged trunk

174. By Federico Gimenez on 2015-09-22

merged lp:~dholbach/snapcraft/snapcraft-example

175. By Federico Gimenez on 2015-09-22

fixed paths to work with the snappy-examples package

176. By Federico Gimenez on 2015-09-22

merged trunk

177. By Federico Gimenez on 2015-09-22

Removed libssl-dev dependency

[Daniel Holbach (dholbach)]

Can you apply the following change?

== modified file 'examples/downloader-with-wiki-parts/snapcraft.yaml'
--- examples/downloader-with-wiki-parts/snapcraft.yaml 2015-09-21 05:04:31 +0000
+++ examples/downloader-with-wiki-parts/snapcraft.yaml 2015-09-22 08:11:03 +0000
@@ -14,3 +14,4 @@
         source: .
         after:
             - curl
+build-packages: [libssl-dev]

review: Needs Fixing

[lp:~fgimenez/snapcraft/build-examples-test]

178. By Federico Gimenez on 2015-09-22

added build dependency on libssl-dev to downloader-withwiki-parts

[Federico Gimenez (fgimenez)]

@Daniel, thanks, done :)

[Daniel Holbach (dholbach)]

Thanks a lot! Good work!

review: Approve

[Leo Arias (elopio)]

There is one lintian error. Fixed with:
http://paste.ubuntu.com/12523307/

review: Needs Fixing

[Leo Arias (elopio)]

Lets land this one, the lintian fix is here: https://code.launchpad.net/~elopio/snapcraft/fix_lintian/+merge/272023

=== modified file 'debian/control'
--- debian/control	2015-09-16 14:26:10 +0000
+++ debian/control	2015-09-22 08:24:43 +0000
@@ -39,3 +39,14 @@
  a single toolset, but instead is a collection of tools that enable the
  natural workflow of an upstream to be extended with a simple release step
  into Snappy.
+
+Package: snapcraft-examples
+Architecture: all
+Depends: snapcraft (= ${binary:Version}), ${misc:Depends}, ${python:Depends}
+Description: examples for snapcraft
+ Snapcraft aims to make upstream developers' lives easier and as such is not
+ a single toolset, but instead is a collection of tools that enable the
+ natural workflow of an upstream to be extended with a simple release step
+ into Snappy.
+ .
+ This package contains examples for snapcraft.

=== modified file 'debian/rules'
--- debian/rules	2015-09-16 14:26:10 +0000
+++ debian/rules	2015-09-22 08:24:43 +0000
@@ -3,3 +3,6 @@
 
 %:
 	dh $@ --with python3 --buildsystem=pybuild
+
+override_dh_compress:
+	dh_compress -X/usr/share/snapcraft/examples

=== added file 'debian/snapcraft-examples.install'
--- debian/snapcraft-examples.install	1970-01-01 00:00:00 +0000
+++ debian/snapcraft-examples.install	2015-09-22 08:24:43 +0000
@@ -0,0 +1,1 @@
+examples		/usr/share/snapcraft

=== added file 'debian/snapcraft.install'
--- debian/snapcraft.install	1970-01-01 00:00:00 +0000
+++ debian/snapcraft.install	2015-09-22 08:24:43 +0000
@@ -0,0 +1,3 @@
+/usr/bin
+/usr/lib/python*
+/usr/share/snapcraft

=== modified file 'debian/source/options'
--- debian/source/options	2015-07-06 16:48:22 +0000
+++ debian/source/options	2015-09-22 08:24:43 +0000
@@ -1,7 +1,2 @@
 # Ignore standard files (like .bzr)
 tar-ignore
-
-# Ignore examples folder because it likely contains a bunch of stuff we don't
-# care about (like downloaded go1.4 binaries and such).  It's not important
-# for it to be distributed with the source.
-tar-ignore = "examples"

=== modified file 'debian/tests/control'
--- debian/tests/control	2015-09-17 14:56:20 +0000
+++ debian/tests/control	2015-09-22 08:24:43 +0000
@@ -1,5 +1,6 @@
-Test-Command: ./runtests.sh
-Restrictions: allow-stderr, isolation-container
+Tests: runtests
+       runexamples
+Restrictions: allow-stderr, isolation-container, rw-build-tree
 Depends: @,
          build-essential,
          pep8,

=== added file 'debian/tests/runexamples'
--- debian/tests/runexamples	1970-01-01 00:00:00 +0000
+++ debian/tests/runexamples	2015-09-22 08:24:43 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh plainbox examples

=== added file 'debian/tests/runtests'
--- debian/tests/runtests	1970-01-01 00:00:00 +0000
+++ debian/tests/runtests	2015-09-22 08:24:43 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh

=== modified file 'examples/downloader-with-wiki-parts/snapcraft.yaml'
--- examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-21 05:04:31 +0000
+++ examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-22 08:24:43 +0000
@@ -14,3 +14,4 @@
         source: .
         after:
             - curl
+build-packages: [libssl-dev]

=== modified file 'integration-tests/runtests.sh'
--- integration-tests/runtests.sh	2015-08-04 23:46:54 +0000
+++ integration-tests/runtests.sh	2015-09-22 08:24:43 +0000
@@ -17,24 +17,45 @@
 
 set -e
 
-if [ -z "$SNAPCRAFT" ]; then
-    export SNAPCRAFT=snapcraft
-fi
-
-# Create a temporary directory so that we can run 'manage.py develop' and
-# create the .provider file there
-temp_dir=$(mktemp -d)
-# Develop the provider, this will let us run tests on it
-./manage.py develop -d $temp_dir
-# Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
-# without installing it.
-export PROVIDERPATH=$PROVIDERPATH:$temp_dir
-# Run the 'normal' test plan
-plainbox run \
-    -T 2015.com.canonical.snapcraft::normal \
-    -f json -o $temp_dir/result.json
-# Analyze the result and fail if there are any failures
-python3 - << __PYTHON__
+parseargs(){
+    export TEST_PLAN=$1
+}
+
+run_test_plan(){
+    if [ -z "$SNAPCRAFT" ]; then
+        export SNAPCRAFT=snapcraft
+    fi
+
+    # Create a temporary directory so that we can run 'manage.py develop' and
+    # create the .provider file there
+    temp_dir=$(mktemp -d)
+    # Develop the provider, this will let us run tests on it
+    ./manage.py develop -d $temp_dir
+    # Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
+    # without installing it.
+    export PROVIDERPATH=$PROVIDERPATH:$temp_dir
+    # create symlink from the provider's data directory to examples for them to
+    # be available to the tests
+    if [ $TEST_PLAN == "examples" ]; then
+        BASEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/.. && pwd )
+        TARGET_PATH=${BASEDIR}/integration-tests/data/examples
+        if [ -z $ADT_ARTIFACTS ]; then
+            EXAMPLES_PATH=${BASEDIR}/examples
+        else
+            EXAMPLES_PATH=/usr/share/snapcraft/examples
+        fi
+        ln -fs ${EXAMPLES_PATH} ${TARGET_PATH}
+    fi
+    # Run the test plan
+    plainbox run \
+             -T 2015.com.canonical.snapcraft::"$TEST_PLAN" \
+             -f json -o $temp_dir/result.json
+    # remove the examples symlink
+    if [ $TEST_PLAN == "examples" ]; then
+        rm ${TARGET_PATH}
+    fi
+    # Analyze the result and fail if there are any failures
+    python3 - << __PYTHON__
 import json
 with open("$temp_dir/result.json", "rt", encoding="utf-8") as stream:
     results = json.load(stream)
@@ -46,3 +67,8 @@
 print("Overall: {0}".format("fail" if failed else "pass"))
 raise SystemExit(failed)
 __PYTHON__
+}
+
+parseargs "$@"
+
+run_test_plan

=== added file 'integration-tests/units/examples.pxu'
--- integration-tests/units/examples.pxu	1970-01-01 00:00:00 +0000
+++ integration-tests/units/examples.pxu	2015-09-22 08:24:43 +0000
@@ -0,0 +1,10 @@
+id: snapcraft/examples/build
+plugin: shell
+command:
+    set -e
+    for example in ${PLAINBOX_PROVIDER_DATA}/examples/*
+    do
+      cp -rT ${example//:} ${PLAINBOX_SESSION_SHARE}
+      cd ${PLAINBOX_SESSION_SHARE} && ${SNAPCRAFT}
+      rm -rf ${PLAINBOX_SESSION_SHARE}/*
+    done

=== modified file 'integration-tests/units/testplans.pxu'
--- integration-tests/units/testplans.pxu	2015-07-02 18:29:47 +0000
+++ integration-tests/units/testplans.pxu	2015-09-22 08:24:43 +0000
@@ -1,4 +1,9 @@
 unit: test plan
 id: normal
-name: Tests for Snapcraft 
+name: Tests for Snapcraft
 include: snapcraft/normal/.*
+
+unit: test plan
+id: examples
+name: Test for building Snapcraft examples
+include: snapcraft/examples/.*

=== modified file 'runtests.sh'
--- runtests.sh	2015-09-17 07:31:24 +0000
+++ runtests.sh	2015-09-22 08:24:43 +0000
@@ -20,39 +20,62 @@
 export PATH=$(pwd)/bin:$PATH
 export PYTHONPATH=$(pwd):$PYTHONPATH
 
-SRC_PATHS="bin snapcraft snapcraft/tests"
-
-# These three checks could easily be done with flake8 in one shot if
-# we had python3-flake8 provide flake8
-# Ignore 501 (line-too-long)
-pep8 $SRC_PATHS --ignore=E501
-
-pyflakes3 $SRC_PATHS
-
-# mccabe in 'warning' mode as we have high complexity
-mccabe_list=
-for unit in $(find snapcraft -type f -name '*.py')
-do
-  output=$(python3 -m mccabe --min 10 "$unit")
-  [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
-done
-
-if [ -n "$mccabe_list" ]; then
-  echo -e "\e[1;31mThe project has gotten complex\e[0m."
-  echo "Here's the list of units exceeding 10:"
-  echo -e "$mccabe_list"
-fi
-
-if which python3-coverage >/dev/null 2>&1; then
-    python3-coverage erase
-    python3-coverage run --branch --source snapcraft -m unittest
-    mv .coverage .coverage.unit
-else
-    python3 -m unittest
-fi
-
-if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ]; then
-(
+parseargs(){
+    if [[ "$#" -eq 0 ]] || [[ "$1" == "all" ]]; then
+        export RUN_UNIT="true"
+        export RUN_PLAINBOX="true"
+        export PLAINBOX_TEST_PLANS="normal"
+    else
+        if [ "$1" == "unit" ] ; then
+            export RUN_UNIT="true"
+        elif [ "$1" == "plainbox" ] ; then
+            export RUN_PLAINBOX="true"
+            if [ "$#" -gt 1 ]; then
+                export PLAINBOX_TEST_PLANS="$2"
+            else
+                export PLAINBOX_TEST_PLANS="normal"
+            fi
+        else
+            echo "Not recognized option, should be one of all, unit or plainbox"
+            exit 1
+        fi
+    fi
+}
+
+run_unit_tests(){
+    SRC_PATHS="bin snapcraft snapcraft/tests"
+
+    # These three checks could easily be done with flake8 in one shot if
+    # we had python3-flake8 provide flake8
+    # Ignore 501 (line-too-long)
+    pep8 $SRC_PATHS --ignore=E501
+
+    pyflakes3 $SRC_PATHS
+
+    # mccabe in 'warning' mode as we have high complexity
+    mccabe_list=
+    for unit in $(find . -type f -name '*.py')
+    do
+        output=$(python3 -m mccabe --min 10 "$unit")
+        [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
+    done
+
+    if [ -n "$mccabe_list" ]; then
+        echo -e "\e[1;31mThe project has gotten complex\e[0m."
+        echo "Here's the list of units exceeding 10:"
+        echo -e "$mccabe_list"
+    fi
+
+    if which python3-coverage >/dev/null 2>&1; then
+        python3-coverage erase
+        python3-coverage run --branch --source snapcraft -m unittest
+        mv .coverage .coverage.unit
+    else
+        python3 -m unittest
+    fi
+}
+
+run_plainbox(){
     # well, well, what can we do
     if ! which plainbox >/dev/null; then
         cat <<EOF
@@ -73,8 +96,17 @@
 
     # Go to the plainbox provider of snapcraft tests
     cd integration-tests
-    ./runtests.sh
-)
+    ./runtests.sh $PLAINBOX_TEST_PLANS
+}
+
+parseargs "$@"
+
+if [ ! -z "$RUN_UNIT" ]; then
+    run_unit_tests
+fi
+
+if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ] && [ ! -z "$RUN_PLAINBOX" ] ; then
+    run_plainbox
 fi
 
 if which python3-coverage >/dev/null 2>&1; then