Merge ~kzapalowicz/snappy-hwe-snaps/+git/jenkins-jobs:feature/make-test-snap-in-sync into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

[Konrad Zapałowicz]

jobs/snap: add hardware testing to the test job

This is based on
https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts/tree/scripts/hardware-test.sh

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:924df366edcce6b15a0305ac1826c82ac7d63897
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1843/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2845
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1751/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3131

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1843/rebuild

review: Approve (continuous-integration)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

I think it would be good to add to the description and the commit message a link to the original sources for snap-test-on-hw.sh

review: Needs Fixing

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:5d262b6542bd60e35be57df298502832e20d61df
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1845/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2847
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1753/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3133

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1845/rebuild

review: Approve (continuous-integration)

[Simon Fels (morphis)]

review: Needs Fixing

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Please see comments below, especially the one about yaml format.

Also, I do not think this should be a separate job. In build-scripts there was a function, set_test_type(), which checked if there were tests defined for real hardware, and in the test job this was checked and real HW tests where run if appropriate. We should do something similar so this gets run by the snap-test.sh job if appropriate. I would also keep the original structure as a function called run_hardware_tests().

Finally, this should be tested using the alsa-utils snap, preferably before merging if possible (we can merge and check after if there is no easy way).

review: Needs Fixing

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:ea3b0280e4c342be1a8915c11f142b58e801b32d
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1850/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/2852
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1758/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3138

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1850/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:1d0cc2005bbf501011c62c59ab1d3e0f5e33ccff
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1884/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3046
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1792/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3324

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1884/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:0ab0583a7a6caa6bd8f06acbe4506fba25b7938f
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1885/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3047
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1793/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3325

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1885/rebuild

review: Approve (continuous-integration)

[Simon Fels (morphis)]

Three comments inline otherwise looks good.

review: Needs Fixing

[Konrad Zapałowicz (kzapalowicz)]

one answer

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:4d7441e425cb3f3cef051238d3575225d462c9a1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1886/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3048
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1794/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3326

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1886/rebuild

review: Approve (continuous-integration)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Thanks for the changes, just one minor request.

review: Needs Fixing

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:81fe8eac832dc8e81f1106580f8831759bfa2f94
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1895/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3127
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1803/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3393

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1895/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:66f1e40dcd0ead212683e57a2de26053f3a587c9
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1933/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3325
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1841/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3586

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1933/rebuild

review: Approve (continuous-integration)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

I just noticed that this recent commit needs to be ported back to this MP:

https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs/commit/?id=cfd48bb7f63b2db05b9f3240905e09f9d0bc7da1

Also, should now tools/hardware-test.sh be removed? Was it being used in the private jenkins instance?

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

And this one as well: https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs/commit/?id=fc653a0e53fec19fa10c6b166da54d75c0e85e9e

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:e2204d18a73522c7e1b5cf480d0ddfef2bf1d32c
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1934/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3326
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1842/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3587

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1934/rebuild

review: Approve (continuous-integration)

[Simon Fels (morphis)]

review: Needs Fixing

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

LGTM

review: Approve

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:a6b1ab3f8a42607a69ac967d32ab9915373e244e
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1939/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/3351
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/1847/console
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/3611

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1939/rebuild

review: Approve (continuous-integration)

[Simon Fels (morphis)]

LGTM

review: Approve

diff --git a/jobs/snap/snap-test-on-hw.sh b/jobs/snap/snap-test-on-hw.sh
new file mode 100644
index 0000000..66ce757
--- /dev/null
+++ b/jobs/snap/snap-test-on-hw.sh
@@ -0,0 +1,149 @@
+#!/bin/sh
+#
+# Copyright (C) 2017 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -ex
+
+. "$WORKSPACE/.build_env"
+
+SSH_PATH="${JENKINS_HOME}/.ssh/"
+SSH_KEY_PATH="${SSH_PATH}/git.launchpad.net/${BOT_USERNAME}"
+
+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
+SCP="scp -i $SSH_KEY_PATH/id_rsa"
+
+REPO=https://git.launchpad.net/~${LAUNCHPAD_TEAM}/${LAUNCHPAD_PROJECT}/+git/$CI_REPO
+BRANCH=$CI_BRANCH
+
+# If no CI repo/branch is set fallback to the source repo/branch set
+# which will be the case for those repositories which don't contain
+# a snap.
+if [ -z "$CI_REPO" ]; then
+    REPO=$SOURCE_GIT_REPO
+    BRANCH=$SOURCE_GIT_REPO_BRANCH
+fi
+
+REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
+REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
+
+tmp_srcdir=$(mktemp -d)
+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
+cd $tmp_srcdir
+
+if ! grep -q "type: adhoc" src/spread.yaml; then
+    echo "INFO: testing on hardware is disabled"
+    rm -rf $tmp_srcdir
+    exit 0
+fi
+
+TEST_TYPE=none
+if [ -e "src/spread.yaml" ]; then
+    TEST_TYPE=spread
+fi
+if [ -e "src/run-tests.sh" ]; then
+    TEST_TYPE=script
+fi
+
+TEST_RESULTS=test_results
+
+# Just dragonboard for the moment
+TEST_SPEC=testflinger-dragonboard.yaml
+
+# We use jq to process testflinger output
+if ! which jq; then
+    sudo apt install --yes jq
+fi
+
+# Initially the device has a password-less ubuntu user. But spread needs a
+# user with a password, so we use the DEVICE_USER/DEVICE_PASSWORD pair to
+# create it.  Note: {device_ip} gets substituted by testflinger.
+DEVICE_USER=test
+DEVICE_PASSWORD=test
+DEVICE_SSH="ssh -q -o UserKnownHostsFile=/dev/null
+    -o StrictHostKeyChecking=no -p 22 ubuntu@{device_ip}"
+
+if [ "$TEST_TYPE" = script ]; then
+TEST_COMMAND="./run-tests.sh --spread-system=hw-ubuntu-core-16
+    --external-address={device_ip}:22 --external-user=$DEVICE_USER
+    --external-password=$DEVICE_PASSWORD --debug"
+else
+TEST_COMMAND="export SPREAD_EXTERNAL_ADDRESS={device_ip}:22 &&
+    export SPREAD_EXTERNAL_USER=$DEVICE_USER &&
+    export SPREAD_EXTERNAL_PASSWORD=$DEVICE_PASSWORD &&
+    ./spread -vv external:hw-ubuntu-core-16"
+fi
+
+# Run testflinger from our bare metal server so we can install it
+
+$SSH mkdir -p "$REMOTE_WORKSPACE"
+
+# If the snap has been built, copy over
+if [ -n "$RESULTS_ID" ]; then
+    set +x
+    # We need to flatten the key here to avoid issues with yaml parsing
+    SSH_KEY_DATA=$(tr '\n:' '?!' < "$SSH_KEY_PATH"/id_rsa)
+    set -x
+fi
+
+cat << EOF > "$WORKSPACE"/"$TEST_SPEC"
+job_queue: dragonboard
+provision_data:
+  channel: stable
+test_data:
+  test_cmds:
+    - sudo apt update
+    - sudo apt install --yes git curl
+    - git clone --depth 1 -b $BRANCH $REPO src
+    - cd src && curl -s -O https://niemeyer.s3.amazonaws.com/spread-amd64.tar.gz && tar xzvf spread-amd64.tar.gz
+    - $DEVICE_SSH "sudo adduser --extrausers --quiet --disabled-password --gecos '' $DEVICE_USER"
+    - $DEVICE_SSH "echo $DEVICE_USER:$DEVICE_PASSWORD | sudo chpasswd"
+    - $DEVICE_SSH "echo '$DEVICE_USER ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/create-user-test"
+    - if [ -n $RESULTS_ID ]; then set +x; echo "$SSH_KEY_DATA" | tr '?!' '\n:' > ssh_key; set -x; chmod 600 ssh_key; scp -i ssh_key $REMOTE_USER@$REMOTE_WORKER:$REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/*.snap src/; fi
+    - cd src && export PATH=$PATH:\$(pwd) && $TEST_COMMAND
+EOF
+
+$SCP "$WORKSPACE"/"$TEST_SPEC" "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/
+
+$SSH << EOF
+#!/bin/sh
+set -ex
+
+cd $REMOTE_WORKSPACE
+if ! which virtualenv; then
+    sudo apt install --yes virtualenv
+fi
+
+git clone -b master https://git.launchpad.net/testflinger-cli
+cd testflinger-cli
+virtualenv -p python3 env
+. env/bin/activate
+./setup.py install
+
+JOB_ID=\$(testflinger-cli submit -q $REMOTE_WORKSPACE/$TEST_SPEC)
+echo "JOB_ID: \${JOB_ID}"
+
+testflinger-cli poll \${JOB_ID}
+testflinger-cli results \${JOB_ID} > $REMOTE_WORKSPACE/$TEST_RESULTS
+EOF
+
+$SCP "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/"$TEST_RESULTS" "$WORKSPACE"/
+
+$SSH sudo rm -rf "$REMOTE_WORKSPACE"
+
+TEST_STATUS=$(jq -r .test_status "$WORKSPACE"/"$TEST_RESULTS")
+echo "Test exit status: $TEST_STATUS"
+
+rm -rf $tmp_srcdir
+exit "$TEST_STATUS"
diff --git a/jobs/snap/snap-test.sh b/jobs/snap/snap-test.sh
index bea5e3a..d673610 100644
--- a/jobs/snap/snap-test.sh
+++ b/jobs/snap/snap-test.sh
@@ -32,7 +32,7 @@ SCP="scp -i $SSH_KEY_PATH/id_rsa"
 REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
 REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
 
-tmp_srcdir=`mktemp -d`
+tmp_srcdir=$(mktemp -d)
 git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
 cd $tmp_srcdir/src
 # This will fail as we have set set -e above when the revision isn't part of
@@ -75,7 +75,7 @@ export PATH=/build/bin:$PATH
 # Once the snapped version includes this backend, then we can
 # change the manual building of spread with making sure the snap
 # package is installed.
-export GOPATH=`mktemp -d`
+export GOPATH=$(mktemp -d)
 go get -d -v github.com/snapcore/spread/...
 go build github.com/snapcore/spread/cmd/spread
 mkdir /build/bin
diff --git a/jobs/snap/snap-test.yaml b/jobs/snap/snap-test.yaml
index 9c15699..a361bbd 100644
--- a/jobs/snap/snap-test.yaml
+++ b/jobs/snap/snap-test.yaml
@@ -62,3 +62,6 @@
         - shell:
             !include-raw-escape:
                 - snap-test.sh
+        - shell:
+            !include-raw-escape:
+                - snap-test-on-hw.sh