Merge ~alexmurray/qa-regression-testing:lpcraft-ci into qa-regression-testing:master

[Alex Murray]

So I am not sure if this is the best way to shoehorn lpcraft for CI of QRT but it is an initial stab at least. I wanted to try and make sure each script on each series gets enumerated as a separate job with the least amount of copy-pasta in the .launchpad.yaml - there may be a better way bit this appears to work for now.

Currently the gcc-security tests fail but should pass once https://code.launchpad.net/~alexmurray/qa-regression-testing/+git/qa-regression-testing/+merge/437462 is merged.

[Alex Murray (alexmurray)]

Note the duplicate test runs and hence job entries seen in the test output is a known bug in Launchpad - https://bugs.launchpad.net/launchpad/+bug/1999591 (ie there should only be one coverage:0 etc - but say for gcc-security we expect there to be 3 instances - gcc-security:0, gcc-security:1 etc for each bionic, focal and jammy - but there should not be multiple gcc-security:0 which is due to the aforementioned bug)

[Alex Murray (alexmurray)]

Ok, I feel like this PoC is ready for review now - tests pass for both sudo and gnupg and once gcc-security changes get merged that should pass too.

Then we can look at extending this to cover other test scripts in QRT.

Can someone please take a look?

diff --git a/.launchpad.yaml b/.launchpad.yaml
new file mode 100644
index 0000000..d5548f2
--- /dev/null
+++ b/.launchpad.yaml
@@ -0,0 +1,57 @@
+pipeline:
+  - coverage
+  -
+    - gcc-security
+    - gnupg
+    - sudo
+
+jobs:
+  coverage:
+    series: jammy
+    architectures: amd64
+    run: |
+      # TODO parse the .launchpad.yaml and for each series check the list of
+      # tests for any missing compared to the scripts in scripts/test-xxx.py
+      true
+
+  # test we can run testlib.require_nonroot() scripts
+  gcc-security:
+    matrix:
+      - series: jammy
+        architectures: amd64
+      - series: focal
+        architectures: amd64
+      - series: bionic
+        architectures: amd64
+    packages:
+      - sudo
+    run: |
+      ./lpcraft-runner gcc-security
+
+  # test we can run testlib.require_sudo() scripts
+  sudo:
+    matrix:
+      - series: jammy
+        architectures: amd64
+      - series: focal
+        architectures: amd64
+      - series: bionic
+        architectures: amd64
+    packages:
+      - sudo
+    run: |
+      ./lpcraft-runner sudo
+
+  # test we can run testlib.require_root() scripts
+  gnupg:
+    matrix:
+      - series: jammy
+        architectures: amd64
+      - series: focal
+        architectures: amd64
+      - series: bionic
+        architectures: amd64
+    packages:
+      - sudo
+    run: |
+      ./lpcraft-runner gnupg
diff --git a/lpcraft-runner b/lpcraft-runner
new file mode 100755
index 0000000..883290d
--- /dev/null
+++ b/lpcraft-runner
@@ -0,0 +1,44 @@
+#!/bin/bash
+set -euo pipefail
+
+test=$1
+
+# setup an unprivileged user
+getent passwd ubuntu >/dev/null || useradd -m -U ubuntu
+# allow sudo access
+grep -q "ubuntu    ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "ubuntu    ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+# one test for each require_nonroot, require_sudo and require_root
+script="./scripts/test-${test}.py"
+tarball="${TMPDIR:-/tmp}/qrt-test-${test}.tar.gz"
+rm -f "$tarball"
+./scripts/make-test-tarball "$script"
+# figure out what user we need to run as
+user=ubuntu
+if grep -q '^\s*testlib.require_sudo()' "$script"; then
+  user=sudo
+elif grep -q '^\s*testlib.require_root()' "$script"; then
+  user=root
+fi
+
+# now work with script directly
+script="./$(basename "$script")"
+if [ $user = ubuntu ] || [ $user = sudo ]; then
+  # work in ~ubuntu
+  cd ~ubuntu || exit
+  runuser -u ubuntu -- tar --overwrite -xf "$tarball"
+  cd "$(basename "${tarball%.tar.gz}")" || exit
+  ./install-packages "$script"
+  # run with correct permissions / user
+  if [ $user = sudo ]; then
+    runuser -u ubuntu -- sudo "$script"
+  else
+    runuser -u ubuntu -- "$script"
+  fi
+else
+  # run as root in $HOME
+  cd "$HOME" || exit
+  tar --overwrite -xf "$tarball"
+  cd "$(basename "${tarball%.tar.gz}")" || exit
+  ./install-packages "$script"
+  "$script"
+fi
diff --git a/scripts/test-gnupg.py b/scripts/test-gnupg.py
index aeaed7f..45311cf 100755
--- a/scripts/test-gnupg.py
+++ b/scripts/test-gnupg.py
@@ -168,7 +168,7 @@ Expire-Date: 0
     def test_03_recv_keys(self):
         '''Test that public keys can be loaded by default from the network.'''
 
-        cmd = '/usr/bin/%s --keyserver keyserver.ubuntu.com --recv-keys 0x8972F4DFDC6DC026' % app
+        cmd = '/usr/bin/%s --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0x8972F4DFDC6DC026' % app
         self.assertShellExitEquals(0, ['/bin/su', '-c', cmd, user.login])
         self.assertShellExitEquals(0, ['/bin/su', '-c', '/usr/bin/%s --fingerprint [e-mail] | fgrep -q -- "A5C3 F68F 229D D60F 723E  6E13 8972 F4DF DC6D C026"' % app, user.login])
 
@@ -272,10 +272,10 @@ wrhmLbkm5ONbPbLWtiPk1HY=
         # been removed from the server. Should find a new one to test.
         #
         # Specifying a short ID should get us three keys
-        #self.assertShellExitEquals(0, ['/bin/su', '-c', '/usr/bin/%s --keyserver keyserver.ubuntu.com --recv-keys 0x70096AD1 2>&1 | grep -q -- "Total number processed: 3"' % app, user.login])
+        #self.assertShellExitEquals(0, ['/bin/su', '-c', '/usr/bin/%s --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0x70096AD1 2>&1 | grep -q -- "Total number processed: 3"' % app, user.login])
 
         # Specifying a long ID should get us only one key
-        self.assertShellExitEquals(0, ['/bin/su', '-c', '/usr/bin/%s --keyserver keyserver.ubuntu.com --recv-keys 0xEC4B033C70096AD1 2>&1 | grep -q -- "Total number processed: 1"' % app, user.login])
+        self.assertShellExitEquals(0, ['/bin/su', '-c', '/usr/bin/%s --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 0xEC4B033C70096AD1 2>&1 | grep -q -- "Total number processed: 1"' % app, user.login])
 
     def test_20_escape_filename_CVE_2018_12020(self):
         '''test escaping of original filename CVE-2018-12020'''