Merge ~sbeattie/qa-regression-testing:moar-ci-tests into qa-regression-testing:master

[Steve Beattie]

tests: add busybox tests to lpci plus infra to skip tests in lpci

Steve Beattie (7):
   * [0012d86b] test-busybox.py: switch to python3
   * [4c70d4a0] test-busybox.py: tests requires root due to udhcpc
   * [fa8ea339] test-busybox.py: older busybox wget doesn't support https
   * [4550df63] test-busybox.py: fix bitwise op test across all releases
   * [175e1a2b] lpcraft-runner: export env var for skipping tests in lpci
   * [841581fe] scripts/README.md: add notes on lpci
   * [14a0bc5f] .launchpad.yaml: add busybox QRT script to CI tests

The overall point of this branch is to enable the busybox qrt scripts in lpci. Along the way, I corrected some issues, including some behavioral differencs in 14.04's busybox, to make sure the tests run across all supported versions and also in lpci; in particular some tests that would work in a local vm would fail to run in the lp test environment.

In particular, the script would attempt to exercise busybox's simple dhcp client (udhcpc) as well as use busybox's wget implementation to access https URLs and the ubuntu archive via ftp. To skip those, I set an environment variable LPCRAFT_QRT_RUNNER in the lpcraft-runner script and then skip tests if that environment variable is set, using python unittest's skipIf decorator (the busybox tests is also converted to python3).

Arguably, this variable could be set in the launchpad.yaml file, and maybe it should be. The idea behind setting it in the runner script was so that local testing with lpci would get the same behavior, but opinions appreciated here.

I also added a bit on lpci and skipping tests to the README for the scripts.

[Alex Murray (alexmurray)]

Thanks Steve, this looks awesome - I would have hoped that setting the variable inside the .launchpad.yaml would then mean that lpci would also set it when running locally - however, this would also mean we would have to duplicate that into every job entry in the yaml as well - so setting it in lpcraft-runner means we only have to put it in one place.

So FWIW I think your current approach is great.

review: Approve

diff --git a/.launchpad.yaml b/.launchpad.yaml
index fa27693..242a4d9 100644
--- a/.launchpad.yaml
+++ b/.launchpad.yaml
@@ -6,6 +6,7 @@ pipeline:
     - sudo
     - git
     - ghostscript
+    - busybox
 
 jobs:
   imagemagick:
@@ -85,3 +86,16 @@ jobs:
       - sudo
     run: |
       ./lpcraft-runner ghostscript
+
+  busybox:
+    matrix:
+      - series: jammy
+        architectures: amd64
+      - series: focal
+        architectures: amd64
+      - series: bionic
+        architectures: amd64
+    packages:
+      - sudo
+    run: |
+      ./lpcraft-runner busybox
diff --git a/lpcraft-runner b/lpcraft-runner
index 0a6f690..ca94a85 100755
--- a/lpcraft-runner
+++ b/lpcraft-runner
@@ -2,16 +2,22 @@
 set -euo pipefail
 
 test=$1
+script="./scripts/test-${test}.py"
+tarball="${TMPDIR:-/tmp}/qrt-test-${test}.tar.gz"
+
+# export this so that QRT test scripts can skip tests that will work in
+# a local VM but not function in the lpci environment
+export LPCRAFT_QRT_RUNNER=true
 
 # setup an unprivileged user
 getent passwd ubuntu >/dev/null || useradd -m -U ubuntu
 # allow sudo access
 grep -q "ubuntu    ALL=(ALL) NOPASSWD: ALL" /etc/sudoers || echo "ubuntu    ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
 # one test for each require_nonroot, require_sudo and require_root
-script="./scripts/test-${test}.py"
-tarball="${TMPDIR:-/tmp}/qrt-test-${test}.tar.gz"
+
 rm -f "$tarball"
 ./scripts/make-test-tarball "$script"
+
 # figure out what user we need to run as
 user=ubuntu
 if grep -q '^\s*testlib.require_sudo()' "$script"; then
diff --git a/scripts/README.md b/scripts/README.md
index 143fae0..41b0a85 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -99,6 +99,34 @@ dependencies explicitly. Eg, `testlib_data.py` has in it:
 
 so you don't need to specify `data` if you specify `testlib_data.py`.
 
+
+Running tests under launchpad CI (aka lpci)
+-------------------------------------------
+
+Reference: <https://lpci.readthedocs.io/en/latest/>
+
+Enabling a given test script in lpci is fairly straightforward... in
+most cases. For the easy cases, editing the top level `.launchpad.yaml`
+will suffice; name a job and create a similar matrix as the existing
+ones. Note that the initial job order definition and the later
+definition should be in matching order.
+
+Some test cases will need to be skipped when run in the lpci
+environment; generally we should try to avid this but sometimes it's
+unavoidable. The lpcraft-runner script defines the environment variable
+`LPCRAFT_QRT_RUNNER` for this reason. Python unitest's `skipIf()`
+decorator can be used to do this:
+
+```python
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'External URLs not allowed in lpci environ'
+    )
+    def test_that_accesses_external_url(self):
+```
+
+
+
 Notes on unittest idioms
 ------------------------
 
diff --git a/scripts/test-busybox.py b/scripts/test-busybox.py
index 48054fd..227f6d2 100755
--- a/scripts/test-busybox.py
+++ b/scripts/test-busybox.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python2
+#!/usr/bin/python3
 #
 #    test-busybox.py quality assurance test script for busybox
 #    Copyright (C) 2019-2021 Canonical Ltd.
@@ -18,6 +18,7 @@
 #
 # packages required for test to run:
 # QRT-Packages: busybox-static busybox-initramfs udhcpc
+# QRT-Privilege: root
 # QRT-Depends: testlib_archive.py testlib_compressor.py data
 
 '''
@@ -254,6 +255,10 @@ class BusyboxUdhcpcTest(testlib.TestlibCase):
     def tearDown(self):
         '''Clean up after each test_* function'''
 
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'unable to attach to interface in lpci environment'
+    )
     def test_dhcpc(self):
         '''Test dhcpc renew'''
 
@@ -278,9 +283,16 @@ class BusyboxWgetTest(testlib.TestlibCase):
 
     def _test_url_wget(self, url="http://localhost/", content="", invert=False, expected=0, extra_opts=[]):
         '''Test the given url with wget'''
+
+        if self.lsb_release['Release'] < 18.04 and url.startswith("https://"):
+            self._skipped('busybox wget does not support https before bionic')
+            expected = 1
+
         report = self._get_page_wget(url, expected=expected, extra_opts=extra_opts)
 
-        if content != "":
+        if self.lsb_release['Release'] < 18.04 and url.startswith("https://"):
+            self._word_find(report, "wget: not an http or ftp url:", invert)
+        elif content != "":
             self._word_find(report, content, invert)
 
     def _get_page_wget(self, url="http://localhost/", expected=0, extra_opts=[]):
@@ -304,21 +316,37 @@ class BusyboxWgetTest(testlib.TestlibCase):
         self._test_url_wget("http://cdimage.ubuntu.com/",
                             "ubuntu-server")
 
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'External https URLs not allowed in lpci environ'
+    )
     def test_https(self):
         '''Test https'''
         self._test_url_wget("https://cdimage.ubuntu.com/",
                             "ubuntu-server")
 
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'External https URLs not allowed in lpci environ'
+    )
     def test_bad_https_selfsigned(self):
         '''Test bad https (self-signed)'''
         self._test_url_wget("https://self-signed.badssl.com/",
                             content="error getting response", expected=1)
 
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'External https URLs not allowed in lpci environ'
+    )
     def test_bad_https_wrong_host(self):
         '''Test bad https (wrong host)'''
         self._test_url_wget("https://wrong.host.badssl.com/",
                             content="error getting response", expected=1)
 
+    @unittest.skipIf(
+        'LPCRAFT_QRT_RUNNER' in os.environ,
+        'External ftp URLs not allowed in lpci environ'
+    )
     def test_ftp(self):
         '''Test ftp (ftp.ubuntu.com)'''
 
@@ -334,10 +362,18 @@ class BusyboxAwkTest(testlib.TestlibCase):
     def test_bitwise_op(self):
         expected = 0
         rc, report = testlib.cmd_pipe(['echo', ''],
-                                      ['busybox', 'awk', '{ print or(4294967295,1) }'])
+                                      ['busybox', 'awk', r'{ printf "%8x\n", or(4294967295,1) }'])
+        result = 'Got exit code %d, expected %d\n' % (rc, expected)
+        self.assertEqual(expected, rc, result + report)
+        self.assertTrue("80000000" in report, "Couldn't find search string.\n" + report)
+
+    def test_bitwise_op_2(self):
+        expected = 0
+        rc, report = testlib.cmd_pipe(['echo', ''],
+                                      ['busybox', 'awk', r'{ printf "%8x\n", or(4294967294,1) }'])
         result = 'Got exit code %d, expected %d\n' % (rc, expected)
         self.assertEqual(expected, rc, result + report)
-        self.assertTrue("4294967295" in report, "Couldn't find search string.\n")
+        self.assertTrue("80000000" in report, "Couldn't find search string.\n" + report)
 
     def test_dash_f(self):
         expected = 0
@@ -354,6 +390,9 @@ class BusyboxAwkTest(testlib.TestlibCase):
         self.assertTrue("2" in report, "Couldn't find search string.\n")
 
 if __name__ == '__main__':
+    # needed for udhcpc tests
+    testlib.require_root()
+
     # more configurable
     suite = unittest.TestSuite()
     suite.addTest(unittest.TestLoader().loadTestsFromTestCase(BusyboxTarTest))