lp:~sbeattie/qa-regression-testing

Author: Steve Beattie
Author Date: 2024-09-10 05:39:29 UTC

testlib.cmd(): don't send data after killing process

When a timeout occurs in testlib.cmd, the subprocess.TimeoutExpired
exception would correctly get raised. To handle that, the script would
then attempt to kill the process and then use communicate() to wait for
it to finish, except that the passed input argument was again given to
it, resulting in python trying to send data to a no-longer existent
process, as seen by the test_testlib.py testcase failure:

======================================================================
ERROR: test_shell_exit_success_timeout (__main__.TestlibTest)
Test simple cmd works
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/steve/git/qa-regression-testing/scripts/testlib.py", line 478, in cmd
    out, outerr = sp.communicate(input, timeout=timeout)
  File "/usr/lib/python3.10/subprocess.py", line 1154, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.10/subprocess.py", line 2022, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
  File "/usr/lib/python3.10/subprocess.py", line 1198, in _check_timeout
    raise TimeoutExpired(
subprocess.TimeoutExpired: Command '['sleep', '3']' timed out after 1.0 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/steve/git/qa-regression-testing/scripts/./test_testlib.py", line 59, in test_shell_exit_success_timeout
    self.assertShellExitSuccess(['sleep', '3'], timeout=1.0)
  File "/home/steve/git/qa-regression-testing/scripts/testlib.py", line 1339, in assertShellExitSuccess
    return self.assertShellExitEquals(
  File "/home/steve/git/qa-regression-testing/scripts/testlib.py", line 1329, in assertShellExitEquals
    rc, report, out = self._testlib_shell_cmd(args, input=input, stdin=stdin, stdout=stdout, stderr=stderr, env=env, timeout=timeout)
  File "/home/steve/git/qa-regression-testing/scripts/testlib.py", line 1320, in _testlib_shell_cmd
    rc, out = cmd(args, input=input, stdin=stdin, stdout=stdout, stderr=stderr, env=env, timeout=timeout)
  File "/home/steve/git/qa-regression-testing/scripts/testlib.py", line 485, in cmd
    sp.communicate(input)
  File "/usr/lib/python3.10/subprocess.py", line 1129, in communicate
    raise ValueError("Cannot send input after starting communication")
ValueError: Cannot send input after starting communication

----------------------------------------------------------------------

Fix this by not passing the input argument to the second invocation of
communicate() and also add information about the timeout occurring to
the output sent back to the method's caller.

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2024-06-13 00:45:10 UTC

lpci: enable apt tests, require root

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2024-06-13 05:26:07 UTC

test-cryptojs.py: fix firefox breakage

cryptojs tests are failing in lpci like so:

:: FAILED (failures=1)
:: [*] Searching for handlers.json in /home/ubuntu/.mozilla/firefox
:: [*] Searching for handlers.json in /home/ubuntu/snap/firefox/common/.mozilla/firefox
:: [!] handlers.json has not been found. This might be because you don't have a firefox profile.
:: Executing Firefox to create a profile and retrying...
:: [*] Executing Firefox with arguments: https://ubuntu.com/
:: [*] Searching for handlers.json in /home/ubuntu/.mozilla/firefox
:: [*] Searching for handlers.json in /home/ubuntu/snap/firefox/common/.mozilla/firefox
:: [!] Found /home/ubuntu/snap/firefox/common/.mozilla/firefox/pm5kjrb5.default/handlers.json
:: [*] Adding the key 'application/json' to handlers.json
:: [*] Executing Firefox with arguments: /home/ubuntu/cryptojs_index_suez28t0.html
:: It was not possible to download the JSON file.
:: [!] Cleaning up dropped HTML file /home/ubuntu/cryptojs_index_suez28t0.html
:: [!] Cleaning up stagged cryptojs directory /home/ubuntu/cryptojs_i_n6tkg5
:: [!] Removing key 'application/json' from handlers.json

FIXME: figure out why

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2024-05-04 16:47:24 UTC

lpci: add noble to job matrices for all tests

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2017-07-07 21:52:36 UTC

TODO: test-cgroup.py: flesh out testcases

Author: Steve Beattie
Author Date: 2024-04-25 06:35:18 UTC

test-ghostscript.py: convert conversion test to subtests

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2024-04-26 18:37:02 UTC

lpci: run `apt upgrade` before exercising tests

The automatic runs of qa-regression-testing were failing in
the util-linux tests, even though the most recent util-linux
security update had been published to the ubuntu archive, because
lpci explicitly does not apply updates to the spawned test
environments that the jobs are run in. Furthermore, there is no
specific job configuration to tell lpci to apply updates first in
https://lpci.readthedocs.io/en/latest/configuration.html#job-definitions

To work around this limitation, add a run-before step in each job to run
`apt upgrade` so that the most up-to-date version of a package is tested
against.

An open question here is whether it would be more appropriate to use
`dist-upgrade` rather than a simple `upgrade`.

- v1: initial attempt
- v2: disable interactive prompts when upgrading

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Bug: https://bugs.launchpad.net/lpci/+bug/2063880
MR: https://code.launchpad.net/~sbeattie/qa-regression-testing/+git/qa-regression-testing/+merge/465105

Author: Steve Beattie
Author Date: 2023-07-23 23:17:10 UTC

test-ecdsautils: disable CVE-2022-24884 in LP CI for bionic

The fix for CVE-2022-24884 is only available in Ubuntu Pro on bionic,
which is not currently available in the LP CI infrastructure. Expect
this test to fail in LP CI in bionic, but run the test so that if
this ever changes, then the test will start unexpectedly succeeding
and cause the CI run to fail, letting us adjust.

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>
Acked-by: Alex Murray <alex.murray@canonical.com>
MR: https://code.launchpad.net/~sbeattie/qa-regression-testing/+git/qa-regression-testing/+merge/448427

Author: Steve Beattie
Author Date: 2023-11-01 04:51:14 UTC

test-kernel-security: add simple shared library for LD_* tests

In jammy and other systemd based releases, preloading libmemusage ends
up conflicting (I suspect) with systemd's allocation routines, in such a
way that the pam_systemd either throws an assert in its json handling
code (wtf?!) or segfaults.

This commit creates a simple library for use as a LD_PRELOAD and
LD_AUDIT target, and switched the test to use it. *Ideally* it
should get temporarily installed in one of the system libraries,
as the loader will do some magic around that for setuid binaries,
but the kernel should get there first.

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2023-08-30 23:16:26 UTC

test-coreutils.py: run in lpci

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2023-07-20 22:49:05 UTC

test-util-linux.py: disable tests that fail in lpci environment

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2023-07-07 07:07:45 UTC

[WIP] .launchpad.yaml: enable util-linux tests

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2023-05-16 03:41:11 UTC

WIP: lpcraft-runner: export env var for skipping tests in lpci

[WIP: trying to see if I can make the udhcpc test work]

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Steve Beattie
Author Date: 2022-06-19 05:54:11 UTC

[WIP] add support for ss(1) output

Signed-off-by: Steve Beattie <steve.beattie@canonical.com>

Author: Colin Ian King
Author Date: 2020-09-15 12:16:47 UTC

test-kernel-security: update test test_190_config_have_stack_protector for riscv64

The new riscv64 architecture does not yet support stack protector so
disable this for test for kernels 5.8 and lower on riscv64. As yet,
it's still to early to know when this will be implemented for this
architecture, so make the test fail for 5.9+ so we can catch this
config setting being disabled for new releases until it gets implemented.

Signed-off-by: Colin Ian King <colin.king@canonical.com>