setup.py: Bump lazr.restfulclient version requirement
Originally, commit 80c3e0b5 specified this to 0.13.5, and commit
9557cf90 bumped it to 0.14.0. But bionic is currently shipping 0.14.2
and this is what pip has been pulling in during jenkins builds, so
update the requirement to match what's actually been in use.
This fixes an issue caught in the recent addition of `pip3 check` to
git-ubuntu-self-test, where the versions were mis-matched.
gitubuntu 0.7.4 has requirement lazr.restfulclient==0.14.0, but you have lazr.restfulclient 0.14.2.
keyring was intended to be pinned to < 16.0.0 due to a regression in
that release that prevented the plaintext keyring from being retrieved.
This was reported upstream[1], and was fixed in a subsequent 16.0.1
release.
Despite this intended pinning, git-ubuntu's snap builds are using a
newer version, 19.2.0, which `pip3 check` is now catching:
+ uvt-kvm ssh --insecure gitubuntu-ci-389 -- bash -l -c git-ubuntu.self-test
gitubuntu 0.7.4 has requirement keyring<16, but you have keyring 19.2.0.
The inconsistency can be resolved by just requiring 16.0.1 or newer for
the keyring package. Since we're already installing a much newer
version this should cause no behavioral change except to pass the test.
Prior to the introduction of git-ubuntu-self-test, bin/integration-test
served as a testsuite runner, and tox was used for some style checks.
These are no longer invoked in the jenkins runs[1,2] and thus are just
vestigial now. In fact bin/integration-test no longer works since it
calls `git-ubuntu` rather than `git ubuntu`; functionally it's just a
smoke test like the one documented in the README.md already. To avoid
future confusion, just drop these two old bits.
Improve testsuite to verify dependencies and provide coverage of python scripts.
The test framework, git-ubuntu-self-test, failed to catch the error
fixed by commit 05897c71 for a couple reasons. First, while setup.py
registered systemd as a dependency, neither the snapcraft.yml nor the
jenkins CI verify that python module requirements are met; even though
we had the wrong systemd package, it didn't even get installed and
nothing detected that. Second, the testsuite only includes unit test
coverage for modules, and since this import statement was in a script no
tests exercised it, so if we *had* installed the incorrect systemd
package, the script would still fail on execution after installation.
This patch improves the testsuite by adding checks for these situations.
To address the first problem with dependency checking, a 'pip3 check' is
added to the self test. This checks both that the required python
modules are installed, and that their versions satisfy requirements in
setup.py; for example this will catch problems that the recent pytest
fix (commit 7de260e6) aims to prevent. Since this is a handy way for
first-time users to check their dependencies, 'pip3 check' is suggested
also in the testing docs.
For the second problem, lack of test coverage for our python scripts,
this adds a couple checks: A basic Python3 syntax check for all bin/*.py
files, and an import check on all bin/*.py that have a __main__ defined;
the latter check would have caught the missing systemd.daemon module
dependency.
As well, git-ubuntu-self-test now passes `shellcheck`.
The previous logic was pulling in the wrong systemd package, and also
excluding its contents. This resulted in a failure executing the
import-source-packages.py script due to a missing systemd.daemon module.
Implement a systemd watchdog daemon to run import-source-packages.py
Git Ubuntu's package importing functionality is invoked via the
import-source-packages.py script. Previously, this script would be
manually started, and on error needed manual intervention.
Instead, wrap the script in a systemd service that starts it up
initially and restarts it on crash. A watchdog timer is used to detect
if the script has hung, and restarts it after a suitable delay.
Another service is added for sending emails when the service crashes,
extracting status from the journal. Errors can also be reviewed using
journalctl normally.
By default, everything is configured to be installable in production,
but configuration considerations are covered in documentation. There
are no unit tests for this, however some testing/validation tips are
identified in the documentation.
This was apparently introduced in astroid 2.3, so hold back from using
it to work around the problem since this is causing our CI to fail.
pylint 2.4 requires astroid 2.3 without declaring it
(https://github.com/PyCQA/pylint/issues/3022) so additionally we also
need to hold back from using pylint 2.4 to avoid hitting this other
failure.
