Improve testsuite to verify dependencies and provide coverage of python scripts.
The test framework, git-ubuntu-self-test, failed to catch the error
fixed by commit 05897c71 for a couple reasons. First, while setup.py
registered systemd as a dependency, neither the snapcraft.yml nor the
jenkins CI verify that python module requirements are met; even though
we had the wrong systemd package, it didn't even get installed and
nothing detected that. Second, the testsuite only includes unit test
coverage for modules, and since this import statement was in a script no
tests exercised it, so if we *had* installed the incorrect systemd
package, the script would still fail on execution after installation.
This patch improves the testsuite by adding checks for these situations.
To address the first problem with dependency checking, a 'pip3 check' is
added to the self test. This checks both that the required python
modules are installed, and that their versions satisfy requirements in
setup.py; for example this will catch problems that the recent pytest
fix (commit 7de260e6) aims to prevent. Since this is a handy way for
first-time users to check their dependencies, 'pip3 check' is suggested
also in the testing docs.
For the second problem, lack of test coverage for our python scripts,
this adds a couple checks: A basic Python3 syntax check for all bin/*.py
files, and an import check on all bin/*.py that have a __main__ defined;
the latter check would have caught the missing systemd.daemon module
dependency.
As well, git-ubuntu-self-test now passes `shellcheck`.
Implement a systemd watchdog daemon to run import-source-packages.py
Git Ubuntu's package importing functionality is invoked via the
import-source-packages.py script. Previously, this script would be
manually started, and on error needed manual intervention.
Instead, wrap the script in a systemd service that starts it up
initially and restarts it on crash. A watchdog timer is used to detect
if the script has hung, and restarts it after a suitable delay.
Another service is added for sending emails when the service crashes,
extracting status from the journal. Errors can also be reviewed using
journalctl normally.
By default, everything is configured to be installable in production,
but configuration considerations are covered in documentation. There
are no unit tests for this, however some testing/validation tips are
identified in the documentation.
This was apparently introduced in astroid 2.3, so hold back from using
it to work around the problem since this is causing our CI to fail.
pylint 2.4 requires astroid 2.3 without declaring it
(https://github.com/PyCQA/pylint/issues/3022) so additionally we also
need to hold back from using pylint 2.4 to avoid hitting this other
failure.
usd-importer/gitubuntu/source_information.py:107:
copy_paste_error: "u_ddi" in "_ddi.codename" looks like a copy-paste error.
usd-importer/gitubuntu/source_information.py:107:
remediation: Should it say "u_udi" instead?
The code is erroneously using the Debian implementation of codename(),
which translates aliases ('unstable', 'testing', etc.) to the release
codenames ('sid', 'buster', etc.) Fortunately, if it doesn't match
anything it just returns the release name passed to it, which is exactly
what the Ubuntu implementation of codename() does.
Thus, even though the code is incorrect, it's behavior will always be
correct in practice. Thus this fix corrects a purely theoretical
problem, not one that would produce invalid behavior.
source_information: Add tests for derive_codename_from_series()
Provide a set of test cases for source_information's
derive_codename_from_series() routine.
The first two test cases check ordinary usage, simple codename->codename
and alias->codename lookups (i.e. 'bionic'->'bionic', and
'unstable'->'sid' respectively). The third test checks that invalid
input triggers an exception as expected. These three tests depend on
the external distro-info and distro-info-data packages.
The last test provides a mock of the DebianDistroInfo class that
includes stricter input checking for its codename() routine. This test
case identifies a code error in derive_codename_from_series() that was
found via Coverity.
