ci deps: Add --test-distro to read-dependencies to install all deps
read-dependencies now takes --test-distro param to indicate we want to install
all system package depenencies to allow for testing and building for our
continous integration environment. It allows us to install all needed deps on
a fresh system with:
python3 ./tools/read-dependencies --distro ubuntu --test-distro [--dry-run].
Additionally read-dependencies now looks at what version of python is running
the script (py2 vs p3) and opts to install python 2 or 3 system deps
respectively. This behavior can still be overridden with
python3 ./tools/read-dependencies ... --python-version 2.
There are also some distro-specific packaging and test dependencies, like
devscripts, tox and libssl-dev on debian or ubuntu. Those pkg dependencies
have now been broken out from common pkg deps to avoid trying to install them
on centos/redhat/suse.
tools/run-centos: cleanups and move to using read-dependencies
These changes are all in an effort to get tools/run-centos using
read-dependencies rather than the 'setup-centos' script with a separate
set of dependencies listed.
- tools/read-dependencies: support taking multiple --requirements
options. This allows run-centos to get both test and build
dependencies. Ultimately, I think it might be nicer for
read-dependencies to take a list of "goals" (build, test, run or
test-tox) rather than having the caller need to know to provide
multiple --requirements.
- packages/pkg-deps.json: drop the version on the sudo package.
centos 6 has newer (1.8.6p3) version than listed, so its not a problem.
- test_handler_disk_setup.py: a test case here was using assertLogs
which is not present in the version of unittest2 that is available in
centos 6 epel. We just adjust it to use with_logs = True.
- tools/run-cents:
- improve usage with example
- add 'inside_as_cd' to provide the dir you want to cd first to.
- avoid the intermediate tarball on disk in the container.
- add 'prep' subcommand and use it to install pre-dependencies.
- use read-dependencies.
pkg build ci: Add make ci-deps-<distro> target to install pkgs
This change adds a couple of makefile targets for ci environments to
install all necessary dependencies for package builds and test runs.
It adds a number of arguments to ./tools/read-dependencies to facilitate
reading pip dependencies, translating pip deps to system package names and
optionally installing needed system-package dependencies on the local
system. This relocates all package dependency and translation logic into
./tools/read-dependencies instead of duplication found in packages/brpm
and packages/bddeb.
In this branch, we also define buildrequires as including all runtime
requires when rendering cloud-init.spec.in and debian/control files
because our package build infrastructure will also be running all unit
test during the package build process so we need runtime deps at build
time.
Additionally, this branch converts
packages/(redhat|suse)/cloud-init.spec.in from cheetah templates to jinja
to allow building python3 envs.
systemd: make cloud-final.service run before apt daily services.
This changes all cloud-init systemd units to run 'Before' the apt processes
that run daily and may cause a lock on the apt database.
apt-daily-upgrade.service contains 'After=apt-daily.service'.
Thus following order is enforced, so we can just be 'Before' the first.
apt-daily.service
apt-daily-upgrade.service
Note that this means only that apt-daily* will not run until
cloud-init has entirely finished. Any other processes running apt-get
operations are still affected by the global lock.
On some systems with python-libselinux a bug[1] related to recursive
restorecon fails but the distro release does not yet include
an update. This change will accept the error and log a warning.
On systems with selinux enabled, some of the networking commands executed
successfully do not return 0. Allow these commands to return 1 since the
output is valid.
Ultimately we need to get this information in some way so that we can
display it correctly. For now, work around the stack trace when selinux
does not allow us to collect it.
In cases where the config json specifies nameserver entries,
if there are interfaces configured to use dhcp, NetworkManager,
if enabled, will clobber the /etc/resolv.conf that cloud-init
has produced, which can break dns. If there are no interfaces
configured to use dhcp, NetworkManager could clobber
/etc/resolv.conf with an empty file.
This patch adds a mechanism for dropping additional configuration
into /etc/NetworkManager/conf.d/ and disables management of
/etc/resolv.conf by NetworkManager when nameserver information is
provided in the config.
Use distro release version to determine if we use systemd in redhat spec
The typical rpm build process will examine the spec file to determine
which packages should be installed in the boot root. This requires
the specfile to declare that it needs system. Provide this information
by checking which version in which the rpm is being built and exporting
requirements for systemd.