Merge ~chad.smith/cloud-init:ci-deps-fixes into cloud-init:master

[Chad Smith]

ci deps: Add --test-distro to read-dependencies to install all deps

read-dependencies now takes --test-distro param to indicate we want to install all system package depenencies to allow for testing and building for our continous integration environment. It allows us to install all needed deps on a fresh system with python3 ./tools/read-depenencies --distro ubuntu --test-distro [--dry-run].

Additionally read-dependencies now looks at what version of python is running the script (py2 vs p3) and opts to install python 2 or 3 system deps respectively. This behavior can still be overridden with python3 ./tools/read-dependencies ... --python-version 2.

There are also some distro-specific packaging dependencies, like devscripts on debian or ubuntu. Those pkg dependencies have now been broken out from common pkg deps to avoid trying to install them on centos/redhat/suse.

ci dependency installs: Add --test-distro param to read-dependencies to install deps

read-dependencies now takes --test-distro param to indicate we want to install all system package depenencies to allow for testing and building for our continous integration environment. It allows us to install all needed deps on a fresh system with python3 ./tools/read-depenencies --distro ubuntu --test-distro [--dry-run].

Additionally read-dependencies now looks at what version of python is running the script (py2 vs p3) and opts to install python 2 or 3 system deps respectively. This behavior can still be overridden with python3 ./tools/read-dependencies ... --python-version 2.

There are also some distro-specific packaging dependencies, like devscripts on debian or ubuntu. Those pkg dependencies have now been broken out from common pkg deps to avoid trying to install them on centos/redhat/suse.

Test instructions:
   lxc launch images:ubuntu/yakkety test-y
   lxc exec test-y bash
   apt update
   apt install git
   git clone -b master https://git.launchpad.net/cloud-init
   python3 ./tools/read-dependencies --distro ubuntu --test-distro
   make deb
   ./packages/bddeb -S

1ae35e5... by Chad Smith on 2017-06-14

now --ci-deps -> --test-distro

d2e7483... by Chad Smith on 2017-06-14

use new --test-distro param for read-dependencies

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:1ae35e5a4944a68b3136e62c2b89d4d91ee951d7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/529/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/529
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/529
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/529
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/529
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/529

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/529/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

2 minor requests.

review: Approve

7e28797... by Chad Smith on 2017-06-14

warn about overriding --requirements-file when --test-distro is provided. Update note with deb bug ref

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:d2e7483380b0e502fc96e879c51204fb74d8050b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/530/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/530
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/530
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/530
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/530
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/530

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/530/rebuild

review: Approve (continuous-integration)

fb499a2... by Chad Smith on 2017-06-14

make --test-distro and --requirements-file a hard error as they should be mutually exclusive

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:fb499a298b0ae22b47b45d3a11af92609e9c6ea8
https://jenkins.ubuntu.com/server/job/cloud-init-ci/531/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/531
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/531
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/531
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/531
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/531

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/531/rebuild

review: Approve (continuous-integration)

f882215... by Chad Smith on 2017-06-14

add other continuous integraion deps one which ci depends

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f882215014999de56ac25c090deaf2107d39482a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/532/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/532
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/532
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/532
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/532
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/532

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/532/rebuild

review: Approve (continuous-integration)

diff --git a/Makefile b/Makefile
index c752530..e9f5498 100644
--- a/Makefile
+++ b/Makefile
@@ -54,12 +54,10 @@ unittest3: clean_pyc
 	nosetests3 $(noseopts) tests/unittests
 
 ci-deps-ubuntu:
-	@$(PYVER) $(CWD)/tools/read-dependencies --distro ubuntu --install --python-version 3
-	@$(PYVER) $(CWD)/tools/read-dependencies --distro ubuntu --requirements-file test-requirements.txt --install --python-version 3
+	@$(PYVER) $(CWD)/tools/read-dependencies --distro-ubuntu --test-distro
 
 ci-deps-centos:
-	@$(PYVER) $(CWD)/tools/read-dependencies --distro centos --install
-	@$(PYVER) $(CWD)/tools/read-dependencies --distro centos --requirements-file test-requirements.txt --install
+	@$(PYVER) $(CWD)/tools/read-dependencies --distro centos --test-distro
 
 pip-requirements:
 	@echo "Installing cloud-init dependencies..."
diff --git a/packages/bddeb b/packages/bddeb
index e45af6e..609a94f 100755
--- a/packages/bddeb
+++ b/packages/bddeb
@@ -72,7 +72,9 @@ def write_debian_folder(root, templ_data, is_python2, cloud_util_deps):
     requires = ['cloud-utils | cloud-guest-utils'] if cloud_util_deps else []
     # We consolidate all deps as Build-Depends as our package build runs all
     # tests so we need all runtime dependencies anyway.
-    requires.extend(reqs + test_reqs + [python])
+    # NOTE: python package was moved to the front after debuild -S would fail with
+    # 'Please add apropriate interpreter' errors (as in debian bug 861132)
+    requires.extend([python] + reqs + test_reqs)
     templater.render_to_file(util.abs_join(find_root(),
                                            'packages', 'debian', 'control.in'),
                              util.abs_join(deb_dir, 'control'),
diff --git a/tools/read-dependencies b/tools/read-dependencies
index 8a58534..2a64868 100755
--- a/tools/read-dependencies
+++ b/tools/read-dependencies
@@ -40,8 +40,13 @@ DISTRO_INSTALL_PKG_CMD = {
 }
 
 
-# List of base system packages required to start using make
-EXTRA_SYSTEM_BASE_PKGS = ['make', 'sudo', 'tar']
+# List of base system packages required to enable ci automation
+CI_SYSTEM_BASE_PKGS = {
+    'common': ['make', 'sudo', 'tar'],
+    'redhat': ['python-tox'],
+    'centos': ['python-tox'],
+    'ubuntu': ['devscripts', 'python3-dev', 'libssl-dev', 'tox', 'sbuild'],
+    'debian': ['devscripts', 'python3-dev', 'libssl-dev', 'tox', 'sbuild']}
 
 
 # JSON definition of distro-specific package dependencies
@@ -70,10 +75,16 @@ def get_parser():
         dest='install',
         help='When specified, install the required system packages.')
     parser.add_argument(
-        '-v', '--python-version', type=str, dest='python_version', default="2",
+        '-t', '--test-distro', action='store_true', default=False,
+        dest='test_distro',
+        help='Additionally install continuous integration system packages '
+             'required for build and test automation.')
+    parser.add_argument(
+        '-v', '--python-version', type=str, dest='python_version', default=None,
         choices=["2", "3"],
-        help='The version of python we want to generate system package '
-             'dependencies for.')
+        help='Override the version of python we want to generate system '
+             'package dependencies for. Defaults to the version of python '
+             'this script is called with')
     return parser
 
 
@@ -114,13 +125,17 @@ def parse_pip_requirements(requirements_path):
     return dep_names
 
 
-def translate_pip_to_system_pkg(pip_requires, renames, python_ver="2"):
+def translate_pip_to_system_pkg(pip_requires, renames, python_ver):
     """Translate pip package names to distro-specific package names.
 
     @param pip_requires: List of versionless pip package names to translate.
     @param renames: Dict containg special case renames from pip name to system
         package name for the distro.
+    @param python_ver: Optional python version string "2" or "3". When None,
+     use the python version that is calling this script via sys.version_info.
     """
+    if python_ver is None:
+        python_ver = str(sys.version_info[0])
     if python_ver == "2":
         prefix = "python-"
     else:
@@ -147,6 +162,16 @@ def main(distro):
     else:
         topd = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
 
+    if args.test_distro:
+        # Give us all the system deps we need for continuous integration
+        if args.req_files:
+            sys.stderr.write(
+                "Parameter --test-distro overrides --requirements-file. Use "
+                "one or the other.\n")
+            sys.exit(1)
+        args.req_files = [os.path.join(topd, DEFAULT_REQUIREMENTS),
+                          os.path.join(topd, 'test-' + DEFAULT_REQUIREMENTS)]
+        args.install = True
     if args.req_files is None:
         args.req_files = [os.path.join(topd, DEFAULT_REQUIREMENTS)]
         if not os.path.isfile(args.req_files[0]):
@@ -179,16 +204,19 @@ def main(distro):
         else:
             all_deps = pip_pkg_names
     if args.install:
-        pkg_install(all_deps, args.distro, args.dry_run)
+        pkg_install(all_deps, args.distro, args.test_distro, args.dry_run)
     else:
         print('\n'.join(all_deps))
 
 
-def pkg_install(pkg_list, distro, dry_run=False):
+def pkg_install(pkg_list, distro, test_distro=False, dry_run=False):
     """Install a list of packages using the DISTRO_INSTALL_PKG_CMD."""
+    if test_distro:
+        pkg_list = list(pkg_list) + CI_SYSTEM_BASE_PKGS['common']
+        distro_base_pkgs = CI_SYSTEM_BASE_PKGS.get(distro, [])
+        pkg_list += distro_base_pkgs
     print('Installing deps: {0}{1}'.format(
           '(dryrun)' if dry_run else '', ' '.join(pkg_list)))
-    pkg_list = list(pkg_list) + EXTRA_SYSTEM_BASE_PKGS
     install_cmd = []
     if dry_run:
        install_cmd.append('echo')
diff --git a/tools/run-centos b/tools/run-centos
index 99ba6be..b10e3bc 100755
--- a/tools/run-centos
+++ b/tools/run-centos
@@ -221,10 +221,7 @@ main() {
     }
 
     inside_as_cd "$name" root "$cdir" \
-        ./tools/read-dependencies \
-            --requirements-file=requirements.txt \
-            --requirements-file=test-requirements.txt \
-            --distro=centos --install || {
+        ./tools/read-dependencies --distro=centos --test-distro || {
         errorrc "FAIL: failed to install dependencies with read-dependencies"
         return
     }