Merge ~powersj/cloud-init:cii-requirements into cloud-init:master

[Joshua Powers]

tests: add integration requirements text file

This adds the specific requirements for integration testing to
a single file that can be referenced in other areas. It also enables
the read-dependencies script to install those packages.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:eed66ea6248d510de87af6ea545bdce2c6e63836
https://jenkins.ubuntu.com/server/job/cloud-init-ci/667/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/667/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b06626f2c5632ad6ce8ed2aced5e04985c535156
https://jenkins.ubuntu.com/server/job/cloud-init-ci/673/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/673/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

I believe I have made the changes we talked about on Friday and this is ready for another review.

[Scott Moser (smoser)]

Things I don't *love*:
 1. there is no way to install test dependencies on ubuntu without installing integration-test dependencies.
 2. installing integration-test dependencies on ubuntu will not get you
    a.) lxd
    b.) qemu-system or cloud-image-utils (for mount-image-callback)

I'm temted to say we should fix '1' by adding a '--integration-test-distro' (-I) flag rather than chaning the behavior of '--test-distro'.

Then that can *fail* on non-ubuntu.

For '2', the '--integration-test-distro' would help in solving it also. Just gives us more information to work from.

[Joshua Powers (powersj)]

I have removed the work done to read-dependencies and updated the commit message: First, all non-ubuntu OSes do not have access to simplestreams as it is not available via system packages or via PyPI. This already limits the usefulness of this to Ubuntu. Second, the version of pylxd, must be the latest from pypi, it cannot be the system package. This is largely why we like using tox to run the tests.

As a result of the two points above, I do not see a strong reason to even have any of the integration requirements inside the read-dependencies script at this time. This commit simple creates a requirements file for managing and documenting specific requirements for integration testing, rather than doing that work inside of the tox.ini file.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:2f5db0ac698cda29bdbb32374b36771114364f4e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/680/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/680/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Ultimately I do want some way to "prepare my system to run c-i".
As I'm going to deploy a stock instance and then want to do just that.

That goal is very similar to what 'read-dependencies --install --test-distro' does, so it seems to make sense to add the functionality there.

I'm not bent on having that done right now though.

Chad, do you have thoughts on that?

[Chad Smith (chad.smith)]

Yeah I'm in the same camp. I think cloud-init should grow the possibility of a one-command developer/ci setup. If we can bake that into make ci-ubuntu-deps via extensions to read-dependencies at some point that'd make me happy, but as Josh suggested in IRC, this might be cloud-init trying to handle too dependency sources.

1. We can't install exclusively from pypi because of bzr-lp dependencies sources defined in tox,
2. We can't install exclusively via system packages because some pips aren't packaged as debs/rpms

So, we might need a hybrid approach for greenfield systems and that'll involve some bikeshedding ;)

review: Approve

diff --git a/integration-requirements.txt b/integration-requirements.txt
new file mode 100644
index 0000000..5c81c2c
--- /dev/null
+++ b/integration-requirements.txt
@@ -0,0 +1,18 @@
+# PyPI requirements for cloud-init integration testing
+# https://cloudinit.readthedocs.io/en/latest/topics/tests.html
+#
+# Note: Changes to this requirements may require updates to
+# the packages/pkg-deps.json file as well.
+#
+
+# ec2 backend
+boto3==1.5.9
+
+# ssh communication
+paramiko==2.4.0
+
+# lxd backend
+pylxd==2.2.4
+
+# finds latest image information
+bzr+lp:simplestreams
diff --git a/tox.ini b/tox.ini
index 88b82dc..bb74853 100644
--- a/tox.ini
+++ b/tox.ini
@@ -132,7 +132,4 @@ basepython = python3
 commands = {envpython} -m tests.cloud_tests {posargs}
 passenv = HOME
 deps =
-    pylxd==2.2.4
-    paramiko==2.3.1
-    boto3==1.4.8
-    bzr+lp:simplestreams
+    -r{toxinidir}/integration-requirements.txt