Merge lp:~allenap/maas-test/use-system-packages into lp:maas-test

[Gavin Panella]

Install production dependencies using system packages, and allow the virtualenv to use them.

[lp:~allenap/maas-test/use-system-packages]

17. By Gavin Panella on 2013-11-11

Help a non-interactive build go smoothly.

[Jeroen T. Vermeulen (jtv)]

Not too keen on being surprised by password prompts... Can we keep package installation out of the default "make" target?

[Raphaël Badin (rvb)]

I agree with what Jeroen said. Which means that building an environment would now be a two-step process. Probably worth updating the README.txt file to explain (really quickly) how this works.

Regarding the changes to requirements.txt, I'm not sure we should manage this file manually like this. Especially now that it's only used for development dependencies, I think we should let pip manage it and generate it with "pip freeze > requirements.txt". Which means that adding a dev dependency would be as simple as:
pip install new-dev
pip freeze > requirements.txt

[Raphaël Badin (rvb)]

Also, again about the change to requirements.txt, shouldn't we drop the production dependencies from there now that we require them to be installed as package? Which means we probably should change how virtualenv+pip is configured to use the system's package as I don't think it's configured like this ATM.

[lp:~allenap/maas-test/use-system-packages]

18. By Gavin Panella on 2013-11-12

Only use Python 2 for now.

19. By Gavin Panella on 2013-11-12

Split out installing packages into a separate target.

[Gavin Panella (allenap)]

> Not too keen on being surprised by password prompts... Can we keep package
> installation out of the default "make" target?

Done.

[Gavin Panella (allenap)]

> I agree with what Jeroen said. Which means that building an environment would
> now be a two-step process. Probably worth updating the README.txt file to
> explain (really quickly) how this works.

Done, and documentation added.

>
> Regarding the changes to requirements.txt, I'm not sure we should manage this
> file manually like this. Especially now that it's only used for development
> dependencies, I think we should let pip manage it and generate it with "pip
> freeze > requirements.txt". Which means that adding a dev dependency would be
> as simple as:
> pip install new-dev
> pip freeze > requirements.txt

When I tried this yesterday (with the --local flag, otherwise it gives you the version of every package on the system) the output wasn't useful, but today it is. I'm not quite sure what I've done to deserve this, but I guess this makes sense. We probably ought to put version information in packages.txt (I think apt-get groks that), but that shouldn't prevent it from working on Trusty and beyond.

[Gavin Panella (allenap)]

> pip install new-dev
> pip freeze > requirements.txt

This causes problems. If the version in requirements.txt differs from
another installed version, you get stuff like this:

$ make test
bin/python setup.py test
running test
Traceback (most recent call last):
  File "setup.py", line 37, in <module>
    "maas-test = maastest.main:main",
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File ".../local/lib/python2.7/site-packages/setuptools/command/test.py", line 128, in run
    self.distribution.fetch_build_eggs(self.distribution.install_requires)
  File ".../local/lib/python2.7/site-packages/setuptools/dist.py", line 289, in fetch_build_eggs
    parse_requirements(requires), installer=self.fetch_build_egg
  File ".../local/lib/python2.7/site-packages/pkg_resources.py", line 630, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (flake8 2.0 (/home/gavin/Library/lib/python2.7/site-packages/flake8-2.0-py2.7.egg), Requirement.parse('flake8==2.1.0'))
make: *** [test] Error 1

Nothing ever just works, does it?

[Gavin Panella (allenap)]

I'm starting to think we don't need a virtualenv here. The only dev package is flake8 (and its deps), and that's packaged for Ubuntu, for Python 3 too.

[lp:~allenap/maas-test/use-system-packages]

20. By Gavin Panella on 2013-11-12

Switch requirements.txt to development-only dependencies.

21. By Gavin Panella on 2013-11-12

Documentation re. dependencies.

22. By Gavin Panella on 2013-11-12

Tweak README.

[Gavin Panella (allenap)]

> We probably ought to put version information in packages.txt (I think
> apt-get groks that), but that shouldn't prevent it from working on
> Trusty and beyond.

apt-get only groks exact versions, so I've left this out.

[Raphaël Badin (rvb)]

Nice! Thanks a lot for doing this!

[0]

10 + apt-get --assume-yes --no-install-recommends

One question: why the "--no-install-recommends"?

review: Approve

[Gavin Panella (allenap)]

> Nice! Thanks a lot for doing this!
>
> [0]
>
> 10 + apt-get --assume-yes --no-install-recommends
>
> One question: why the "--no-install-recommends"?

Copied from MAAS's install-dependencies target.

Thanks for the review!

=== modified file 'Makefile'
--- Makefile	2013-11-05 10:15:44 +0000
+++ Makefile	2013-11-12 13:44:42 +0000
@@ -3,6 +3,12 @@
 
 PYTHON := python
 
+define apt-get
+sudo DEBIAN_FRONTEND=noninteractive \
+    apt-get --assume-yes --no-install-recommends
+endef
+
+
 build: bin/python bin/pip requirements.txt
 	bin/pip install --requirement=requirements.txt
 
@@ -26,6 +32,10 @@
 	@bin/pip install -q flake8
 
 bin/python bin/pip:
-	virtualenv --python=$(PYTHON) $(PWD)
-
-.PHONY: build dist clean lint test
+	virtualenv --python=$(PYTHON) --system-site-packages $(PWD)
+
+install-dependencies: packages.txt
+	@xargs --verbose --no-run-if-empty $(apt-get) install < packages.txt
+
+
+.PHONY: build dist clean install-dependencies lint test

=== modified file 'README.txt'
--- README.txt	2013-10-30 12:46:38 +0000
+++ README.txt	2013-11-12 13:44:42 +0000
@@ -10,3 +10,40 @@
 For more information see the `Launchpad project page`_.
 
 .. _Launchpad project page: https://launchpad.net/maas-test
+
+
+Setting up for development
+--------------------------
+
+To get maas-test running should be fairly easy, assuming you're on
+Ubuntu 13.10 or later::
+
+  bzr branch lp:maas-test
+  cd maas-test
+  make install-dependencies
+  make build
+
+The ``make install-dependencies`` step ensures that maas-test's
+dependencies are installed as system packages, and will not later be
+installed by pip.
+
+
+Dependencies
+------------
+
+The policy is:
+
+  * All production dependencies must be satisfied from system packages.
+
+  * Development-only dependencies can be installed from PyPI.
+
+That means that production dependencies need to go in ``packages.txt``
+and development-only ones into ``requirements.txt``. The latter can be
+maintained like so::
+
+  bin/pip install some-new-dev-only-library
+  bin/pip freeze --local > requirements.txt
+
+Then **carefully** review the changes; libraries installed elsewhere on
+the system can cause ``freeze --local`` to come up with radically
+different answers to those on another machine.

=== added file 'packages.txt'
--- packages.txt	1970-01-01 00:00:00 +0000
+++ packages.txt	2013-11-12 13:44:42 +0000
@@ -0,0 +1,6 @@
+python-fixtures
+python-lxml
+python-mock
+python-netaddr
+python-testresources
+python-testtools

=== modified file 'requirements.txt'
--- requirements.txt	2013-11-08 14:26:53 +0000
+++ requirements.txt	2013-11-12 13:44:42 +0000
@@ -1,7 +1,4 @@
-lxml >= 3.2.3
-mock >= 1.0.1
-netaddr >= 0.7
-fixtures >= 0.3.8
-flake8 >= 2.1.0
-testresources >= 0.2.7
-testtools >= 0.9.14
+flake8==2.1.0
+mccabe==0.2.1
+pep8==1.4.6
+pyflakes==0.7.3