Merge lp:~flacoste/maas/buildout-site-packages into lp:~maas-committers/maas/trunk

[Francis J. Lacoste]

Add explicit versioning for the buildout dependencies. Integrate buildout-versions to manage versions.cfg. Load runtime dependencies from site packages.

This branch adds explicit versioning the the MaaS dependencies.

It adds buildout-versions which makes it easy to manage the versions.cfg
file.

All runtime dependencies are loaded from site packages (well, if they are
available, apart Django, there is not much harm done if you didn't install
the package, as it will still pull it from PyPI).

I turned off allow-picked-versions which will make buildout complain if any
new dependencies are introduced without adding the correct entry to
versions.cfg This should make us more mindful of introducing dep and make us
pause to see if this is a runtime dep (should be packaged and loaded from
site-packages) or a dev-only dep.

I also updated setup.py with the dependencies.

[Gavin Panella (allenap)]

Looks good. Lots of comments, mostly trivial. However, setup.py is
broken, so Needs Fixing.

[1]

+You will need to manually install Postgres 9.1 (postgresql-9.1),
+, RabbitMQ (rabbitmq-server), python-dev and make::

s/^, //

[2]

+Additionally, you probably want to install the following python libraries
+for development convenience::

s/probably want/need/, at least for python-lxml.

[3]

+possible. You'll need to add the dependency to the
+``allowed-eggs-from-site-packages`` option in the ``buildout.cfg`` file. (And
+don't forget to add the version to ``versions.cfg`` as we run with
+``allowed-picked-version`` set to false.)

Might be worth mentioning that it also needs to be added to setup.py.

It would be nice if we could auto-generate versions.cfg from setup.py,
or vice-versa. Actually, doesn't buildout use dependency information
from setup.py?

[4]

+If it is a development-only depency (i.e. only needed for the test suite, or

s/depency/dependency/

[5]

+buildout_versions_file = versions.cfg

Grr, everything else uses hyphens. /me emails buildout-versions
author.

[6]

+download-cache = download-cache

/^download-cache/d

We've chosen to *not* define this in <branch>/buildout.cfg; we're
using ~/.buildout/default.cfg to use a shared cache. This is much
easier than following the Launchpad model while MaaS's deps are small
enough.

[7]

+allowed-eggs-from-site-packages =
+ Django
+ South
...

I know it's trivial, but the rest of the file uses 2 spaces for
indentation :)

[8]

+ # Convenient developer dependencies
+ Jinja2
+ Pygments

Are these dependencies of Sphinx? They're not direct dependencies I
think, so can they be left out here?

[9]

+ install_requires=[
+ 'setuptools',
+ 'Django' = 1.3.1,

This doesn't work. This does:

        'Django == 1.3.1',

[10]

+ extra_requires=dict(

s/extra_requires/extras_require/

Also, there's a missing brace at the end of setup.py.

[11]

+# Automatic versions in Precise

Is this something that buildout-versions provides?

[12]

We could structure versions.cfg like so:

  [versions]
  <= versions-run
     versions-dev
     versions-doc
     versions-auto

  [versions-run]
  Django = ...

  [versions-dev]
  lxml = ...

  [version-doc]
  Sphinx = ...

  [versions-auto]
  buildout-versions = ...

Then we could pull these out in setup.py:

  versions = ConfigParser.ConfigParser()
  versions.read("versions.cfg")

  def gen_deps(section):
      for option in versions.options(section):
          value = versions.get(section, option)
          if options[-1] in "<!>":
              yield b"%s=%s" % (option, value)
          else:
              yield b"%s==%s" % (option, value)

  deps = lambda section: list(gen_deps(section))

  ...
      install_requires=(
          deps("versions-run") +
          deps("versions-auto")),
  ...
      extras_require={
          b'doc': deps("versions-doc"),
          b'tests': deps("versions-run"),
          },

And, after using buildout-versions to update versions.cfg, we should
probably merge the automatically added things into their appropriate
section in versions.cfg.

[13]

Does buildout-versions help us notice depende...

review: Needs Fixing

[Francis J. Lacoste (flacoste)]

On 12-02-24 06:41 PM, Gavin Panella wrote:
> Review: Needs Fixing
>
> Looks good. Lots of comments, mostly trivial. However, setup.py is
> broken, so Needs Fixing.

Thanks for the review, I've fixed everything you asked for.

>
> It would be nice if we could auto-generate versions.cfg from setup.py,
> or vice-versa. Actually, doesn't buildout use dependency information
> from setup.py?

It does, but versions.cfg is applied on top of those. The general rule
of setup.py is that you don't pin version unless your package won't work
with any other version, whereas we use version pinning in buildout for
repeatable builds.

>
> [5]
>
> +buildout_versions_file = versions.cfg
>
> Grr, everything else uses hyphens. /me emails buildout-versions
> author.

:-)

>
>
> [6]
>
> +download-cache = download-cache
>
> /^download-cache/d
>
> We've chosen to *not* define this in <branch>/buildout.cfg; we're
> using ~/.buildout/default.cfg to use a shared cache. This is much
> easier than following the Launchpad model while MaaS's deps are small
> enough.

Well, ok. But I disagree, it's not easier than the Launchpad model if
you have a lot of different projects managed by buildout. In which case,
you end up with a massive eggs and download-cache where it's very hard
to know what comes from where.

>
> [8]
>
> + # Convenient developer dependencies
> + Jinja2
> + Pygments
>
> Are these dependencies of Sphinx? They're not direct dependencies I
> think, so can they be left out here?

They are, but they still need to be added here otherwise buildout will
install them from PyPI.

>
>
> [11]
>
> +# Automatic versions in Precise
>
> Is this something that buildout-versions provides?
>

Unfortunately, no, buildout-versions doesn't pick up the dependency
coming from system python. I need to report a bug (probably submit a
fix) upstream. The future version of buildout (2.0) actually drop the
system eggs support that Gary added. It seems that nobody but us is
using that and supporting it requires isn't trivial, so upstream decided
to drop it.

>
> [12]
>
> We could structure versions.cfg like so:
>
> [versions]
> <= versions-run
> versions-dev
> versions-doc
> versions-auto
>
> [versions-run]
> Django = ...
>
> [versions-dev]
> lxml = ...
>
> [version-doc]
> Sphinx = ...
>
> [versions-auto]
> buildout-versions = ...
>
> Then we could pull these out in setup.py:
>
> versions = ConfigParser.ConfigParser()
> versions.read("versions.cfg")
>
> def gen_deps(section):
> for option in versions.options(section):
> value = versions.get(section, option)
> if options[-1] in "<!>":
> yield b"%s=%s" % (option, value)
> else:
> yield b"%s==%s" % (option, value)
>
> deps = lambda section: list(gen_deps(section))
>
> ...
> install_requires=(
> deps("versions-run") +
> deps("versions-auto")),
> ...
> extras_require={
> b'doc': deps("versions-doc"),
> b'tests': deps("versions-run"),
> },
>
> And, after using buildout-versions to update versions.cfg, we should
> probably merge the au...

[Gavin Panella (allenap)]

> Thanks for the review, I've fixed everything you asked for.

I'm very sorry I didn't reply yesterday; I had forgotten I marked it
Needs Fixing (and I obviously can't read email).

> > [6]
> >
> > +download-cache = download-cache
> >
> > /^download-cache/d
> >
> > We've chosen to *not* define this in <branch>/buildout.cfg; we're
> > using ~/.buildout/default.cfg to use a shared cache. This is much
> > easier than following the Launchpad model while MaaS's deps are small
> > enough.
>
> Well, ok. But I disagree, it's not easier than the Launchpad model if
> you have a lot of different projects managed by buildout. In which case,
> you end up with a massive eggs and download-cache where it's very hard
> to know what comes from where.

I agree with having a separate download-cache and eggs directory when
deploying from a tree built using buildout, like Launchpad. Then you
can't go and nuke the shared caches because you'll break things. But
for MaaS, buildout is only for development, so using a shared cache -
so that all branches build quickly without having to think about it -
is a win.

> > [12]
...
> I've reordered the file versions.cfg file as you suggested, but I'm not
> sure that generating the setup.py dependency from versions.cfg is such a
> good idea. This will basically mean that the version will be pinned like
> in buildout, which is good for repeatable builds, but not so much for
> general use. Given that we are not going to release that package on
> PyPI, maybe that's kind of moot (and mean that setup.py isn't that
> interesting either).

Why no release on PyPI?

Looks good to go, +1.

review: Approve

[Francis J. Lacoste (flacoste)]

On 12-02-28 08:39 AM, Gavin Panella wrote:
> Review: Approve
>
>> Thanks for the review, I've fixed everything you asked for.
>
> I'm very sorry I didn't reply yesterday; I had forgotten I marked it
> Needs Fixing (and I obviously can't read email).
>
>>> [6]
>>>
>>> +download-cache = download-cache
>>>
>>> /^download-cache/d
>>>
>>> We've chosen to *not* define this in <branch>/buildout.cfg; we're
>>> using ~/.buildout/default.cfg to use a shared cache. This is much
>>> easier than following the Launchpad model while MaaS's deps are small
>>> enough.
>>
>> Well, ok. But I disagree, it's not easier than the Launchpad model if
>> you have a lot of different projects managed by buildout. In which case,
>> you end up with a massive eggs and download-cache where it's very hard
>> to know what comes from where.
>
> I agree with having a separate download-cache and eggs directory when
> deploying from a tree built using buildout, like Launchpad. Then you
> can't go and nuke the shared caches because you'll break things. But
> for MaaS, buildout is only for development, so using a shared cache -
> so that all branches build quickly without having to think about it -
> is a win.

Right, I'm all for using a shared cache, and I symlink download-cache to
that shared cache. My beef is just that if the way you set your shared
cache is ~/.buildout/defaults.cfg then that's shared cache is actually
used for all of your buildout-managed development projects (which don't
override it locally). And that's what I personally don't like. But like
I said, it's more a personal preferences than anything else.

>
>>> [12]
> ...
>> I've reordered the file versions.cfg file as you suggested, but I'm not
>> sure that generating the setup.py dependency from versions.cfg is such a
>> good idea. This will basically mean that the version will be pinned like
>> in buildout, which is good for repeatable builds, but not so much for
>> general use. Given that we are not going to release that package on
>> PyPI, maybe that's kind of moot (and mean that setup.py isn't that
>> interesting either).
>
> Why no release on PyPI?
>

Well, that's not a binding decision. We could upload it to PyPI I guess,
but given that this is more an application than a library and that it's
targeted at deployment in Ubuntu... why bother?

> Looks good to go, +1.
>

Thanks!

--
Francis J. Lacoste
<email address hidden>

[Dave Walker (davewalker)]

Bouncing approved status to see if tarmac catches it.

review: Approve

[Raphaël Badin (rvb)]

Sorry but this broke the build so after a discussion with the team we decided to revert this change.

See: https://code.launchpad.net/~rvb/maas/maas-revert-buildout-chang/+merge/95140 for details.

=== modified file 'HACKING.txt'
--- HACKING.txt	2012-02-24 11:16:36 +0000
+++ HACKING.txt	2012-02-27 21:22:17 +0000
@@ -23,20 +23,32 @@
 
 Prerequisites
 =============
-You will need to manually install Postgres 9.1 (postgresql-9.1 and
-libpq-dev), RabbitMQ (rabbitmq-server), python-dev and make::
-
-    $ sudo apt-get install postgresql-9.1 libpq-dev rabbitmq-server python-dev make
+You will need to manually install Postgres 9.1 (postgresql-9.1),
+RabbitMQ (rabbitmq-server), python-dev and make::
+
+    $ sudo apt-get install postgresql-9.1 rabbitmq-server python-dev make
+
 
 Also, you might want to install Bazaar (bzr) to grab the source code directly
 from Launchpad::
 
     $ sudo apt-get install bzr
 
-If you intend to run the test suite, you also need libxslt1-dev, libxml2-dev,
-xvfb and firefox::
-
-    $ sudo apt-get install libxslt1-dev libxml2-dev xvfb firefox
+This is the list of runtime dependencies that you'll need to install::
+
+    $ sudo apt-get install python-django python-django-piston \
+        python-django-south python-twisted python-txamqp python-amqplib \
+        python-formencode python-oauth python-oops python-oops-datedir-repo \
+        python-twisted python-oops-wsgi python-psycopg2 python-yaml
+
+Additionally, you need to install the following python libraries
+for development convenience::
+
+    $ sudo apt-get install python-sphinx python-lxml
+
+If you intend to run the test suite, you also need xvfb and firefox::
+
+    $ sudo apt-get install xvfb firefox
 
 All other development dependencies are pulled automatically from `PyPI`_
 when buildout runs.
@@ -107,6 +119,22 @@
 
     $ make distclean
 
+Adding new dependencies
+=======================
+
+Since MaaS is distributed mainly as Ubuntu package, all runtime dependencies
+should be packaged and we should develop with the packaged version if
+possible. You'll need to add the dependency to the
+``allowed-eggs-from-site-packages`` option in the ``buildout.cfg`` file. You
+also need to add it to setup.py (And don't forget to add the version to
+``versions.cfg`` as we run with ``allowed-picked-version`` set to false.)
+
+If it is a development-only dependency (i.e. only needed for the test suite, or
+for developers' convenience), simply running ``buildout`` like this will make
+the necessary updates to ``versions.cfg``::
+
+    $ ./bin/buildout -v buildout:allow-picked-versions=true
+
 
 Adding new source files
 =======================
@@ -126,3 +154,4 @@
 
 .. _convention for headings as used in the Python documentation:
   http://sphinx.pocoo.org/rest.html#sections
+

=== modified file 'buildout.cfg'
--- buildout.cfg	2012-02-24 10:54:15 +0000
+++ buildout.cfg	2012-02-27 21:22:17 +0000
@@ -7,12 +7,43 @@
   pserv-test
   repl
   sphinx
+extensions = buildout-versions
+buildout_versions_file = versions.cfg
+versions = versions
 extends = versions.cfg
-versions = versions
-include-site-packages = false
-# Don't always check for newer packages; use `bin/buildout -n` to
-# override this and check explicitly.
-newest = false
+install-from-cache = false
+
+# Since MaaS's main deployment target is Ubuntu, all
+# runtime dependencies should come from python packages.
+# Only development-time dependencies should come from eggs.
+# For convenience, we allow some of those to come from site-packages,
+# mainly those which contains C extensions like lxml
+include-site-packages = true
+allowed-eggs-from-site-packages =
+  Django
+  South
+  amqplib
+  django-piston
+  FormEncode
+  oauth
+  oops
+  oops-datedir-repo
+  oops-twisted
+  oops-wsgi
+  psycopg2
+  PyYAML
+  Twisted
+  txAMQP
+  # Convenient developer dependencies
+  Jinja2
+  Pygments
+  Sphinx
+  docutils
+  lxml
+  ipython
+
+prefer-final = true
+allow-picked-versions = false
 
 [common]
 extra-paths =

=== modified file 'setup.py'
--- setup.py	2012-01-24 12:53:05 +0000
+++ setup.py	2012-02-27 21:22:17 +0000
@@ -40,7 +40,7 @@
     name="maas",
     version=__version__,
     url="https://launchpad.net/maas",
-    license="GPL",
+    license="AGPLv3",
     description="Metal as as Service",
     long_description=read('README.txt'),
 
@@ -56,8 +56,22 @@
         ),
     package_dir={'': b'src'},
 
-    install_requires=['setuptools'],
-
+    install_requires=[
+        'setuptools',
+        'Django == 1.3.1',
+        'psycopg2',
+        'amqplib',
+        'django-piston',
+        'FormEncode',
+        'oauth',
+        'oops',
+        'oops-datedir-repo',
+        'oops-twisted',
+        'PyYAML',
+        'South',
+        'Twisted',
+        'txAMQP',
+        ],
     classifiers=[
         'Development Status :: 4 - Beta',
         'Framework :: Django',
@@ -67,5 +81,24 @@
         'Operating System :: OS Independent',
         'Programming Language :: Python',
         'Topic :: Internet :: WWW/HTTP',
-        ]
+        ],
+    extras_require=dict(
+        doc=[
+            'collective.recipe.sphinxbuilder',
+            'Sphinx',
+            ],
+        tests=[
+            'coverage',
+            'django-nose',
+            'lxml',
+            'sst',
+            'fixtures',
+            'nose',
+            'nose-subunit',
+            'python-subunit',
+            'rabbitfixture',
+            'testresources',
+            'testtools',
+            ],
     )
+)

=== modified file 'versions.cfg'
--- versions.cfg	2012-02-24 10:54:15 +0000
+++ versions.cfg	2012-02-27 21:22:17 +0000
@@ -1,31 +1,102 @@
 [versions]
-coverage =
-django = 1.3.1
+<= versions-run
+   versions-dev
+   versions-doc
+   versions-auto
+
+[versions-run]
+# Actually, we depend on the version of django in Ubuntu precise
+# which contains the backported fix for
+# https://code.djangoproject.com/ticket/16250
+# Otherwise, psycopg2 > 2.4.1 is problematic with Django 1.3.1.
+Django = 1.3.1
+# psycopg2 = 2.4.1
+psycopg2 = 2.4.4
+
+# Versions in Precise
+amqplib = 1.0.0
+django-piston = 0.2.3
+FormEncode = 1.2.4
+oauth = 1.0.1
+oops = 0.0.10
+oops-datedir-repo = 0.0.15
+oops-twisted = 0.0.6
+oops-wsgi = 0.0.9
+PyYAML = 3.10
+South = 0.7.3
+Twisted = 11.1.0
+txAMQP = 0.5
+
+[versions-doc]
+# Versions in Precise
+docutils = 0.8.1
+Jinja2 = 2.6
+Pygments = 1.4
+Sphinx = 1.0.8
+lxml = 2.3.2
+
+[versions-dev]
+ipython = 0.12
 # Bug 251 is problematic in 0.9.2.
 django-debug-toolbar = 0.9.1
-django-nose =
-django-piston =
-docutils =
-fixtures =
-formencode = 1.2.4
-ipython =
-lxml =
-nose =
-nose-subunit =
-oauth =
-oops =
-oops-datedir-repo =
-oops-twisted =
-oops-wsgi =
-# psycopg2 > 2.4.1 is problematic with Django 1.3.1; see
-# https://code.djangoproject.com/ticket/16250
-psycopg2 = 2.4.1
-python-subunit =
-pyyaml = 3.10
+
+[versions-auto]
+# Added by Buildout Versions at 2012-02-24 15:51:04.865203
+buildout-versions = 1.7
+collective.recipe.sphinxbuilder = 0.7.0
+coverage = 3.5.1
+distribute = 0.6.24
+django-nose = 0.1.3
+fixtures = 0.3.8
+flake8 = 1.1
+nose = 1.1.2
+nose-subunit = 0.2
+python-subunit = 0.0.7
 rabbitfixture = 0.3.2
-South =
-sst =
-testresources >= 0.2.4-r58
-testtools =
-twisted =
-txamqp =
+testresources = 0.2.5
+testtools = 0.9.14
+z3c.recipe.scripts = 1.0.1
+
+# Required by:
+# collective.recipe.sphinxbuilder==0.7.0
+zc.buildout = 1.5.2
+
+# Required by:
+# collective.recipe.sphinxbuilder==0.7.0
+zc.recipe.egg = 1.3.2
+
+# Added by Buildout Versions at 2012-02-24 16:56:06.100791
+PyVirtualDisplay = 0.0.9
+sst = 0.1.0
+
+# Required by:
+# PyVirtualDisplay==0.0.9
+EasyProcess = 0.1.3
+
+# Required by:
+# entrypoint2==0.0.4
+argparse = 1.2.1
+
+# Required by:
+# entrypoint2==0.0.4
+decorator = 3.3.2
+
+# Required by:
+# PyVirtualDisplay==0.0.9
+entrypoint2 = 0.0.4
+
+# Required by:
+# sst==0.1.0
+junitxml = 0.6
+
+# Required by:
+# PyVirtualDisplay==0.0.9
+path.py = 2.2.2
+
+# Required by:
+# sst==0.1.0
+selenium = 2.19.1
+
+# Required by:
+# sst==0.1.0
+unittest2 = 0.5.1