Merge lp:~lvh/divmod.org/modernize-axiom-packaging into lp:divmod.org

[Laurens Van Houtven]

A number of changes to make Axiom a bit more present-day Python packaging compatible :)

setup.py now works without relying on non-stdlib projects. This makes it easier to pip install Axiom.

Also, tox has been configured to automagically make virtualenvs to run the tests in. This will hopefully make test runs more reproducible, instead of relying. It has the added benefit that our dependency specifications (previously DEPS.txt, now requirements.txt, for consistency with current packaging standards) are actually tested. As a free bonus, tox does coverage testing using coverage.py, too.

[Laurens Van Houtven (lvh)]

Also, Axiom's version is now also available as __version__. This appears to be the more common place to specify the version. axiom.version still works, so this is not a backwards incompatible change. I still need to write a test to check that the two are identical, but I'm not sure where to put it: test_listversions seems the closest one, but not exactly a match :)

[Laurens Van Houtven (lvh)]

(Also, this patch doesn't actually make Axiom pip installable yet on a clean machine, as you can tell from tox.ini. To do that, I'm going to have to give Epsilon similar treatment.)

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2728. By Laurens Van Houtven on 2013-08-15

Don't use setuptools

[Jean-Paul Calderone (exarkun)]

Thanks.

  1) The way setup.py and axiom/_version.py interact to convey version information seems more complex than necessary. This is of particular interest since there is no test coverage of setup.py's idea of the project version, I think. Since assembling strings is easier than parsing them, perhaps the version information should be available as a string literal in _version.py - whatever form is the absolute simplest for setup.py to get at - and it can be parsed into the Version object in _version.py (by unit tested code).

Oh, apparently that's my only comment. Please address, verify the result is sufficiently successful on continuous integration, and then merge.

Thanks again!

review: Needs Fixing

[Laurens Van Houtven (lvh)]

I'm a bit at a loss of how to make this better. A version number tuple (i.e. (1, 2, 3)) makes the regex essentially the same. An explicit version string would require parsing the string again (granted, *map(int, v.split(".") isn't the hardest parse in the world).

[Tristan Seligmann (mithrandi)]

+1 on this branch, aside from the issue exarkun raised above, but I'd like to suggest something else: dropping support for Python 2.4 and Python 2.5. This is probably a little out of scope for this branch, but since you're adding a requirements-py24.txt in this branch, I thought it might be easier just to get rid of it.

We don't actually seem to have an explict statement of Python version requirement anywhere now that DEPS.txt is gone; I guess the way to indicate this is with the appropriate classifiers?

review: Needs Fixing

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2729. By Laurens Van Houtven on 2013-08-15

Address JP's version issues

2730. By Laurens Van Houtven on 2013-08-15

Remove Python 2.4 requirements file

2731. By Laurens Van Houtven on 2013-08-15

Update existing requirements file

2732. By Laurens Van Houtven on 2013-08-15

Test against Twisted trunk

2733. By Laurens Van Houtven on 2013-08-15

Add trove classifiers for supported Python versions

[Laurens Van Houtven (lvh)]

I've added the trove classifiers.

The tox.ini file now tests against py26 and py27, both for Twisted's latest release and Twisted trunk.

[Tristan Seligmann (mithrandi)]

I've noticed another issue with switching away from epsilon.setuphelper; it has a regeneratePluginCache hook to regenerate the Twisted plugin cache (actually, it looks for any directory of the form foo/plugins/ where foo/plugins/__init__.py does not exist). The new setup.py does not have this functionality; on the other hand, after reading https://twistedmatrix.com/trac/ticket/2410 I start to wonder if this functionality shouldn't just be dropped after all.

review: Needs Fixing

[Tristan Seligmann (mithrandi)]

To summarize: the new setup.py doesn't regenerate the Twisted plugin cache post-install the way epsilon.setuphelper does. This doesn't matter for distro packaging or user installs, but represents a regression for system-wide installs as the administrator will need to regenerate the plugin cache manually.

(Also, I don't know of a better way of doing this than the hack present in epsilon.setuphelper)

[Jean-Paul Calderone (exarkun)]

A better implementation than epsilon.setuphelper is probably to customize the `install` command. http://stackoverflow.com/questions/1321270/how-to-extend-distutils-with-a-simple-post-install-script shows how this can be done.

[Laurens Van Houtven (lvh)]

I'm trying to do this, but this section of distutils isn't really documented[1]. For one, epsilon.setuphelper.regeneratePluginCaches wants the distobj, but I'm not sure what the appropriate way to give it a reference to that is. Do I just do distobj = setup(...) and use that from in setup.py?

Either way, that thing seems pretty general, and I think it can all be boiled down to:

list(plugin.getPlugins(plugin.IPlugin, "axiom.plugins"))

maybe wrapped in a try except if modules have been removed. I'm not sure what happens if pip is dragging in Twisted as a dependency when installing Axiom.

[1]: http://docs.python.org/2/distutils/apiref.html#module-distutils.command

[Laurens Van Houtven (lvh)]

I am saddened:

https://gist.github.com/lvh/8ced76c8ba794e42fac3

I'm not entirely sure what the hell is going on there, but it appears that by default, setup.py install is using some *setuptools* stuff; that's the only way I can explain that that flag works, since it's something distutils doesn't know about.

[Laurens Van Houtven (lvh)]

Realized why. tox installs axiom into the virtualenv using pip, which uses setuptools. I guess when I said earlier that I could do this with distutils, that wasn't really true. Sorry.

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2734. By Laurens Van Houtven on 2013-09-01

Explicitly use setuptools

2735. By Laurens Van Houtven on 2013-09-02

Post-install cache regeneration

2736. By Laurens Van Houtven on 2013-09-02

Merge trunk

2737. By Laurens Van Houtven on 2013-09-02

Merge new trunk (with removed performance tests that were causing failures on py27 + new sqlites)

[Laurens Van Houtven (lvh)]

I've fixed this by customizing the equivalent setuptools command.

This regenerates the cache using the command Tristan suggested. The only downside is that it now requires setuptools. On the plus side running the tests using tox now no longer relies on pip's deep distutils-masking magic :)

Please review.

[Laurens Van Houtven (lvh)]

Also, my new branch: https://code.launchpad.net/~lvh/divmod.org/axiom-docs adds a bunch of documentation and relies on this (or at least will) for automatically building and testing the documentation.

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2738. By Laurens Van Houtven on 2013-09-02

Use Launchpad page as URL

[Tristan Seligmann (mithrandi)]

> list(plugin.getPlugins(plugin.IPlugin, "axiom.plugins"))

I haven't actually tried running this code yet, but I'm pretty sure it raises an exception. The second parameter to getPlugins() is an actual python module object, not a string. In other words, this code should be something like:

import axiom.plugins
list(plugin.getPlugins(plugin.IPlugin))
list(plugin.getPlugins(plugin.IPlugin, axiom.plugins))

(Note the two calls to getPlugins; we want to regenerate the cache for twisted.plugins as well as for axiom.plugins)

Alas, I suspect this approach may work fine when running 'setup.py install' as the old setup.py did, but fail during a pip installation (or during the building of a Debian/etc. package) as Axiom is not being installed into a location that is on the sys.path of the running Python interpreter.

review: Needs Fixing

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2739. By Laurens Van Houtven on 2013-09-03

Fix cache regeneration logic

2740. By Laurens Van Houtven on 2013-09-03

Actually use the install class. Oops.

[Laurens Van Houtven (lvh)]

My bad. I've fixed it in the current version.

pip installs into new virtualenvs appear to work fine, now. I'm 100% sure the _regenerateCache function is being called (I know because I put a raise statement in there).

I'm trying now, rebuilding *all* the virtualenvs to make sure it's not a spuriour success due to a stale install of Axiom somewhere.

[Laurens Van Houtven (lvh)]

(tox -r succeeded too.)

[lp:~lvh/divmod.org/modernize-axiom-packaging]

2741. By Laurens Van Houtven on 2013-10-14

setup.py doesn't actually rely on the strip or triple-quoting anymore

2742. By Laurens Van Houtven on 2013-10-14

Add PyOpenSSL to install_requires

2743. By Laurens Van Houtven on 2013-10-14

Add note to eventually remove PyOpenSSL from install_requires

=== added file 'Axiom/.coveragerc'
--- Axiom/.coveragerc	1970-01-01 00:00:00 +0000
+++ Axiom/.coveragerc	2013-10-14 15:41:13 +0000
@@ -0,0 +1,8 @@
+[run]
+branch = True
+source =
+    axiom 
+
+[report]
+exclude_lines =
+    pragma: no cover

=== modified file 'Axiom/axiom/__init__.py'
--- Axiom/axiom/__init__.py	2008-03-09 21:03:23 +0000
+++ Axiom/axiom/__init__.py	2013-10-14 15:41:13 +0000
@@ -1,5 +1,8 @@
 # -*- test-case-name: axiom.test -*-
-
-from axiom._version import version
-version                         # tell pyflakes we're exporting it.
-
+from twisted.python import versions
+from axiom._version import __version__
+
+def _asTwistedVersion(packageName, versionString):
+    return versions.Version(packageName, *map(int, versionString.split(".")))
+
+version = _asTwistedVersion("axiom", __version__)

=== modified file 'Axiom/axiom/_version.py'
--- Axiom/axiom/_version.py	2009-11-30 01:08:55 +0000
+++ Axiom/axiom/_version.py	2013-10-14 15:41:13 +0000
@@ -1,3 +1,1 @@
-# This is an auto-generated file. Use Epsilon/bin/release-divmod to update.
-from twisted.python import versions
-version = versions.Version(__name__[:__name__.rfind('.')], 0, 6, 0)
+__version__ = "0.6.0"

=== added file 'Axiom/requirements-testing.txt'
--- Axiom/requirements-testing.txt	1970-01-01 00:00:00 +0000
+++ Axiom/requirements-testing.txt	2013-10-14 15:41:13 +0000
@@ -0,0 +1,2 @@
+# Requirements for running the test suite using tox
+coverage

=== renamed file 'Axiom/DEPS.txt' => 'Axiom/requirements.txt'
--- Axiom/DEPS.txt	2006-06-14 11:54:41 +0000
+++ Axiom/requirements.txt	2013-10-14 15:41:13 +0000
@@ -1,7 +1,5 @@
-Python 2.4
-SQLite 3.2.1
-Twisted 2.4.0
-PySQLite 2.0
-NPTL 2.3.5 or later (LinuxThreads not supported: see
-                     <http://twistedmatrix.com/bugs/issue1251>)
-Epsilon 0.5.0
+Twisted>=13.1.0
+Epsilon>=0.6.0
+PyOpenSSL>=0.13 # epsilon.juice depends on this
+# You may also optionally install pysqlite2. If you don't, the stdlib
+# sqlite module will be used.

=== modified file 'Axiom/setup.py'
--- Axiom/setup.py	2009-11-30 01:08:55 +0000
+++ Axiom/setup.py	2013-10-14 15:41:13 +0000
@@ -1,23 +1,59 @@
-from epsilon.setuphelper import autosetup
-
-import axiom
-
-distobj = autosetup(
+from setuptools import setup, find_packages
+from setuptools.command.install import install as Install
+import re
+
+versionPattern = re.compile(r"""^__version__ = ['"](.*?)['"]$""", re.M)
+with open("axiom/_version.py", "rt") as f:
+    version = versionPattern.search(f.read()).group(1)
+
+
+
+class InstallAndRegenerate(Install):
+    def run(self):
+        """
+        Runs the usual install logic, then regenerates the plugin cache.
+        """
+        Install.run(self)
+        _regenerateCache()
+
+
+
+def _regenerateCache():
+    from twisted import plugin
+    from axiom import plugins
+    list(plugin.getPlugins(plugin.IPlugin)) # Twisted
+    list(plugin.getPlugins(plugin.IPlugin, plugins)) # Axiom
+
+
+setup(
     name="Axiom",
-    version=axiom.version.short(),
+    version=version,
+    description="An in-process object-relational database",
+    url="https://launchpad.net/divmod.org",
+
     maintainer="Divmod, Inc.",
     maintainer_email="[e-mail]",
-    url="http://divmod.org/trac/wiki/DivmodAxiom",
+
+    install_requires=["twisted", "epsilon", "PyOpenSSL"],
+    # XXX: Remove PyOpenSSL install_requires as soon as epsilon
+    # specifies this dependency
+    packages=find_packages() + ['twisted.plugins'],
+    scripts=['bin/axiomatic'],
+    cmdclass={
+        "install": InstallAndRegenerate,
+    },
+
     license="MIT",
     platforms=["any"],
-    description="An in-process object-relational database",
+
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Framework :: Twisted",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: MIT License",
         "Programming Language :: Python",
-        "Topic :: Database"],
-
-    scripts=['bin/axiomatic'])
-
+        "Programming Language :: Python :: 2",
+        "Programming Language :: Python :: 2.6",
+        "Programming Language :: Python :: 2.7",
+        "Programming Language :: Python :: 2 :: Only",
+        "Topic :: Database"])

=== added file 'Axiom/tox.ini'
--- Axiom/tox.ini	1970-01-01 00:00:00 +0000
+++ Axiom/tox.ini	2013-10-14 15:41:13 +0000
@@ -0,0 +1,24 @@
+[tox]
+envlist = py26,py27,py26-trunkdeps,py27-trunkdeps
+
+[testenv]
+deps =
+    Twisted
+commands =
+    pip install \
+        -r {toxinidir}/requirements.txt \
+        -r {toxinidir}/requirements-testing.txt
+    coverage run {envdir}/bin/trial \
+        --temp-directory={envdir}/_trial {posargs:axiom}
+    coverage report --show-missing
+    coverage html --directory {envdir}/_coverage
+
+[testenv:py26-trunkdeps]
+basepython = python2.6
+deps =
+     --editable=svn+svn://svn.twistedmatrix.com/svn/Twisted/trunk
+
+[testenv:py27-trunkdeps]
+basepython = python2.7
+deps =
+     --editable=svn+svn://svn.twistedmatrix.com/svn/Twisted/trunk