Merge lp:~james-w/pkgme-service-python/buildout into lp:pkgme-service-python

[James Westby]

Switch to buildout.

Hi,

Here's a stab at moving a project to buildout.

Some issues:

  * buildout is very tempremental trying to bootstrap.
    - If you have it installed system-wide then you can't use the
      bootstrap.py. You can work around this by doing the bootstrap
      in a virtualenv, but that's not particularly nice.
    - This branch uses the system-wide one. I think the drawback of that
      is that everyone will have to use exactly the same version.
    - I can't really work out how LP is avoiding these issues. I suspect
      they aren't.
  * The Makefile dependencies are a little off, e.g. after a bootstrap
    a check will do a quick buildout run, and subsequent runs won't.
  * There's no depenencies branch to use yet.
  * I don't know if we can speed things up with sharing the eggs dir between
    branches. As it stands this branch is a few seconds slower to bootstrap
    than trunk. I don't know if that will reverse on larger projects (I
    suspect it will though.)

I'm mainly posting this for discussion at this point.

Thanks,

James

[lp:~james-w/pkgme-service-python/buildout]

18. By James Westby on 2012-07-27

Switch to distribute and add a note about site-wide buildout.

[James Westby (james-w)]

I investigated the site-wide thing some more, and filed

  https://bugs.launchpad.net/zc.buildout/+bug/1029715

The workaround I'm using for now is to require v1.5.1 of buildout,
so it works for precise users, even if they have buildout installed
via apt.

Thanks,

James

[lp:~james-w/pkgme-service-python/buildout]

19. By James Westby on 2012-07-27

Symlink the eggs by default.

[James Westby (james-w)]

and yes we can share eggs (link-external-sourcecode does this).

It speeds up subsequent bootstraps immensely, and there's little
risk of contamination.

I've taken a stab at doing it by default with no extra steps, but
I'm not sure if using the parent dir will be appreciated.

Thanks,

James

[Jonathan Lange (jml)]

Looks good. Thanks for exploring this! I've pinged benji & gary_poster on IRC to take a look as well.

Is the shared eggs directory is inherited from the environment? ISTR buildout has a facility for this already, but I can't recall exactly?

Also, in case you haven't read it, http://launchpad.readthedocs.org/en/latest/buildout.html

Thanks,
jml

review: Needs Information

[Jonathan Lange (jml)]

 * EGGS_DIR being set works.

 * Works for me.

 * Probably should add lp:ca-download-cache step to either Makefile or documentation

review: Approve

[lp:~james-w/pkgme-service-python/buildout]

20. By James Westby on 2012-07-30

Checkout the download-cache too, and use more specific variable names.

21. By James Westby on 2012-07-30

Explain the env vars in the HACKING doc.

[James Westby (james-w)]

On Mon, 30 Jul 2012 16:12:20 -0000, Jonathan Lange <email address hidden> wrote:
> * Probably should add lp:ca-download-cache step to either Makefile or documentation

Added to both. I also made the names of the env vars more specific.

Thanks,

James

=== modified file '.bzrignore'
--- .bzrignore	2012-07-19 16:09:51 +0000
+++ .bzrignore	2012-07-30 20:14:17 +0000
@@ -1,7 +1,9 @@
-_trial_temp
-build
-virtualenv
-virtualenv.tarmac
 log
 pkgme_service_python.egg-info
 credentials
+.installed.cfg
+bin
+develop-eggs
+download-cache
+eggs
+parts

=== modified file 'HACKING'
--- HACKING	2012-07-19 14:22:06 +0000
+++ HACKING	2012-07-30 20:14:17 +0000
@@ -14,7 +14,7 @@
 Running the tests
 =================
 
-Development of pkgme-service-python is done within a virtualenv build
+Development of pkgme-service-python is done within a buildout build
 environment.  You will need to create one of these in order to run the tests.
 
 In a checkout of `lp:pkgme-service-python
@@ -23,9 +23,25 @@
   $ make bootstrap
   $ make check
 
-You may wish to run ``. virtualenv/bin/activate`` to be able to run e.g.
-``python`` and have the project and all dependencies importable.
-
+The bootstrap will fail if you have a system-wide install of buildout that
+is the same version as the one in use by this project. (You will see
+``DistributionNotFound: zc.buildout==<version>``). If you encounter
+that then you can either remove the site-wide install, or use a virtualenv
+to run the bootstrap step.
+
+You can get a shell to try code interactively by running ``./bin/py``.
+
+Buildout uses two directories as caches that can be shared between branches.
+The first is the ``download-cache`` directory. This contains all of the
+distributions of the Python dependencies. You can get this from
+``lp:ca-download-cache``, but the Makefile will grab it for you.
+
+The other directory is the ``eggs`` directory that holds built versions
+of the dependencies.
+
+The default for both of these is to symlink them from the parent directory,
+but if you wish to put them somewhere else you can set the locations with
+the ``CA_DOWNLOAD_CACHE_DIR`` and ``CA_EGGS_DIR`` environment variables.
 
 Bug tracker
 ===========
@@ -94,8 +110,14 @@
 Dependencies are tracked in two different places:
 
 1. ``setup.py`` lists the key dependencies of the package
-1. ``test-dependencies.txt`` lists dependencies that are only used by the
-   suite.
+1. ``buildout.cfg`` lists dependencies that are only used by the
+   test suite.
+
+The versions of all dependencies are chosen in ``versions.cfg``.
+To change the version in use you should
+
+1. Change the version in ``versions.cfg``.
+1. Run ``make update-deps``.
 
 
 How to release

=== modified file 'Makefile'
--- Makefile	2012-07-11 12:02:57 +0000
+++ Makefile	2012-07-30 20:14:17 +0000
@@ -12,27 +12,69 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-VIRTUALENV=virtualenv
-VIRTUALENV_DIR?=virtualenv
-VIRTUALENV_PIP=$(VIRTUALENV_DIR)/bin/pip
-VIRTUALENV_PYTHON=$(VIRTUALENV_DIR)/bin/python
+EXTERNAL_PY?=python2.7
+PY=./bin/py
+
+EXTERNAL_BUILDOUT?=$(EXTERNAL_PY) bootstrap.py
+BUILDOUT=./bin/buildout
+
+CA_EGGS_DIR?=../eggs
+CA_DOWNLOAD_CACHE_DIR?=../download-cache
 
 clean:
-	[ ! -d $(VIRTUALENV_DIR) ] || rm -r $(VIRTUALENV_DIR)
-
-fetch-deps: $(VIRTUALENV_DIR)
-	$(VIRTUALENV_PIP) install -r test-dependencies.txt
-	$(VIRTUALENV_PYTHON) setup.py develop
-
-$(VIRTUALENV_DIR):
-	$(VIRTUALENV) --no-site-packages $(VIRTUALENV_DIR)
-
-bootstrap: clean fetch-deps
-
-test: $(VIRTUALENV_DIR)
-	$(VIRTUALENV_PYTHON) -m testtools.run discover pkgme_service_client
+	$(RM) -r bin
+	$(RM) -r develop-eggs
+ifeq ($(CA_EGGS_DIR),eggs)
+	$(RM) -r eggs
+else
+	$(RM) eggs
+endif
+ifeq ($(CA_DOWNLOAD_CACHE_DIR),download-cache)
+	$(RM) -r download-cache
+else
+	$(RM) download-cache
+endif
+	$(RM) -r parts
+	$(RM) -r .installed.cfg
+
+eggs:
+ifeq ($(CA_EGGS_DIR),$@)
+	mkdir $@
+else
+	[ -d $(CA_EGGS_DIR) ] || mkdir $(CA_EGGS_DIR)
+	ln -s $(CA_EGGS_DIR) $@
+endif
+
+download-cache:
+ifeq ($(CA_DOWNLOAD_CACHE_DIR),$@)
+	bzr checkout lp:ca-download-cache $@
+else
+	[ -d $(CA_DOWNLOAD_CACHE_DIR) ] || bzr checkout lp:ca-download-cache $(CA_DOWNLOAD_CACHE_DIR)
+	ln -s $(CA_DOWNLOAD_CACHE_DIR) $@
+endif
+
+$(BUILDOUT): bootstrap.py distribute_setup.py eggs download-cache
+	$(EXTERNAL_BUILDOUT) --distribute --setup-source distribute_setup.py \
+				--download-base=download-cache/dist --eggs=eggs \
+				--version 1.5.1
+	touch --no-create $@
+
+$(PY): $(BUILDOUT)
+	$(BUILDOUT)
+	touch --no-create $@
+
+bootstrap: clean $(PY)
+
+test: $(PY)
+	$(PY) -m testtools.run discover pkgme_service_client
 
 check: test
 
+# Can be used to update the dependencies when you change something
+# in versions.cfg.
+update-deps: $(BUILDOUT)
+	$(BUILDOUT) buildout:install-from-cache=false
+
 
 .PHONY: bootstrap test check clean fetch-deps
+.DEFAULT_GOAL := $(PY)

=== added file 'bootstrap.py'
--- bootstrap.py	1970-01-01 00:00:00 +0000
+++ bootstrap.py	2012-07-30 20:14:17 +0000
@@ -0,0 +1,262 @@
+##############################################################################
+#
+# Copyright (c) 2006 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Bootstrap a buildout-based project
+
+Simply run this script in a directory containing a buildout.cfg.
+The script accepts buildout command-line options, so you can
+use the -c option to specify an alternate configuration file.
+"""
+
+import os, shutil, sys, tempfile, urllib, urllib2, subprocess
+from optparse import OptionParser
+
+if sys.platform == 'win32':
+    def quote(c):
+        if ' ' in c:
+            return '"%s"' % c  # work around spawn lamosity on windows
+        else:
+            return c
+else:
+    quote = str
+
+# See zc.buildout.easy_install._has_broken_dash_S for motivation and comments.
+stdout, stderr = subprocess.Popen(
+    [sys.executable, '-Sc',
+     'try:\n'
+     '    import ConfigParser\n'
+     'except ImportError:\n'
+     '    print 1\n'
+     'else:\n'
+     '    print 0\n'],
+    stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+has_broken_dash_S = bool(int(stdout.strip()))
+
+# In order to be more robust in the face of system Pythons, we want to
+# run without site-packages loaded.  This is somewhat tricky, in
+# particular because Python 2.6's distutils imports site, so starting
+# with the -S flag is not sufficient.  However, we'll start with that:
+if not has_broken_dash_S and 'site' in sys.modules:
+    # We will restart with python -S.
+    args = sys.argv[:]
+    args[0:0] = [sys.executable, '-S']
+    args = map(quote, args)
+    os.execv(sys.executable, args)
+# Now we are running with -S.  We'll get the clean sys.path, import site
+# because distutils will do it later, and then reset the path and clean
+# out any namespace packages from site-packages that might have been
+# loaded by .pth files.
+clean_path = sys.path[:]
+import site  # imported because of its side effects
+sys.path[:] = clean_path
+for k, v in sys.modules.items():
+    if k in ('setuptools', 'pkg_resources') or (
+        hasattr(v, '__path__') and
+        len(v.__path__) == 1 and
+        not os.path.exists(os.path.join(v.__path__[0], '__init__.py'))):
+        # This is a namespace package.  Remove it.
+        sys.modules.pop(k)
+
+is_jython = sys.platform.startswith('java')
+
+setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
+distribute_source = 'http://python-distribute.org/distribute_setup.py'
+
+
+# parsing arguments
+def normalize_to_url(option, opt_str, value, parser):
+    if value:
+        if '://' not in value:  # It doesn't smell like a URL.
+            value = 'file://%s' % (
+                urllib.pathname2url(
+                    os.path.abspath(os.path.expanduser(value))),)
+        if opt_str == '--download-base' and not value.endswith('/'):
+            # Download base needs a trailing slash to make the world happy.
+            value += '/'
+    else:
+        value = None
+    name = opt_str[2:].replace('-', '_')
+    setattr(parser.values, name, value)
+
+usage = '''\
+[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
+
+Bootstraps a buildout-based project.
+
+Simply run this script in a directory containing a buildout.cfg, using the
+Python that you want bin/buildout to use.
+
+Note that by using --setup-source and --download-base to point to
+local resources, you can keep this script from going over the network.
+'''
+
+parser = OptionParser(usage=usage)
+parser.add_option("-v", "--version", dest="version",
+                          help="use a specific zc.buildout version")
+parser.add_option("-d", "--distribute",
+                   action="store_true", dest="use_distribute", default=False,
+                   help="Use Distribute rather than Setuptools.")
+parser.add_option("--setup-source", action="callback", dest="setup_source",
+                  callback=normalize_to_url, nargs=1, type="string",
+                  help=("Specify a URL or file location for the setup file. "
+                        "If you use Setuptools, this will default to " +
+                        setuptools_source + "; if you use Distribute, this "
+                        "will default to " + distribute_source + "."))
+parser.add_option("--download-base", action="callback", dest="download_base",
+                  callback=normalize_to_url, nargs=1, type="string",
+                  help=("Specify a URL or directory for downloading "
+                        "zc.buildout and either Setuptools or Distribute. "
+                        "Defaults to PyPI."))
+parser.add_option("--eggs",
+                  help=("Specify a directory for storing eggs.  Defaults to "
+                        "a temporary directory that is deleted when the "
+                        "bootstrap script completes."))
+parser.add_option("-t", "--accept-buildout-test-releases",
+                  dest='accept_buildout_test_releases',
+                  action="store_true", default=False,
+                  help=("Normally, if you do not specify a --version, the "
+                        "bootstrap script and buildout gets the newest "
+                        "*final* versions of zc.buildout and its recipes and "
+                        "extensions for you.  If you use this flag, "
+                        "bootstrap and buildout will get the newest releases "
+                        "even if they are alphas or betas."))
+parser.add_option("-c", None, action="store", dest="config_file",
+                   help=("Specify the path to the buildout configuration "
+                         "file to be used."))
+
+options, args = parser.parse_args()
+
+# if -c was provided, we push it back into args for buildout's main function
+if options.config_file is not None:
+    args += ['-c', options.config_file]
+
+if options.eggs:
+    eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
+else:
+    eggs_dir = tempfile.mkdtemp()
+
+if options.setup_source is None:
+    if options.use_distribute:
+        options.setup_source = distribute_source
+    else:
+        options.setup_source = setuptools_source
+
+if options.accept_buildout_test_releases:
+    args.append('buildout:accept-buildout-test-releases=true')
+args.append('bootstrap')
+
+try:
+    import pkg_resources
+    import setuptools  # A flag.  Sometimes pkg_resources is installed alone.
+    if not hasattr(pkg_resources, '_distribute'):
+        raise ImportError
+except ImportError:
+    ez_code = urllib2.urlopen(
+        options.setup_source).read().replace('\r\n', '\n')
+    ez = {}
+    exec ez_code in ez
+    setup_args = dict(to_dir=eggs_dir, download_delay=0)
+    if options.download_base:
+        setup_args['download_base'] = options.download_base
+    if options.use_distribute:
+        setup_args['no_fake'] = True
+    ez['use_setuptools'](**setup_args)
+    if 'pkg_resources' in sys.modules:
+        reload(sys.modules['pkg_resources'])
+    import pkg_resources
+    # This does not (always?) update the default working set.  We will
+    # do it.
+    for path in sys.path:
+        if path not in pkg_resources.working_set.entries:
+            pkg_resources.working_set.add_entry(path)
+
+cmd = [quote(sys.executable),
+       '-c',
+       quote('from setuptools.command.easy_install import main; main()'),
+       '-mqNxd',
+       quote(eggs_dir)]
+
+if not has_broken_dash_S:
+    cmd.insert(1, '-S')
+
+find_links = options.download_base
+if not find_links:
+    find_links = os.environ.get('bootstrap-testing-find-links')
+if find_links:
+    cmd.extend(['-f', quote(find_links)])
+
+if options.use_distribute:
+    setup_requirement = 'distribute'
+else:
+    setup_requirement = 'setuptools'
+ws = pkg_resources.working_set
+setup_requirement_path = ws.find(
+    pkg_resources.Requirement.parse(setup_requirement)).location
+env = dict(
+    os.environ,
+    PYTHONPATH=setup_requirement_path)
+
+requirement = 'zc.buildout'
+version = options.version
+if version is None and not options.accept_buildout_test_releases:
+    # Figure out the most recent final version of zc.buildout.
+    import setuptools.package_index
+    _final_parts = '*final-', '*final'
+
+    def _final_version(parsed_version):
+        for part in parsed_version:
+            if (part[:1] == '*') and (part not in _final_parts):
+                return False
+        return True
+    index = setuptools.package_index.PackageIndex(
+        search_path=[setup_requirement_path])
+    if find_links:
+        index.add_find_links((find_links,))
+    req = pkg_resources.Requirement.parse(requirement)
+    if index.obtain(req) is not None:
+        best = []
+        bestv = None
+        for dist in index[req.project_name]:
+            distv = dist.parsed_version
+            if _final_version(distv):
+                if bestv is None or distv > bestv:
+                    best = [dist]
+                    bestv = distv
+                elif distv == bestv:
+                    best.append(dist)
+        if best:
+            best.sort()
+            version = best[-1].version
+if version:
+    requirement = '=='.join((requirement, version))
+cmd.append(requirement)
+
+if is_jython:
+    import subprocess
+    exitcode = subprocess.Popen(cmd, env=env).wait()
+else:  # Windows prefers this, apparently; otherwise we would prefer subprocess
+    exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
+if exitcode != 0:
+    sys.stdout.flush()
+    sys.stderr.flush()
+    print ("An error occurred when trying to install zc.buildout. "
+           "Look above this message for any errors that "
+           "were output by easy_install.")
+    sys.exit(exitcode)
+
+ws.add_entry(eggs_dir)
+ws.require(requirement)
+import zc.buildout.buildout
+zc.buildout.buildout.main(args)
+if not options.eggs:  # clean up temporary egg directory
+    shutil.rmtree(eggs_dir)

=== added file 'buildout.cfg'
--- buildout.cfg	1970-01-01 00:00:00 +0000
+++ buildout.cfg	2012-07-30 20:14:17 +0000
@@ -0,0 +1,36 @@
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+[buildout]
+parts =
+    scripts
+unzip = true
+eggs-directory = eggs
+download-cache = download-cache
+relative-paths = true
+
+# Disable this option temporarily if you want buildout to find software
+# dependencies *other* than those in our download-cache.  Once you have the
+# desired software, reenable this option (and check in the new software to
+# lp:ca-dependencies if this is going to be reviewed/merged/deployed.)
+install-from-cache = true
+
+# This also will need to be temporarily disabled or changed for package
+# upgrades.  Newly-added packages should also add their desired version number
+# to versions.cfg.
+extends = versions.cfg
+
+allow-picked-versions = false
+
+prefer-final = true
+
+develop = .
+
+[scripts]
+recipe = z3c.recipe.scripts
+# Test dependencies get added here
+eggs = pkgme-service-python
+       mock
+       testtools
+include-site-packages = false
+interpreter = py

=== added file 'distribute_setup.py'
--- distribute_setup.py	1970-01-01 00:00:00 +0000
+++ distribute_setup.py	2012-07-30 20:14:17 +0000
@@ -0,0 +1,515 @@
+#!python
+"""Bootstrap distribute installation
+
+If you want to use setuptools in your package's setup.py, just include this
+file in the same directory with it, and add this to the top of your setup.py::
+
+    from distribute_setup import use_setuptools
+    use_setuptools()
+
+If you want to require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, you can do so by supplying
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import os
+import sys
+import time
+import fnmatch
+import tempfile
+import tarfile
+from distutils import log
+
+try:
+    from site import USER_SITE
+except ImportError:
+    USER_SITE = None
+
+try:
+    import subprocess
+
+    def _python_cmd(*args):
+        args = (sys.executable,) + args
+        return subprocess.call(args) == 0
+
+except ImportError:
+    # will be used for python 2.3
+    def _python_cmd(*args):
+        args = (sys.executable,) + args
+        # quoting arguments if windows
+        if sys.platform == 'win32':
+            def quote(arg):
+                if ' ' in arg:
+                    return '"%s"' % arg
+                return arg
+            args = [quote(arg) for arg in args]
+        return os.spawnl(os.P_WAIT, sys.executable, *args) == 0
+
+DEFAULT_VERSION = "0.6.28"
+DEFAULT_URL = "http://pypi.python.org/packages/source/d/distribute/"
+SETUPTOOLS_FAKED_VERSION = "0.6c11"
+
+SETUPTOOLS_PKG_INFO = """\
+Metadata-Version: 1.0
+Name: setuptools
+Version: %s
+Summary: xxxx
+Home-page: xxx
+Author: xxx
+Author-email: xxx
+License: xxx
+Description: xxx
+""" % SETUPTOOLS_FAKED_VERSION
+
+
+def _install(tarball, install_args=()):
+    # extracting the tarball
+    tmpdir = tempfile.mkdtemp()
+    log.warn('Extracting in %s', tmpdir)
+    old_wd = os.getcwd()
+    try:
+        os.chdir(tmpdir)
+        tar = tarfile.open(tarball)
+        _extractall(tar)
+        tar.close()
+
+        # going in the directory
+        subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
+        os.chdir(subdir)
+        log.warn('Now working in %s', subdir)
+
+        # installing
+        log.warn('Installing Distribute')
+        if not _python_cmd('setup.py', 'install', *install_args):
+            log.warn('Something went wrong during the installation.')
+            log.warn('See the error message above.')
+    finally:
+        os.chdir(old_wd)
+
+
+def _build_egg(egg, tarball, to_dir):
+    # extracting the tarball
+    tmpdir = tempfile.mkdtemp()
+    log.warn('Extracting in %s', tmpdir)
+    old_wd = os.getcwd()
+    try:
+        os.chdir(tmpdir)
+        tar = tarfile.open(tarball)
+        _extractall(tar)
+        tar.close()
+
+        # going in the directory
+        subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
+        os.chdir(subdir)
+        log.warn('Now working in %s', subdir)
+
+        # building an egg
+        log.warn('Building a Distribute egg in %s', to_dir)
+        _python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir)
+
+    finally:
+        os.chdir(old_wd)
+    # returning the result
+    log.warn(egg)
+    if not os.path.exists(egg):
+        raise IOError('Could not build the egg.')
+
+
+def _do_download(version, download_base, to_dir, download_delay):
+    egg = os.path.join(to_dir, 'distribute-%s-py%d.%d.egg'
+                       % (version, sys.version_info[0], sys.version_info[1]))
+    if not os.path.exists(egg):
+        tarball = download_setuptools(version, download_base,
+                                      to_dir, download_delay)
+        _build_egg(egg, tarball, to_dir)
+    sys.path.insert(0, egg)
+    import setuptools
+    setuptools.bootstrap_install_from = egg
+
+
+def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
+                   to_dir=os.curdir, download_delay=15, no_fake=True):
+    # making sure we use the absolute path
+    to_dir = os.path.abspath(to_dir)
+    was_imported = 'pkg_resources' in sys.modules or \
+        'setuptools' in sys.modules
+    try:
+        try:
+            import pkg_resources
+            if not hasattr(pkg_resources, '_distribute'):
+                if not no_fake:
+                    _fake_setuptools()
+                raise ImportError
+        except ImportError:
+            return _do_download(version, download_base, to_dir, download_delay)
+        try:
+            pkg_resources.require("distribute>=" + version)
+            return
+        except pkg_resources.VersionConflict:
+            e = sys.exc_info()[1]
+            if was_imported:
+                sys.stderr.write(
+                "The required version of distribute (>=%s) is not available,\n"
+                "and can't be installed while this script is running. Please\n"
+                "install a more recent version first, using\n"
+                "'easy_install -U distribute'."
+                "\n\n(Currently using %r)\n" % (version, e.args[0]))
+                sys.exit(2)
+            else:
+                del pkg_resources, sys.modules['pkg_resources']    # reload ok
+                return _do_download(version, download_base, to_dir,
+                                    download_delay)
+        except pkg_resources.DistributionNotFound:
+            return _do_download(version, download_base, to_dir,
+                                download_delay)
+    finally:
+        if not no_fake:
+            _create_fake_setuptools_pkg_info(to_dir)
+
+
+def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
+                        to_dir=os.curdir, delay=15):
+    """Download distribute from a specified location and return its filename
+
+    `version` should be a valid distribute version number that is available
+    as an egg for download under the `download_base` URL (which should end
+    with a '/'). `to_dir` is the directory where the egg will be downloaded.
+    `delay` is the number of seconds to pause before an actual download
+    attempt.
+    """
+    # making sure we use the absolute path
+    to_dir = os.path.abspath(to_dir)
+    try:
+        from urllib.request import urlopen
+    except ImportError:
+        from urllib2 import urlopen
+    tgz_name = "distribute-%s.tar.gz" % version
+    url = download_base + tgz_name
+    saveto = os.path.join(to_dir, tgz_name)
+    src = dst = None
+    if not os.path.exists(saveto):  # Avoid repeated downloads
+        try:
+            log.warn("Downloading %s", url)
+            src = urlopen(url)
+            # Read/write all in one block, so we don't create a corrupt file
+            # if the download is interrupted.
+            data = src.read()
+            dst = open(saveto, "wb")
+            dst.write(data)
+        finally:
+            if src:
+                src.close()
+            if dst:
+                dst.close()
+    return os.path.realpath(saveto)
+
+
+def _no_sandbox(function):
+    def __no_sandbox(*args, **kw):
+        try:
+            from setuptools.sandbox import DirectorySandbox
+            if not hasattr(DirectorySandbox, '_old'):
+                def violation(*args):
+                    pass
+                DirectorySandbox._old = DirectorySandbox._violation
+                DirectorySandbox._violation = violation
+                patched = True
+            else:
+                patched = False
+        except ImportError:
+            patched = False
+
+        try:
+            return function(*args, **kw)
+        finally:
+            if patched:
+                DirectorySandbox._violation = DirectorySandbox._old
+                del DirectorySandbox._old
+
+    return __no_sandbox
+
+
+def _patch_file(path, content):
+    """Will backup the file then patch it"""
+    existing_content = open(path).read()
+    if existing_content == content:
+        # already patched
+        log.warn('Already patched.')
+        return False
+    log.warn('Patching...')
+    _rename_path(path)
+    f = open(path, 'w')
+    try:
+        f.write(content)
+    finally:
+        f.close()
+    return True
+
+_patch_file = _no_sandbox(_patch_file)
+
+
+def _same_content(path, content):
+    return open(path).read() == content
+
+
+def _rename_path(path):
+    new_name = path + '.OLD.%s' % time.time()
+    log.warn('Renaming %s into %s', path, new_name)
+    os.rename(path, new_name)
+    return new_name
+
+
+def _remove_flat_installation(placeholder):
+    if not os.path.isdir(placeholder):
+        log.warn('Unkown installation at %s', placeholder)
+        return False
+    found = False
+    for file in os.listdir(placeholder):
+        if fnmatch.fnmatch(file, 'setuptools*.egg-info'):
+            found = True
+            break
+    if not found:
+        log.warn('Could not locate setuptools*.egg-info')
+        return
+
+    log.warn('Removing elements out of the way...')
+    pkg_info = os.path.join(placeholder, file)
+    if os.path.isdir(pkg_info):
+        patched = _patch_egg_dir(pkg_info)
+    else:
+        patched = _patch_file(pkg_info, SETUPTOOLS_PKG_INFO)
+
+    if not patched:
+        log.warn('%s already patched.', pkg_info)
+        return False
+    # now let's move the files out of the way
+    for element in ('setuptools', 'pkg_resources.py', 'site.py'):
+        element = os.path.join(placeholder, element)
+        if os.path.exists(element):
+            _rename_path(element)
+        else:
+            log.warn('Could not find the %s element of the '
+                     'Setuptools distribution', element)
+    return True
+
+_remove_flat_installation = _no_sandbox(_remove_flat_installation)
+
+
+def _after_install(dist):
+    log.warn('After install bootstrap.')
+    placeholder = dist.get_command_obj('install').install_purelib
+    _create_fake_setuptools_pkg_info(placeholder)
+
+
+def _create_fake_setuptools_pkg_info(placeholder):
+    if not placeholder or not os.path.exists(placeholder):
+        log.warn('Could not find the install location')
+        return
+    pyver = '%s.%s' % (sys.version_info[0], sys.version_info[1])
+    setuptools_file = 'setuptools-%s-py%s.egg-info' % \
+            (SETUPTOOLS_FAKED_VERSION, pyver)
+    pkg_info = os.path.join(placeholder, setuptools_file)
+    if os.path.exists(pkg_info):
+        log.warn('%s already exists', pkg_info)
+        return
+
+    if not os.access(pkg_info, os.W_OK):
+        log.warn("Don't have permissions to write %s, skipping", pkg_info)
+
+    log.warn('Creating %s', pkg_info)
+    f = open(pkg_info, 'w')
+    try:
+        f.write(SETUPTOOLS_PKG_INFO)
+    finally:
+        f.close()
+
+    pth_file = os.path.join(placeholder, 'setuptools.pth')
+    log.warn('Creating %s', pth_file)
+    f = open(pth_file, 'w')
+    try:
+        f.write(os.path.join(os.curdir, setuptools_file))
+    finally:
+        f.close()
+
+_create_fake_setuptools_pkg_info = _no_sandbox(
+    _create_fake_setuptools_pkg_info
+)
+
+
+def _patch_egg_dir(path):
+    # let's check if it's already patched
+    pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO')
+    if os.path.exists(pkg_info):
+        if _same_content(pkg_info, SETUPTOOLS_PKG_INFO):
+            log.warn('%s already patched.', pkg_info)
+            return False
+    _rename_path(path)
+    os.mkdir(path)
+    os.mkdir(os.path.join(path, 'EGG-INFO'))
+    pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO')
+    f = open(pkg_info, 'w')
+    try:
+        f.write(SETUPTOOLS_PKG_INFO)
+    finally:
+        f.close()
+    return True
+
+_patch_egg_dir = _no_sandbox(_patch_egg_dir)
+
+
+def _before_install():
+    log.warn('Before install bootstrap.')
+    _fake_setuptools()
+
+
+def _under_prefix(location):
+    if 'install' not in sys.argv:
+        return True
+    args = sys.argv[sys.argv.index('install') + 1:]
+    for index, arg in enumerate(args):
+        for option in ('--root', '--prefix'):
+            if arg.startswith('%s=' % option):
+                top_dir = arg.split('root=')[-1]
+                return location.startswith(top_dir)
+            elif arg == option:
+                if len(args) > index:
+                    top_dir = args[index + 1]
+                    return location.startswith(top_dir)
+        if arg == '--user' and USER_SITE is not None:
+            return location.startswith(USER_SITE)
+    return True
+
+
+def _fake_setuptools():
+    log.warn('Scanning installed packages')
+    try:
+        import pkg_resources
+    except ImportError:
+        # we're cool
+        log.warn('Setuptools or Distribute does not seem to be installed.')
+        return
+    ws = pkg_resources.working_set
+    try:
+        setuptools_dist = ws.find(
+            pkg_resources.Requirement.parse('setuptools', replacement=False)
+            )
+    except TypeError:
+        # old distribute API
+        setuptools_dist = ws.find(
+            pkg_resources.Requirement.parse('setuptools')
+        )
+
+    if setuptools_dist is None:
+        log.warn('No setuptools distribution found')
+        return
+    # detecting if it was already faked
+    setuptools_location = setuptools_dist.location
+    log.warn('Setuptools installation detected at %s', setuptools_location)
+
+    # if --root or --preix was provided, and if
+    # setuptools is not located in them, we don't patch it
+    if not _under_prefix(setuptools_location):
+        log.warn('Not patching, --root or --prefix is installing Distribute'
+                 ' in another location')
+        return
+
+    # let's see if its an egg
+    if not setuptools_location.endswith('.egg'):
+        log.warn('Non-egg installation')
+        res = _remove_flat_installation(setuptools_location)
+        if not res:
+            return
+    else:
+        log.warn('Egg installation')
+        pkg_info = os.path.join(setuptools_location, 'EGG-INFO', 'PKG-INFO')
+        if (os.path.exists(pkg_info) and
+            _same_content(pkg_info, SETUPTOOLS_PKG_INFO)):
+            log.warn('Already patched.')
+            return
+        log.warn('Patching...')
+        # let's create a fake egg replacing setuptools one
+        res = _patch_egg_dir(setuptools_location)
+        if not res:
+            return
+    log.warn('Patched done.')
+    _relaunch()
+
+
+def _relaunch():
+    log.warn('Relaunching...')
+    # we have to relaunch the process
+    # pip marker to avoid a relaunch bug
+    _cmd = ['-c', 'install', '--single-version-externally-managed']
+    if sys.argv[:3] == _cmd:
+        sys.argv[0] = 'setup.py'
+    args = [sys.executable] + sys.argv
+    sys.exit(subprocess.call(args))
+
+
+def _extractall(self, path=".", members=None):
+    """Extract all members from the archive to the current working
+       directory and set owner, modification time and permissions on
+       directories afterwards. `path' specifies a different directory
+       to extract to. `members' is optional and must be a subset of the
+       list returned by getmembers().
+    """
+    import copy
+    import operator
+    from tarfile import ExtractError
+    directories = []
+
+    if members is None:
+        members = self
+
+    for tarinfo in members:
+        if tarinfo.isdir():
+            # Extract directories with a safe mode.
+            directories.append(tarinfo)
+            tarinfo = copy.copy(tarinfo)
+            tarinfo.mode = 448  # decimal for oct 0700
+        self.extract(tarinfo, path)
+
+    # Reverse sort directories.
+    if sys.version_info < (2, 4):
+        def sorter(dir1, dir2):
+            return cmp(dir1.name, dir2.name)
+        directories.sort(sorter)
+        directories.reverse()
+    else:
+        directories.sort(key=operator.attrgetter('name'), reverse=True)
+
+    # Set correct owner, mtime and filemode on directories.
+    for tarinfo in directories:
+        dirpath = os.path.join(path, tarinfo.name)
+        try:
+            self.chown(tarinfo, dirpath)
+            self.utime(tarinfo, dirpath)
+            self.chmod(tarinfo, dirpath)
+        except ExtractError:
+            e = sys.exc_info()[1]
+            if self.errorlevel > 1:
+                raise
+            else:
+                self._dbg(1, "tarfile: %s" % e)
+
+
+def _build_install_args(argv):
+    install_args = []
+    user_install = '--user' in argv
+    if user_install and sys.version_info < (2, 6):
+        log.warn("--user requires Python 2.6 or later")
+        raise SystemExit(1)
+    if user_install:
+        install_args.append('--user')
+    return install_args
+
+
+def main(argv, version=DEFAULT_VERSION):
+    """Install or upgrade setuptools and EasyInstall"""
+    tarball = download_setuptools()
+    _install(tarball, _build_install_args(argv))
+
+
+if __name__ == '__main__':
+    main(sys.argv[1:])

=== modified file 'tarmac_tests.sh'
--- tarmac_tests.sh	2012-07-11 12:02:57 +0000
+++ tarmac_tests.sh	2012-07-30 20:14:17 +0000
@@ -16,10 +16,7 @@
 set -e
 
 LOG_FILE=log
-VIRTUALENV_PATH=virtualenv.tarmac
-
-rm -rf $VIRTUALENV_PATH
 
 echo "Running pkgme-service-python tests in tarmac" > $LOG_FILE
-make bootstrap VIRTUALENV_DIR=$VIRTUALENV_PATH >> $LOG_FILE
-make test VIRTUALENV_DIR=$VIRTUALENV_PATH
+make bootstrap >> $LOG_FILE
+make test

=== removed file 'test-dependencies.txt'
--- test-dependencies.txt	2012-07-19 18:00:03 +0000
+++ test-dependencies.txt	1970-01-01 00:00:00 +0000
@@ -1,5 +0,0 @@
-fixtures
-mock==0.8
-testtools
-# These dependencies are for tarmac's benefit
-discover

=== added file 'versions.cfg'
--- versions.cfg	1970-01-01 00:00:00 +0000
+++ versions.cfg	2012-07-30 20:14:17 +0000
@@ -0,0 +1,24 @@
+[buildout]
+versions = versions
+
+# Update the version number here to upgrade to a new version.
+# You'll likely want to run 'make update-deps' when you have
+# changed something here.
+#
+# If you are forking a dependency then make sure you include a pointer
+# to the branch where the fork is maintained.
+#
+# Alphabetical, case-insensitive, please! :-)
+[versions]
+fixtures = 0.3.9
+discover = 0.4
+distribute = 0.6.28
+httplib2 = 0.7.4
+oauth = 1.0.1
+mock = 0.8
+piston-mini-client = 0.7.2
+testtools = 0.9.15
+z3c.recipe.scripts = 1.0.1
+# Also upgrade the zc.buildout version in the Makefile's bin/buildout section.
+zc.buildout = 1.5.1
+zc.recipe.egg = 1.3.2