Merge lp:~cjwatson/launchpad/virtualenv-pip into lp:launchpad

[Colin Watson]

Convert build system to virtualenv and pip.

Convert build system to virtualenv and pip.

This is a more or less complete rewrite of the build system to get us to the
point where we can upgrade packages with non-trivial uses of setup_requires.
It will only work on xenial (precise would be possible in principle, but
requires somewhat more complicated bootstrapping of the virtualenv that
doesn't seem worth it now).

This requires some package upgrades:

Twisted: 13.0.0-p2 -> 13.0.0post3
  Use PEP 440-compliant version.
d2to1: 0.2.10 -> 0.2.12
  Cope with modern setuptools.
distribute: 0.6.36 -> 0.7.3
  Switch to legacy wrapper for modern setuptools.
launchpad-buildd: 136 -> 157
  Normalise Python packaging to avoid buildd-slave.tac being installed at
  the root of the virtualenv.
pip: 1.4 -> 9.0.1
  Modern pip.
setuptools: 0.6c11 -> 36.4.0
  Modern setuptools.
setuptools-git: 1.0 -> 1.2
  Avoid a significant performance regression.
virtualenv-tools3: None -> 2.0.0
  Used to relocate virtualenvs.
wheel: None -> 0.29.0
  For now we're still mostly using sdists, but some eggs need to be replaced
  with corresponding wheels.
z3c.recipe.tag: 0.6 -> None
  No longer needed. The ID Makefile target was already broken so I removed
  it, and it's relatively easy to provide good-enough reimplementations of
  TAGS and tags.
zope.pagetemplate: 3.5.0-p1 -> 3.5.0.post2
  Cherry-pick dependency tweak; use PEP 440-compliant version.

Some bits of application code need minor adjustments to cope with the
changes to the Python module path.

[Colin Watson (cjwatson)]

We may need a bit of extra care to make this work properly with deployments. At the moment we do "make build_eggs" (which also runs buildout in the old world, or builds the virtualenv in the new world), sync that out everywhere, and then run "make compile" on each deployment target. That isn't quite good enough here though, because "make compile" won't rebuild the virtualenv to account for changed paths, and in any case if we can avoid building a virtualenv on each deployment target then we wouldn't have to cope with deployments being significantly slower.

I think the best approach is probably to use something like virtualenv-tools to relocate the environment. I'm not thrilled about this, but it may be the least bad answer for now. If that doesn't work, we can probably arrange for "make compile" to check whether the virtualenv's paths are correct and rebuild it if they aren't.

[Colin Watson (cjwatson)]

This seems to do the job. There's a bit of care needed because virtualenv-tools3 misses a couple of things, but it's close enough that we can fix the rest up manually.

[William Grant (wgrant)]

I've played with it locally and can't obviously break it. I'm sure it'll break the build process and production in unknown ways, but we can hardly avoid that.

review: Approve (code)

=== modified file '.bzrignore'
--- .bzrignore	2017-07-24 12:47:04 +0000
+++ .bzrignore	2017-12-19 09:48:36 +0000
@@ -33,15 +33,14 @@
 lib/canonical/launchpad/icing/build/*
 lib/canonical/launchpad/icing/combo.css
 bin
-develop-eggs
 .installed.cfg
 parts
 *.egg-info
 build
-*.egg
 dist
 ./eggs
 ./download-cache
+./env
 ./production-configs
 bzr.dev
 _trial_temp
 
=== modified file 'Makefile'
--- Makefile	2017-10-20 13:03:40 +0000
+++ Makefile	2017-12-19 09:48:36 +0000
@@ -6,9 +6,24 @@
 WD:=$(shell pwd)
 PY=$(WD)/bin/py
 PYTHONPATH:=$(WD)/lib:${PYTHONPATH}
-BUILDOUT_CFG=buildout.cfg
 VERBOSITY=-vv
 
+# virtualenv and pip fail if setlocale fails, so force a valid locale.
+VIRTUALENV := LC_ALL=C.UTF-8 virtualenv
+PIP := PYTHONPATH= LC_ALL=C.UTF-8 env/bin/pip
+# Run with "make PIP_NO_INDEX=" if you want pip to find software
+# dependencies *other* than those in our download-cache.  Once you have the
+# desired software, commit it to lp:lp-source-dependencies if it is going to
+# be reviewed/merged/deployed.
+# Although --ignore-installed is slower, we need it to avoid confusion with
+# system-installed Python packages.  If we ever manage to remove the need
+# for virtualenv --system-site-packages, then we can remove this too.
+PIP_NO_INDEX := --no-index
+PIP_INSTALL_ARGS := \
+	$(PIP_NO_INDEX) \
+	--ignore-installed \
+	--find-links=file://$(WD)/download-cache/dist/ \
+
 TESTFLAGS=-p $(VERBOSITY)
 TESTOPTS=
 
@@ -40,22 +55,32 @@
 APIDOC_TMPDIR = $(APIDOC_DIR).tmp/
 API_INDEX = $(APIDOC_DIR)/index.html
 
-# Do not add bin/buildout to this list.
-# It is impossible to get buildout to tell us all the files it would
-# build, since each egg's setup.py doesn't tell us that information.
+# It is impossible to get pip to tell us all the files it would build, since
+# each package's setup.py doesn't tell us that information.
 #
-# NB: It's important BUILDOUT_BIN only mentions things genuinely produced by
-# buildout.
-BUILDOUT_BIN = \
-    $(PY) bin/build-twisted-plugin-cache bin/bzr \
-    bin/combine-css bin/fl-build-report \
-    bin/fl-credential-ctl bin/fl-install-demo bin/fl-monitor-ctl \
-    bin/fl-record bin/fl-run-bench bin/fl-run-test bin/googletestservice \
-    bin/harness bin/iharness bin/ipy bin/jsbuild bin/lpjsmin\
-    bin/killservice bin/kill-test-services bin/retest \
-    bin/run bin/run-testapp bin/sprite-util bin/start_librarian \
-    bin/tags bin/test bin/tracereport bin/twistd \
-    bin/watch_jsbuild bin/with-xvfb
+# NB: It's important PIP_BIN only mentions things genuinely produced by pip.
+PIP_BIN = \
+    $(PY) \
+    bin/build-twisted-plugin-cache \
+    bin/combine-css \
+    bin/googletestservice \
+    bin/harness \
+    bin/iharness \
+    bin/ipy \
+    bin/jsbuild \
+    bin/lpjsmin \
+    bin/killservice \
+    bin/kill-test-services \
+    bin/retest \
+    bin/run \
+    bin/run-testapp \
+    bin/sprite-util \
+    bin/start_librarian \
+    bin/test \
+    bin/tracereport \
+    bin/twistd \
+    bin/watch_jsbuild \
+    bin/with-xvfb
 
 # DO NOT ALTER : this should just build by default
 default: inplace
@@ -131,7 +156,7 @@
 build: compile apidoc jsbuild css_combine
 
 # LP_SOURCEDEPS_PATH should point to the sourcecode directory, but we
-# want the parent directory where the download-cache and eggs directory
+# want the parent directory where the download-cache and env directories
 # are. We re-use the variable that is using for the rocketfuel-get script.
 download-cache:
 ifdef LP_SOURCEDEPS_PATH
@@ -189,55 +214,48 @@
 	build/js/lp/meta.js >/dev/null
 	utilities/check-js-deps
 
-eggs:
-	# Usually this is linked via link-external-sourcecode, but in
-	# deployment we create this ourselves.
-	mkdir eggs
-
-buildonce_eggs: $(PY)
-	find eggs -name '*.pyc' -exec rm {} \;
-
-# The download-cache dependency comes *before* eggs so that developers get the
-# warning before the eggs directory is made.  The target for the eggs
-# directory is only there for deployment convenience.
-# Note that the buildout version must be maintained here and in versions.cfg
-# to make sure that the build does not go over the network.
-#
-# buildout won't touch files that would have the same contents, but for Make's
-# sake we need them to get fresh timestamps, so we touch them after building.
-bin/buildout: download-cache eggs
-	$(SHHH) PYTHONPATH= $(PYTHON) bootstrap.py\
-		--setup-source=ez_setup.py \
-		--download-base=download-cache/dist --eggs=eggs \
-		--version=1.7.1
-	touch --no-create $@
-
 # This target is used by LOSAs to prepare a build to be pushed out to
-# destination machines.  We only want eggs: they are the expensive bits,
+# destination machines.  We only want wheels: they are the expensive bits,
 # and the other bits might run into problems like bug 575037.  This
-# target runs buildout, and then removes everything created except for
-# the eggs.
-build_eggs: $(BUILDOUT_BIN) clean_buildout
-
-# This builds bin/py and all the other bin files except bin/buildout.
-# Remove the target before calling buildout to ensure that buildout
-# updates the timestamp.
-buildout_bin: $(BUILDOUT_BIN)
-
-# buildout won't touch files that would have the same contents, but for Make's
-# sake we need them to get fresh timestamps, so we touch them after building.
+# target runs pip, and then removes everything created except for the
+# wheels.
+build_wheels: $(PIP_BIN) clean_pip
+
+# Compatibility.
+build_eggs: build_wheels
+
+# setuptools won't touch files that would have the same contents, but for
+# Make's sake we need them to get fresh timestamps, so we touch them after
+# building.
 #
 # If we listed every target on the left-hand side, a parallel make would try
 # multiple copies of this rule to build them all.  Instead, we nominally build
 # just $(PY), and everything else is implicitly updated by that.
-$(PY): bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
-	$(SHHH) PYTHONPATH= ./bin/buildout \
-                configuration:instance_name=${LPCONFIG} -c $(BUILDOUT_CFG)
+$(PY): download-cache constraints.txt setup.py
+	rm -rf env
+	mkdir -p env
+	(echo '[easy_install]'; \
+	 echo "allow_hosts = ''"; \
+	 echo 'find_links = file://$(WD)/download-cache/dist/') \
+		>env/.pydistutils.cfg
+	$(VIRTUALENV) \
+		--python=$(PYTHON) --system-site-packages --never-download \
+		--extra-search-dir=$(WD)/download-cache/dist/ \
+		env
+	ln -sfn env/bin bin
+	$(SHHH) $(PIP) install $(PIP_INSTALL_ARGS) \
+		-r pip-requirements.txt
+	$(SHHH) LPCONFIG=$(LPCONFIG) $(PIP) \
+		--cache-dir=$(WD)/download-cache/ \
+		install $(PIP_INSTALL_ARGS) \
+		-c pip-requirements.txt -c constraints.txt -e . \
+		|| { code=$$?; rm -f $@; exit $$code; }
 	touch $@
 
-$(subst $(PY),,$(BUILDOUT_BIN)): $(PY)
+$(subst $(PY),,$(PIP_BIN)): $(PY)
 
 compile: $(PY) $(VERSION_INFO)
+	${SHHH} utilities/relocate-virtualenv env
 	${SHHH} $(MAKE) -C sourcecode build PYTHON=${PYTHON} \
 	    LPCONFIG=${LPCONFIG}
 	${SHHH} bin/build-twisted-plugin-cache
@@ -345,14 +363,16 @@
 	$(RM) -r $(JS_BUILD_DIR)
 	$(RM) -r yarn/node_modules
 
-clean_buildout:
+clean_pip:
 	$(RM) -r build
 	if [ -d $(CONVOY_ROOT) ]; then $(RM) -r $(CONVOY_ROOT) ; fi
 	$(RM) -r bin
 	$(RM) -r parts
-	$(RM) -r develop-eggs
 	$(RM) .installed.cfg
 
+# Compatibility.
+clean_buildout: clean_pip
+
 clean_logs:
 	$(RM) logs/thread*.request
 
@@ -365,7 +385,7 @@
 	$(RM) -r lib/mailman
 endif
 
-lxc-clean: clean_js clean_mailman clean_buildout clean_logs
+lxc-clean: clean_js clean_mailman clean_pip clean_logs
 	# XXX: BradCrittenden 2012-05-25 bug=1004514:
 	# It is important for parallel tests inside LXC that the
 	# $(CODEHOSTING_ROOT) directory not be completely removed.
@@ -381,10 +401,7 @@
 	if test -f sourcecode/mailman/Makefile; then \
 		$(MAKE) -C sourcecode/mailman clean; \
 	fi
-	find . -path ./eggs -prune -false -o \
-		-type f \( -name '*.o' -o -name '*.so' -o -name '*.la' -o \
-	    -name '*.lo' -o -name '*.py[co]' -o -name '*.dll' \) \
-	    -print0 | xargs -r0 $(RM)
+	$(RM) -r env
 	$(RM) -r lib/subvertpy/*.so
 	$(RM) -r $(LP_BUILT_JS_ROOT)/*
 	$(RM) -r $(CODEHOSTING_ROOT)/*
@@ -466,15 +483,15 @@
 
 TAGS: compile
 	# emacs tags
-	bin/tags -e
+	ctags -R -e --languages=-JavaScript --python-kinds=-i -f $@.new \
+		$(CURDIR)/lib $(CURDIR)/env/lib/$(PYTHON)/site-packages
+	mv $@.new $@
 
 tags: compile
 	# vi tags
-	bin/tags -v
-
-ID: compile
-	# idutils ID file
-	bin/tags -i
+	ctags -R --languages=-JavaScript --python-kinds=-i -f $@.new \
+		$(CURDIR)/lib $(CURDIR)/env/lib/$(PYTHON)/site-packages
+	mv $@.new $@
 
 PYDOCTOR = pydoctor
 PYDOCTOR_OPTIONS =
@@ -485,11 +502,10 @@
 		--docformat restructuredtext --verbose-about epytext-summary \
 		$(PYDOCTOR_OPTIONS)
 
-.PHONY: apidoc build_eggs buildonce_eggs buildout_bin check \
-	check_config check_mailman clean clean_buildout clean_js	\
-	clean_logs compile css_combine debug default doc ftest_build	\
-	ftest_inplace hosted_branches jsbuild jsbuild_widget_css	\
-	launchpad.pot pagetests pull_branches pydoctor realclean	\
-	reload-apache run run-testapp runner scan_branches schema	\
-	sprite_css sprite_image start stop sync_branches TAGS tags	\
-	test_build test_inplace $(LP_JS_BUILD)
+.PHONY: apidoc build_eggs build_wheels check check_config check_mailman	\
+	clean clean_buildout clean_js clean_logs clean_pip compile	\
+	css_combine debug default doc ftest_build ftest_inplace		\
+	hosted_branches jsbuild jsbuild_widget_css launchpad.pot	\
+	pydoctor realclean reload-apache run run-testapp runner schema	\
+	sprite_css sprite_image start stop TAGS tags test_build		\
+	test_inplace $(LP_JS_BUILD)
 
=== modified file '_pythonpath.py'
--- _pythonpath.py	2017-05-08 12:39:30 +0000
+++ _pythonpath.py	2017-12-19 09:48:36 +0000
@@ -19,11 +19,12 @@
 # project root.
 top = os.path.dirname(os.path.abspath(os.path.realpath(filename)))
 
-site_dir = os.path.join(top, 'parts', 'scripts')
+env = os.path.join(top, 'env')
+stdlib_dir = os.path.join(env, 'lib', 'python%s' % sys.version[:3])
 
 if ('site' in sys.modules and
     not sys.modules['site'].__file__.startswith(
-        os.path.join(site_dir, 'site.py'))):
+        os.path.join(stdlib_dir, 'site.py'))):
     # We have the wrong site.py, so our paths are not set up correctly.
     # We blow up, with a hopefully helpful error message.
     raise RuntimeError(
@@ -31,13 +32,44 @@
         'Scripts should usually be '
         "started with Launchpad's bin/py, or with a Python invoked with "
         'the -S flag.' % (
-        sys.modules['site'].__file__, os.path.join(site_dir, 'site.py')))
-
-if site_dir not in sys.path:
-    sys.path.insert(0, site_dir)
-elif 'site' not in sys.modules:
-    # XXX 2010-05-04 gary bug 575206
-    # This one line is to support Mailman 2, which does something unexpected
-    # to set up its paths.
-    sys.path[:] = [p for p in sys.path if 'site-packages' not in p]
-import site  # sets up paths
+        sys.modules['site'].__file__, os.path.join(stdlib_dir, 'site.py')))
+
+# Ensure that the virtualenv's standard library directory is in sys.path;
+# activate_this will not put it there.
+if stdlib_dir not in sys.path and (stdlib_dir + os.sep) not in sys.path:
+    sys.path.insert(0, stdlib_dir)
+
+if not sys.executable.startswith(top + os.sep) or 'site' not in sys.modules:
+    # Activate the virtualenv.  Avoid importing lp_sitecustomize here, as
+    # activate_this imports site before it's finished setting up sys.path.
+    orig_disable_sitecustomize = os.environ.get('LP_DISABLE_SITECUSTOMIZE')
+    os.environ['LP_DISABLE_SITECUSTOMIZE'] = '1'
+    # This is a bit like env/bin/activate_this.py, but to help namespace
+    # packages work properly we change sys.prefix before importing site
+    # rather than after.
+    sys.real_prefix = sys.prefix
+    sys.prefix = env
+    os.environ['PATH'] = (
+        os.path.join(env, 'bin') + os.pathsep + os.environ.get('PATH', ''))
+    site_packages = os.path.join(
+        env, 'lib', 'python%s' % sys.version[:3], 'site-packages')
+    import site
+    site.addsitedir(site_packages)
+    if orig_disable_sitecustomize is not None:
+        os.environ['LP_DISABLE_SITECUSTOMIZE'] = orig_disable_sitecustomize
+    else:
+        del os.environ['LP_DISABLE_SITECUSTOMIZE']
+
+# Move all our own directories to the front of the path.
+new_sys_path = []
+for item in list(sys.path):
+    if item == top or item.startswith(top + os.sep):
+        new_sys_path.append(item)
+        sys.path.remove(item)
+sys.path[:0] = new_sys_path
+
+# Initialise the Launchpad environment.
+if 'LP_DISABLE_SITECUSTOMIZE' not in os.environ:
+    if 'lp_sitecustomize' not in sys.modules:
+        import lp_sitecustomize
+        lp_sitecustomize.main()
 
=== removed file 'bootstrap.py'
--- bootstrap.py	2010-08-23 14:40:19 +0000
+++ bootstrap.py	1970-01-01 00:00:00 +0000
@@ -1,258 +0,0 @@
-##############################################################################
-#
-# Copyright (c) 2006 Zope Foundation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-"""Bootstrap a buildout-based project
-
-Simply run this script in a directory containing a buildout.cfg.
-The script accepts buildout command-line options, so you can
-use the -c option to specify an alternate configuration file.
-"""
-
-import os, shutil, sys, tempfile, textwrap, urllib, urllib2, subprocess
-from optparse import OptionParser
-
-if sys.platform == 'win32':
-    def quote(c):
-        if ' ' in c:
-            return '"%s"' % c # work around spawn lamosity on windows
-        else:
-            return c
-else:
-    quote = str
-
-# See zc.buildout.easy_install._has_broken_dash_S for motivation and comments.
-stdout, stderr = subprocess.Popen(
-    [sys.executable, '-Sc',
-     'try:\n'
-     '    import ConfigParser\n'
-     'except ImportError:\n'
-     '    print 1\n'
-     'else:\n'
-     '    print 0\n'],
-    stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
-has_broken_dash_S = bool(int(stdout.strip()))
-
-# In order to be more robust in the face of system Pythons, we want to
-# run without site-packages loaded.  This is somewhat tricky, in
-# particular because Python 2.6's distutils imports site, so starting
-# with the -S flag is not sufficient.  However, we'll start with that:
-if not has_broken_dash_S and 'site' in sys.modules:
-    # We will restart with python -S.
-    args = sys.argv[:]
-    args[0:0] = [sys.executable, '-S']
-    args = map(quote, args)
-    os.execv(sys.executable, args)
-# Now we are running with -S.  We'll get the clean sys.path, import site
-# because distutils will do it later, and then reset the path and clean
-# out any namespace packages from site-packages that might have been
-# loaded by .pth files.
-clean_path = sys.path[:]
-import site
-sys.path[:] = clean_path
-for k, v in sys.modules.items():
-    if (hasattr(v, '__path__') and
-        len(v.__path__)==1 and
-        not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
-        # This is a namespace package.  Remove it.
-        sys.modules.pop(k)
-
-is_jython = sys.platform.startswith('java')
-
-setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
-distribute_source = 'http://python-distribute.org/distribute_setup.py'
-
-# parsing arguments
-def normalize_to_url(option, opt_str, value, parser):
-    if value:
-        if '://' not in value: # It doesn't smell like a URL.
-            value = 'file://%s' % (
-                urllib.pathname2url(
-                    os.path.abspath(os.path.expanduser(value))),)
-        if opt_str == '--download-base' and not value.endswith('/'):
-            # Download base needs a trailing slash to make the world happy.
-            value += '/'
-    else:
-        value = None
-    name = opt_str[2:].replace('-', '_')
-    setattr(parser.values, name, value)
-
-usage = '''\
-[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
-
-Bootstraps a buildout-based project.
-
-Simply run this script in a directory containing a buildout.cfg, using the
-Python that you want bin/buildout to use.
-
-Note that by using --setup-source and --download-base to point to
-local resources, you can keep this script from going over the network.
-'''
-
-parser = OptionParser(usage=usage)
-parser.add_option("-v", "--version", dest="version",
-                          help="use a specific zc.buildout version")
-parser.add_option("-d", "--distribute",
-                   action="store_true", dest="use_distribute", default=False,
-                   help="Use Distribute rather than Setuptools.")
-parser.add_option("--setup-source", action="callback", dest="setup_source",
-                  callback=normalize_to_url, nargs=1, type="string",
-                  help=("Specify a URL or file location for the setup file. "
-                        "If you use Setuptools, this will default to " +
-                        setuptools_source + "; if you use Distribute, this "
-                        "will default to " + distribute_source +"."))
-parser.add_option("--download-base", action="callback", dest="download_base",
-                  callback=normalize_to_url, nargs=1, type="string",
-                  help=("Specify a URL or directory for downloading "
-                        "zc.buildout and either Setuptools or Distribute. "
-                        "Defaults to PyPI."))
-parser.add_option("--eggs",
-                  help=("Specify a directory for storing eggs.  Defaults to "
-                        "a temporary directory that is deleted when the "
-                        "bootstrap script completes."))
-parser.add_option("-t", "--accept-buildout-test-releases",
-                  dest='accept_buildout_test_releases',
-                  action="store_true", default=False,
-                  help=("Normally, if you do not specify a --version, the "
-                        "bootstrap script and buildout gets the newest "
-                        "*final* versions of zc.buildout and its recipes and "
-                        "extensions for you.  If you use this flag, "
-                        "bootstrap and buildout will get the newest releases "
-                        "even if they are alphas or betas."))
-parser.add_option("-c", None, action="store", dest="config_file",
-                   help=("Specify the path to the buildout configuration "
-                         "file to be used."))
-
-options, args = parser.parse_args()
-
-# if -c was provided, we push it back into args for buildout's main function
-if options.config_file is not None:
-    args += ['-c', options.config_file]
-
-if options.eggs:
-    eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
-else:
-    eggs_dir = tempfile.mkdtemp()
-
-if options.setup_source is None:
-    if options.use_distribute:
-        options.setup_source = distribute_source
-    else:
-        options.setup_source = setuptools_source
-
-if options.accept_buildout_test_releases:
-    args.append('buildout:accept-buildout-test-releases=true')
-args.append('bootstrap')
-
-try:
-    import pkg_resources
-    import setuptools # A flag.  Sometimes pkg_resources is installed alone.
-    if not hasattr(pkg_resources, '_distribute'):
-        raise ImportError
-except ImportError:
-    ez_code = urllib2.urlopen(
-        options.setup_source).read().replace('\r\n', '\n')
-    ez = {}
-    exec ez_code in ez
-    setup_args = dict(to_dir=eggs_dir, download_delay=0)
-    if options.download_base:
-        setup_args['download_base'] = options.download_base
-    if options.use_distribute:
-        setup_args['no_fake'] = True
-    ez['use_setuptools'](**setup_args)
-    reload(sys.modules['pkg_resources'])
-    import pkg_resources
-    # This does not (always?) update the default working set.  We will
-    # do it.
-    for path in sys.path:
-        if path not in pkg_resources.working_set.entries:
-            pkg_resources.working_set.add_entry(path)
-
-cmd = [quote(sys.executable),
-       '-c',
-       quote('from setuptools.command.easy_install import main; main()'),
-       '-mqNxd',
-       quote(eggs_dir)]
-
-if not has_broken_dash_S:
-    cmd.insert(1, '-S')
-
-find_links = options.download_base
-if not find_links:
-    find_links = os.environ.get('bootstrap-testing-find-links')
-if find_links:
-    cmd.extend(['-f', quote(find_links)])
-
-if options.use_distribute:
-    setup_requirement = 'distribute'
-else:
-    setup_requirement = 'setuptools'
-ws = pkg_resources.working_set
-setup_requirement_path = ws.find(
-    pkg_resources.Requirement.parse(setup_requirement)).location
-env = dict(
-    os.environ,
-    PYTHONPATH=setup_requirement_path)
-
-requirement = 'zc.buildout'
-version = options.version
-if version is None and not options.accept_buildout_test_releases:
-    # Figure out the most recent final version of zc.buildout.
-    import setuptools.package_index
-    _final_parts = '*final-', '*final'
-    def _final_version(parsed_version):
-        for part in parsed_version:
-            if (part[:1] == '*') and (part not in _final_parts):
-                return False
-        return True
-    index = setuptools.package_index.PackageIndex(
-        search_path=[setup_requirement_path])
-    if find_links:
-        index.add_find_links((find_links,))
-    req = pkg_resources.Requirement.parse(requirement)
-    if index.obtain(req) is not None:
-        best = []
-        bestv = None
-        for dist in index[req.project_name]:
-            distv = dist.parsed_version
-            if _final_version(distv):
-                if bestv is None or distv > bestv:
-                    best = [dist]
-                    bestv = distv
-                elif distv == bestv:
-                    best.append(dist)
-        if best:
-            best.sort()
-            version = best[-1].version
-if version:
-    requirement = '=='.join((requirement, version))
-cmd.append(requirement)
-
-if is_jython:
-    import subprocess
-    exitcode = subprocess.Popen(cmd, env=env).wait()
-else: # Windows prefers this, apparently; otherwise we would prefer subprocess
-    exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
-if exitcode != 0:
-    sys.stdout.flush()
-    sys.stderr.flush()
-    print ("An error occurred when trying to install zc.buildout. "
-           "Look above this message for any errors that "
-           "were output by easy_install.")
-    sys.exit(exitcode)
-
-ws.add_entry(eggs_dir)
-ws.require(requirement)
-import zc.buildout.buildout
-zc.buildout.buildout.main(args)
-if not options.eggs: # clean up temporary egg directory
-    shutil.rmtree(eggs_dir)
 
=== removed file 'buildout.cfg'
--- buildout.cfg	2017-09-23 03:13:41 +0000
+++ buildout.cfg	1970-01-01 00:00:00 +0000
@@ -1,53 +0,0 @@
-# Copyright 2009-2017 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-[buildout]
-parts =
-    scripts
-    tags
-unzip = true
-eggs-directory = eggs
-download-cache = download-cache
-relative-paths = true
-
-# Disable this option temporarily if you want buildout to find software
-# dependencies *other* than those in our download-cache.  Once you have the
-# desired software, reenable this option (and check in the new software to
-# lp:lp-source-dependencies if this is going to be reviewed/merged/deployed.)
-install-from-cache = true
-
-# This also will need to be temporarily disabled or changed for package
-# upgrades.  Newly-added packages should also add their desired version number
-# to versions.cfg.
-extends = versions.cfg
-
-allow-picked-versions = false
-
-prefer-final = true
-
-develop = .
-
-[configuration]
-instance_name = development
-
-[scripts]
-recipe = z3c.recipe.scripts
-eggs = lp
-    celery
-    jsautobuild
-    lazr.jobrunner
-    lpjsmin
-    pyinotify
-    zc.zservertracelog
-include-site-packages = true
-allowed-eggs-from-site-packages =
-interpreter = py
-# Note that any indentation is lost in initialization blocks.
-initialization =
-    # See buildout.cfg, [scripts] section, "initialization" key.
-    from lp_sitecustomize import main
-    main('${configuration:instance_name}') # Initializes LP environment.
-
-[tags]
-recipe = z3c.recipe.tag:tags
-eggs = lp
 
=== removed file 'bzr-trunk-buildout.cfg'
--- bzr-trunk-buildout.cfg	2009-07-24 03:39:09 +0000
+++ bzr-trunk-buildout.cfg	1970-01-01 00:00:00 +0000
@@ -1,12 +0,0 @@
-# A custom buildout.cfg file for testing against a newer bzr.
-# Checkout or symlink the new bzr as 'bzr.dev' in the root of the
-# launchpad tree then run 'make BUILDOUT_CFG=bzr-trunk-buildout.cfg'.
-
-[buildout]
-extends = buildout.cfg
-
-develop = .
- bzr.dev
-
-[versions]
-bzr =
 
=== modified file 'bzrplugins/lpserve/test_lpserve.py'
--- bzrplugins/lpserve/test_lpserve.py	2014-01-30 09:58:18 +0000
+++ bzrplugins/lpserve/test_lpserve.py	2017-12-19 09:48:36 +0000
@@ -1,4 +1,4 @@
-# Copyright 2010-2011 Canonical Ltd.  This software is licensed under the
+# Copyright 2010-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 import errno
@@ -24,7 +24,6 @@
     get_bzr_path,
     get_BZR_PLUGIN_PATH_for_subprocess,
     )
-from lp.services.config import config
 from lp.testing.fakemethod import FakeMethod
 
 
@@ -319,10 +318,6 @@
     same as the bzrlib.tests.TestCase version.
     """
 
-    def get_python_path(self):
-        """Return the path to the Python interpreter."""
-        return '%s/bin/py' % config.root
-
     def start_bzr_subprocess(self, process_args, env_changes=None,
                              working_dir=None):
         """Start bzr in a subprocess for testing.
@@ -353,15 +348,12 @@
             cwd = osutils.getcwd()
             os.chdir(working_dir)
 
-        # LAUNCHPAD: Because of buildout, we need to get a custom Python
-        # binary, not sys.executable.
-        python_path = self.get_python_path()
         # LAUNCHPAD: We can't use self.get_bzr_path(), since it'll find
-        # lib/bzrlib, rather than the path to sourcecode/bzr/bzr.
+        # lib/bzrlib, rather than the path to bin/bzr.
         bzr_path = get_bzr_path()
         try:
             cleanup_environment()
-            command = [python_path, bzr_path]
+            command = [bzr_path]
             command.extend(process_args)
             process = self._popen(
                 command, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
 
=== added file 'constraints.txt'
--- constraints.txt	1970-01-01 00:00:00 +0000
+++ constraints.txt	2017-12-19 09:48:36 +0000
@@ -0,0 +1,356 @@
+# ztk-versions.cfg from ZTK 1.1.6, with some upgrades
+# ---------------------------------------------------
+
+zope.annotation==3.6.0
+zope.applicationcontrol==3.5.5
+zope.authentication==3.7.1
+zope.broken==3.6.0
+zope.browser==1.3
+zope.browsermenu==3.9.1
+zope.browserpage==3.12.2
+zope.browserresource==3.12.0
+zope.cachedescriptors==3.5.1
+zope.catalog==3.8.2
+#zope.component==3.10.0
+# Tell pip about extras to work around https://github.com/pypa/pip/issues/3046
+# XXX cjwatson 2017-09-03: This should be fixed in pip 9.0.0, but apparently
+# isn't.
+# Upgrade from ZTK 1.1.6 for ZCML registration performance.
+zope.component[hook,zcml]==3.11.0
+zope.componentvocabulary==1.0.1
+zope.configuration==3.7.4
+zope.container==3.12.0
+zope.contentprovider==3.7.2
+zope.contenttype==3.5.5
+zope.copy==3.5.0
+zope.copypastemove==3.8.0
+zope.datetime==3.4.1
+zope.deferredimport==3.5.3
+zope.deprecation==3.4.1
+zope.dottedname==3.4.6
+zope.dublincore==3.8.2
+zope.error==3.7.4
+zope.event==3.5.2
+zope.exceptions==3.6.2
+zope.filerepresentation==3.6.1
+zope.formlib==4.0.6
+zope.hookable==3.4.1
+zope.i18n==3.7.4
+zope.i18nmessageid==3.5.3
+zope.index==3.6.4
+#zope.interface==3.7.0
+# Upgrade from ZTK 1.1.6 for ZCML registration performance.
+zope.interface==4.4.3
+zope.intid==3.7.2
+zope.keyreference==3.6.4
+zope.lifecycleevent==3.6.2
+zope.location==3.9.1
+zope.login==1.0.0
+zope.mimetype==1.3.1
+zope.minmax==1.1.2
+#zope.pagetemplate==3.5.2
+# Build of lp:~wallyworld/zope.pagetemplate/fix-isinstance
+# p1    This version adds a small change to the traversal logic so that the
+#       optimisation which applies if the object is a dict also works for
+#       subclasses of dict. This patch has been merged in 4.2.0, so we can
+#       drop it when we upgrade.
+# post2 Cherry-pick zope.security [untrustedpython] dependency from 3.5.1.
+#       Use PEP 440-compliant version.
+zope.pagetemplate==3.5.0.post2
+zope.password==3.6.1
+zope.pluggableauth==1.2
+zope.principalannotation==3.6.1
+zope.principalregistry==3.7.1
+zope.processlifetime==1.0
+zope.proxy==3.6.1
+zope.ptresource==3.9.0
+zope.publisher==3.12.6
+zope.ramcache==1.0
+zope.schema==3.7.1
+#zope.security==3.8.3
+# Tell pip about extras to work around https://github.com/pypa/pip/issues/3046
+# XXX cjwatson 2017-09-03: This should be fixed in pip 9.0.0, but apparently
+# isn't.
+zope.security[untrustedpython]==3.8.3
+zope.securitypolicy==3.7.0
+zope.sendmail==3.7.5
+zope.sequencesort==3.4.0
+zope.server==3.8.6
+#zope.session==3.9.5
+# XXX: downgraded to avoid 3.9.2 cookie calculation changes
+zope.session==3.9.1
+zope.site==3.9.2
+zope.size==3.4.1
+zope.structuredtext==3.5.1
+zope.tal==3.5.2
+zope.tales==3.5.3
+#zope.testing==3.10.3
+# p1 Build of lp:~mars/zope.testing/3.9.4-p1.  Fixes bugs 570380 and 587886.
+# p2 With patch for thread leaks to make them skips, fixes windmill errors
+#    with 'new threads' in hudson/ec2 builds.
+# p3 And always tear down layers, because thats the Right Thing To Do.
+# p4 fixes --subunit --list to really just list the tests.
+# p5 Build of lp:~launchpad/zope.testing/3.9.4-p5. Fixes bug #609986.
+# p6 reinstates fix from p4.  Build of lp:~launchpad/zope.testing/3.9.4-fork
+#    revision 26.
+# p7 was unused
+# p8 redirects stdout and stderr to a black hole device when --subunit is used
+# p9 adds the redirection of __stderr__ to a black hole device
+# p10 changed the test reporting to use test.id() rather than
+#     str(test) since only the id is unique.
+# p11 reverts p9.
+# p12 reverts p11, restoring p9.
+# p13 Add a new --require-unique flag to the testrunner. When set,
+#     this will cause the testrunner to check all tests IDs to ensure they
+#     haven't been loaded before. If it encounters a duplicate, it will
+#     raise an error and quit.
+# p14 Adds test data written to stderr and stdout into the subunit output.
+# p15 Fixed internal tests.
+# p16 Adds support for skips in Python 2.7.
+# p17 Fixes skip support for Python 2.6.
+# To build (use Python 2.6) run "python bootstrap.py; ./bin/buildout".  Then to
+#    build the distribution run "bin/buildout setup . sdist"
+# Make sure you have subunit installed.
+zope.testing==3.9.4-p17
+zope.testrunner==4.0.4
+zope.traversing==3.14.0
+zope.viewlet==3.7.2
+
+# Deprecating
+
+# Dependencies
+#distribute==0.6.36
+distribute==0.7.3
+docutils==0.7
+Jinja2==2.5.5
+mechanize==0.2.5
+Paste==1.7.5.1
+PasteDeploy==1.3.4
+PasteScript==1.7.5
+py==1.4.8
+#Pygments==1.4
+Pygments==1.6
+python-gettext==1.0
+#python-subunit==0.0.7
+python-subunit==0.0.8beta
+#pytz==2014.10
+pytz==2017.2
+RestrictedPython==3.6.0
+#setuptools==0.6c11
+setuptools==36.4.0
+Sphinx==1.0.8
+#testtools==0.9.12
+testtools==0.9.30
+transaction==1.1.1
+z3c.recipe.sphinxdoc==0.0.8
+zc.buildout==1.7.1
+zc.lockfile==1.0.2
+#ZConfig==2.8.0
+ZConfig==2.9.1dev-20110728
+zc.recipe.egg==1.3.2
+zc.recipe.testrunner==1.4.0
+zc.resourcelibrary==1.3.4
+zdaemon==2.0.7
+ZODB3==3.10.5
+zope.mkzeoinstance==3.9.5
+
+# toolchain
+#argparse==1.1
+argparse==1.2.1
+coverage==3.5.2
+lxml==2.2.8
+mr.developer==1.25
+nose==1.1.2
+tl.eggdeps==0.4
+z3c.checkversions==0.4.1
+z3c.recipe.compattest==0.13.1
+z3c.recipe.depgraph==0.5
+z3c.recipe.scripts==1.0.1
+zope.kgs==1.2.0
+
+# zopeapp-versions.cfg from ZTK 1.1.6, with some upgrades
+# -------------------------------------------------------
+
+# ZopeApp
+zc.sourcefactory==0.7.0
+zope.app.applicationcontrol==3.5.10
+zope.app.appsetup==3.15.0
+zope.app.debug==3.4.1
+zope.app.http==3.9.0
+zope.app.publication==3.12.0
+zope.app.wsgi==3.10.0
+zope.testbrowser==3.10.4
+
+# Deprecated
+roman==1.4.0
+#wsgi-intercept==0.4
+# Upgrade from ZTK 1.1.5 to intercept lazr.restfulclient.
+wsgi-intercept==0.5.1
+zope.app.authentication==3.9.0
+zope.app.basicskin==3.5.1
+zope.app.broken==3.6.0
+zope.app.component==3.9.3
+zope.app.container==3.9.2
+zope.app.content==3.5.1
+zope.app.dependable==3.5.1
+zope.app.error==3.5.3
+zope.app.exception==3.6.3
+zope.app.folder==3.5.2
+zope.app.form==4.0.2
+zope.app.generations==3.7.1
+zope.app.i18n==3.6.4
+zope.app.locales==3.6.2
+zope.app.localpermission==3.7.2
+zope.app.pagetemplate==3.11.2
+zope.app.principalannotation==3.7.0
+zope.app.publisher==3.10.2
+zope.app.renderer==3.5.1
+zope.app.rotterdam==3.5.3
+zope.app.schema==3.5.0
+zope.app.security==3.7.5
+zope.app.testing==3.8.1
+zope.app.zcmlfiles==3.7.1
+zope.app.zopeappgenerations==3.6.1
+zope.generations==3.7.1
+
+# Launchpad
+# ---------
+
+# Alphabetical, case-insensitive, please! :-)
+
+# lp:~launchpad/ampoule/lp
+# post1 Don't add a process back to the ready set if it received an error
+# such as a timeout.
+ampoule==0.2.0.post1
+amqp==1.4.9
+amqplib==1.0.2
+anyjson==0.3.3
+auditor==0.0.3
+auditorclient==0.0.4
+auditorfixture==0.0.7
+backports.lzma==0.0.3
+BeautifulSoup==3.2.1
+billiard==3.3.0.20
+bson==0.3.3
+bzr==2.6.0.lp.2
+celery==3.1.18
+Chameleon==2.11
+cssselect==0.9.1
+cssutils==0.9.10
+d2to1==0.2.12
+Django==1.4
+dkimpy==0.5.4
+# Required by dkimpy
+dnspython==1.10.0
+elementtree==1.2.6-20050316
+epydoc==3.0.1
+extras==0.0.3
+feedparser==4.1
+feedvalidator==0.0.0DEV-r1049
+fixtures==0.3.9
+FormEncode==1.2.4
+grokcore.component==1.6
+html5browser==0.0.9
+httmock==1.2.3
+httplib2==0.8
+importlib==1.0.2
+ipython==0.13.2
+iso8601==0.1.4
+jsautobuild==0.2
+keyring==0.6.2
+kombu==3.0.30
+launchpad-buildd==157
+launchpadlib==1.10.5
+lazr.authentication==0.1.1
+lazr.batchnavigator==1.2.11
+lazr.config==2.2.1
+lazr.delegates==2.0.4
+lazr.enum==1.1.3
+lazr.jobrunner==0.13
+lazr.lifecycle==1.1
+lazr.restful==0.20.0
+lazr.restfulclient==0.13.2
+lazr.smtptest==1.3
+lazr.sshserver==0.1.3
+lazr.testing==0.1.1
+lazr.uri==1.0.3
+libnacl==1.3.6
+lpjsmin==0.5
+manuel==1.7.2
+Markdown==2.3.1
+martian==0.11
+meliae==0.2.0.final.0
+mock==1.0.1
+mocker==1.1.1
+oauth==1.0
+oops==0.0.13
+oops-amqp==0.0.8b1
+oops-datedir-repo==0.0.23
+oops-timeline==0.0.1
+oops-twisted==0.0.7
+oops-wsgi==0.0.8
+ordereddict==1.1
+oslo.config==1.1.1
+paramiko==1.7.7.2
+pbr==0.5.20
+pgbouncer==0.0.8
+prettytable==0.7.2
+psycopg2==2.6.1
+pyasn1==0.1.6
+pycrypto==2.6
+pygpgme==0.2
+pyinotify==0.9.4
+pymacaroons==0.9.2
+pyOpenSSL==0.13
+pystache==0.5.3
+python-dateutil==1.5
+python-debian==0.1.23
+python-keystoneclient==0.3.1
+python-memcached==1.58
+python-mimeparse==0.1.4
+# XXX: deryck 2012-08-10
+# See lp:~deryck/python-openid/python-openid-fix1034376 which
+# reapplied a patch from wgrant to get codehosting going again.
+python-openid==2.2.5-fix1034376
+python-swiftclient==1.5.0
+PyYAML==3.10
+rabbitfixture==0.3.6
+requests==2.7.0
+requests-toolbelt==0.6.2
+setproctitle==1.1.7
+setuptools-git==1.2
+simplejson==3.8.2
+SimpleTAL==4.3
+six==1.9.0
+soupmatchers==0.2
+# lp:~launchpad-committers/storm/with-without-datetime
+storm==0.19.0.99-lpwithnodatetime-r408
+subprocess32==3.2.6
+subvertpy==0.9.1
+testresources==0.2.7
+testscenarios==0.4
+timeline==0.0.3
+# Build of lp:~canonical-launchpad-branches/twisted:lp-backport.
+# p1    Support diffie-hellman-group14-sha1 key exchange in conch.ssh.
+# p2    Add diffie-hellman-group-exchange-sha256 to twisted.conch.ssh.
+#       Add support in twisted.conch.ssh for hmac-sha2-256 and hmac-sha2-512.
+# post3 Use PEP 440-compliant version.
+Twisted==13.0.0post3
+txAMQP==0.6.2
+txfixtures==0.1.4
+txlongpoll==0.2.12
+txlongpollfixture==0.1.3
+txpkgupload==0.2
+unittest2==0.5.1
+van.testing==3.0.0
+virtualenv-tools3==2.0.0
+wadllib==1.3.2
+wheel==0.29.0
+wsgiref==0.1.2
+z3c.pt==2.2.3
+z3c.ptcompat==0.5.7
+zc.zservertracelog==1.3.2
+# Not in ZTK 1.1.5
+zope.app.server==3.6.0
+# Not in ZTK 1.1.5 (extracted from zope.app.schema)
+zope.vocabularyregistry==1.0.0
 
=== modified file 'doc/index.txt'
--- doc/index.txt	2011-03-31 16:41:05 +0000
+++ doc/index.txt	2017-12-19 09:48:36 +0000
@@ -27,7 +27,7 @@
 .. toctree::
    :maxdepth: 1
 
-   buildout
+   pip
 
 Possibly out-of-date
 --------------------
 
=== renamed file 'doc/buildout.txt' => 'doc/pip.txt'
--- doc/buildout.txt	2017-11-18 09:49:07 +0000
+++ doc/pip.txt	2017-12-19 09:48:36 +0000
@@ -1,9 +1,8 @@
-Launchpad Buildout
-******************
+Launchpad pip integration
+*************************
 
-Launchpad uses the buildout_ (or "zc.buildout") build system.
-Buildout's biggest strength is managing Python packages.  That is also
-our focus for it.
+Launchpad uses the pip_ build system for managing Python packages (replacing
+`zc.buildout`, which we used for many years).
 
 We have at least two other ways of managing dependencies.  Apt
 manages our Python language installation, as well as many of our
@@ -12,151 +11,15 @@
 supposed to only contain dependencies that are not standard Python
 packages.  bzr plugins and Javascript libraries are existing examples.
 
-If you are not interested in our `Motivations`_  or in an `Introduction to
-zc.buildout`_, all developers will at least want to read the very brief
-sections about the `Set Up`_ and the `Everyday Usage`_.
+All developers will at least want to read the very brief sections about the
+`Set Up`_ and the `Everyday Usage`_.
 
 Developers who manage source dependencies probably should read the general
 information about `Managing Dependencies and Scripts`_, but will also find
 detailed instructions to `Add a Package`_, to `Upgrade a Package`_, to `Add a
 Script`_, and to `Work with Unreleased or Forked Packages`_.
 
-.. _buildout: http://www.buildout.org/
-
-===========
-Motivations
-===========
-
-These motivations are labeled as "[INTERNAL]" or "[EXTERNAL]" to indicate
-whether it is pertinent for internal dependencies, which we on the Launchpad
-team create and release ourselves; or external dependencies, which other
-parties, in and out of Canonical, create and release.
-
-* We want more careful specification of our dependencies across branches.
-  [INTERNAL] [EXTERNAL]
-
-  This is a real concern pertinent both for our "trunks" (devel,
-  stable, db-devel, db-stable) and for our development boxes. For
-  instance, before incorporating buildout, in our trunks, when we want
-  to update a dependency, we needed to make sure that *all* the
-  current Launchpad trunks work with the dependency initially; then
-  submit a new Launchpad branch that uses the change dependency.  A
-  mistake can even potentially break one or both of the db-* trunks,
-  since PQM only tests against one branch (usually devel), and
-  sourcecode changes affect all branches at once.  For simplicity,
-  speed, and safety, we want to be able to submit a single branch that
-  incorporates the source dependencies and the associated changes at
-  once.
-
-  This is also true, if less critical and easier to work around, on developer
-  boxes.  Without care, changes to sourcecode when working on dependencies will
-  affect all a developer's branches, polluting test results with false
-  negatives or false positives.
-
-* We want to default to using released versions of our software dependencies.
-  [EXTERNAL]
-
-  A significant number of projects do not always have a pristine trunk, and
-  many also spend extra effort on polish, bug fixes, and compatibility before a
-  release.  If we do not desperately need a new feature on trunk, using a
-  release is generally regarded as a safer, better practice. Our earlier usage
-  of bzr branches of the development trunks does not encourage this practice.
-
-* We want to be encouraged to make the effort to interact with upstream
-  projects to have our patches integrated. [EXTERNAL]
-
-  Interacting and negotiating with upstream is undeniably more
-  time-consuming than our previous practice of maintaining local bzr
-  branches with our patches, especially short-term.  But our previous
-  use of bzr branches is not good open-source community behaviour--an
-  ironic characteristic for a project like Launchpad. It also can
-  cause problems down the road, for instance, if the patch becomes
-  stale and we want to migrate to new releases.
-
-* We want to be protected from changes and differences in our operating system.
-  [INTERNAL] [EXTERNAL]
-
-  This is a concern both over time and across different Launchpad environments.
-
-  First, our operating system, Ubuntu, is driven by many needs and
-  goals.  Launchpad is among them, but generally Launchpad serves
-  Ubuntu, not the reverse.  For instance, Jaunty dropped Launchpad's
-  Python version.  The Ubuntu developers had good reason--Python 2.4
-  has not been supported by the Python developers for some time--but
-  it caused a significant inconvenience to the Launchpad
-  team. Managing our dependencies, particularly the Python library
-  dependencies, can help alleviate these problems.
-
-  Second, Launchpad developers run a significantly different version of the
-  operating system than that run in production. Maintaining our Python library
-  dependencies ourselves can also help alleviate these concerns.
-
-* We want to be able to easily use standard packages of our primary
-  programming language, Python. [EXTERNAL]
-
-  Our Python library dependencies are distributed for many operating systems--
-  Windows, Mac, and other flavors of Linux--in a unified location and format:
-  PyPI, using distutils.  Using Python library dependencies in their standard
-  distributions makes it easier for us to reuse code.
-
-* We want to be encouraged to release Python packages of our open-source
-  code. [INTERNAL]
-
-  We are beginning to realize our aspirations of abstracting and releasing some
-  of our code.  Much of that code is in Python.  Consuming standard Python
-  packages encourages us to follow good practice in releasing our own Python
-  packages.  Dogfooding can help keep us honest, and good official releases can
-  help us support and encourage a community of users.
-
-Meanwhile, we want to be able to keep certain aspects of the legacy story.
-
-* We need deployment to not need network access.
-
-* We need to be able to use custom releases of our Python dependencies, if
-  absolutely necessary.
-
-* We would like to be able to follow security releases in our operating system.
-
-We will be able to support the first two of these aspects entirely.
-
-As to the third, we will continue to follow operating system security releases
-for most or all of the dependencies that are not Python packages--a very
-similar situation to the one before Buildout integration.
-
-===========================
-Introduction to zc.buildout
-===========================
-
-Buildout is a relatively simple system that increases in complexity as it is
-extended via "recipes".  It works on top of distutils_ and setuptools_.  It
-uses declarative ini-style files to direct its work.  The `Buildout site`_
-points to a variety of documents describing and documenting its use.
-
-For Launchpad, Buildout is hidden behind a Makefile as of this writing.  If
-the Makefile is removed, developers will typically run ``bootstrap.py`` in the
-branch, and then run ``bin/buildout``.  After that, the ``bin`` directory will
-have the commands to start, stop, test, and debug the software.  If you are
-interested in example direct usage of Buildout, you may want to read `the
-"Hacking" document in the Launchpad wiki`_ that describes the usage patterns in
-``lazr.*`` packages.
-
-.. _`the "Hacking" document in the Launchpad wiki`: https://dev.launchpad.net/Hacking
-
-Launchpad's Buildout usage is roughly of medium complexity.  It is more
-complex than that needed by a package with few dependencies and simple usage
-(see lazr.uri, for instance), but less complex than that of other large
-applications that use the buildout system.  More complexity can come by
-building more non-Python tools and by having multiple configuration variations,
-for instance.
-
-The documentation below will focus on using Launchpad's buildout.  See the
-links given above for a more thorough general review.
-
-.. _distutils: http://docs.python.org/distutils/index.html
-
-.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
-
-.. _`Buildout site`: http://www.buildout.org/
+.. _pip: https://pip.pypa.io/
 
 ======
 Set Up
@@ -192,21 +55,20 @@
 ======================
 
 If you typically use ``rocketfuel-get``, and you don't change source
-dependencies, you should not have any further changes, except that ``bin/test``
-has replaced ``test.py``.  ``rocketfuel-branch`` and
-``link-external-dependencies`` will Do the Right Thing.
+dependencies, then you don't need to know any more.  ``rocketfuel-branch``
+and ``link-external-dependencies`` will Do the Right Thing.
 
 Manual
 ======
 
 If you don't use the rocketfuel scripts, you will still use
-``link-external-dependencies`` as before.  When a buildout complains that it
+``link-external-dependencies`` as before.  When ``pip`` complains that it
 cannot find a version of a dependency, do the following, from within the
 branch::
 
     git -C download-cache pull
 
-After this, retry your make (or run ``bin/buildout`` from the branch).
+After this, retry your ``make``.
 
 That's it for everyday usage.
 
@@ -220,120 +82,42 @@
 
 First let's talk a little about the anatomy of what we have set up.  To be
 clear, much of this is based on our own decisions of what to do.  If you see
-something problematic, bring it up with the Foundations team.  Maybe together
-we can come up with another approach that meets our needs better.
-
-If you saw the top-level Launchpad directory before we started using Buildout,
-you might notice seven new items in the checkout.
-
-``bootstrap.py``
-    This is the standard bootstrapping file provided by the Buildout
-    distribution.  As of this writing, the Makefile uses this file, and you do
-    not have to modify it or call it directly.
-
-    Without a Makefile, the first step of developing a source tree that uses
-    Buildout is to run this file in the top level directory with the Python
-    executable that you wish to use for the source tree.  It creates a ``bin``
-    directory, if it does not already exist, and puts a ``buildout`` executable
-    there.  The next step is to run ``bin/buildout``, which, unless you supply
-    a ``-c`` option to specify a configuration file, will look for a
-    buildout.cfg file by default to discover what to do.
-
-    Again, as of this writing, the Makefile uses this file, and it does not
-    need to be modified, so you need not concern yourself with it further at
-    this time.
-
-``ez_setup.py``
-    This is another standard file from another project.  In this case, it is
-    the file provided by the setuptools project to install setuptools.  It is
-    used by the ``setup.py`` file, described below.  It does not need to be
-    modified or called directly.
+something problematic, bring it up with other Launchpad developers.  Maybe
+together we can come up with another approach that meets our needs better.
+
+These are the items in the top-level Launchpad directory associated with
+pip:
 
 ``setup.py``
     This is the file that uses ``distutils``, extended by ``setuptools``, to
     specify direct dependencies, scripts, and other elements of the local
     source tree.
 
-    Buildout uses it, but the reverse is not true: ``setup.py`` does not know
-    about Buildout.  This means that packages that use Buildout for development
-    do not have to require it when they are being installed in other software
-    as a dependency.
+    pip uses it, but the reverse is not true: ``setup.py`` does not know
+    about pip.
 
     Describing this file in full is well beyond the scope of this document.  We
     will give recipes for modifying it for certain tasks below. For more
     information beyond these recipes, see the setuptools and distutils
     documentation.
 
-``buildout.cfg``
-    This is the default configuration file that ``bin/buildout`` will look to
-    for instructions.
-
-    Describing it in full is well beyond the scope of this document.  However,
-    we will give an overview here.
-
-    Configuration files for Buildout are comprised of sections with key-value
-    pairs.
-
-    The key-value pairs are separated with new lines, when the subsequent line
-    is not indented.  The key and value are each separated with an equals sign.
-
-    ::
-
-        foo = bar
-        baz = bing
-              bah
-              boo
-        sha = zam
-
-    That example shows three keys, 'foo', 'baz', and 'sha'.  The 'baz' key has
-    a string with two new lines (which might be interpreted one several ways,
-    as defined for that key).
-
-    The ``[buildout]`` section is the starting point for Buildout to determine
-    what to do.  It looks for an ``extends`` key to find any additional files
-    to merge in; we use this for ``versions.cfg``, discussed below.
-
-    In addition to general configuration and initialization such as this, it
-    looks in the ``develop`` key to find source trees to develop as part of the
-    buildout. In the standard Launchpad configuration, we develop only
-    Launchpad itself (the current directory, or '.').  This means that the
-    local ``setup.py`` will be run.  If you want to develop Launchpad while you
-    develop another dependency, you can link another source tree in, and
-    specify an additional ``develop`` directory in another line::
-
-        [buildout]
-        develop = .
-                  lazr_uri_branch
-
-    See `Developing a Dependent Library In Parallel`_ for more on this.
-
-    The other basic key in the ``[buildout]`` section that we'll highlight here
-    is ``parts``.  This key identifies the other sections in buildout.cfg that
-    will be processed.  A section that is not identified in the ``[buildout]``
-    sections ``parts`` key will usually be ignored (unless chosen for another
-    role by another key elsewhere).
-
-    Sections other than ``[buildout]`` that are specified as parts always must
-    specify a ``recipe``: an identifier that determines what code should
-    process that section.  You'll see a variety of recipes in Launchpad's
-    buildout.cfg, including ``zc.recipe.egg`` and others.
-
-``versions.cfg``
-    As mentioned above, ``buildout.cfg`` extends ``versions.cfg``  by
-    specifying it in the ``extends`` key of the ``[buildout]`` section.
-    Versions.cfg specifies the precise versions of the dependencies we use.
-    This means that we can have several versions of a dependency available
-    locally, but we only build the precise one we specify.  We give an
-    example of its use below.  To read about the mechanism used, see the
-    zc.buildout documentation of the ``versions`` option in the ``[buildout]``
-    section.
-
-``eggs``
-    The ``eggs`` directory holds the eggs built from the downloaded
-    distributions. Unless you set it up differently yourself, this directory is
-    shared by all your branches. This directory is local to your system--we do
-    not manage it in a branch. One reason for this is that eggs are often
-    platform-specific.
+``pip-requirements.txt``
+    This is a `requirements file`_ used to upgrade pip itself to a
+    reasonably recent version.  We use this so that we aren't confined to
+    features of pip supported by the version supplied by the operating
+    system.
+
+``constraints.txt``
+    This is a `constraints file`_ that specifies the precise versions of the
+    dependencies we use.  This means that we can have several versions of a
+    dependency available locally, but we only build the precise one we
+    specify.  We give an example of its use below.
+
+``env``
+    The ``env`` directory holds a virtualenv built from the downloaded
+    distributions.  You have one of these per branch (virtualenvs do not
+    relocate especially well).  This directory is local to your system--we
+    do not manage it in a branch.
 
 ``download-cache``
     The ``download-cache`` directory is a set of downloaded distributions--that
@@ -342,20 +126,19 @@
     download cache as a shared resource across all of our developers with a git
     branch in a Launchpad project called ``lp-source-dependencies``.
 
-    When we run buildout, Buildout reads a special key and value in the
-    ``[buildout]`` section: ``install-from-cache = true``.  This means that, by
-    default, Buildout will *not* use network access to find packages, but
-    *only* look in the download cache.  This has many advantages.
+    We run pip with the ``--no-index`` and ``--find-links`` options, which
+    cause it to *not* use network access to find packages, but *only* look
+    in the download cache.  This has many advantages.
 
     - First, it helps us keep our deployment boxes from needing network access
       out to PyPI and other download sites.
 
-    - Second, it makes the buildout much faster, because it does not have to
+    - Second, it makes the build much faster, because it does not have to
       look out on the net for every dependency.
 
-    - Third, it makes the buildout more repeatable, because we are more
-      insulated from outages at download sites such as PyPI, and poor release
-      management.
+    - Third, it makes the build more repeatable, because we are more
+      insulated from outages at download sites such as PyPI, and poor
+      release management.
 
     - Fourth, it makes our deployments more auditable, because we can tell
       exactly what we are deploying.
@@ -366,24 +149,25 @@
     The downside is that adding and upgrading packages takes a small additional
     step, as we'll see below.
 
-In addition to these seven listings, after you have run the Makefile (or
-``bin/buildout``), you will see an additional listing:
+In addition to these directory entries, after you have run the Makefile, you
+will see an additional entry:
 
 ``bin``
-    The ``bin`` directory has already been discussed many times.  After running
-    the bootstrap.py, it holds the ``buildout`` script which can be used to
-    process Buildout configuration files.  In Launchpad's case, after running
-    the buildout, it also holds many executables, including scripts to test
-    Launchpad; to run it; to run Python or IPython with Launchpad's sourcetree
-    and dependencies available; to run harness or iharness (with IPython) with
-    the sourcetree, dependencies, and database connections; or to perform
-    several other tasks.  For now, the Makefile provides aliases for many of
-    these.
-
-Now that you have an introduction to the pertinent files and directories, we'll
-move on to trying to perform tasks in the buildout.  We'll discuss adding a
-dependency, upgrading a dependency, adding a script, adding an arbitrary file,
-and working with unreleased packages.
+    The ``bin`` directory has already been discussed many times.  After
+    running the build, it also holds many executables, including scripts to
+    test Launchpad; to run it; to run Python or IPython with Launchpad's
+    sourcetree and dependencies available; to run harness or iharness (with
+    IPython) with the sourcetree, dependencies, and database connections; or
+    to perform several other tasks.  For now, the Makefile provides aliases
+    for many of these.
+
+Now that you have an introduction to the pertinent files and directories,
+we'll move on to trying to perform maintenance tasks.  We'll discuss adding
+a dependency, upgrading a dependency, adding a script, adding an arbitrary
+file, and working with unreleased packages.
+
+.. _`requirements file`: https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format
+.. _`constraints file`: https://pip.pypa.io/en/stable/user_guide/#constraints-files
 
 Add a Package
 =============
@@ -393,63 +177,81 @@
 1.  Add the new package to the ``setup.py`` file in the ``install_requires``
     list.
 
-    Generally, our policy is to only set minimum version numbers in this file,
-    or none at all.  It doesn't really matter for an application like
-    Launchpad, but it a good rule for library packages, so we follow it for
-    consistency.  Therefore, we might simply add ``'lazr.foo'`` to
-    install_requires, or ``'lazr.foo >= 1.1'`` if we know that we are depending
-    on features introduced in version 1.1 of lazr.foo.
-
-2.  [OPTIONAL] If you know it, add the desired version number to versions.cfg
-    now.
-
-    For instance, if you know you want lazr.foo 1.1.2, add this line to the
-    ``[versions]`` section of ``versions.cfg``::
-
-      lazr.foo = 1.1.2
-
-3.  [OPTIONAL] Add the desired distribution of lazr.foo 1.1.2 to the
-    ``download-cache/dist`` directory.
-
-4.  Run the following command (or your variation)::
-
-        ./bin/buildout -v buildout:install-from-cache=false | tee out.txt | grep 'Picked'
-
-    The first part (``./bin/buildout -v
-    buildout:install-from-cache=false``) will run buildout, allowing
-    it to download source packages from the Internet to
-    ``download-cache/dist``. The second part (``tee out.txt``) will
-    dump the full output to the ``out.txt`` file in case you need to
-    debug a problem. The last part (``grep 'Picked'``) will filter the
-    output so that only additional packages--dependencies of your
-    dependency--will be listed.  You need to see if it means that you
-    have dependencies, some of which may be indirect
-    dependencies. We'll see how to do this with an example.  Here's an
-    imaginary output::
-
-        Picked: ipython = 0.9.1
-        Picked: lazr.foom = 1.4
-        Picked: zope.bar = 3.6.1
-        Picked: z3c.shazam = 2.0.1
-
-    In our example, this means that these packages have been added to
-    your ``download-cache/dist`` directory. You now need to add those
-    versions to the ``versions.cfg`` file::
-
-        ipython = 0.9.1
-        lazr.foom = 1.4
-        zope.bar = 3.6.1
-        z3c.shazam = 2.0.1
-
-    Note that the output might include at least one, and possibly
-    more, spurious "Picked:" listings.  ipython, in particular, has
-    shown up in the past incorrectly--that is, when you try to add the
-    file to the download-cache/dist directory, you'll discover that it
-    is already there; and you'll see that versions.cfg already
-    specifies the version.  As long as the test passes (see step 5,
-    below), you can ignore this.
-
-5.  Test.
+    Generally, our policy is to only set minimum version numbers in this
+    file, or none at all.  It doesn't really matter for an application like
+    Launchpad, but it's a good rule for library packages, so we follow it
+    for consistency.  Therefore, we might simply add ``'lazr.foo'`` to
+    install_requires, or ``'lazr.foo>=1.1'`` if we know that we are
+    depending on features introduced in version 1.1 of lazr.foo.
+
+2.  [OPTIONAL] Add the desired package to the ``download-cache/dist``
+    directory.
+
+    You should only need to do this if the package is one that doesn't exist
+    on PyPI at all (which should be unusual).  Otherwise, it's less
+    error-prone to fetch the desired package from PyPI along with any new
+    dependencies it may have.
+
+3.  Run the following command (or your variation)::
+
+        bin/pip install --no-binary :all: lazr.foo
+
+    This will either produce some errors which you'll need to fix, or it
+    will succeed and finish with a line such as this::
+
+        Successfully installed lazr-foo-1.1.2 z3c.shazam-2.0.1 zope.bar-3.6.1
+
+    You can use `requirements specifiers`_ on this command line, so, for
+    instance, if you already know you want lazr.foo 1.1.2, you might run
+    this command instead::
+
+        bin/pip install --no-binary :all: lazr.foo==1.1.2
+
+    You may need to use the ``--ignore-installed`` option if some of the
+    packages involved happen to be installed system-wide on your build
+    system, but we want to consume them as Python packages.  You can spot
+    this situation by looking for output like this from ``pip install``:
+
+        Requirement already satisfied: netaddr>=0.7.6 in /usr/lib/python2.7/dist-packages (from python-keystoneclient==0.7.1)
+
+    If you do this, check the output very carefully, as it can easily be
+    wrong.  (This will stop being a problem once we stop using system
+    site-packages.)
+
+4.  Add the successfully-installed packages to the shared download cache for
+    future use.
+
+        bin/pip download -d download-cache/dist/ --no-deps \
+          --no-binary :all: ...
+
+    You'll need to copy the list of packages from the "Successfully
+    installed" line above, replacing the ``-`` immediately before each
+    version number with ``==`` to turn each package/version pair into a
+    requirements specifier.  So, in the case above, you would run:
+
+        bin/pip download -d download-cache/dist/ --no-deps \
+          --no-binary :all: \
+          lazr-foo==1.1.2 z3c.shazam==2.0.1 zope.bar==3.6.1
+
+    This will normally be able to fetch package files that were saved to
+    your ``pip`` cache directory (``~/.cache/pip/`` by default) by ``pip
+    install``, so it shouldn't need to download them from PyPI again.
+
+    We use ``--no-deps`` here because ``pip install`` has already done the
+    hard work of resolving dependencies and told us the result, and because
+    ``pip download`` doesn't consider what's currently installed and so is
+    liable to download too much otherwise.
+
+5.  Add the new versions to the ``constraints.txt`` file, still using the
+    requirements specifier syntax::
+
+        lazr.foo==1.1.2
+        z3c.shazam==2.0.1
+        zope.bar==3.6.1
+
+6.  Run ``make``.  If it breaks, go back to step 3.
+
+7.  Test.
 
     Note that you can tell ``ec2 test`` to include all uncommitted
     distributions from the local download-cache in its tests with the
@@ -459,7 +261,7 @@
     *not* explicitly tell ec2 test to include or ignore the
     uncommitted distributions, it will refuse to start an instance.
 
-6.  Check old versions in the download-cache.  If you are sure that
+8.  Check old versions in the download-cache.  If you are sure that
     they are not in use any more, *anywhere*, then remove them to save
     checkout space.  More explicitly, check with the LOSAs to see if
     they are in use in production and send an email to
@@ -470,7 +272,7 @@
     information by using ``git log`` on the newer (replacement)
     download-cache/dist file for the particular package.
 
-7.  Now you need to share your package changes with the rest of the
+9.  Now you need to share your package changes with the rest of the
     team.  You must do this before submitting your Launchpad branch to
     PQM or else your branch will not build properly anywhere else,
     including buildbot.  Commit the changes (``cd download-cache``,
@@ -482,32 +284,32 @@
 change in version number, or else very bad inconsistencies and
 confusion across build environments will happen.
 
+.. _`requirements specifiers`: https://pip.pypa.io/en/stable/reference/pip_install/#requirement-specifiers
+
 
 Upgrade a Package
 =================
 
 Sometimes you need to upgrade a dependency.  This may require additional
-dependency additions or upgrades.
-
-If you already know what version you want, the simplest thing to try is to
-modify versions.cfg to specify the new version and run steps 4, 5, and 6 of the
-`Add a Package`_ instructions.
-
-If you don't know what version you want, but just want to see what happens when
-you upgrade to the most recent revision, you can clear out the versions of the
-packages for upgrade and give it a try (that is, run steps 4, 5, and 6 of the
-`Add a Package`_ instructions).  Note that, when not given an explicit version
-number, our buildout is set to prefer final releases over alpha and beta
-releases.  If you want to temporarily override this behaviour, include
-``buildout:prefer-final=false`` as another argument to ``bin/buildout``.
+dependency additions or upgrades.  In general, this works just like adding a
+new package, so follow the `Add a Package`_ instructions above.
+
+If you know what version you want, specify it explicitly on the ``pip
+install`` line.
+
+If you don't know what version you want, but just want to see what happens
+when you upgrade to the most recent version, then omit the version and
+specify the ``--upgrade`` option to ``pip install``.  Note that, when not
+given an explicit version number, pip prefers final releases over alpha and
+beta releases.  If you want to temporarily override this behaviour, use the
+``--pre`` option to ``pip``.
 
 Add a Script
 ============
 
 We often need scripts that are run in a certain environment defined by Python
 dependencies, and sometimes even different Python executables.  Several of the
-scripts we have are specified using the setuptools-based spelling that the
-``zc.recipe.egg`` recipe supports.
+scripts we have are specified using setuptools.
 
 For the common case, in ``setup.py``, add a string in the ``console_scripts``
 list of the ``entry_points`` argument. Here's an example string::
@@ -518,11 +320,6 @@
 ``start_launchpad`` function in the
 ``lp.scripts.runlaunchpad`` module.
 
-See the `zc.recipe.egg documentation`_ for more information on how to add
-scripts using this method.
-
-.. _`zc.recipe.egg documentation`: http://pypi.python.org/pypi/zc.recipe.egg
-
 Work with Unreleased or Forked Packages
 =======================================
 
@@ -531,110 +328,48 @@
 access.  Similarly, we may require a patched or unreleased version of a package
 for some purpose.  Hopefully, these situations will be rare, but they do occur.
 
-While `other answers`_ are available for Buildout, our solution is to use the
-download-cache.  Basically, make a custom source distribution with a unique
-suffix in the name, and use it (and its version name) for the normal process of
-adding or updating a package, as described above.  Because the custom package
-is in the download-cache, it will be found and used.
-
-Here's an example of making a custom distribution of FeedValidator.
-
-FeedValidator is a Subversion project.  We check it out::
-
-    svn co http://feedvalidator.googlecode.com/svn/trunk/feedvalidator/src feedvalidator
-
-Next, we ``cd feedvalidator``, and, using a Python that has setuptools
-installed, we run the following command::
-
-    python setup.py egg_info -r -bDEV sdist
-
-For this example, imagine that the current revision of the repository is 1049.
-Because setuptools has built-in Subversion support, the command above will
-create a tar.gz in the ``dist`` directory named
-``feedvalidator-0.0.0DEV-r1049.tar.gz``. The -r option specifies that the
-subversion revision should be part of the package name.  The -bDEV option
-specifies that the 'DEV' suffix should be added to the version number.
-
-We could then put the tar.gz file in Launchpad's ``download-cache/dist``
-directory, specify ``feedvalidator = 0.0.0DEV-r1049`` in the ``versions.cfg``
-file, and proceed with the usual steps to update or add a new package.
-
-If you use a bzr branch, you might use the ``-d`` option instead of the ``-r``
-option when you create the distribution.  This will add the date instead of the
-revision::
-
-    python setup.py egg_info -d -bDEV sdist
-
-For instance, this might produce a distribution for the ``lazr.restful``
-project with a name like this: ``lazr.restful-0.9.1DEV-20090512.tar.gz``.
-
-See the setuptools documentation for more information about `the egg_info
-command`_.
-
-It is also possible that setup.py does not support the egg_info command and
-it is sufficient to just run the sdist command. It adds the current revision
-of the bzr branch to the version number::
-
-    python setup.py sdist
-
-For instance, this might produce a distribution for the ``testtools``
-project with a name like this: ``testtools-0.9.12-r228.tar.gz``.
+At the moment, our solution is to use the download-cache.  Basically, make a
+custom source distribution with a unique suffix in the name, and use it (and
+its version name) for the normal process of adding or updating a package, as
+described above.  Because the custom package is in the download-cache, it
+will be found and used.
+
+In general, the suffix should comply with `PEP 440`_; in the case of a
+forked package, you should use ``lp`` as a local version identifier.  For
+example, you might start by appending ``+lp1``, followed by ``+lp2`` and so
+on for further revisions.
 
 .. _FeedValidator: http://feedvalidator.org/
 
-.. _`other answers`: http://pypi.python.org/pypi/zc.buildoutsftp
-
-.. _`the egg_info command`: http://peak.telecommunity.com/DevCenter/setuptools#tagging-and-daily-build-or-snapshot-releases
+.. _`PEP 440`: https://www.python.org/dev/peps/pep-0440/
 
 Developing a Dependent Library In Parallel
 ==========================================
 
-Sometimes you need to iterate on change to a library used by Launchpad
-that is managed by buildout as an egg. You could just edit what it is in
-the ``eggs`` directory, but it is harder to produce a patch while doing
-this. You could instead grab a branch of the libarary and produce an egg
-everytime you make a change and make buildout use the new egg, but this is
-slow.
-
-buildout defaults to only using the current directory as code that will
-be used without creating a distribution. We can arrange for it to use other
-paths as well, so we can use a checkout of any code we like, with changes
-being picked up instantly without us having to create a distribution.
-
-To do this add the extra paths to the ``develop`` key in the ``[buildout]``
-section of ``buildout.cfg``::
-
-    [buildout]
-    develop = .
-              path/to/branch
-
-and re-run ``make``.
+Sometimes you need to iterate on change to a library used by Launchpad that
+is managed by pip.  You could just edit what is in the ``env`` directory,
+but it is harder to produce a patch while doing this.  You could instead
+grab a branch of the library and produce an sdist every time you make a
+change and make pip use the new sdist, but this is slow.
+
+Instead, we can use "editable mode" so that changes are picked up instantly
+without us having to create a distribution.  For example:
+
+        bin/pip install -e /path/to/branch
 
 Now any changes you make in that path will be picked up, and you are free
 to make the changes you need and test them in the Launchpad environment.
 
 Once you are finished you can produce a distribution as above for inclusion
-in to Launchpad, as well as sending your patch upstream. At that point you
-are free to revert the configuration to only develop Launchpad. You should
-not submit your branch with this change in the configuration as it will
-not work for others.
-
-Be aware that you may have to change the version for the package in
-``versions.cfg`` if there is a difference between the version in the
-``eggs`` directory and the one in the ``setup.py`` that you pointed to
-in the ``develop`` key.
-
-One thing to be wary of is that setuptools treats "develop eggs" created
-by this process with the same precedence as system packages. That means
-that if the version in the ``setup.py`` at the path that you put in the
-``develop`` key is the same as the version installed system wide, setuptools
-may pick the wrong one. If that happens then increase the version in
-setup.py and setuptools will take it.
+in to Launchpad, as well as sending your patch upstream.  At that point you
+are free to revert the configuration to only develop Launchpad.  Make sure
+to test with the final distribution before submitting your branch.
 
 =====================
 Possible Future Goals
 =====================
 
+- Use wheels.
 - No longer use system site-packages.
 - No longer use make.
 - Get rid of the sourcecode directory.
 
=== removed file 'ez_setup.py'
--- ez_setup.py	2012-06-29 08:40:05 +0000
+++ ez_setup.py	1970-01-01 00:00:00 +0000
@@ -1,288 +0,0 @@
-#!python
-
-# NOTE TO LAUNCHPAD DEVELOPERS: This is a bootstrapping file from the
-# setuptools project.  It is imported by our setup.py.
-
-"""Bootstrap setuptools installation
-
-If you want to use setuptools in your package's setup.py, just include this
-file in the same directory with it, and add this to the top of your setup.py::
-
-    from ez_setup import use_setuptools
-    use_setuptools()
-
-If you want to require a specific version of setuptools, set a download
-mirror, or use an alternate download directory, you can do so by supplying
-the appropriate options to ``use_setuptools()``.
-
-This file can also be run as a script to install or upgrade setuptools.
-"""
-import sys
-DEFAULT_VERSION = "0.6c11"
-DEFAULT_URL     = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
-
-md5_data = {
-    'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca',
-    'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb',
-    'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b',
-    'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a',
-    'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618',
-    'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac',
-    'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5',
-    'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4',
-    'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c',
-    'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b',
-    'setuptools-0.6c10-py2.3.egg': 'ce1e2ab5d3a0256456d9fc13800a7090',
-    'setuptools-0.6c10-py2.4.egg': '57d6d9d6e9b80772c59a53a8433a5dd4',
-    'setuptools-0.6c10-py2.5.egg': 'de46ac8b1c97c895572e5e8596aeb8c7',
-    'setuptools-0.6c10-py2.6.egg': '58ea40aef06da02ce641495523a0b7f5',
-    'setuptools-0.6c11-py2.3.egg': '2baeac6e13d414a9d28e7ba5b5a596de',
-    'setuptools-0.6c11-py2.4.egg': 'bd639f9b0eac4c42497034dec2ec0c2b',
-    'setuptools-0.6c11-py2.5.egg': '64c94f3bf7a72a13ec83e0b24f2749b2',
-    'setuptools-0.6c11-py2.6.egg': 'bfa92100bd772d5a213eedd356d64086',
-    'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27',
-    'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277',
-    'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa',
-    'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e',
-    'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e',
-    'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f',
-    'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2',
-    'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc',
-    'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167',
-    'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64',
-    'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d',
-    'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20',
-    'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab',
-    'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
-    'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2',
-    'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e',
-    'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372',
-    'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902',
-    'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de',
-    'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b',
-    'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03',
-    'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a',
-    'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6',
-    'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a',
-}
-
-import sys, os
-try: from hashlib import md5
-except ImportError: from md5 import md5
-
-def _validate_md5(egg_name, data):
-    if egg_name in md5_data:
-        digest = md5(data).hexdigest()
-        if digest != md5_data[egg_name]:
-            print >>sys.stderr, (
-                "md5 validation of %s failed!  (Possible download problem?)"
-                % egg_name
-            )
-            sys.exit(2)
-    return data
-
-def use_setuptools(
-    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
-    download_delay=15
-):
-    """Automatically find/download setuptools and make it available on sys.path
-
-    `version` should be a valid setuptools version number that is available
-    as an egg for download under the `download_base` URL (which should end with
-    a '/').  `to_dir` is the directory where setuptools will be downloaded, if
-    it is not already available.  If `download_delay` is specified, it should
-    be the number of seconds that will be paused before initiating a download,
-    should one be required.  If an older version of setuptools is installed,
-    this routine will print a message to ``sys.stderr`` and raise SystemExit in
-    an attempt to abort the calling script.
-    """
-    was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules
-    def do_download():
-        egg = download_setuptools(version, download_base, to_dir, download_delay)
-        sys.path.insert(0, egg)
-        import setuptools; setuptools.bootstrap_install_from = egg
-    try:
-        import pkg_resources
-    except ImportError:
-        return do_download()
-    try:
-        pkg_resources.require("setuptools>="+version); return
-    except pkg_resources.VersionConflict as e:
-        if was_imported:
-            print >>sys.stderr, (
-            "The required version of setuptools (>=%s) is not available, and\n"
-            "can't be installed while this script is running. Please install\n"
-            " a more recent version first, using 'easy_install -U setuptools'."
-            "\n\n(Currently using %r)"
-            ) % (version, e.args[0])
-            sys.exit(2)
-        else:
-            del pkg_resources, sys.modules['pkg_resources']    # reload ok
-            return do_download()
-    except pkg_resources.DistributionNotFound:
-        return do_download()
-
-def download_setuptools(
-    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
-    delay = 15
-):
-    """Download setuptools from a specified location and return its filename
-
-    `version` should be a valid setuptools version number that is available
-    as an egg for download under the `download_base` URL (which should end
-    with a '/'). `to_dir` is the directory where the egg will be downloaded.
-    `delay` is the number of seconds to pause before an actual download attempt.
-    """
-    import urllib2, shutil
-    egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3])
-    url = download_base + egg_name
-    saveto = os.path.join(to_dir, egg_name)
-    src = dst = None
-    if not os.path.exists(saveto):  # Avoid repeated downloads
-        try:
-            from distutils import log
-            if delay:
-                log.warn("""
----------------------------------------------------------------------------
-This script requires setuptools version %s to run (even to display
-help).  I will attempt to download it for you (from
-%s), but
-you may need to enable firewall access for this script first.
-I will start the download in %d seconds.
-
-(Note: if this machine does not have network access, please obtain the file
-
-   %s
-
-and place it in this directory before rerunning this script.)
----------------------------------------------------------------------------""",
-                    version, download_base, delay, url
-                ); from time import sleep; sleep(delay)
-            log.warn("Downloading %s", url)
-            src = urllib2.urlopen(url)
-            # Read/write all in one block, so we don't create a corrupt file
-            # if the download is interrupted.
-            data = _validate_md5(egg_name, src.read())
-            dst = open(saveto,"wb"); dst.write(data)
-        finally:
-            if src: src.close()
-            if dst: dst.close()
-    return os.path.realpath(saveto)
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-def main(argv, version=DEFAULT_VERSION):
-    """Install or upgrade setuptools and EasyInstall"""
-    try:
-        import setuptools
-    except ImportError:
-        egg = None
-        try:
-            egg = download_setuptools(version, delay=0)
-            sys.path.insert(0,egg)
-            from setuptools.command.easy_install import main
-            return main(list(argv)+[egg])   # we're done here
-        finally:
-            if egg and os.path.exists(egg):
-                os.unlink(egg)
-    else:
-        if setuptools.__version__ == '0.0.1':
-            print >>sys.stderr, (
-            "You have an obsolete version of setuptools installed.  Please\n"
-            "remove it from your system entirely before rerunning this script."
-            )
-            sys.exit(2)
-
-    req = "setuptools>="+version
-    import pkg_resources
-    try:
-        pkg_resources.require(req)
-    except pkg_resources.VersionConflict:
-        try:
-            from setuptools.command.easy_install import main
-        except ImportError:
-            from easy_install import main
-        main(list(argv)+[download_setuptools(delay=0)])
-        sys.exit(0) # try to force an exit
-    else:
-        if argv:
-            from setuptools.command.easy_install import main
-            main(argv)
-        else:
-            print "Setuptools version",version,"or greater has been installed."
-            print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)'
-
-def update_md5(filenames):
-    """Update our built-in md5 registry"""
-
-    import re
-
-    for name in filenames:
-        base = os.path.basename(name)
-        f = open(name,'rb')
-        md5_data[base] = md5(f.read()).hexdigest()
-        f.close()
-
-    data = ["    %r: %r,\n" % it for it in md5_data.items()]
-    data.sort()
-    repl = "".join(data)
-
-    import inspect
-    srcfile = inspect.getsourcefile(sys.modules[__name__])
-    f = open(srcfile, 'rb'); src = f.read(); f.close()
-
-    match = re.search("\nmd5_data = {\n([^}]+)}", src)
-    if not match:
-        print >>sys.stderr, "Internal error!"
-        sys.exit(2)
-
-    src = src[:match.start(1)] + repl + src[match.end(1):]
-    f = open(srcfile,'w')
-    f.write(src)
-    f.close()
-
-
-if __name__=='__main__':
-    if len(sys.argv)>2 and sys.argv[1]=='--md5update':
-        update_md5(sys.argv[2:])
-    else:
-        main(sys.argv[1:])
-
-
-
-
-
-
 
=== modified file 'lib/lp/codehosting/__init__.py'
--- lib/lp/codehosting/__init__.py	2012-09-06 00:01:38 +0000
+++ lib/lp/codehosting/__init__.py	2017-12-19 09:48:36 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Launchpad code-hosting system.
@@ -26,15 +26,7 @@
 
 def get_bzr_path():
     """Find the path to the copy of Bazaar for this rocketfuel instance"""
-    bzr_in_egg_path = os.path.join(
-        os.path.dirname(os.path.dirname(bzrlib.__file__)),
-        'EGG-INFO/scripts/bzr')
-    if os.path.exists(bzr_in_egg_path):
-        return bzr_in_egg_path
-    else:
-        return os.path.join(
-            os.path.dirname(os.path.dirname(bzrlib.__file__)),
-            'bzr')
+    return os.path.join(config.root, 'bin', 'bzr')
 
 
 def _get_bzr_plugins_path():
 
=== modified file 'lib/lp/scripts/harness.py'
--- lib/lp/scripts/harness.py	2013-06-20 05:50:00 +0000
+++ lib/lp/scripts/harness.py	2017-12-19 09:48:36 +0000
@@ -1,4 +1,4 @@
-# Copyright 2004-2012 Canonical Ltd.  This software is licensed under the
+# Copyright 2004-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Scripts for starting a Python prompt with Launchpad initialized.
@@ -12,10 +12,8 @@
 __metaclass__ = type
 __all__ = ['python', 'ipython']
 
-# This has setup.py scripts.  It is usually installed via buildout.
-#
+# This has entry points with corresponding scripts installed by setup.py.
 
-#
 import os
 import readline
 import rlcompleter
 
=== removed file 'lib/lp/scripts/utilities/bzr.py'
--- lib/lp/scripts/utilities/bzr.py	2017-01-18 00:55:27 +0000
+++ lib/lp/scripts/utilities/bzr.py	1970-01-01 00:00:00 +0000
@@ -1,13 +0,0 @@
-# Copyright 2010-2017 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-import pkg_resources
-
-
-def main():
-    # Run the script.
-    bzr_distribution = pkg_resources.get_distribution(
-        pkg_resources.Requirement.parse('bzr'))
-    namespace = globals().copy()
-    namespace['__name__'] = '__main__'
-    bzr_distribution.run_script('bzr', namespace)
 
=== modified file 'lib/lp/scripts/utilities/importfascist.py'
--- lib/lp/scripts/utilities/importfascist.py	2016-07-16 07:53:54 +0000
+++ lib/lp/scripts/utilities/importfascist.py	2017-12-19 09:48:36 +0000
@@ -31,6 +31,10 @@
     'json.decoder': set(['JSONDecodeError']),
     'openid.fetchers': set(['Urllib2Fetcher']),
     'openid.message': set(['NamespaceAliasRegistrationError']),
+    # Exported as shlex.quote in Python 3.
+    'pipes': set(['quote']),
+    # Exported in Python 3, but missing and so not exported in Python 2.
+    'shlex': set(['quote']),
     'storm.database': set(['STATE_DISCONNECTED']),
     'textwrap': set(['dedent']),
     'testtools.testresult.real': set(['_details_to_str']),
 
=== modified file 'lib/lp/services/job/runner.py'
--- lib/lp/services/job/runner.py	2017-06-14 11:40:16 +0000
+++ lib/lp/services/job/runner.py	2017-12-19 09:48:36 +0000
@@ -462,7 +462,8 @@
         if 'LPCONFIG' in os.environ:
             env['LPCONFIG'] = os.environ['LPCONFIG']
         env['PYTHONPATH'] = os.pathsep.join(sys.path)
-        starter = main.ProcessStarter(env=env)
+        starter = main.ProcessStarter(
+            bootstrap="import _pythonpath\n" + main.BOOTSTRAP, env=env)
         super(TwistedJobRunner, self).__init__(logger, error_utility)
         self.job_source = job_source
         self.import_name = '%s.%s' % (
 
=== modified file 'lib/lp/services/mailman/monkeypatches/mm_cfg.py.in'
--- lib/lp/services/mailman/monkeypatches/mm_cfg.py.in	2011-11-11 21:59:00 +0000
+++ lib/lp/services/mailman/monkeypatches/mm_cfg.py.in	2017-12-19 09:48:36 +0000
@@ -1,9 +1,8 @@
 # Automatically generated by runlaunchpad.py
 
 # Initialize sys.path so that the Mailman processes, which use the standard
-# system Python instead of buildout's bin/py, can find all the necessary
-# packages and modules.  Some of these are in sourcecode and some are in
-# buildout eggs.
+# system Python instead of bin/py, can find all the necessary packages and
+# modules.  Some of these are in sourcecode and some are in the virtualenv.
 #
 # This is a two-step process.  First, we hack sys.path in order to find a
 # directory containing _pythonpath.  Then the _pythonpath module does all the
 
=== modified file 'lib/lp/services/scripts/tests/test_logger.txt'
--- lib/lp/services/scripts/tests/test_logger.txt	2011-12-29 05:29:36 +0000
+++ lib/lp/services/scripts/tests/test_logger.txt	2017-12-19 09:48:36 +0000
@@ -7,12 +7,6 @@
     ...     import sys
     ...     import subprocess
     ...     from lp.services.config import config
-    ...     if 'env' in kw:
-    ...         # We want to make sure that the subprocess will have the
-    ...         # benefit of all of the dependency paths inserted by the
-    ...         # buildout.  This is already set up in our environment's
-    ...         # PYTHONPATH, so use it.
-    ...         kw['env']['PYTHONPATH'] = os.environ['PYTHONPATH']
     ...     test_script_path = os.path.join(
     ...         config.root, 'lib', 'lp', 'services',
     ...         'scripts', 'tests', 'loglevels.py')
 
=== modified file 'lib/lp_sitecustomize.py'
--- lib/lp_sitecustomize.py	2017-10-05 12:45:46 +0000
+++ lib/lp_sitecustomize.py	2017-12-19 09:48:36 +0000
@@ -1,14 +1,15 @@
 # Copyright 2009-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-# This file is imported by parts/scripts/sitecustomize.py, as set up in our
-# buildout.cfg (see the "initialization" key in the "[scripts]" section).
+# This file is imported by _pythonpath.py and by the standard Launchpad
+# script preamble (see LPScriptWriter in setup.py).
 
 from collections import defaultdict
 import itertools
 import logging
 import os
 import warnings
+import sys
 
 from twisted.internet.defer import (
     Deferred,
@@ -172,14 +173,15 @@
     silence_swiftclient_logger()
 
 
-def main(instance_name):
-    # This is called by our custom buildout-generated sitecustomize.py
-    # in parts/scripts/sitecustomize.py. The instance name is sent to
-    # buildout from the Makefile, and then inserted into
-    # sitecustomize.py.  See buildout.cfg in the "initialization" value
-    # of the [scripts] section for the code that goes into this custom
-    # sitecustomize.py.  We do all actual initialization here, in a more
-    # visible place.
+def main(instance_name=None):
+    # This is called by _pythonpath.py and by the standard Launchpad script
+    # preamble (see LPScriptWriter in setup.py).  The instance name is sent
+    # to setup.py from the Makefile, and then written to env/instance_name.
+    # We do all actual initialization here, in a more visible place.
+    if instance_name is None:
+        instance_name_path = os.path.join(sys.prefix, 'instance_name')
+        with open(instance_name_path) as instance_name_file:
+            instance_name = instance_name_file.read().rstrip('\n')
     if instance_name and instance_name != 'development':
         # See bug 656213 for why we do this carefully.
         os.environ.setdefault('LPCONFIG', instance_name)
 
=== added file 'pip-requirements.txt'
--- pip-requirements.txt	1970-01-01 00:00:00 +0000
+++ pip-requirements.txt	2017-12-19 09:48:36 +0000
@@ -0,0 +1,1 @@
+pip==9.0.1
 
=== removed file 'setup.cfg'
--- setup.cfg	2014-01-30 15:04:06 +0000
+++ setup.cfg	1970-01-01 00:00:00 +0000
@@ -1,12 +0,0 @@
-[easy_install]
-# These settings control the behaviour of the bootstrap.py script.  They keep
-# the version of setuptools and zc.buildout found in the download-cache as the
-# ones used to make the bin/buildout script.  Therefore, if your system Python
-# already has setuptools installed, this file will keep the system from going
-# out on the network for any distributions during the zc.buildout
-# bootstrapping.  Also, therefore, if you want to use a newer version of
-# setuptools or zc.buildout, you need to put it in ./download-cache/dist and
-# commit there, and change the specification in versions.cfg, as described
-# generally for dependencies in ./doc/buildout.txt.
-allow_hosts =
-
 
=== modified file 'setup.py'
--- setup.py	2017-12-07 12:05:13 +0000
+++ setup.py	2017-12-19 09:48:36 +0000
@@ -3,12 +3,129 @@
 # Copyright 2009, 2010 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-import ez_setup
-
-
-ez_setup.use_setuptools()
-
-from setuptools import setup, find_packages
+from __future__ import print_function
+
+from distutils.sysconfig import get_python_lib
+import imp
+import os.path
+from string import Template
+import sys
+from textwrap import dedent
+
+from setuptools import (
+    find_packages,
+    setup,
+    )
+from setuptools.command.develop import develop
+from setuptools.command.easy_install import ScriptWriter
+
+
+class LPScriptWriter(ScriptWriter):
+    """A modified ScriptWriter that uses Launchpad's boilerplate.
+
+    Any script written using this class will set up its environment using
+    `lp_sitecustomize` before calling its entry point.
+
+    The standard setuptools handling of entry_points uses
+    `pkg_resources.load_entry_point` to resolve requirements at run-time.
+    This involves walking Launchpad's entire dependency graph, which is
+    rather slow, and we always build all of our "optional" features anyway,
+    so we might as well just take the simplified approach of importing the
+    modules we need directly.  If we ever want to start using the "extras"
+    feature of setuptools then we may want to revisit this.
+    """
+
+    template = Template(dedent("""
+        import sys
+
+        import ${module_name}
+
+        if __name__ == '__main__':
+            sys.exit(${module_name}.${attrs}())
+        """))
+
+    @classmethod
+    def get_args(cls, dist, header=None):
+        """See `ScriptWriter`."""
+        if header is None:
+            header = cls.get_header()
+        for name, ep in dist.get_entry_map("console_scripts").items():
+            cls._ensure_safe_name(name)
+            script_text = cls.template.substitute({
+                "attrs": ".".join(ep.attrs),
+                "module_name": ep.module_name,
+                })
+            args = cls._get_script_args("console", name, header, script_text)
+            for res in args:
+                yield res
+
+
+class lp_develop(develop):
+    """A modified develop command to handle LP script generation."""
+
+    def _get_orig_sitecustomize(self):
+        env_top = os.path.join(os.path.dirname(__file__), "env")
+        system_paths = [
+            path for path in sys.path if not path.startswith(env_top)]
+        try:
+            fp, orig_sitecustomize_path, _ = (
+                imp.find_module("sitecustomize", system_paths))
+            if fp:
+                fp.close()
+        except ImportError:
+            return ""
+        if orig_sitecustomize_path.endswith(".py"):
+            with open(orig_sitecustomize_path) as orig_sitecustomize_file:
+                orig_sitecustomize = orig_sitecustomize_file.read()
+                return dedent("""
+                    # The following is from
+                    # %s
+                    """ % orig_sitecustomize_path) + orig_sitecustomize
+        else:
+            return ""
+
+    def install_wrapper_scripts(self, dist):
+        if not self.exclude_scripts:
+            for args in LPScriptWriter.get_args(dist):
+                self.write_script(*args)
+
+            # Write bin/py for compatibility.  This is much like
+            # env/bin/python, but if we just symlink to it and try to
+            # execute it as bin/py then the virtualenv doesn't get
+            # activated.  We use -S to avoid importing sitecustomize both
+            # before and after the execve.
+            py_header = LPScriptWriter.get_header("#!python -S")
+            py_script_text = dedent("""\
+                import os
+                import sys
+
+                os.execv(sys.executable, [sys.executable] + sys.argv[1:])
+                """)
+            self.write_script("py", py_header + py_script_text)
+
+            env_top = os.path.join(os.path.dirname(__file__), "env")
+            stdlib_dir = get_python_lib(standard_lib=True, prefix=env_top)
+            orig_sitecustomize = self._get_orig_sitecustomize()
+            sitecustomize_path = os.path.join(stdlib_dir, "sitecustomize.py")
+            with open(sitecustomize_path, "w") as sitecustomize_file:
+                sitecustomize_file.write(dedent("""\
+                    import os
+                    import sys
+
+                    if "LP_DISABLE_SITECUSTOMIZE" not in os.environ:
+                        if "lp_sitecustomize" not in sys.modules:
+                            import lp_sitecustomize
+                            lp_sitecustomize.main()
+                    """))
+                if orig_sitecustomize:
+                    sitecustomize_file.write(orig_sitecustomize)
+
+            # Write out the build-time value of LPCONFIG so that it can be
+            # used by scripts as the default instance name.
+            instance_name_path = os.path.join(env_top, "instance_name")
+            with open(instance_name_path, "w") as instance_name_file:
+                print(os.environ["LPCONFIG"], file=instance_name_file)
+
 
 __version__ = '2.2.3'
 
@@ -62,7 +179,8 @@
         'Markdown',
         'mechanize',
         'meliae',
-        'mock',
+        # Pin version for now to avoid confusion with system site-packages.
+        'mock==1.0.1',
         'oauth',
         'oops',
         'oops_amqp',
@@ -105,6 +223,7 @@
         'txlongpoll',
         'txlongpollfixture',
         'txpkgupload',
+        'virtualenv-tools3',
         'wadllib',
         'z3c.pt',
         'z3c.ptcompat',
@@ -125,6 +244,7 @@
         'zope.exceptions',
         'zope.formlib',
         'zope.i18n',
+        'zope.i18nmessageid',
         'zope.interface',
         'zope.lifecycleevent',
         'zope.location',
@@ -158,17 +278,13 @@
         "Intended Audience :: Developers",
         "Programming Language :: Python",
     ],
-    extras_require=dict(
-        docs=[
-            'Sphinx',
-            'z3c.recipe.sphinxdoc',
-        ]
-    ),
+    cmdclass={
+        'develop': lp_develop,
+    },
     entry_points=dict(
         console_scripts=[  # `console_scripts` is a magic name to setuptools
             'build-twisted-plugin-cache = '
                 'lp.services.twistedsupport.plugincache:main',
-            'bzr = lp.scripts.utilities.bzr:main',
             'combine-css = lp.scripts.utilities.js.combinecss:main',
             'googletestservice = '
                 'lp.services.googlesearch.googletestservice:main',
 
=== modified file 'utilities/link-external-sourcecode'
--- utilities/link-external-sourcecode	2013-04-15 02:36:07 +0000
+++ utilities/link-external-sourcecode	2017-12-19 09:48:36 +0000
@@ -1,6 +1,6 @@
 #!/usr/bin/python
 #
-# Copyright 2009-2011 Canonical Ltd. This software is licensed under the GNU
+# Copyright 2009-2017 Canonical Ltd. This software is licensed under the GNU
 # Affero General Public License version 3 (see the file LICENSE).
 
 import optparse
@@ -138,7 +138,7 @@
     for source, destination in missing_files:
         link(source, destination)
 
-    for folder_name in ('download-cache', 'eggs'):
+    for folder_name in ('download-cache',):
         source = abspath(join(options.parent, folder_name))
         destination = abspath(join(options.target, folder_name))
         if not exists(destination):
 
=== added file 'utilities/relocate-virtualenv'
--- utilities/relocate-virtualenv	1970-01-01 00:00:00 +0000
+++ utilities/relocate-virtualenv	2017-12-19 09:48:36 +0000
@@ -0,0 +1,25 @@
+#! /bin/sh
+set -e
+
+# Ensure that a virtualenv is relocated to its current path.  This does not
+# cope with moving a virtualenv to a different version of the base operating
+# system or a different architecture; it only copes with moving it to a
+# different filesystem path (perhaps on a different machine as long as it is
+# similar enough).
+
+if [ -z "$1" ]; then
+    echo "Usage: $0 ENV_PATH" 2>&1
+    exit 1
+fi
+
+# virtualenv-tools does most of the hard work.  We must explicitly invoke it
+# with the virtualenv's Python, as its #! line is probably wrong.
+"$1/bin/python" "$1/bin/virtualenv-tools" --update-path=auto "$1"
+
+# Fix up a few things that virtualenv-tools doesn't handle.
+top="$(readlink -f "$(dirname "$0")/..")"
+for path in "$1"/lib/*/site-packages/lp.egg-link; do
+    printf '%s/lib\n../' "$top" >"$path"
+done
+sed -i "s#^find_links = .*#find_links = file://$top/download-cache/dist/#" \
+    "$1"/.pydistutils.cfg
 
=== modified file 'utilities/rocketfuel-branch'
--- utilities/rocketfuel-branch	2009-06-24 20:15:50 +0000
+++ utilities/rocketfuel-branch	2017-12-19 09:48:36 +0000
@@ -1,6 +1,6 @@
 #! /bin/bash
 #
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 #
 # Create a new branch of LP called "foo" in $LP_PROJECT_PATH/foo, with all the
@@ -12,11 +12,6 @@
     exit 1
 fi
 
-LP_DOWNLOAD_CACHE_PATH=$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR/download-cache
-LP_EGGS_PATH=$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR/eggs
-LP_DOWNLOAD_CACHE_PATH=$(eval echo ${LP_DOWNLOAD_CACHE_PATH})
-LP_EGGS_PATH=$(eval echo ${LP_EGGS_PATH})
-
 if [ "x$1" == "x" ]; then
     echo "Usage: $0 new-branch-name"
     echo "Example: '$0 fixes-bug-54356'"
 
=== modified file 'utilities/rocketfuel-get'
--- utilities/rocketfuel-get	2017-11-18 09:49:07 +0000
+++ utilities/rocketfuel-get	2017-12-19 09:48:36 +0000
@@ -1,6 +1,6 @@
 #! /bin/bash
 #
-# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2017 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 #
 # Update your copy of trunk and the necessary source dependencies, and make
@@ -29,10 +29,8 @@
 fi
 
 LP_DOWNLOAD_CACHE_PATH="$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR/download-cache"
-LP_EGGS_PATH="$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR/eggs"
 YUI_PATH="$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR/yui"
 LP_DOWNLOAD_CACHE_PATH="$(eval echo ${LP_DOWNLOAD_CACHE_PATH})"
-LP_EGGS_PATH="$(eval echo ${LP_EGGS_PATH})"
 
 # Pull launchpad devel from launchpad.
 INITIAL_REV=$(bzr revno "$LP_TRUNK_PATH")
@@ -40,7 +38,7 @@
 FINAL_REV=$(bzr revno "$LP_TRUNK_PATH")
 
 # Make sure our directories are around.
-mkdir -p "$LP_SOURCEDEPS_PATH" "$LP_EGGS_PATH" "$YUI_PATH"
+mkdir -p "$LP_SOURCEDEPS_PATH" "$YUI_PATH"
 
 # Get/update the download cache.
 if [ -d "$LP_DOWNLOAD_CACHE_PATH" ]
 
=== removed file 'versions.cfg'
--- versions.cfg	2017-12-07 12:05:13 +0000
+++ versions.cfg	1970-01-01 00:00:00 +0000
@@ -1,193 +0,0 @@
-[buildout]
-extends =
-    ztk-versions.cfg
-    zopeapp-versions.cfg
-versions = versions
-
-[versions]
-# Alphabetical, case-insensitive, please! :-)
-
-# lp:~launchpad/ampoule/lp
-# post1 Don't add a process back to the ready set if it received an error
-# such as a timeout.
-ampoule = 0.2.0.post1
-amqp = 1.4.9
-amqplib = 1.0.2
-anyjson = 0.3.3
-argparse = 1.2.1
-auditor = 0.0.3
-auditorclient = 0.0.4
-auditorfixture = 0.0.7
-backports.lzma = 0.0.3
-BeautifulSoup = 3.2.1
-billiard = 3.3.0.20
-bson = 0.3.3
-bzr = 2.6.0.lp.2
-celery = 3.1.18
-Chameleon = 2.11
-cssselect = 0.9.1
-cssutils = 0.9.10
-d2to1 = 0.2.10
-Django = 1.4
-dkimpy = 0.5.4
-# Required by dkimpy
-dnspython = 1.10.0
-elementtree = 1.2.6-20050316
-epydoc = 3.0.1
-extras = 0.0.3
-FeedParser = 4.1
-feedvalidator = 0.0.0DEV-r1049
-fixtures = 0.3.9
-FormEncode = 1.2.4
-grokcore.component = 1.6
-html5browser = 0.0.9
-httmock = 1.2.3
-httplib2 = 0.8
-importlib = 1.0.2
-ipython = 0.13.2
-iso8601 = 0.1.4
-jsautobuild = 0.2
-keyring = 0.6.2
-kombu = 3.0.30
-launchpad-buildd = 136
-launchpadlib = 1.10.5
-lazr.authentication = 0.1.1
-lazr.batchnavigator = 1.2.11
-lazr.config = 2.2.1
-lazr.delegates = 2.0.4
-lazr.enum = 1.1.3
-lazr.jobrunner = 0.13
-lazr.lifecycle = 1.1
-lazr.restful = 0.20.0
-lazr.restfulclient = 0.13.2
-lazr.smtptest = 1.3
-lazr.sshserver = 0.1.3
-lazr.testing = 0.1.1
-lazr.uri = 1.0.3
-libnacl = 1.3.6
-lpjsmin = 0.5
-manuel = 1.7.2
-Markdown = 2.3.1
-martian = 0.11
-meliae = 0.2.0.final.0
-mock = 1.0.1
-mocker = 1.1.1
-oauth = 1.0
-oops = 0.0.13
-oops-amqp = 0.0.8b1
-oops-datedir-repo = 0.0.23
-oops-timeline = 0.0.1
-oops-twisted = 0.0.7
-oops-wsgi = 0.0.8
-ordereddict = 1.1
-oslo.config = 1.1.1
-paramiko = 1.7.7.2
-pbr = 0.5.20
-pgbouncer = 0.0.8
-pip = 1.4
-prettytable = 0.7.2
-psycopg2 = 2.6.1
-pyasn1 = 0.1.6
-pycrypto = 2.6
-Pygments = 1.6
-pygpgme = 0.2
-pyinotify = 0.9.4
-pymacaroons = 0.9.2
-pyOpenSSL = 0.13
-pystache = 0.5.3
-python-dateutil = 1.5
-python-debian = 0.1.23
-python-keystoneclient = 0.3.1
-python-memcached = 1.58
-python-mimeparse = 0.1.4
-# XXX: deryck 2012-08-10
-# See lp:~deryck/python-openid/python-openid-fix1034376 which
-# reapplied a patch from wgrant to get codehosting going again.
-python-openid = 2.2.5-fix1034376
-python-subunit = 0.0.8beta
-python-swiftclient = 1.5.0
-PyYAML = 3.10
-pytz = 2017.2
-rabbitfixture = 0.3.6
-requests = 2.7.0
-requests-toolbelt = 0.6.2
-setproctitle = 1.1.7
-setuptools-git = 1.0
-simplejson = 3.8.2
-SimpleTAL = 4.3
-six = 1.9.0
-soupmatchers = 0.2
-# lp:~launchpad-committers/storm/with-without-datetime
-storm = 0.19.0.99-lpwithnodatetime-r408
-subprocess32 = 3.2.6
-subvertpy = 0.9.1
-testresources = 0.2.7
-testscenarios = 0.4
-testtools = 0.9.30
-timeline = 0.0.3
-# Build of lp:~canonical-launchpad-branches/twisted:lp-backport.
-# p1 Support diffie-hellman-group14-sha1 key exchange in conch.ssh.
-# p2 Add diffie-hellman-group-exchange-sha256 to twisted.conch.ssh.
-#    Add support in twisted.conch.ssh for hmac-sha2-256 and hmac-sha2-512.
-Twisted = 13.0.0-p2
-txAMQP = 0.6.2
-txfixtures = 0.1.4
-txlongpoll = 0.2.12
-txlongpollfixture = 0.1.3
-txpkgupload = 0.2
-unittest2 = 0.5.1
-van.testing = 3.0.0
-wadllib = 1.3.2
-# Upgrade from ZTK 1.1.5 to intercept lazr.restfulclient.
-wsgi-intercept = 0.5.1
-wsgiref = 0.1.2
-z3c.pt = 2.2.3
-z3c.ptcompat = 0.5.7
-z3c.recipe.tag = 0.6
-# Also upgrade the zc.buildout version in the Makefile's bin/buildout section.
-zc.buildout = 1.7.1
-zc.zservertracelog = 1.3.2
-ZConfig = 2.9.1dev-20110728
-# Not in ZTK 1.1.5
-zope.app.server = 3.6.0
-# Upgrade from ZTK 1.1.6 for ZCML registration performance.
-zope.component = 3.11.0
-# Upgrade from ZTK 1.1.6 for ZCML registration performance.
-zope.interface = 4.4.3
-# Build of lp:~wallyworld/zope.pagetemplate/fix-isinstance
-# This version adds a small change to the traversal logic so that the
-# optimisation which applies if the object is a dict also works for subclasses
-# of dict. The change has been approved for merge into the official zope code
-# base. This patch is a temporary fix until the next official release.
-zope.pagetemplate = 3.5.0-p1
-# XXX: downgraded to avoid 3.9.2 cookie calculation changes
-zope.session = 3.9.1
-# p1 Build of lp:~mars/zope.testing/3.9.4-p1.  Fixes bugs 570380 and 587886.
-# p2 With patch for thread leaks to make them skips, fixes windmill errors
-#    with 'new threads' in hudson/ec2 builds.
-# p3 And always tear down layers, because thats the Right Thing To Do.
-# p4 fixes --subunit --list to really just list the tests.
-# p5 Build of lp:~launchpad/zope.testing/3.9.4-p5. Fixes bug #609986.
-# p6 reinstates fix from p4.  Build of lp:~launchpad/zope.testing/3.9.4-fork
-#    revision 26.
-# p7 was unused
-# p8 redirects stdout and stderr to a black hole device when --subunit is used
-# p9 adds the redirection of __stderr__ to a black hole device
-# p10 changed the test reporting to use test.id() rather than
-#     str(test) since only the id is unique.
-# p11 reverts p9.
-# p12 reverts p11, restoring p9.
-# p13 Add a new --require-unique flag to the testrunner. When set,
-#     this will cause the testrunner to check all tests IDs to ensure they
-#     haven't been loaded before. If it encounters a duplicate, it will
-#     raise an error and quit.
-# p14 Adds test data written to stderr and stdout into the subunit output.
-# p15 Fixed internal tests.
-# p16 Adds support for skips in Python 2.7.
-# p17 Fixes skip support for Python 2.6.
-# To build (use Python 2.6) run "python bootstrap.py; ./bin/buildout".  Then to
-#    build the distribution run "bin/buildout setup . sdist"
-# Make sure you have subunit installed.
-zope.testing = 3.9.4-p17
-# Not in ZTK 1.1.5 (extracted from zope.app.schema)
-zope.vocabularyregistry = 1.0.0
 
=== removed file 'zopeapp-versions.cfg'
--- zopeapp-versions.cfg	2013-05-09 23:35:44 +0000
+++ zopeapp-versions.cfg	1970-01-01 00:00:00 +0000
@@ -1,40 +0,0 @@
-[versions]
-# ZopeApp
-zc.sourcefactory = 0.7.0
-zope.app.applicationcontrol = 3.5.10
-zope.app.appsetup = 3.15.0
-zope.app.debug = 3.4.1
-zope.app.http = 3.9.0
-zope.app.publication = 3.12.0
-zope.app.wsgi = 3.10.0
-zope.testbrowser = 3.10.4
-
-# Deprecated
-roman = 1.4.0
-wsgi-intercept = 0.4
-zope.app.authentication = 3.9.0
-zope.app.basicskin = 3.5.1
-zope.app.broken = 3.6.0
-zope.app.component = 3.9.3
-zope.app.container = 3.9.2
-zope.app.content = 3.5.1
-zope.app.dependable = 3.5.1
-zope.app.error = 3.5.3
-zope.app.exception = 3.6.3
-zope.app.folder = 3.5.2
-zope.app.form = 4.0.2
-zope.app.generations = 3.7.1
-zope.app.i18n = 3.6.4
-zope.app.locales = 3.6.2
-zope.app.localpermission = 3.7.2
-zope.app.pagetemplate = 3.11.2
-zope.app.principalannotation = 3.7.0
-zope.app.publisher = 3.10.2
-zope.app.renderer = 3.5.1
-zope.app.rotterdam = 3.5.3
-zope.app.schema = 3.5.0
-zope.app.security = 3.7.5
-zope.app.testing = 3.8.1
-zope.app.zcmlfiles = 3.7.1
-zope.app.zopeappgenerations = 3.6.1
-zope.generations = 3.7.1
 
=== removed file 'ztk-versions.cfg'
--- ztk-versions.cfg	2016-06-06 07:58:13 +0000
+++ ztk-versions.cfg	1970-01-01 00:00:00 +0000
@@ -1,112 +0,0 @@
-[versions]
-# ZTK
-zope.annotation = 3.6.0
-zope.applicationcontrol = 3.5.5
-zope.authentication = 3.7.1
-zope.broken = 3.6.0
-zope.browser = 1.3
-zope.browsermenu = 3.9.1
-zope.browserpage = 3.12.2
-zope.browserresource = 3.12.0
-zope.cachedescriptors = 3.5.1
-zope.catalog = 3.8.2
-zope.component = 3.10.0
-zope.componentvocabulary = 1.0.1
-zope.configuration = 3.7.4
-zope.container = 3.12.0
-zope.contentprovider = 3.7.2
-zope.contenttype = 3.5.5
-zope.copy = 3.5.0
-zope.copypastemove = 3.8.0
-zope.datetime = 3.4.1
-zope.deferredimport = 3.5.3
-zope.deprecation = 3.4.1
-zope.dottedname = 3.4.6
-zope.dublincore = 3.8.2
-zope.error = 3.7.4
-zope.event = 3.5.2
-zope.exceptions = 3.6.2
-zope.filerepresentation = 3.6.1
-zope.formlib = 4.0.6
-zope.hookable = 3.4.1
-zope.i18n = 3.7.4
-zope.i18nmessageid = 3.5.3
-zope.index = 3.6.4
-zope.interface = 3.7.0
-zope.intid = 3.7.2
-zope.keyreference = 3.6.4
-zope.lifecycleevent = 3.6.2
-zope.location = 3.9.1
-zope.login = 1.0.0
-zope.mimetype = 1.3.1
-zope.minmax = 1.1.2
-zope.pagetemplate = 3.5.2
-zope.password = 3.6.1
-zope.pluggableauth = 1.2
-zope.principalannotation = 3.6.1
-zope.principalregistry = 3.7.1
-zope.processlifetime = 1.0
-zope.proxy = 3.6.1
-zope.ptresource = 3.9.0
-zope.publisher = 3.12.6
-zope.ramcache = 1.0
-zope.schema = 3.7.1
-zope.security = 3.8.3
-zope.securitypolicy = 3.7.0
-zope.sendmail = 3.7.5
-zope.sequencesort = 3.4.0
-zope.server = 3.8.6
-zope.session = 3.9.5
-zope.site = 3.9.2
-zope.size = 3.4.1
-zope.structuredtext = 3.5.1
-zope.tal = 3.5.2
-zope.tales = 3.5.3
-zope.testing = 3.10.3
-zope.testrunner = 4.0.4
-zope.traversing = 3.14.0
-zope.viewlet = 3.7.2
-
-# Deprecating
-
-# Dependencies
-distribute = 0.6.36
-docutils = 0.7
-Jinja2 = 2.5.5
-mechanize = 0.2.5
-Paste = 1.7.5.1
-PasteDeploy = 1.3.4
-PasteScript = 1.7.5
-py = 1.4.8
-Pygments = 1.4
-python-gettext = 1.0
-python-subunit = 0.0.7
-pytz = 2013b
-RestrictedPython = 3.6.0
-setuptools = 0.6c11
-Sphinx = 1.0.8
-testtools = 0.9.12
-transaction = 1.1.1
-z3c.recipe.sphinxdoc = 0.0.8
-zc.buildout = 1.7.1
-zc.lockfile = 1.0.2
-ZConfig = 2.8.0
-zc.recipe.egg = 1.3.2
-zc.recipe.testrunner = 1.4.0
-zc.resourcelibrary = 1.3.4
-zdaemon = 2.0.7
-ZODB3 = 3.10.5
-zope.mkzeoinstance = 3.9.5
-
-# toolchain
-argparse = 1.1
-coverage = 3.5.2
-lxml = 2.2.8
-mr.developer = 1.25
-nose = 1.1.2
-tl.eggdeps = 0.4
-z3c.checkversions = 0.4.1
-z3c.recipe.compattest = 0.13.1
-z3c.recipe.depgraph = 0.5
-z3c.recipe.scripts = 1.0.1
-zope.kgs = 1.2.0