Merge lp:~cjwatson/turnip/virtualenv into lp:turnip

[Colin Watson]

Add targets to build a virtualenv and to build a tarball with included pip cache.

Add targets to build a virtualenv and to build a tarball with included pip cache.

This is the first step towards making our deployment system more sensible: it will eventually allow us to have a mojo manifest that builds the entire code asset in an appropriate container, virtualenv and all, run tests the whole thing, and upload it to swift for the charm to deploy. This will involve less downtime on upgrades and will support the autodeployment scheme currently being trialled by Canonical IS. As a bonus, since the intended use of this is to run the build step in an environment that matches the deployment target, we can switch to installing pygit2 from pip like everything else; 0.22.1 includes our current set of local patches.

This of course requires some additions to turnip-dependencies (pbr-1.8.1.tar.gz, pip-7.1.2.tar.gz, pygit2-0.22.1.tar.gz, setuptools-18.4.tar.gz), which I'll commit if this is accepted, and I'll also commit a similar change to turnipcake in that case so that the charms can remain consistent.

The turnip charm will need to be updated to use this, and that update will involve quite a bit of refactoring, but this part of it can be landed in advance of that, and as a bonus it makes "make check" much more reliable.

[lp:~cjwatson/turnip/virtualenv]

185. By Colin Watson on 2015-10-19

Ship a pip cache in the tarball rather than a full virtualenv.

186. By Colin Watson on 2015-10-20

Switch order of env and pip-cache construction, since we need to do the downloads with a modern pip.

187. By Colin Watson on 2015-10-21

Quieten pip.

188. By Colin Watson on 2015-10-22

Use revision IDs rather than revnos to construct build labels.

189. By Colin Watson on 2015-10-22

Install turnip itself in editable mode, for the sake of gunicorn.

190. By Colin Watson on 2015-10-23

Fix turnip dependencies URL.

[Kit Randel (blr)]

This is great, and I'll be updating rutabaga to reflect this shortly!

I did wonder however if it would be worth providing a similar warning to `make build-tarball`, when PIP_SOURCE_DIR is not set for the `make` target? The `pip install` will fail without it.

review: Approve

[Colin Watson (cjwatson)]

Oh, right, I was thinking it'd use PyPI in that case but indeed it fails when it gets to our python-openid fork. I'll add the warning.

[lp:~cjwatson/turnip/virtualenv]

191. By Colin Watson on 2015-10-29

Move PIP_SOURCE_DIR check from build-tarball to $(ENV), since we need the dependencies checkout for our python-openid fork.

=== modified file '.bzrignore'
--- .bzrignore	2015-05-20 23:32:18 +0000
+++ .bzrignore	2015-10-29 10:59:31 +0000
@@ -4,10 +4,12 @@
 dist
 download-cache
 eggs
+env
+pip-cache
 *.egg*
 *.egg-info
 .installed.cfg
 *.log
 parts
 tags
-TAGS
\ No newline at end of file
+TAGS

=== modified file 'Makefile'
--- Makefile	2015-03-12 07:51:28 +0000
+++ Makefile	2015-10-29 10:59:31 +0000
@@ -1,18 +1,57 @@
 # Copyright 2005-2015 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
-PYTHON=python
-PSERVE=pserve
-FLAKE8=flake8
-
-check:
+ENV = $(CURDIR)/env
+PIP_CACHE = $(CURDIR)/pip-cache
+
+PYTHON = $(ENV)/bin/python
+PSERVE = $(ENV)/bin/pserve
+FLAKE8 = $(ENV)/bin/flake8
+
+PIP_CACHE_ARGS := -q --no-use-wheel
+ifneq ($(PIP_SOURCE_DIR),)
+PIP_CACHE_ARGS += --no-index --find-links=file://$(realpath $(PIP_SOURCE_DIR))/
+endif
+
+# Create archives in labelled directories (e.g.
+# <rev-id>/$(PROJECT_NAME).tar.gz)
+TARBALL_BUILD_LABEL ?= $(shell bzr log -rlast: --show-ids | sed -n 's/^revision-id: //p')
+TARBALL_FILE_NAME = turnip.tar.gz
+TARBALL_BUILDS_DIR ?= build
+TARBALL_BUILD_DIR = $(TARBALL_BUILDS_DIR)/$(TARBALL_BUILD_LABEL)
+TARBALL_BUILD_PATH = $(TARBALL_BUILD_DIR)/$(TARBALL_FILE_NAME)
+
+build: $(ENV)
+
+$(ENV):
+ifeq ($(PIP_SOURCE_DIR),)
+	@echo "Set PIP_SOURCE_DIR to the path of a checkout of" >&2
+	@echo "lp:~canonical-launchpad-branches/turnip/dependencies." >&2
+	@exit 1
+endif
+	mkdir -p $(ENV)
+	(echo '[easy_install]'; \
+	 echo "allow_hosts = ''"; \
+	 echo 'find_links = file://$(realpath $(PIP_SOURCE_DIR))/') \
+		>$(ENV)/.pydistutils.cfg
+	virtualenv $(ENV)
+	$(ENV)/bin/pip install $(PIP_CACHE_ARGS) \
+		-r bootstrap-requirements.txt
+	$(ENV)/bin/pip install $(PIP_CACHE_ARGS) \
+		-r requirements.txt \
+		-r deploy-requirements.txt \
+		-r test-requirements.txt \
+		-e .
+
+check: $(ENV)
 	$(PYTHON) -m unittest discover turnip
 
 clean:
 	find turnip -name '*.py[co]' -exec rm '{}' \;
+	rm -rf $(ENV) $(PIP_CACHE)
 
 dist:
-	$(PYTHON) ./setup.py sdist
+	python ./setup.py sdist
 
 TAGS:
 	ctags -e -R turnip
@@ -20,10 +59,31 @@
 tags:
 	ctags -R turnip
 
-lint:
+lint: $(ENV)
 	@$(FLAKE8) turnip
 
-run-api:
+run-api: $(ENV)
 	$(PSERVE) api.ini --reload
 
-.PHONY: check clean dist lint run-api
+$(PIP_CACHE): $(ENV)
+	mkdir -p $(PIP_CACHE)
+	$(ENV)/bin/pip install $(PIP_CACHE_ARGS) -d $(PIP_CACHE) \
+		-r bootstrap-requirements.txt \
+		-r requirements.txt \
+		-r deploy-requirements.txt \
+		-r test-requirements.txt
+
+# XXX cjwatson 2015-10-16: limit to only interesting files
+build-tarball:
+	@echo "Creating deployment tarball at $(TARBALL_BUILD_PATH)"
+	rm -rf $(PIP_CACHE)
+	$(MAKE) $(PIP_CACHE)
+	mkdir -p $(TARBALL_BUILD_DIR)
+	tar -czf $(TARBALL_BUILD_PATH) \
+		--exclude-vcs \
+		--exclude build \
+		--exclude dist \
+		--exclude env \
+		./
+
+.PHONY: build check clean dist lint run-api build-tarball

=== added file 'bootstrap-requirements.txt'
--- bootstrap-requirements.txt	1970-01-01 00:00:00 +0000
+++ bootstrap-requirements.txt	2015-10-29 10:59:31 +0000
@@ -0,0 +1,2 @@
+pip==7.1.2
+setuptools==18.4

=== added file 'deploy-requirements.txt'
--- deploy-requirements.txt	1970-01-01 00:00:00 +0000
+++ deploy-requirements.txt	2015-10-29 10:59:31 +0000
@@ -0,0 +1,5 @@
+# This will be merged into requirements.txt once the turnip charm uses the
+# new tarball construction code.
+envdir==0.7
+gunicorn==19.3.0
+pygit2==0.22.1

=== modified file 'test-requirements.txt'
--- test-requirements.txt	2015-06-16 07:03:32 +0000
+++ test-requirements.txt	2015-10-29 10:59:31 +0000
@@ -4,6 +4,7 @@
 flake8==2.4.0
 linecache2==1.0.0
 mccabe==0.3
+pbr==1.8.1
 pep8==1.5.7
 pyflakes==0.8.1
 python-mimeparse==0.1.4