Merge lp:~allenap/txlongpoll/bootstrap-without-net into lp:txlongpoll

[Gavin Panella]

Many changes related to improving deployment of txlongpoll as a working service.

Several fixes relating to making deployment easier for the LOSAs, and
reliably repeatable:

- Make txlongpoll bootstrappable locally, without network, and build
  from cache.

- `make build` now only builds what's needed to _run_ txlongpoll.

- `make bin/test` installs the test runner and dependencies.

- `make tags TAGS` builds tag files.

- Build with relative paths to make deployment a bit easier, which
  goes in hand with the next change...

- `make build-update-paths` is for the use of people deploying
  txlongpoll, i.e. the LOSAs. They will build txlongpoll on one
  machine and sync it to another, but at a different fs location.
  `make build-update-paths` fixes the last few remaining absolute
  paths in the buildout. I suspect that the need to do this indicates
  a deficiency or bug in buildout or one of the recipes.

- Updated the README to reflect much of the above.

- Depend on txAMQP >= 0.5 because lower versions cause test failures.

[lp:~allenap/txlongpoll/bootstrap-without-net]

82. By Gavin Panella on 2011-11-07

Correct README.

83. By Gavin Panella on 2011-11-07

Get rid of ez_setup.py and instead get it from the download cache.

84. By Gavin Panella on 2011-11-07

Update README to explain how to run online.

[Raphaël Badin (rvb)]

Looks good.

[0]

146 +- Remove the --setup-source and --download-base options in Makefile to
147 + allow bootstrap.py to download setuptools.
148 +
149 +- Comment out the ``install-from-cache = true`` line in buildout.cfg
150 + to allow bin/buildout to download dependencies.

I wish it would be more straight forward to do this (like running make with a simple option) but I'm not sure it's feasible/simple.

[0]

47 +TAGS: bin/tags
48 + bin/tags --ctags-emacs
49 +
50 +
51 +tags: bin/tags
52 + bin/tags --ctags-vi

Just curious here, is this standard (the name of the targets I mean)?

review: Approve

=== modified file 'Makefile'
--- Makefile	2011-11-04 14:35:34 +0000
+++ Makefile	2011-11-07 12:07:26 +0000
@@ -2,54 +2,71 @@
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 PYTHON = python
-PY = $(PWD)/bin/py
-
-BUILDOUT_CFG = buildout.cfg
-
-# Do not add bin/buildout to this list. It is impossible to get
-# buildout to tell us all the files it would build, since each egg's
-# setup.py doesn't tell us that information.
-#
-# NB: It's important BUILDOUT_BIN only mentions things genuinely
-# produced by buildout.
-BUILDOUT_BIN = $(PY) bin/tags bin/test bin/txlongpoll
+
+BOOTSTRAP_BIN := bootstrap.py
+BOOTSTRAP = PYTHONPATH= $(PYTHON) $(BOOTSTRAP_BIN)
+
+BUILDOUT_BIN := bin/buildout
+BUILDOUT_CFG := buildout.cfg
+BUILDOUT = PYTHONPATH= $(BUILDOUT_BIN) -qc $(BUILDOUT_CFG)
 
 
 default: check
 
 
+build: bin/twistd
+
+
+# When a built tree is moved this updates absolute paths.
+build-update-paths:
+	$(BUILDOUT)
+
+
+check: bin/test
+	bin/test -vv
+
+
+dist: $(BUILDOUT_BIN)
+	$(BUILDOUT) setup setup.py egg_info -r sdist
+
+
+TAGS: bin/tags
+	bin/tags --ctags-emacs
+
+
+tags: bin/tags
+	bin/tags --ctags-vi
+
+
 download-cache:
-	mkdir download-cache
-
-
-bin/buildout: download-cache
-	$(PYTHON) bootstrap.py
+	mkdir -p download-cache
+
+
+eggs:
+	mkdir -p eggs
+
+
+$(BUILDOUT_BIN): download-cache eggs
+	$(BOOTSTRAP) \
+	    --setup-source=download-cache/dist/ez_setup.py \
+	    --download-base=download-cache/dist \
+	    --eggs=eggs --version=1.5.2
 	touch --no-create $@
 
 
-$(PY): bin/buildout $(BUILDOUT_CFG) setup.py
-	PYTHONPATH=. ./bin/buildout -c $(BUILDOUT_CFG)
-
-
-$(subst $(PY),,$(BUILDOUT_BIN)): $(PY)
-
-
-build: $(BUILDOUT_BIN)
-
-
-update-paths: bin/buildout
-	bin/buildout -oN
-
-
-dist: $(PY)
-	$(PY) setup.py egg_info -r sdist
-
-
-check: bin/test
-	./bin/test -vv
-
-
-clean_buildout: 
+bin/twistd: $(BUILDOUT_BIN) $(BUILDOUT_CFG) setup.py
+	$(BUILDOUT) install runtime
+
+
+bin/test: $(BUILDOUT_BIN) $(BUILDOUT_CFG) setup.py
+	$(BUILDOUT) install test
+
+
+bin/tags: $(BUILDOUT_BIN) $(BUILDOUT_CFG) setup.py
+	$(BUILDOUT) install tags
+
+
+clean_buildout:
 	$(RM) -r bin
 	$(RM) -r parts
 	$(RM) -r develop-eggs
@@ -65,10 +82,11 @@
 
 
 clean: clean_buildout
+	find txlongpoll twisted -name '*.py[co]' -print0 | xargs -r0 $(RM)
 
 clean_all: clean_buildout clean_eggs
 
 
 .PHONY: \
-  build check clean clean_all clean_buildout \
-  clean_eggs default dist update-paths
+    build build-update-paths check clean clean_all clean_buildout \
+    clean_eggs default dist

=== modified file 'README'
--- README	2011-10-04 13:03:49 +0000
+++ README	2011-11-07 12:07:26 +0000
@@ -1,23 +1,50 @@
 Generic long poll server used by Launchpad and Landscape.
 
 
+Dependencies
+------------
+
+By default txlongpoll attempts to build without going to the network;
+instead it tries to find all dependencies from download-cache/dist,
+and you must download all the dependencies yourself.
+
+If you prefer to use resources from the 'net, PyPI for example, then:
+
+- Remove the --setup-source and --download-base options in Makefile to
+  allow bootstrap.py to download setuptools.
+
+- Comment out the ``install-from-cache = true`` line in buildout.cfg
+  to allow bin/buildout to download dependencies.
+
+To run the tests you may also need lp:rabbitfixture (bzr branch
+lp:rabbitfixture). Build an sdist tarball and drop it into
+download-cache/dist. You may also need to do the same for
+testresources-0.2.4_r58 until it reaches PyPI.
+
+txlongpoll needs a recent RabbitMQ with support for rejection. Maverick's
+1.8.0 is not sufficient, but 2.1.1 works fine, so >= Natty is OK.
+
+
 Building
 --------
 
-txlongpoll needs a recent RabbitMQ with support for rejection. Maverick's
-1.8.0 is not sufficient, but 2.1.1 works fine, so >= Natty is OK.
-
-You also need to get lp:rabbitfixture, build it and drop the egg in
-./download-cache/dist in order for txlongpoll to build.  You will also possibly
-need to do the same for testresources-0.2.4_r58 until it reaches PyPI.
-
  $ make build
 
+will build only those parts needed to run txlongpoll. No support for
+tags or testing.
+
 
 Testing
 -------
 
- $ bin/test
+ $ make check
+
+will build all the test-related parts of txlongpoll and then do a full
+test run, but
+
+ $ make bin/test
+
+will just do the first part.
 
 
 Running

=== modified file 'buildout.cfg'
--- buildout.cfg	2011-11-04 14:31:08 +0000
+++ buildout.cfg	2011-11-07 12:07:26 +0000
@@ -2,10 +2,7 @@
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 [buildout]
-parts =
-    interpreter
-    test
-    tags
+parts = runtime
 use-dependency-links = false
 unzip = true
 include-site-packages = false
@@ -13,20 +10,24 @@
 develop = .
 download-cache = download-cache
 eggs-directory = eggs
-# Relative paths are required for deployment.
+# Relative paths are useful when deploying built trees.
 relative-paths = true
+# Disable this option temporarily if you want buildout to find
+# software dependencies *other* than those in our download-cache.
+# Once you have the desired software, reenable this option.
+install-from-cache = true
+
+[runtime]
+recipe = zc.recipe.egg:scripts
+eggs = txlongpoll
+entry-points =  twistd=twisted.scripts.twistd:run
+interpreter = py
 
 [test]
 recipe = zc.recipe.testrunner
 eggs = txlongpoll [test]
 defaults = '--tests-pattern ^tests --exit-with-status'.split()
 
-[interpreter]
-recipe = zc.recipe.egg:scripts
-interpreter = py
-eggs = txlongpoll
-entry-points =  twistd=twisted.scripts.twistd:run
-
 [tags]
 recipe = z3c.recipe.tag:tags
 eggs = txlongpoll

=== modified file 'setup.py'
--- setup.py	2011-11-04 14:51:49 +0000
+++ setup.py	2011-11-07 12:07:26 +0000
@@ -22,7 +22,7 @@
         'oops_twisted >= 0.0.3',
         'setproctitle',
         'Twisted',
-        'txamqp',
+        'txAMQP >= 0.5',
         'zope.interface',
         ],
     extras_require=dict(