Merge lp:~jtv/launchpad/parallel-make into lp:launchpad

[Jeroen T. Vermeulen]

[r=benji,edwin-grubbs][ui=none][no-qa] Support "make -j2"

= Parallel make =

This fixes the Makefile so that the default target (or "build") can be built in parallel. So instead of "make" you can do "make -j2" and get a nice speedup: on my system right after a "make clean" it takes the build time from 2:43 to 1:32. Should be a welcome change.

What changes? A single buildout run in the makefile produces bin/py and a bunch of other files. But "make" isn't aware that there's no point to running two buildout instances at once, and indeed it's likely to break.

So I channeled all the files produced by buildout through a single virtual target. Now "make" realizes when it's about to run two buildout instances in parallel that they are actually producing the same target—and wait for the existing run to complete instead of starting a redundant one.

(Actually I had to start out with completing dependencies on bin/py, because not all targets that used it depended on it. These things become more noticeable when you run parallel builds.)

The good news is that the build parallelizes almost perfectly to two processes; the bad news is that that's where the ride ends. This is because one single-process job, the generation of WADL files, takes up just about half the build time. With two processors, the two halvse will complete almost simultaneously at just over half the single-process build time. Adding a third CPU won't help the WADL target, and so can't do much for overall build time. At best it gets you to the point where everything waits for that one big job a little faster. On my dual-core SMT laptop, a third process even slows the build down a little.

Jeroen

[Benji York (benji)]

This branch looks good. Edwin, my review mentor, will also be reviewing this branch.

The transformation you applied is straightforward. (And welcome, my first day working on LP I ran into this issue).

It's probably not worth adding, because changes are rare, but for completeness I'll note that the buildout_bin rule has an unlisted dependency: buildout.cfg uses template files in buildout-templates (which at the moment is just _pythonpath.py.in).

review: Approve (code*)

[Jeroen T. Vermeulen (jtv)]

Thanks. I'm tempted to add that dependency right now, but a quick glance doesn't tell me enough to see how it fits in so I'll leave that for someone else who does see it.

[Edwin Grubbs (edwin-grubbs)]

Jeroen,

This branch looks good. Can you add the dependencies that Benji asked for, and next time, can you submit a really difficult branch for Benji to review so he gets more practice? Oh wait, that means I have to review it, too. Nevermind.

-Edwin

review: Approve (code)

[Jeroen T. Vermeulen (jtv)]

I added the dependency, and broke some over-length lines because, surprisingly, "make lint" also checks the makefile!

=== modified file 'Makefile'
--- Makefile	2010-11-02 01:34:05 +0000
+++ Makefile	2010-11-18 04:52:18 +0000
@@ -45,6 +45,8 @@
     bin/start_librarian bin/stxdocs bin/tags bin/test bin/tracereport \
     bin/twistd bin/update-download-cache bin/windmill
 
+BUILDOUT_TEMPLATES = buildout-templates/_pythonpath.py.in
+
 # DO NOT ALTER : this should just build by default
 default: inplace
 
@@ -55,10 +57,10 @@
 newsampledata:
 	$(MAKE) -C database/schema newsampledata
 
-hosted_branches: $(PY)
+hosted_branches: buildout_bin
 	$(PY) ./utilities/make-dummy-hosted-branches
 
-$(API_INDEX): $(BZR_VERSION_INFO)
+$(API_INDEX): $(BZR_VERSION_INFO) buildout_bin
 	mkdir -p $(APIDOC_DIR).tmp
 	LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl-and-apidoc.py --force "$(WADL_TEMPLATE)"
 	mv $(APIDOC_DIR).tmp $(APIDOC_DIR)
@@ -66,12 +68,12 @@
 apidoc: compile $(API_INDEX)
 
 # Run by PQM.
-check_merge: $(PY)
+check_merge: buildout_bin
 	[ `PYTHONPATH= bzr status -S database/schema/ | \
 		grep -v "\(^P\|pending\|security.cfg\|Makefile\|unautovacuumable\|_pythonpath.py\)" | wc -l` -eq 0 ]
 	${PY} lib/canonical/tests/test_no_conflict_marker.py
 
-check_db_merge: $(PY)
+check_db_merge: buildout_bin
 	${PY} lib/canonical/tests/test_no_conflict_marker.py
 
 check_config: build
@@ -109,16 +111,16 @@
 	${PY} -t ./test_on_merge.py $(VERBOSITY) $(TESTOPTS) \
 		--layer=MailmanLayer
 
-lint: ${PY}
+lint: buildout_bin
 	@bash ./bin/lint.sh
 
-lint-verbose: ${PY}
+lint-verbose: buildout_bin
 	@bash ./bin/lint.sh -v
 
-xxxreport: $(PY)
+xxxreport: buildout_bin
 	${PY} -t ./utilities/xxxreport.py -f csv -o xxx-report.csv ./
 
-check-configs: $(PY)
+check-configs: buildout_bin
 	${PY} utilities/check-configs.py
 
 pagetests: build
@@ -140,12 +142,14 @@
 	${SHHH} bin/sprite-util create-image
 
 jsbuild_lazr: bin/jsbuild
-	# We absolutely do not want to include the lazr.testing module and its
-	# jsTestDriver test harness modifications in the lazr.js and launchpad.js
-	# roll-up files.  They fiddle with built-in functions!  See Bug 482340.
-	${SHHH} bin/jsbuild $(JSFLAGS) -b $(LAZR_BUILT_JS_ROOT) -x testing/ -c $(LAZR_BUILT_JS_ROOT)/yui
+	# We absolutely do not want to include the lazr.testing module and
+	# its jsTestDriver test harness modifications in the lazr.js and
+	# launchpad.js roll-up files.  They fiddle with built-in functions!
+	# See Bug 482340.
+	${SHHH} bin/jsbuild $(JSFLAGS) -b $(LAZR_BUILT_JS_ROOT) -x testing/ \
+	-c $(LAZR_BUILT_JS_ROOT)/yui
 
-jsbuild: jsbuild_lazr bin/jsbuild bin/jssize
+jsbuild: jsbuild_lazr bin/jsbuild bin/jssize buildout_bin
 	${SHHH} bin/jsbuild \
 		$(JSFLAGS) \
 		-n launchpad \
@@ -173,12 +177,12 @@
 	@exit 1
 endif
 
-buildonce_eggs: $(PY)
+buildonce_eggs: buildout_bin
 	find eggs -name '*.pyc' -exec rm {} \;
 
 # The download-cache dependency comes *before* eggs so that developers get the
-# warning before the eggs directory is made.  The target for the eggs directory
-# is only there for deployment convenience.
+# warning before the eggs directory is made.  The target for the eggs
+# directory is only there for deployment convenience.
 # Note that the buildout version must be maintained here and in versions.cfg
 # to make sure that the build does not go over the network.
 bin/buildout: download-cache eggs
@@ -192,19 +196,22 @@
 # and the other bits might run into problems like bug 575037.  This
 # target runs buildout, and then removes everything created except for
 # the eggs.
-build_eggs: $(BUILDOUT_BIN) clean_buildout
+build_eggs: buildout_bin clean_buildout
+
+$(BUILDOUT_BIN): buildout_bin
 
 # This builds bin/py and all the other bin files except bin/buildout.
 # Remove the target before calling buildout to ensure that buildout
 # updates the timestamp.
-$(BUILDOUT_BIN): bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
+buildout_bin: bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py \
+		$(BUILDOUT_TEMPLATES)
 	$(RM) $@
 	$(SHHH) PYTHONPATH= ./bin/buildout \
                 configuration:instance_name=${LPCONFIG} -c $(BUILDOUT_CFG)
 
 # bin/compile_templates is responsible for building all chameleon templates,
 # of which there is currently one, but of which many more are coming.
-compile: $(PY) $(BZR_VERSION_INFO)
+compile: buildout_bin $(BZR_VERSION_INFO)
 	mkdir -p /var/tmp/vostok-archive
 	${SHHH} $(MAKE) -C sourcecode build PYTHON=${PYTHON} \
 	    LPCONFIG=${LPCONFIG}
@@ -405,7 +412,8 @@
 	# We insert the absolute path to the branch-rewrite script
 	# into the Apache config as we copy the file into position.
 	sed -e 's,%BRANCH_REWRITE%,$(shell pwd)/scripts/branch-rewrite.py,' configs/development/local-launchpad-apache > /etc/apache2/sites-available/local-launchpad
-	cp configs/development/local-vostok-apache /etc/apache2/sites-available/local-vostok
+	cp configs/development/local-vostok-apache \
+		/etc/apache2/sites-available/local-vostok
 	touch /var/tmp/bazaar.launchpad.dev/rewrite.log
 	chown $(SUDO_UID):$(SUDO_GID) /var/tmp/bazaar.launchpad.dev/rewrite.log
 
@@ -430,8 +438,9 @@
 
 lp.sfood:
 	# Generate import dependency graph
-	sfood -i -u -I lib/sqlobject -I lib/schoolbell -I lib/devscripts -I lib/contrib \
-	-I lib/canonical/not-used lib/canonical lib/lp 2>/dev/null | grep -v contrib/ \
+	sfood -i -u -I lib/sqlobject -I lib/schoolbell -I lib/devscripts \
+	-I lib/contrib -I lib/canonical/not-used lib/canonical \
+	lib/lp 2>/dev/null | grep -v contrib/ \
 	| grep -v sqlobject | grep -v BeautifulSoup | grep -v psycopg \
 	| grep -v schoolbell > lp.sfood.tmp
 	mv lp.sfood.tmp lp.sfood
@@ -463,10 +472,10 @@
 		--docformat restructuredtext --verbose-about epytext-summary \
 		$(PYDOCTOR_OPTIONS)
 
-.PHONY: apidoc check tags TAGS zcmldocs realclean clean debug stop\
-	start run ftest_build ftest_inplace test_build test_inplace pagetests\
-	check check_merge \
-	schema default launchpad.pot check_merge_ui pull scan sync_branches\
-	reload-apache hosted_branches check_db_merge check_mailman check_config\
-	jsbuild jsbuild_lazr clean_js clean_buildout buildonce_eggs build_eggs\
-	sprite_css sprite_image css_combine compile check_schema pydoctor
+.PHONY: apidoc buildout_bin check tags TAGS zcmldocs realclean clean debug \
+	stop start run ftest_build ftest_inplace test_build test_inplace \
+	pagetests check check_merge schema default launchpad.pot \
+	check_merge_ui pull scan sync_branches reload-apache hosted_branches \
+	check_db_merge check_mailman check_config jsbuild jsbuild_lazr \
+	clean_js clean_buildout buildonce_eggs build_eggs sprite_css \
+	sprite_image css_combine compile check_schema pydoctor