Merge lp:~gary/launchpad/bug559128 into lp:launchpad

[Gary Poster]

Switch most of the buildout-generated code to using relative paths, to address bug 559128

This branch switches most of the buildout-generated code to using relative paths, to address bug 559128. Better distribution of compiled eggs will be a better solution to the underlying causes, but for now this fixes the problem and gets closer to making builds themselves use relative paths.

To test this for the current use case, get this branch in a directory with a unique name (like "bug559128" and run ``make bin/buildout && bin/buildout``. You can then grep the results for the string "bug559128". I only see them in the tags and the i18n scripts, which deployment does not use.

As an aside, in the future, LOSAs might like to be able to build an entire tree and just sync them out. If you run make build, currently only mailman adds further absolute paths. We probably won't be able to address that until we switch to Mailman 3 (after Python 2.6) but then it might be very easy. Further, if you run ``make run`` then +config-overrides.zcml gets an absolute path. Cleaning those up might give us a built tree that could be rsynced and run. That might be nice.

In any case, this branch accomplishes the relative path changes using new versions of z3c.recipe.filetemplate and our zc.buildout branches that support relative-paths. These have been reviewed separately. (https://code.edge.launchpad.net/~gary/z3c.recipe.filetemplate/relative-paths/+merge/23701 and https://code.edge.launchpad.net/~gary/zc.buildout/python-support-9-relative-paths/+merge/23696 , respectively). The syntax changes in the templates are described here: http://pypi.python.org/pypi/z3c.recipe.filetemplate#path-extensions and http://pypi.python.org/pypi/z3c.recipe.filetemplate#filters .

[Francis J. Lacoste (flacoste)]

Looks good. I suppose we will QA this by removing the work-around from edge deployment config and seeing if the edge update breaks?

review: Approve

[Gary Poster (gary)]

On Apr 21, 2010, at 4:44 PM, Francis J. Lacoste wrote:

> Review: Approve
> Looks good. I suppose we will QA this by removing the work-around from edge deployment config and seeing if the edge update breaks?

Yeah. I also QA'd it by doing the following:

- (get branch and dependencies)
- make bin/buildout && bin/buildout
- cd ..
- mv (branch) (new location)
- cd (new location)
- make build
- make run

That worked. That's the closest thing I can figure on doing locally.

Gary

=== modified file 'Makefile'
--- Makefile	2010-04-13 10:35:03 +0000
+++ Makefile	2010-04-21 20:27:25 +0000
@@ -334,6 +334,7 @@
 	$(RM) -rf $(APIDOC_DIR).tmp
 	$(RM) $(BZR_VERSION_INFO)
 	$(RM) _pythonpath.py
+	$(RM) +config-overrides.zcml
 	$(RM) -rf \
 			  /var/tmp/builddmaster \
 			  /var/tmp/bzrsync \

=== modified file 'buildout-templates/_pythonpath.py.in'
--- buildout-templates/_pythonpath.py.in	2010-02-25 16:49:46 +0000
+++ buildout-templates/_pythonpath.py.in	2010-04-21 20:27:25 +0000
@@ -7,10 +7,14 @@
 # This file works if the Python has been started with -S, or if bin/py
 # has been used.
 
+# Auto-generated code to handle relative paths
+${python-relative-path-setup}
+
 import os
 import sys
 
-site_dir = "${scripts:parts-directory}"
+
+site_dir = ${scripts:parts-directory|path-repr}
 
 if ('site' in sys.modules and
     not sys.modules['site'].__file__.startswith(
@@ -18,9 +22,11 @@
     # We have the wrong site.py, so our paths are not set up correctly.
     # We blow up, with a hopefully helpful error message.
     raise RuntimeError(
-        'The wrong site.py is imported. Scripts should usually be '
+        'The wrong site.py is imported (%r imported, %r expected). '
+        'Scripts should usually be '
         "started with Launchpad's bin/py, or with a Python invoked with "
-        'the -S flag.')
+        'the -S flag.' % (
+        sys.modules['site'].__file__, os.path.join(site_dir, 'site.py')))
 
 if site_dir not in sys.path:
     sys.path.insert(0, site_dir)

=== modified file 'buildout-templates/bin/combine-css.in'
--- buildout-templates/bin/combine-css.in	2010-02-17 21:59:32 +0000
+++ buildout-templates/bin/combine-css.in	2010-04-21 20:27:25 +0000
@@ -1,4 +1,10 @@
-#!/usr/bin/env ${buildout:directory}/bin/py
+#!${buildout:executable} -S
+
+# Initialize our paths.
+${python-relative-path-setup}
+import sys
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
 
 import os
 
@@ -6,7 +12,7 @@
 from lazr.js.combo import combine_files
 
 
-root = '${buildout:directory}'
+root = ${buildout:directory|path-repr}
 icing = os.path.join(root, 'lib/canonical/launchpad/icing')
 target = os.path.join(icing, 'combo.css')
 # It'd probably be nice to have this script find all the CSS files we might

=== modified file 'buildout-templates/bin/jstest.in'
--- buildout-templates/bin/jstest.in	2010-02-16 15:25:52 +0000
+++ buildout-templates/bin/jstest.in	2010-04-21 20:27:25 +0000
@@ -1,4 +1,4 @@
-#! /usr/bin/env ${buildout:directory}/bin/py
+#!${buildout:executable} -S
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
@@ -14,6 +14,12 @@
 
 """
 
+# Initialize our paths.
+${python-relative-path-setup}
+import sys
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
+
 import subprocess
 import os
 from lp.scripts.utilities.lpwindmill import runLaunchpad

=== modified file 'buildout-templates/bin/kill-test-services.in'
--- buildout-templates/bin/kill-test-services.in	2010-02-03 13:33:24 +0000
+++ buildout-templates/bin/kill-test-services.in	2010-04-21 20:27:25 +0000
@@ -1,9 +1,14 @@
-#! /usr/bin/env ${buildout:directory}/bin/py
+#!${buildout:executable} -S
 #
 # Copyright 2009 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 """Kill all the test services that may persist between test runs."""
 
+# Initialize our paths.
+${python-relative-path-setup}
+import sys
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
 
 # Tell canonical.config to use the testrunner config instance, so that
 # we don't kill the real services.

=== modified file 'buildout-templates/bin/lint.sh.in'
--- buildout-templates/bin/lint.sh.in	2010-04-14 22:45:46 +0000
+++ buildout-templates/bin/lint.sh.in	2010-04-21 20:27:25 +0000
@@ -6,7 +6,9 @@
 # Runs xmlint, pyflakes and pylint on files changed from parent branch.
 # Use '-v' to run pylint under stricter conditions with additional messages.
 
-utilitiesdir=`dirname $0`/../utilities
+${shell-relative-path-setup}
+
+utilitiesdir=${buildout:directory/utilities|shell-path}
 [ -z "$utilitiesdir" ] && utilitiesdir=.
 
 # Fail if any of the required tools are not installed.
@@ -153,7 +155,7 @@
     echo "        cp $4 $1"
     echo "    Run make schema again to update the test/dev database."
 }
-    
+
 if [ -n "$sql_diff" -o -n "$sql_dev_diff" -o -n "$karma_bombs" ]; then
     echo ""
     echo ""
@@ -161,7 +163,7 @@
     echo ""
 fi
 
-# 
+#
 if [ -n "$sql_diff" -o -n "$karma_bombs" ]; then
     echo "$current_sql"
 fi
@@ -279,7 +281,7 @@
 fi
 
 extra_path="/usr/share/pyshared:/usr/share/pycentral/pylint/site-packages"
-export PYTHONPATH="${os-paths}:$extra_path:$PYTHONPATH"
+export PYTHONPATH=${scripts:parts-directory|shell-path}:"$extra_path":"$PYTHONPATH"
 pylint="${buildout:executable} -Wi::DeprecationWarning `which pylint`"
 
 # XXX sinzui 2007-10-18 bug=154140:

=== modified file 'buildout-templates/bin/retest.in'
--- buildout-templates/bin/retest.in	2009-10-01 10:12:28 +0000
+++ buildout-templates/bin/retest.in	2010-04-21 20:27:25 +0000
@@ -31,9 +31,10 @@
 from itertools import takewhile
 from pprint import pprint
 
+${python-relative-path-setup}
 
 # The test script for this branch.
-TEST = '${buildout:directory}/bin/test'
+TEST = ${buildout:directory/bin/test|path-repr}
 
 # Regular expression to match numbered stories.
 STORY_RE = re.compile("(.*)/\d{2}-.*")

=== modified file 'buildout-templates/bin/sprite-util.in'
--- buildout-templates/bin/sprite-util.in	2010-02-10 06:02:33 +0000
+++ buildout-templates/bin/sprite-util.in	2010-04-21 20:27:25 +0000
@@ -1,8 +1,13 @@
-#!/usr/bin/env ${buildout:directory}/bin/py
+#!${buildout:executable} -S
 
 import os
 import sys
 
+# Initialize our paths.
+${python-relative-path-setup}
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
+
 from lp.services.spriteutils import SpriteUtil
 
 command_options = ('create-image', 'create-css')
@@ -21,8 +26,7 @@
         print >> sys.stderr, usage()
         sys.exit(2)
 
-root = '${buildout:directory}'
-icing = os.path.join(root, 'lib/canonical/launchpad/icing')
+icing = ${buildout:directory/lib/canonical/launchpad/icing|path-repr}
 combined_image_file = os.path.join(icing, 'icon-sprites')
 positioning_file = os.path.join(icing, 'icon-sprites.positioning')
 css_template_file = os.path.join(icing, 'style-3-0.css.in')

=== modified file 'buildout-templates/bin/test.in'
--- buildout-templates/bin/test.in	2010-04-07 12:03:10 +0000
+++ buildout-templates/bin/test.in	2010-04-21 20:27:25 +0000
@@ -1,4 +1,4 @@
-#! /usr/bin/env ${buildout:directory}/bin/py
+#!${buildout:executable} -S
 ##############################################################################
 #
 # Copyright (c) 2004 Zope Corporation and Contributors.
@@ -20,7 +20,14 @@
 
 import logging, os, re, sys, time, warnings
 
-BUILD_DIR = '${buildout:directory}'
+# Initialize our paths.
+${python-relative-path-setup}
+import sys
+sys.path.insert(0, ${scripts:parts-directory|path-repr})
+import site
+
+BUILD_DIR = ${buildout:directory|path-repr}
+CUSTOM_SITE_DIR = ${scripts:parts-directory|path-repr}
 
 if os.getsid(0) == os.getsid(os.getppid()):
     # We need to become the process group leader so test_on_merge.py
@@ -43,7 +50,7 @@
 assert os.environ['STORM_CEXTENSIONS'] == '1'
 
 # Make sure our site.py is the one that subprocesses use.
-os.environ['PYTHONPATH'] = "${scripts:parts-directory}"
+os.environ['PYTHONPATH'] = CUSTOM_SITE_DIR
 
 # Set a flag if this is the main testrunner process
 if len(sys.argv) > 1 and sys.argv[1] == '--resume-layer':
@@ -143,7 +150,7 @@
 defaults = {
     # Find tests in the tests and ftests directories
     'tests_pattern': '^f?tests$',
-    'test_path': ['${buildout:directory}/lib'],
+    'test_path': [${buildout:directory/lib|path-repr}],
     'package': ['canonical', 'lp', 'devscripts'],
     'layer': ['!(MailmanLayer)'],
     }
@@ -219,7 +226,7 @@
     # e.g. run the test that you are currently editing.
     try:
         there = os.getcwd()
-        os.chdir('${buildout:directory}')
+        os.chdir(BUILD_DIR)
         result = testrunner.run([])
     finally:
         os.chdir(there)

=== modified file 'buildout-templates/bin/update-download-cache.in'
--- buildout-templates/bin/update-download-cache.in	2009-05-13 00:29:48 +0000
+++ buildout-templates/bin/update-download-cache.in	2010-04-21 20:27:25 +0000
@@ -1,2 +1,4 @@
-bzr up ${buildout:directory}/buildout/download-cache
+${shell-relative-path-setup}
+
+bzr up ${buildout:directory/buildout/download-cache|shell-path}
 

=== modified file 'buildout.cfg'
--- buildout.cfg	2010-04-07 12:03:10 +0000
+++ buildout.cfg	2010-04-21 20:27:25 +0000
@@ -11,6 +11,7 @@
 unzip = true
 eggs-directory = eggs
 download-cache = download-cache
+relative-paths = true
 
 # Disable this option temporarily if you want buildout to find software
 # dependencies *other* than those in our download-cache.  Once you have the

=== modified file 'versions.cfg'
--- versions.cfg	2010-04-16 13:51:06 +0000
+++ versions.cfg	2010-04-21 20:27:25 +0000
@@ -94,7 +94,7 @@
 z3c.pagelet = 1.0.2
 z3c.pt = 1.0b16
 z3c.ptcompat = 0.5.3
-z3c.recipe.filetemplate = 2.0.2
+z3c.recipe.filetemplate = 2.1.0
 z3c.recipe.i18n = 0.5.3
 z3c.recipe.scripts = 1.0.0dev-gary-r110068
 z3c.recipe.tag = 0.2.0
@@ -106,7 +106,7 @@
 z3c.viewlet = 1.0.0
 z3c.viewtemplate = 0.3.2
 z3c.zrtresource = 1.0.1
-zc.buildout = 1.5.0dev-gary-r110073
+zc.buildout = 1.5.0dev-gary-r111190
 zc.catalog = 1.2.0
 zc.datetimewidget = 0.5.2
 zc.i18n = 0.5.2