Launchpad itself

Merge lp:~gary/launchpad/zbuildout into lp:launchpad/db-devel

zbuildout
Merge into db-devel

Proposed by Gary Poster on 2009-08-14

Status:

Merged

Approved by:

Francis J. Lacoste on 2009-08-18

Approved revision:

no longer in the source branch.

Merged at revision:

not available

Proposed branch:

lp:~gary/launchpad/zbuildout

Merge into:

lp:launchpad/db-devel

Diff against target:

None lines

To merge this branch:

bzr merge lp:~gary/launchpad/zbuildout

Related bugs:

Bug #408897: Use latest zope.testing	Undecided	Fix Released
Bug #409480: Use source distributions (via zc.buildout) for zope dependencies	High	Fix Released

Link a bug report

Reviewer	Review Type	Date Requested	Status
Francis J. Lacoste (community)		2009-08-14	Approve on 2009-08-18
Review via email: mp+10181@code.launchpad.net

Revision history for this message

Gary Poster (gary) wrote on 2009-08-14:

Download full text (32.6 KiB)

This branch changes us to use source distributions for our zope-related
dependencies.

Because we still are using some packages in sourcecode that want to be
egg-based distributions--particularly Twisted--we filter out pkg_resources
warning in _pythonpath and other scripts. This prevents some test failures
that do not want to see any output from running a command.

There are many other changes, of course.

- Many are driven by changes in reprs of various objects--validation errors
(LaunchpadValidationError) and ClientForm response wrappers
(httperror_seek_wrapper) in particular.

- Others are for API changes, like in zope.testing
(lib/canonical/testing/__init__.py and ).

- I commented out the retry tests, as we have discussed.

- Some tests expected zope files to be in the tree, so I had to switch to the
pkg_resources API, or simply use more proper zcml in some cases.

- There's a case or two of tokens needing to be converted from unicode to
  binary strings before they go into the URL, as with the
  lp:~gary/canonical-identity-provider/zbuildout branch that accompanies this
  one.

- zope.schema set fields no longer support sets.Set, only the built-in set,
which also resulted in some changes.

- The transaction.begin API now does not expect to encounter an in-progress
transaction (lib/canonical/launchpad/doc/storm.txt and
lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt).

- I found a couple of tests that failed in isolation. Related changes are in
lib/canonical/launchpad/doc/batch_navigation.txt and
lib/canonical/launchpad/scripts/ftests/test_oops_prune.py .

- I'm not entirely sure how
lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape.txt
ever passed before. The doctest syntax there was wrong.

- I switched back to the feedvalidator solution that we had discussed way back
when, rather than using the hacked version. I left a comment to describe
the situation.

- Some tests did a browser reload. That now has a different behavior,
resubmitting forms and so on, so I reload the same URL instead.

- I did a fly-by on lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt
because that was a [testfix] problem I diagnosed yesterday, and using
queryMultiAdapter hid the problem.

versions.cfg comments out version numbers only when my changes represent a
*previous* version that what was specified before. Happy to remove those as
well.

For this branch, I made changes and releases in the following packages:

- zope.testing (modified trunk, released and used as zope.testing 3.8.1)
- zope.security (modified trunk, released and used as zope.security 3.7.1)
- zope.app.publication (modified trunk, eventually to be 3.8.2; also modified 3.4.3, released and used as zope.app.publication 3.4.4)

Diffs to the dependencies are below. For zope.testing and zope.security, I
only show the commit that made the substantive changes, rather than the commits
generated by the release process. zope.testing has the biggest changes from
our branch because it significantly changed how it handled subprocesses from
the version we were using in order to support being able to run layers in
...

This branch changes us to use source distributions for our zope-related
dependencies.

Because we still are using some packages in sourcecode that want to be
egg-based distributions--particularly Twisted--we filter out pkg_resources
warning in _pythonpath and other scripts.  This prevents some test failures
that do not want to see any output from running a command.

There are many other changes, of course.

- Many are driven by changes in reprs of various objects--validation errors
  (LaunchpadValidationError) and ClientForm response wrappers
  (httperror_seek_wrapper) in particular.

- Others are for API changes, like in zope.testing
  (lib/canonical/testing/__init__.py and ).

- I commented out the retry tests, as we have discussed.

- Some tests expected zope files to be in the tree, so I had to switch to the
  pkg_resources API, or simply use more proper zcml in some cases.

- zope.schema set fields no longer support sets.Set, only the built-in set,
  which also resulted in some changes.

- The transaction.begin API now does not expect to encounter an in-progress
  transaction (lib/canonical/launchpad/doc/storm.txt and
  lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt).

- I found a couple of tests that failed in isolation.  Related changes are in
  lib/canonical/launchpad/doc/batch_navigation.txt and
  lib/canonical/launchpad/scripts/ftests/test_oops_prune.py .

- I'm not entirely sure how
  lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape.txt
  ever passed before.  The doctest syntax there was wrong.

- I switched back to the feedvalidator solution that we had discussed way back
  when, rather than using the hacked version.  I left a comment to describe
  the situation.

- Some tests did a browser reload.  That now has a different behavior,
  resubmitting forms and so on, so I reload the same URL instead.

- I did a fly-by on lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt
  because that was a [testfix] problem I diagnosed yesterday, and using
  queryMultiAdapter hid the problem.

versions.cfg comments out version numbers only when my changes represent a
*previous* version that what was specified before.  Happy to remove those as
well.

For this branch, I made changes and releases in the following packages:

Diffs to the dependencies are below.  For zope.testing and zope.security, I
only show the commit that made the substantive changes, rather than the commits
generated by the release process.  zope.testing has the biggest changes from
our branch because it significantly changed how it handled subprocesses from
the version we were using in order to support being able to run layers in
multiple processes, taking advantage of multiple cores.

############
zope.testing
############

Log message for revision 102634:
 reduce batching and improve output for subprocesses

Changed:
 U   zope.testing/trunk/CHANGES.txt
 U   zope.testing/trunk/src/zope/testing/testrunner/filter.py
 U   zope.testing/trunk/src/zope/testing/testrunner/formatter.py
 U   zope.testing/trunk/src/zope/testing/testrunner/process.py
 U   zope.testing/trunk/src/zope/testing/testrunner/runner.py
 A   zope.testing/trunk/src/zope/testing/testrunner/testrunner-ex/sampletests_buffering.py
 A   zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt
 U   zope.testing/trunk/src/zope/testing/testrunner/tests.py

-=-
Modified: zope.testing/trunk/CHANGES.txt
===================================================================
--- zope.testing/trunk/CHANGES.txt	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/CHANGES.txt	2009-08-10 20:57:09 UTC (rev 102634)
@@ -4,13 +4,23 @@
3.8.1 (unreleased)
==================

-- Avoid to hardcode sys.argv[0] as script; 
-  allow for instance Zope 2 `bin/instance test` (LP#407916).
+- Avoid hardcoding sys.argv[0] as script; 
+  allow, for instance, Zope 2's `bin/instance test` (LP#407916).

-- Produce a clear error message when subprocess doesn't follow the
+- Produce a clear error message when a subprocess doesn't follow the
  zope.testing.testrunner protocol (LP#407916).

+- Do not unnecessarily squelch verbose output in a subprocess when there are
+  not multiple subprocesses.

+- Do not unnecessarily batch subprocess output, which can stymie automated and
+  human processes for identifying hung tests.
+
+- Include incremental output when there are multiple subprocesses and a
+  verbosity of -vv or greater is requested.  This again is not batched,
+  supporting automated processes and humans looking for hung tests. 
+
+
3.8.0 (2009-07-24)
==================

Modified: zope.testing/trunk/src/zope/testing/testrunner/filter.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/filter.py	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/src/zope/testing/testrunner/filter.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -65,7 +65,8 @@
                    # No pattern matched this name so we remove it
                    layers.pop(name)

-        if self.runner.options.verbose:
+        if (self.runner.options.verbose and
+            not self.runner.options.resume_layer):
            if self.runner.options.all:
                msg = "Running tests at all levels"
            else:

Modified: zope.testing/trunk/src/zope/testing/testrunner/formatter.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/formatter.py	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/src/zope/testing/testrunner/formatter.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -52,6 +52,9 @@

progress = property(lambda self: self.options.progress)
    verbose = property(lambda self: self.options.verbose)
+    in_subprocess = property(
+        lambda self: self.options.resume_layer is not None and
+                     self.options.processes > 1)

def compute_max_width(self):
        """Try to determine the terminal width."""
@@ -263,6 +266,12 @@

elif self.verbose == 1:
            sys.stdout.write('.' * test.countTestCases())
+        
+        elif self.in_subprocess:
+            sys.stdout.write('.' * test.countTestCases())
+            # Give the parent process a new line so it sees the progress
+            # in a timely manner.
+            sys.stdout.write('\n')

if self.verbose > 1:
            s = str(test)

Modified: zope.testing/trunk/src/zope/testing/testrunner/process.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/process.py	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/src/zope/testing/testrunner/process.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -31,7 +31,13 @@
    def global_setup(self):
        self.original_stderr = sys.stderr
        sys.stderr = sys.stdout
-        self.runner.options.verbose = False
+        if self.runner.options.processes > 1:
+            # If we only have one subprocess, there's absolutely
+            # no reason to squelch.  We will let the messages through in a
+            # timely manner, if they have been requested. On the other hand, if
+            # there are multiple processes, we do squelch to 0.
+            self.runner.options.verbose = 0
+        self.progress = False

def report(self):
        sys.stdout.close()

Modified: zope.testing/trunk/src/zope/testing/testrunner/runner.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/runner.py	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/src/zope/testing/testrunner/runner.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -20,6 +20,7 @@

import cStringIO
import gc
+import Queue
import re
import sys
import threading
@@ -229,11 +230,12 @@
        if should_resume:
            setup_layers = None
            if layers_to_run:
-                self.ran += resume_tests(self.script_parts, self.options, self.features,
+                self.ran += resume_tests(
+                    self.script_parts, self.options, self.features,
                    layers_to_run, self.failures, self.errors)

if setup_layers:
-            if self.options.resume_layer == None:
+            if self.options.resume_layer is None:
                self.options.output.info("Tearing down left over layers:")
            tear_down_unneeded(self.options, (), setup_layers, True)

@@ -355,7 +357,7 @@
        output.info("Running %s tests:" % layer_name)
    tear_down_unneeded(options, needed, setup_layers)

-    if options.resume_layer != None:
+    if options.resume_layer is not None:
        output.info_suboptimal("  Running in a subprocess.")

try:
@@ -378,8 +380,9 @@
        "Layer set up failure."

-def spawn_layer_in_subprocess(result, script_parts, options, features, layer_name, layer,
-        failures, errors, resume_number):
+def spawn_layer_in_subprocess(result, script_parts, options, features,
+                              layer_name, layer, failures, errors,
+                              resume_number):
    try:
        # BBB
        if script_parts is None:
@@ -408,9 +411,34 @@
        child = subprocess.Popen(args, shell=False, stdin=subprocess.PIPE,
            stdout=subprocess.PIPE, stderr=subprocess.PIPE,
            close_fds=not sys.platform.startswith('win'))
-        subout, suberr = child.communicate()

-        erriter = iter(suberr.splitlines())
+        while True:
+            try:
+                while True:
+                    # We use readline() instead of iterating over stdout
+                    # because it appears that iterating over stdout causes a
+                    # lot more buffering to take place (probably so it can
+                    # return its lines as a batch). We don't want too much
+                    # buffering because this foils automatic and human monitors
+                    # trying to verify that the subprocess is still alive.
+                    l = child.stdout.readline()
+                    if not l:
+                        break
+                    result.write(l)
+            except IOError, e:
+                if e.errno == errno.EINTR:
+                    # If the subprocess dies before we finish reading its
+                    # output, a SIGCHLD signal can interrupt the reading.
+                    # The correct thing to to in that case is to retry.
+                    continue
+                output.error(
+                    "Error reading subprocess output for %s" % layer_name)
+                output.info(str(e))
+            else:
+                break
+
+        # Now stderr should be ready to read the whole thing.
+        erriter = iter(child.stderr.read().splitlines())
        nfail = nerr = 0
        for line in erriter:
            try:
@@ -431,37 +459,88 @@
            nerr -= 1
            errors.append((erriter.next().strip(), None))

-        result.stdout.append(subout)
-
    finally:
        result.done = True

-class SubprocessResult(object):
+class AbstractSubprocessResult(object):
+    """A result of a subprocess layer run."""

-    def __init__(self):
-        self.num_ran = 0
+    num_ran = 0
+    done = False
+
+    def __init__(self, layer_name, queue):
+        self.layer_name = layer_name
+        self.queue = queue
        self.stdout = []
-        self.done = False

+    def write(self, out):
+        """Receive a line of the subprocess out."""

+
+class DeferredSubprocessResult(AbstractSubprocessResult):
+    """Keeps stdout around for later processing,"""
+    
+    def write(self, out):
+        if not _is_dots(out):
+            self.stdout.append(out)
+
+
+class ImmediateSubprocessResult(AbstractSubprocessResult):
+    """Sends complete output to queue."""
+
+    def write(self, out):
+        sys.stdout.write(out)
+        # Help keep-alive monitors (human or automated) keep up-to-date.
+        sys.stdout.flush()
+
+
+_is_dots = re.compile(r'\.+\n').match
+class KeepaliveSubprocessResult(AbstractSubprocessResult):
+    "Keeps stdout for later processing; sends marks to queue to show activity."
+
+    _done = False
+
+    def _set_done(self, value):
+        self._done = value
+        assert value, 'Internal error: unexpectedly setting done to False'
+        self.queue.put((self.layer_name, ' LAYER FINISHED'))
+    done = property(lambda self: self._done, _set_done) 
+
+    def write(self, out):
+        if _is_dots(out):
+            self.queue.put((self.layer_name, out.strip()))
+        else:
+            self.stdout.append(out)
+
+
def resume_tests(script_parts, options, features, layers, failures, errors):
    results = []
+    stdout_queue = None
+    if options.processes == 1:
+        result_factory = ImmediateSubprocessResult
+    elif options.verbose > 1:
+        result_factory = KeepaliveSubprocessResult
+        stdout_queue = Queue.Queue()
+    else:
+        result_factory = DeferredSubprocessResult
    resume_number = int(options.processes > 1)
    ready_threads = []
    for layer_name, layer, tests in layers:
-        result = SubprocessResult()
+        result = result_factory(layer_name, stdout_queue)
        results.append(result)
        ready_threads.append(threading.Thread(
            target=spawn_layer_in_subprocess,
-            args=(result, script_parts, options, features, layer_name, layer, failures,
-                errors, resume_number)))
+            args=(result, script_parts, options, features, layer_name, layer,
+                  failures, errors, resume_number)))
        resume_number += 1

# Now start a few threads at a time.
    running_threads = []
    results_iter = iter(results)
    current_result = results_iter.next()
+    last_layer_intermediate_output = None
+    output = None
    while ready_threads or running_threads:
        while len(running_threads) < options.processes and ready_threads:
            thread = ready_threads.pop(0)
@@ -472,15 +551,36 @@
            if not thread.isAlive():
                del running_threads[index]

-        # We want to display results in the order they would have been
-        # displayed, had the work not been done in parallel.
+        # Clear out any messages in queue
+        while stdout_queue is not None:
+            previous_output = output
+            try:
+                layer_name, output = stdout_queue.get(False)
+            except Queue.Empty:
+                break
+            if layer_name != last_layer_intermediate_output:
+                # Clarify what layer is reporting activity.
+                if previous_output is not None:
+                    sys.stdout.write(']\n')
+                sys.stdout.write(
+                    '[Parallel tests running in %s:\n  ' % (layer_name,))
+                last_layer_intermediate_output = layer_name
+            sys.stdout.write(output)
+        # Display results in the order they would have been displayed, had the
+        # work not been done in parallel.
        while current_result and current_result.done:
+            if output is not None:
+                sys.stdout.write(']\n')
+                output = None
            map(sys.stdout.write, current_result.stdout)

try:
                current_result = results_iter.next()
            except StopIteration:
                current_result = None
+
+        # Help keep-alive monitors (human or automated) keep up-to-date.
+        sys.stdout.flush()
        time.sleep(0.01) # Keep the loop from being too tight.

# Return the total number of tests run.
@@ -488,8 +588,8 @@

def tear_down_unneeded(options, needed, setup_layers, optional=False):
-    # Tear down any layers not needed for these tests. The unneeded
-    # layers might interfere.
+    # Tear down any layers not needed for these tests. The unneeded layers
+    # might interfere.
    unneeded = [l for l in setup_layers if l not in needed]
    unneeded = order_by_bases(unneeded)
    unneeded.reverse()

Added: zope.testing/trunk/src/zope/testing/testrunner/testrunner-ex/sampletests_buffering.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/testrunner-ex/sampletests_buffering.py	                       (rev 0)
+++ zope.testing/trunk/src/zope/testing/testrunner/testrunner-ex/sampletests_buffering.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -0,0 +1,70 @@
+##############################################################################
+#
+# Copyright (c) 2003 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Sample tests with sleep and layers that can't be torn down
+
+$Id$
+"""
+
+import unittest, time
+
+class Layer1:
+
+    def setUp(self):
+        pass
+    setUp = classmethod(setUp)
+
+    def tearDown(self):
+        raise NotImplementedError
+    tearDown = classmethod(tearDown)
+
+
+class Layer2:
+
+    def setUp(self):
+        pass
+    setUp = classmethod(setUp)
+
+    def tearDown(self):
+        raise NotImplementedError
+    tearDown = classmethod(tearDown)
+
+
+class TestSomething1(unittest.TestCase):
+
+    layer = Layer1
+
+    def test_something(self):
+        pass
+
+
+class TestSomething2(unittest.TestCase):
+
+    layer = Layer2
+
+    def test_something(self):
+        time.sleep(0.5)
+
+    def test_something2(self):
+        time.sleep(0.5)
+
+
+def test_suite():
+    suite = unittest.TestSuite()
+    suite.addTest(unittest.makeSuite(TestSomething1))
+    suite.addTest(unittest.makeSuite(TestSomething2))
+    return suite
+
+
+if __name__ == '__main__':
+    unittest.main()
\ No newline at end of file

Added: zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt	                       (rev 0)
+++ zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt	2009-08-10 20:57:09 UTC (rev 102634)
@@ -0,0 +1,142 @@
+This is a test for a fix in buffering of output from a layer in a subprocess.
+
+Prior to the change that this tests, output from within a test layer in a
+subprocess would be buffered.  This could wreak havoc on supervising processes
+(or human) that would kill a test run if no output was seen for some period of
+time.
+
+First, we wrap stdout with an object that instruments it. It notes the time at
+which a given line was written.
+
+    >>> import os, sys, datetime
+    >>> class RecordingStreamWrapper:
+    ...     def __init__(self, wrapped):
+    ...         self.record = []
+    ...         self.wrapped = wrapped
+    ...     def write(self, out):
+    ...         self.record.append((out, datetime.datetime.now()))
+    ...         self.wrapped.write(out)
+    ...     def flush(self):
+    ...         self.wrapped.flush()
+    ...
+    >>> sys.stdout = RecordingStreamWrapper(sys.stdout)
+
+Now we actually call our test.  If you open the file to which we are referring
+here (zope/testing/testrunner-ex/sampletests_buffering.py) you will see two test
+suites, each with its own layer that does not know how to tear down.  This
+forces the second suite to be run in a subprocess.
+
+That second suite has two tests.  Both sleep for half a second each.
+
+    >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
+    >>> from zope.testing import testrunner
+    >>> defaults = [
+    ...     '--path', directory_with_tests,
+    ...     ]
+    >>> argv = [sys.argv[0],
+    ...         '-vv', '--tests-pattern', '^sampletests_buffering.*']
+
+    >>> try:
+    ...     testrunner.run_internal(defaults, argv)
+    ...     record = sys.stdout.record
+    ... finally:
+    ...     sys.stdout = sys.stdout.wrapped
+    ...
+    Running tests at level 1
+    Running sampletests_buffering.Layer1 tests:
+      Set up sampletests_buffering.Layer1 in N.NNN seconds.
+      Running:
+     test_something (sampletests_buffering.TestSomething1)
+      Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
+    Running sampletests_buffering.Layer2 tests:
+      Tear down sampletests_buffering.Layer1 ... not supported
+      Running in a subprocess.
+      Set up sampletests_buffering.Layer2 in N.NNN seconds.
+      Running:
+     test_something (sampletests_buffering.TestSomething2)
+     test_something2 (sampletests_buffering.TestSomething2)
+      Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
+      Tear down sampletests_buffering.Layer2 ... not supported
+    Total: 3 tests, 0 failures, 0 errors in N.NNN seconds.
+    False
+
+Now we actually check the results we care about.  We should see that there are
+two pauses of about half a second, one around the first test and one around the
+second.  Before the change that this test verfies, there was a single pause of
+more than a second after the second suite ran.
+
+    >>> pause = datetime.timedelta(seconds=0.3)
+    >>> last_line, last_time = record.pop(0)
+    >>> for line, time in record:
+    ...     if (time-last_time >= pause and
+    ...         line != '  Running in a subprocess.\n'):
+    ...         # We paused!
+    ...         print 'PAUSE FOUND BETWEEN THESE LINES:'
+    ...         print ''.join([last_line, line, '-'*70])
+    ...     last_line, last_time = line, time
+    PAUSE FOUND BETWEEN THESE LINES:
+      Running:
+     test_something (sampletests_buffering.TestSomething2)
+    ----------------------------------------------------------------------
+    PAUSE FOUND BETWEEN THESE LINES:
+     test_something (sampletests_buffering.TestSomething2)
+     test_something2 (sampletests_buffering.TestSomething2)
+    ----------------------------------------------------------------------
+
+Because this is a test based on timing, it may be somewhat fragile.  However,
+on a relatively slow machine, this timing works out fine; I'm hopeful that this
+test will not be a source of spurious errors.  If it is, we will have to
+readdress.
+
+Now let's do the same thing, but with multiple processes at once.  We'll get
+a different result that has similar characteristics.
+
+    >>> sys.stdout = RecordingStreamWrapper(sys.stdout)
+    >>> argv.extend(['-j', '2'])
+    >>> try:
+    ...     testrunner.run_internal(defaults, argv)
+    ...     record = sys.stdout.record
+    ... finally:
+    ...     sys.stdout = sys.stdout.wrapped
+    ...
+    Running tests at level 1
+    Running sampletests_buffering.Layer1 tests:
+      Set up sampletests_buffering.Layer1 in N.NNN seconds.
+      Running:
+     test_something (sampletests_buffering.TestSomething1)
+      Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
+    [Parallel tests running in sampletests_buffering.Layer2:
+      .. LAYER FINISHED]
+    Running sampletests_buffering.Layer2 tests:
+      Running in a subprocess.
+      Set up sampletests_buffering.Layer2 in N.NNN seconds.
+      Ran 2 tests with 0 failures and 0 errors in N.NNN seconds.
+    Total: 3 tests, 0 failures, 0 errors in N.NNN seconds.
+    False
+
+Notice that, with a -vv (or greater) verbosity, the parallel test run includes
+a progress report to keep track of tests run in the various layers.  Because
+the actual results are saved to be displayed assembled in the original test
+order, the progress report shows up before we are given the news that the
+testrunner is starting Layer2.  This is counterintuitive, but lets us keep the
+primary reporting information for the given layer in one location, while also
+giving us progress reports that can be used for keepalive analysis by a human or
+automated agent. In particular for the second point, notice below that, as
+before, the progress output is not buffered.
+
+    >>> last_line, last_time = record.pop(0)
+    >>> for line, time in record:
+    ...     if (time-last_time >= pause and
+    ...         line != '  Running in a subprocess.\n'):
+    ...         # We paused!
+    ...         print 'PAUSE FOUND BETWEEN THIS OUTPUT:'
+    ...         print '\n'.join([last_line, line, '-'*70])
+    ...     last_line, last_time = line, time
+    PAUSE FOUND BETWEEN THIS OUTPUT:
+    .
+    .
+    ----------------------------------------------------------------------
+    PAUSE FOUND BETWEEN THIS OUTPUT:
+    .
+     LAYER FINISHED
+    ----------------------------------------------------------------------

Modified: zope.testing/trunk/src/zope/testing/testrunner/tests.py
===================================================================
--- zope.testing/trunk/src/zope/testing/testrunner/tests.py	2009-08-10 18:23:10 UTC (rev 102633)
+++ zope.testing/trunk/src/zope/testing/testrunner/tests.py	2009-08-10 20:57:09 UTC (rev 102634)
@@ -150,6 +150,7 @@
        'testrunner-debugging.txt',
        'testrunner-edge-cases.txt',
        'testrunner-errors.txt',
+        'testrunner-layers-buff.txt',
        'testrunner-layers-ntd.txt',
        'testrunner-layers.txt',
        'testrunner-layers-api.txt',

#####################################################
zope.security (commit with substantive changes shown)
#####################################################

Log message for revision 102691:
 fix for potential segfault (LP 181833)

Changed:
 U   zope.security/trunk/CHANGES.txt
 U   zope.security/trunk/src/zope/security/_proxy.c

-=-
Modified: zope.security/trunk/CHANGES.txt
===================================================================
--- zope.security/trunk/CHANGES.txt	2009-08-11 21:11:26 UTC (rev 102690)
+++ zope.security/trunk/CHANGES.txt	2009-08-11 21:32:45 UTC (rev 102691)
@@ -5,7 +5,13 @@
3.7.1 (unreleased)
------------------

-- TBD
+- Fix for LP bug 181833 (from Gustavo Niemeyer). Before "visiting" a
+  sub-object, a check should be made to ensure the object is still valid.
+  Because garbage collection may involve loops, if you garbage collect an
+  object, it is possible that the actions done on this object may modify the
+  state of other objects. This may cause another round of garbage collection,
+  eventually generating a segfault (see LP bug). The Py_VISIT macro does the
+  necessary checks, so it is used instead of the previous code.

3.7.0 (2009-05-13)
------------------
@@ -17,7 +23,7 @@
3.6.3 (2009-03-23)
------------------

-- Ensure that simple zope.schema's VocabularyRegistry is used for 
+- Ensure that simple zope.schema's VocabularyRegistry is used for
  PermissionVocabulary tests, because it's replaced implicitly in
  environments with zope.app.schema installed that makes that tests
  fail.
@@ -65,14 +71,14 @@

- Don't define security checkers for deprecated set types from the
  "sets" module on Python 2.6. It's discouraged to use them and
-  `set` and `frozenset` built-in types should be used instead. 
+  `set` and `frozenset` built-in types should be used instead.

- Change package's mailng list address to zope-dev at zope.org as
  zope3-dev at zope.org is now retired.

- Remove old zpkg-related files.

-3.6.0 (2009-01-31) 
+3.6.0 (2009-01-31)
------------------

- Install decorated security checker support on LocationProxy from the

Modified: zope.security/trunk/src/zope/security/_proxy.c
===================================================================
--- zope.security/trunk/src/zope/security/_proxy.c	2009-08-11 21:11:26 UTC (rev 102690)
+++ zope.security/trunk/src/zope/security/_proxy.c	2009-08-11 21:32:45 UTC (rev 102691)
@@ -302,10 +302,8 @@
static int
proxy_traverse(SecurityProxy *self, visitproc visit, void *arg)
{
-  if (visit(self->proxy.proxy_object, arg) < 0)
-    return -1;
-  if (visit(self->proxy_checker, arg) < 0)
-    return -1;
+  Py_VISIT(self->proxy.proxy_object);
+  Py_VISIT(self->proxy_checker);
  return 0;
}

#############################################
zope.app.publication (changes to trunk shown)
#############################################

log problems that cause an abort to fail.

Changed:
 U   zope.app.publication/trunk/CHANGES.txt
 U   zope.app.publication/trunk/src/zope/app/publication/tests/test_zopepublication.py
 U   zope.app.publication/trunk/src/zope/app/publication/zopepublication.py

-=-
Modified: zope.app.publication/trunk/CHANGES.txt
===================================================================
--- zope.app.publication/trunk/CHANGES.txt	2009-08-11 21:32:45 UTC (rev 102691)
+++ zope.app.publication/trunk/CHANGES.txt	2009-08-11 22:34:23 UTC (rev 102692)
@@ -2,10 +2,16 @@
CHANGES
=======

+3.8.2 (unreleased)
+------------------
+
+- An abort within handleExceptions could have failed without logging what
+  caused the error. It now logs the original problem.
+
3.8.1 (2009-06-21)
------------------

-- Bug fix: The publication traverseName methoid used ProxyFactory
+- Bug fix: The publication traverseName method used ProxyFactory
  rather than the publication proxy method.

3.8.0 (2009-06-20)

Modified: zope.app.publication/trunk/src/zope/app/publication/tests/test_zopepublication.py
===================================================================
--- zope.app.publication/trunk/src/zope/app/publication/tests/test_zopepublication.py	2009-08-11 21:32:45 UTC (rev 102691)
+++ zope.app.publication/trunk/src/zope/app/publication/tests/test_zopepublication.py	2009-08-11 22:34:23 UTC (rev 102692)
@@ -414,7 +414,46 @@
        self.assert_(isinstance(error_info[1], FooError))
        self.assert_(request is self.request)

+    def testLogBeforeAbort(self):
+        # If we get an exception, and then (a catastrophe, but one that has
+        # been experienced) transaction.abort fails, we really want to know
+        # what happened before that abort.
+        # (Set up:)
+        zope.component.provideUtility(ErrorReportingUtility())
+        abort = transaction.abort
+        class AbortError(Exception):
+            pass
+        class AnEarlierError(Exception):
+            pass
+        def faux_abort():
+            raise AbortError
+        try:
+            raise AnEarlierError()
+        except AnEarlierError:
+            pass
+        transaction.abort = faux_abort
+        try:
+            # (Test:)
+            try:
+                self.publication.handleException(
+                    self.object, self.request, sys.exc_info(),
+                    retry_allowed=False)
+            except AbortError:
+                pass
+            else:
+                self.fail('Aborting should have failed')
+            # we expect a message in our logging utility
+            error_log = zope.component.getUtility(IErrorReportingUtility)
+            self.assertEqual(len(error_log.exceptions), 1)
+            error_info, request = error_log.exceptions[0]
+            self.assertEqual(error_info[0], AnEarlierError)
+            self.failUnless(isinstance(error_info[1], AnEarlierError))
+            self.failUnless(request is self.request)
+        finally:
+            # (Tear down:)
+            transaction.abort = abort

+
class ZopePublicationTests(BasePublicationTests):

def testGlobalAuth(self):

Modified: zope.app.publication/trunk/src/zope/app/publication/zopepublication.py
===================================================================
--- zope.app.publication/trunk/src/zope/app/publication/zopepublication.py	2009-08-11 21:32:45 UTC (rev 102691)
+++ zope.app.publication/trunk/src/zope/app/publication/zopepublication.py	2009-08-11 22:34:23 UTC (rev 102692)
@@ -285,7 +285,12 @@
    def handleException(self, object, request, exc_info, retry_allowed=True):
        # This transaction had an exception that reached the publisher.
        # It must definitely be aborted.
-        transaction.abort()
+        try:
+            transaction.abort()
+        except:
+            # Hm, a catastrophe.  We might want to know what preceded it.
+            self._logErrorWithErrorReportingUtility(object, request, exc_info)
+            raise

# Reraise Retry exceptions for the publisher to deal with.
        if retry_allowed and isinstance(exc_info[1], Retry):
@@ -305,7 +310,7 @@
        # handling determine whether a retry is allowed or not?
        # Assume not for now.

-        # Record the error with the ErrorReportingUtility
+        # Record the error with the ErrorReportingUtility.
        self._logErrorWithErrorReportingUtility(object, request, exc_info)

response = request.response

Revision history for this message

Francis J. Lacoste (flacoste) wrote on 2009-08-18:

Download full text (5.3 KiB)

On August 14, 2009, Gary Poster wrote:
> Gary Poster has proposed merging lp:~gary/launchpad/zbuildout into
> lp:launchpad.
>
> Requested reviews:
> Francis J. Lacoste (flacoste)
>
> This branch changes us to use source distributions for our zope-related
> dependencies.
>
> Because we still are using some packages in sourcecode that want to be
> egg-based distributions--particularly Twisted--we filter out pkg_resources
> warning in _pythonpath and other scripts. This prevents some test failures
> that do not want to see any output from running a command.
>
> There are many other changes, of course.
>
> - Many are driven by changes in reprs of various objects--validation errors
> (LaunchpadValidationError) and ClientForm response wrappers
> (httperror_seek_wrapper) in particular.
>
> - Others are for API changes, like in zope.testing
> (lib/canonical/testing/__init__.py and ).
>
> - I commented out the retry tests, as we have discussed.
>
> - Some tests expected zope files to be in the tree, so I had to switch to
> the pkg_resources API, or simply use more proper zcml in some cases.
>
> - There's a case or two of tokens needing to be converted from unicode to
> binary strings before they go into the URL, as with the
> lp:~gary/canonical-identity-provider/zbuildout branch that accompanies
> this one.
>
> - zope.schema set fields no longer support sets.Set, only the built-in set,
> which also resulted in some changes.
>
> - The transaction.begin API now does not expect to encounter an in-progress
> transaction (lib/canonical/launchpad/doc/storm.txt and
> lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt).
>
> - I found a couple of tests that failed in isolation. Related changes are
> in lib/canonical/launchpad/doc/batch_navigation.txt and
> lib/canonical/launchpad/scripts/ftests/test_oops_prune.py .
>
> - I'm not entirely sure how
>
> lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape
>.txt ever passed before. The doctest syntax there was wrong.
>
> - I switched back to the feedvalidator solution that we had discussed way
> back when, rather than using the hacked version. I left a comment to
> describe the situation.
>
> - Some tests did a browser reload. That now has a different behavior,
> resubmitting forms and so on, so I reload the same URL instead.
>
> - I did a fly-by on
> lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt because that was
> a [testfix] problem I diagnosed yesterday, and using queryMultiAdapter hid
> the problem.
>
> versions.cfg comments out version numbers only when my changes represent a
> *previous* version that what was specified before. Happy to remove those
> as well.
>
> For this branch, I made changes and releases in the following packages:
>
> - zope.testing (modified trunk, released and used as zope.testing 3.8.1)
> - zope.security (modified trunk, released and used as zope.security 3.7.1)
> - zope.app.publication (modified trunk, eventually to be 3.8.2; also
> modified 3.4.3, released and used as zope.app.publication 3.4.4)
>
> Diffs to the dependencies are below. For zope.testing and zope.security, I
> only show the commit that ...

On August 14, 2009, Gary Poster wrote:
> Gary Poster has proposed merging lp:~gary/launchpad/zbuildout into
> lp:launchpad.
>
> Requested reviews:
>     Francis J. Lacoste (flacoste)
>
> This branch changes us to use source distributions for our zope-related
> dependencies.
>
> Because we still are using some packages in sourcecode that want to be
> egg-based distributions--particularly Twisted--we filter out pkg_resources
> warning in _pythonpath and other scripts.  This prevents some test failures
> that do not want to see any output from running a command.
>
> There are many other changes, of course.
>
> - Many are driven by changes in reprs of various objects--validation errors
>   (LaunchpadValidationError) and ClientForm response wrappers
>   (httperror_seek_wrapper) in particular.
>
> - Others are for API changes, like in zope.testing
>   (lib/canonical/testing/__init__.py and ).
>
> - I commented out the retry tests, as we have discussed.
>
> - Some tests expected zope files to be in the tree, so I had to switch to
> the pkg_resources API, or simply use more proper zcml in some cases.
>
> - There's a case or two of tokens needing to be converted from unicode to
>   binary strings before they go into the URL, as with the
>   lp:~gary/canonical-identity-provider/zbuildout branch that accompanies
> this one.
>
> - zope.schema set fields no longer support sets.Set, only the built-in set,
>   which also resulted in some changes.
>
> - The transaction.begin API now does not expect to encounter an in-progress
>   transaction (lib/canonical/launchpad/doc/storm.txt and
>   lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt).
>
> - I found a couple of tests that failed in isolation.  Related changes are
> in lib/canonical/launchpad/doc/batch_navigation.txt and
>   lib/canonical/launchpad/scripts/ftests/test_oops_prune.py .
>
> - I'm not entirely sure how
>  
> lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape
>.txt ever passed before.  The doctest syntax there was wrong.
>
> - I switched back to the feedvalidator solution that we had discussed way
> back when, rather than using the hacked version.  I left a comment to
> describe the situation.
>
> - Some tests did a browser reload.  That now has a different behavior,
>   resubmitting forms and so on, so I reload the same URL instead.
>
> - I did a fly-by on
> lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt because that was
> a [testfix] problem I diagnosed yesterday, and using queryMultiAdapter hid
> the problem.
>
> versions.cfg comments out version numbers only when my changes represent a
> *previous* version that what was specified before.  Happy to remove those
> as well.
>
> For this branch, I made changes and releases in the following packages:
>
> - zope.testing (modified trunk, released and used as zope.testing 3.8.1)
> - zope.security (modified trunk, released and used as zope.security 3.7.1)
> - zope.app.publication (modified trunk, eventually to be 3.8.2; also
> modified 3.4.3, released and used as zope.app.publication 3.4.4)
>
> Diffs to the dependencies are below.  For zope.testing and zope.security, I
> only show the commit that made the substantive changes, rather than the
> commits generated by the release process.  zope.testing has the biggest
> changes from our branch because it significantly changed how it handled
> subprocesses from the version we were using in order to support being able
> to run layers in multiple processes, taking advantage of multiple cores.
>
> ############
> zope.testing
> ############
>
> Log message for revision 102634:
>  reduce batching and improve output for subprocesses
>

>
> Modified: zope.testing/trunk/src/zope/testing/testrunner/formatter.py
> ===================================================================
> --- zope.testing/trunk/src/zope/testing/testrunner/formatter.py	2009-08-10
> 18:23:10 UTC (rev 102633) +++
> zope.testing/trunk/src/zope/testing/testrunner/formatter.py	2009-08-10
> 20:57:09 UTC (rev 102634) @@ -52,6 +52,9 @@
>
>     progress = property(lambda self: self.options.progress)
>     verbose = property(lambda self: self.options.verbose)
> +    in_subprocess = property(
> +        lambda self: self.options.resume_layer is not None and
> +                     self.options.processes > 1)
>
>     def compute_max_width(self):
>         """Try to determine the terminal width."""
> @@ -263,6 +266,12 @@
>
>         elif self.verbose == 1:
>             sys.stdout.write('.' * test.countTestCases())
> +
> +        elif self.in_subprocess:
> +            sys.stdout.write('.' * test.countTestCases())
> +            # Give the parent process a new line so it sees the progress
> +            # in a timely manner.
> +            sys.stdout.write('\n')

Why not use sys.stdout.flush() here instead of using a newline?

> Added:
> zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt
> =================================================================== ---
> zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers-buff.txt	 
>                      (rev 0) +++

> +
> +Now we actually check the results we care about.  We should see that there
> are +two pauses of about half a second, one around the first test and one
> around the +second.  Before the change that this test verfies, there was a

'verifies'

Nice tests btw!

-- 
Francis J. Lacoste
francis.lacoste@canonical.com

Revision history for this message

Francis J. Lacoste (flacoste) wrote on 2009-08-18:

Download full text (5.9 KiB)

Hi Gary,

OK, I managed to go through the changes. I have a bunch of trival comment,
nothing that should prevent you from merging this.

Finally! Good work!

status approved
review approve

> === modified file 'buildout-templates/_pythonpath.py.in'
> --- buildout-templates/_pythonpath.py.in 2009-06-24 20:22:29 +0000
> +++ buildout-templates/_pythonpath.py.in 2009-08-18 21:42:18 +0000
> @@ -11,3 +11,10 @@
> sys.path[0:0] = [${string-paths}]
> # Enable Storm's C extensions
> os.environ['STORM_CEXTENSIONS'] = '1'
> +
> +# we don't want to bother tests or logs with these.

Capitalization.

> +import warnings
> +warnings.filterwarnings(
> + 'ignore',
> + 'Module .+ was already imported from .+, but .+ is being added.*',
> + UserWarning)
> \ No newline at end of file

Missing newline.

> === modified file 'buildout.cfg'
> --- buildout.cfg 2009-08-05 03:30:57 +0000
> +++ buildout.cfg 2009-08-18 21:42:18 +0000
> @@ -56,6 +56,12 @@
> initialization = import os
> os.environ['STORM_CEXTENSIONS'] = '1'
> os.environ.setdefault('LPCONFIG',
'${configuration:instance_name}')
> + # this can hopefully be removed when Twisted is used as an
egg.

This should be an XXX pointing to a bug about using twisted as an egg. (And
please fix the capitalization of the comment.)

> === modified file 'lib/canonical/config/__init__.py'
> @@ -194,8 +195,10 @@
>
> def _setZConfig(self):
> """Modify the config, adding automatically generated settings"""
> - schemafile = os.path.join(
> - self.root, 'lib/zope/app/server/schema.xml')
> + self.root = TREE_ROOT

Why did you need to add that line back? I mean self.root should already be
defined properly.

> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'

Instead of commenting everything here, why didn't you simply comment out the
line adding the MockDBTestCase to the suite in suite() ? I think this would
basically have the same effect. (With a comment explaining why this is
disabled.

> === modified file 'lib/canonical/testing/tests/test_mockdb.py'

Same kind of comments here.

> === modified file 'setup.py'
> --- setup.py 2009-07-24 11:14:47 +0000
> +++ setup.py 2009-08-18 21:42:18 +0000
> @@ -20,26 +20,78 @@
> maintainer='Launchpad Developers',
> description=('A unique collaboration and Bazaar code hosting platform '
> 'for software projects.'),
> - license='LGPL v3',
> + license='Affero GPL v3',
> + # this list should only contain direct dependencies--things imported or
> + # used in zcml.
> install_requires=[
> 'bzr',
> + 'chameleon.core',
> + 'chameleon.zpt',
> 'feedvalidator',
> 'funkload',
> 'launchpadlib',
> 'lazr.smtptest',
> 'lazr.uri',
> + 'mechanize',

Do we require mechanize? Isn't this an indirect dependency of
zope.testbrowser?

> > 'mocker',
> > 'oauth',
> > 'python-openid',
> 'pytz',
> + # This appears to be a broken indirect dependency from
zope.security:
> + 'RestrictedPython',
> 'setuptools',
> 'sour...

Hi Gary,

OK, I managed to go through the changes. I have a bunch of trival comment,
nothing that should prevent you from merging this.

Finally! Good work!

status approved
  review approve

> === modified file 'buildout-templates/_pythonpath.py.in'
> --- buildout-templates/_pythonpath.py.in	2009-06-24 20:22:29 +0000
> +++ buildout-templates/_pythonpath.py.in	2009-08-18 21:42:18 +0000
> @@ -11,3 +11,10 @@
>  sys.path[0:0] = [${string-paths}]
>  # Enable Storm's C extensions
>  os.environ['STORM_CEXTENSIONS'] = '1'
> +
> +# we don't want to bother tests or logs with these.

Capitalization.

> +import warnings
> +warnings.filterwarnings(
> +    'ignore',
> +    'Module .+ was already imported from .+, but .+ is being added.*',
> +    UserWarning)
> \ No newline at end of file

Missing newline.

> === modified file 'buildout.cfg'
> --- buildout.cfg	2009-08-05 03:30:57 +0000
> +++ buildout.cfg	2009-08-18 21:42:18 +0000
> @@ -56,6 +56,12 @@
>  initialization = import os
>                   os.environ['STORM_CEXTENSIONS'] = '1'
>                   os.environ.setdefault('LPCONFIG', 
'${configuration:instance_name}')
> +                 # this can hopefully be removed when Twisted is used as an 
egg.

This should be an XXX pointing to a bug about using twisted as an egg. (And
please fix the capitalization of the comment.)

> === modified file 'lib/canonical/config/__init__.py'
> @@ -194,8 +195,10 @@
>  
>      def _setZConfig(self):
>          """Modify the config, adding automatically generated settings"""
> -        schemafile = os.path.join(
> -            self.root, 'lib/zope/app/server/schema.xml')
> +        self.root = TREE_ROOT

Why did you need to add that line back? I mean self.root should already be
defined properly.

> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'

> === modified file 'lib/canonical/testing/tests/test_mockdb.py'

Same kind of comments here.

> === modified file 'setup.py'
> --- setup.py	2009-07-24 11:14:47 +0000
> +++ setup.py	2009-08-18 21:42:18 +0000
> @@ -20,26 +20,78 @@
>      maintainer='Launchpad Developers',
>      description=('A unique collaboration and Bazaar code hosting platform '
>                   'for software projects.'),
> -    license='LGPL v3',
> +    license='Affero GPL v3',
> +    # this list should only contain direct dependencies--things imported or
> +    # used in zcml.
>      install_requires=[
>          'bzr',
> +        'chameleon.core',
> +        'chameleon.zpt',
>          'feedvalidator',
>          'funkload',
>          'launchpadlib',
>          'lazr.smtptest',
>          'lazr.uri',
> +        'mechanize',

Do we require mechanize? Isn't this an indirect dependency of
zope.testbrowser?

> >          'mocker',
> >          'oauth',
> >          'python-openid',
>          'pytz',
> +        # This appears to be a broken indirect dependency from 
zope.security:
> +        'RestrictedPython',
>          'setuptools',
>          'sourcecodegen',
>          'storm',
> -        'chameleon.core',
> -        'chameleon.zpt',
> +        'transaction',
> +        'wadllib',
>          'z3c.pt',
>          'z3c.ptcompat',
> -        'wadllib',
> +        'zc.zservertracelog',
> +        'zope.app.appsetup',
> +        'zope.app.component',
> +        'zope.app.dav', # ./package-includes/dav-configure.zcml

Really!?!

> +        'zope.app.error',
> +        'zope.app.exception',
> +        'zope.app.file',
> +        'zope.app.form',
> +        'zope.app.pagetemplate',
> +        'zope.app.pluggableauth',
> +        'zope.app.publication',
> +        'zope.app.publisher',
> +        'zope.app.security',
> +        'zope.app.securitypolicy',
> +        'zope.app.server',
> +        'zope.app.session',
> +        'zope.app.testing',
> +        'zope.app.wsgi',
> +        'zope.app.zapi',
> +        'zope.contenttype',

We use that directly?

> +        'zope.component[zcml]',
> +        'zope.datetime',
> +        'zope.thread',

And this?

> +        'zope.error',
> +        'zope.event',
> +        'zope.exceptions',
> +        'zope.formlib',
> +        'zope.i18n',
> +        'zope.interface',
> +        'zope.hookable', # indirect, via zope.app.component

> +        'zope.lifecycleevent',
> +        'zope.location',
> +        'zope.pagetemplate',
> +        'zope.publisher',
> +        'zope.proxy',
> +        'zope.schema',
> +        'zope.security',
> +        'zope.sendmail',
> +        'zope.server',
> +        'zope.session',
> +        'zope.tal',
> +        'zope.tales',
> +        'zope.testbrowser',
> +        'zope.testing',
> +        'zope.traversing',
> +        'zope.viewlet', # only fixing a broken dependency

Isn't possible to trim that list a little, to contain only what we actually
requires and let the rest be pulled in automatically?

> === modified file 'versions.cfg'
> --- versions.cfg	2009-08-12 12:35:51 +0000
> +++ versions.cfg	2009-08-18 21:42:18 +0000
> @@ -2,17 +2,20 @@
>  versions = versions
>  
>  [versions]
> -# Alphabetical, please! :-)
> +# Alphabetical, case-insensitive, please! :-)
>  bzr = 1.17
>  chameleon.core = 1.0b35
>  chameleon.zpt = 1.0b17
> +ClientForm = 0.2.10
>  # Required by Windmill to run on 2.4
>  ctypes = 1.0.2
>  docutils = 0.5
>  elementtree = 1.2.6-20050316
> -# We use a version of feedvalidator that has been changed to not
> -# change the default socket timeout on import.
> -feedvalidator = 0.0.0DEV-r1049-hacked
> +feedvalidator = 0.0.0DEV-r1049
> +# We could use a locally hacked version of feedvalidator that has been 
changed
> +# to not change the default socket timeout on import.  We are currently 
handling
> +# that problem elsewhere.
> +# feedvalidator = 0.0.0DEV-r1049-hacked

Why not simply remove the comments and line altogether?

review: Approve

Revision history for this message

Gary Poster (gary) wrote on 2009-08-21:

Download full text (6.2 KiB)

On Aug 18, 2009, at 4:57 PM, Francis J. Lacoste wrote:

On Aug 18, 2009, at 4:57 PM, Francis J. Lacoste wrote:

> On August 14, 2009, Gary Poster wrote:
>> Gary Poster has proposed merging lp:~gary/launchpad/zbuildout into
>> lp:launchpad.
>>
>> Requested reviews:
>>    Francis J. Lacoste (flacoste)
>>
>> This branch changes us to use source distributions for our zope- 
>> related
>> dependencies.
>>
>> Because we still are using some packages in sourcecode that want to  
>> be
>> egg-based distributions--particularly Twisted--we filter out  
>> pkg_resources
>> warning in _pythonpath and other scripts.  This prevents some test  
>> failures
>> that do not want to see any output from running a command.
>>
>> There are many other changes, of course.
>>
>> - Many are driven by changes in reprs of various objects-- 
>> validation errors
>>  (LaunchpadValidationError) and ClientForm response wrappers
>>  (httperror_seek_wrapper) in particular.
>>
>> - Others are for API changes, like in zope.testing
>>  (lib/canonical/testing/__init__.py and ).
>>
>> - I commented out the retry tests, as we have discussed.
>>
>> - Some tests expected zope files to be in the tree, so I had to  
>> switch to
>> the pkg_resources API, or simply use more proper zcml in some cases.
>>
>> - There's a case or two of tokens needing to be converted from  
>> unicode to
>>  binary strings before they go into the URL, as with the
>>  lp:~gary/canonical-identity-provider/zbuildout branch that  
>> accompanies
>> this one.
>>
>> - zope.schema set fields no longer support sets.Set, only the built- 
>> in set,
>>  which also resulted in some changes.
>>
>> - The transaction.begin API now does not expect to encounter an in- 
>> progress
>>  transaction (lib/canonical/launchpad/doc/storm.txt and
>>  lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt).
>>
>> - I found a couple of tests that failed in isolation.  Related  
>> changes are
>> in lib/canonical/launchpad/doc/batch_navigation.txt and
>>  lib/canonical/launchpad/scripts/ftests/test_oops_prune.py .
>>
>> - I'm not entirely sure how
>>
>> lib/lp/translations/stories/standalone/xx-pofile-translate-html- 
>> tags-escape
>> .txt ever passed before.  The doctest syntax there was wrong.
>>
>> - I switched back to the feedvalidator solution that we had  
>> discussed way
>> back when, rather than using the hacked version.  I left a comment to
>> describe the situation.
>>
>> - Some tests did a browser reload.  That now has a different  
>> behavior,
>>  resubmitting forms and so on, so I reload the same URL instead.
>>
>> - I did a fly-by on
>> lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt because  
>> that was
>> a [testfix] problem I diagnosed yesterday, and using  
>> queryMultiAdapter hid
>> the problem.
>>
>> versions.cfg comments out version numbers only when my changes  
>> represent a
>> *previous* version that what was specified before.  Happy to remove  
>> those
>> as well.
>>
>> For this branch, I made changes and releases in the following  
>> packages:
>>
>> - zope.testing (modified trunk, released and used as zope.testing  
>> 3.8.1)
>> - zope.security (modified trunk, released and used as zope.security  
>> 3.7.1)
>> - zope.app.publication (modified trunk, eventually to be 3.8.2; also
>> modified 3.4.3, released and used as zope.app.publication 3.4.4)
>>
>> Diffs to the dependencies are below.  For zope.testing and  
>> zope.security, I
>> only show the commit that made the substantive changes, rather than  
>> the
>> commits generated by the release process.  zope.testing has the  
>> biggest
>> changes from our branch because it significantly changed how it  
>> handled
>> subprocesses from the version we were using in order to support  
>> being able
>> to run layers in multiple processes, taking advantage of multiple  
>> cores.
>>
>> ############
>> zope.testing
>> ############
>>
>> Log message for revision 102634:
>> reduce batching and improve output for subprocesses
>>
>
>>
>> Modified: zope.testing/trunk/src/zope/testing/testrunner/formatter.py
>> ===================================================================
>> --- zope.testing/trunk/src/zope/testing/testrunner/formatter.py	 
>> 2009-08-10
>> 18:23:10 UTC (rev 102633) +++
>> zope.testing/trunk/src/zope/testing/testrunner/formatter.py	 
>> 2009-08-10
>> 20:57:09 UTC (rev 102634) @@ -52,6 +52,9 @@
>>
>>    progress = property(lambda self: self.options.progress)
>>    verbose = property(lambda self: self.options.verbose)
>> +    in_subprocess = property(
>> +        lambda self: self.options.resume_layer is not None and
>> +                     self.options.processes > 1)
>>
>>    def compute_max_width(self):
>>        """Try to determine the terminal width."""
>> @@ -263,6 +266,12 @@
>>
>>        elif self.verbose == 1:
>>            sys.stdout.write('.' * test.countTestCases())
>> +
>> +        elif self.in_subprocess:
>> +            sys.stdout.write('.' * test.countTestCases())
>> +            # Give the parent process a new line so it sees the  
>> progress
>> +            # in a timely manner.
>> +            sys.stdout.write('\n')
>
> Why not use sys.stdout.flush() here instead of using a newline?

Because this feeds into the same new code in the parent process that  
now has the readline behavior.  I don't think a flush is going to help  
if the listener is doing a readline.  Maybe I could change the  
listener behavior for this case so that it no longer does a readline,  
and then I could do a flush here, but just sending a newline seemed  
like a nice one-line fix with no downside that I saw.

>> Added:
>> zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers- 
>> buff.txt
>> ===================================================================  
>> ---
>> zope.testing/trunk/src/zope/testing/testrunner/testrunner-layers- 
>> buff.txt	
>>                     (rev 0) +++
>
>> +
>> +Now we actually check the results we care about.  We should see  
>> that there
>> are +two pauses of about half a second, one around the first test  
>> and one
>> around the +second.  Before the change that this test verfies,  
>> there was a
>
> 'verifies'

Ah, good catch. :-)  Changed in the packages trunk (though I'm not  
bothering to make a new release :-) ).

> Nice tests btw!

Thanks.  Still a chance that they will have intermittent failures, but  
maybe I'll be lucky ;-)

Gary

Revision history for this message

Francis J. Lacoste (flacoste) wrote on 2009-08-21:

On August 21, 2009, Gary Poster wrote:
> > Why not use sys.stdout.flush() here instead of using a newline?
>
> Because this feeds into the same new code in the parent process that
> now has the readline behavior. I don't think a flush is going to help
> if the listener is doing a readline. Maybe I could change the
> listener behavior for this case so that it no longer does a readline,
> and then I could do a flush here, but just sending a newline seemed
> like a nice one-line fix with no downside that I saw.

That's fine.

--
Francis J. Lacoste
<email address hidden>

Revision history for this message

Gary Poster (gary) wrote on 2009-08-21:

Download full text (9.3 KiB)

On Aug 18, 2009, at 5:57 PM, Francis J. Lacoste wrote:

> Review: Approve
> Hi Gary,
>
> OK, I managed to go through the changes. I have a bunch of trival
> comment,
> nothing that should prevent you from merging this.
>
> Finally! Good work!

Thanks!

Responses below. I deleted the sections that I simply changed as you
suggested.

...

>> === modified file 'lib/canonical/config/__init__.py'
>> @@ -194,8 +195,10 @@
>>
>> def _setZConfig(self):
>> """Modify the config, adding automatically generated
>> settings"""
>> - schemafile = os.path.join(
>> - self.root, 'lib/zope/app/server/schema.xml')
>> + self.root = TREE_ROOT
>
> Why did you need to add that line back? I mean self.root should
> already be
> defined properly.

I *think* this must be a conflict resolution mistake of mine. I
removed the line again, and ec2test will tell me whether it is OK.

>
>> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'
>
> Instead of commenting everything here, why didn't you simply comment
> out the
> line adding the MockDBTestCase to the suite in suite() ? I think
> this would
> basically have the same effect. (With a comment explaining why this is
> disabled.

The reason why was that there were many tests that were still able to
run. All of the ones that do not rely on the retry stuff still are
registered. Is that not worthwhile for some reason?

>> === modified file 'lib/canonical/testing/tests/test_mockdb.py'
>
> Same kind of comments here.

Likewise. :-)

>> === modified file 'setup.py'
>> --- setup.py 2009-07-24 11:14:47 +0000
>> +++ setup.py 2009-08-18 21:42:18 +0000
>> @@ -20,26 +20,78 @@
>> maintainer='Launchpad Developers',
>> description=('A unique collaboration and Bazaar code hosting
>> platform '
>> 'for software projects.'),
>> - license='LGPL v3',
>> + license='Affero GPL v3',
>> + # this list should only contain direct dependencies--things
>> imported or
>> + # used in zcml.
>> install_requires=[
>> 'bzr',
>> + 'chameleon.core',
>> + 'chameleon.zpt',
>> 'feedvalidator',
>> 'funkload',
>> 'launchpadlib',
>> 'lazr.smtptest',
>> 'lazr.uri',
>> + 'mechanize',
>
> Do we require mechanize? Isn't this an indirect dependency of
> zope.testbrowser?

We have it here because of testbrowser, but we import it directly.

Here and in the other usage lists below I show you the uses I found
quickly today, reviewing your questions. There may be more.

lib/lp/code/browser/tests/test_product.py:11:from mechanize import
LinkNotFoundError
./lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-
redirection.txt:137: >>> from mechanize._clientcookie import Cookie
./lib/lp/translations/stories/standalone/xx-sourcepackage-export.txt:
31: >>> from mechanize import LinkNotFoundError
./lib/lp/soyuz/stories/ppa/xx-ppa-files.txt:127: >>> from mechanize
import LinkNotFoundError

>
>>> 'mocker',
>>> 'oauth',
>>> 'python-openid',
>> 'pytz',
>> + # This appears to be a broken indirect dependency from
> zope.security:
>> + ...

On Aug 18, 2009, at 5:57 PM, Francis J. Lacoste wrote:

> Review: Approve
> Hi Gary,
>
> OK, I managed to go through the changes. I have a bunch of trival  
> comment,
> nothing that should prevent you from merging this.
>
> Finally! Good work!

Thanks!

Responses below.  I deleted the sections that I simply changed as you  
suggested.

...

>> === modified file 'lib/canonical/config/__init__.py'
>> @@ -194,8 +195,10 @@
>>
>>     def _setZConfig(self):
>>         """Modify the config, adding automatically generated  
>> settings"""
>> -        schemafile = os.path.join(
>> -            self.root, 'lib/zope/app/server/schema.xml')
>> +        self.root = TREE_ROOT
>
> Why did you need to add that line back? I mean self.root should  
> already be
> defined properly.

I *think* this must be a conflict resolution mistake of mine.  I  
removed the line again, and ec2test will tell me whether it is OK.

>
>> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'
>
> Instead of commenting everything here, why didn't you simply comment  
> out the
> line adding the MockDBTestCase to the suite in suite() ? I think  
> this would
> basically have the same effect. (With a comment explaining why this is
> disabled.

The reason why was that there were many tests that were still able to  
run.  All of the ones that do not rely on the retry stuff still are  
registered.  Is that not worthwhile for some reason?

>> === modified file 'lib/canonical/testing/tests/test_mockdb.py'
>
> Same kind of comments here.

Likewise. :-)

>> === modified file 'setup.py'
>> --- setup.py	2009-07-24 11:14:47 +0000
>> +++ setup.py	2009-08-18 21:42:18 +0000
>> @@ -20,26 +20,78 @@
>>     maintainer='Launchpad Developers',
>>     description=('A unique collaboration and Bazaar code hosting  
>> platform '
>>                  'for software projects.'),
>> -    license='LGPL v3',
>> +    license='Affero GPL v3',
>> +    # this list should only contain direct dependencies--things  
>> imported or
>> +    # used in zcml.
>>     install_requires=[
>>         'bzr',
>> +        'chameleon.core',
>> +        'chameleon.zpt',
>>         'feedvalidator',
>>         'funkload',
>>         'launchpadlib',
>>         'lazr.smtptest',
>>         'lazr.uri',
>> +        'mechanize',
>
> Do we require mechanize? Isn't this an indirect dependency of
> zope.testbrowser?

We have it here because of testbrowser, but we import it directly.

Here and in the other usage lists below I show you the uses I found  
quickly today, reviewing your questions.  There may be more.

lib/lp/code/browser/tests/test_product.py:11:from mechanize import  
LinkNotFoundError
./lib/canonical/launchpad/pagetests/standalone/xx-beta-testers- 
redirection.txt:137:    >>> from mechanize._clientcookie import Cookie
./lib/lp/translations/stories/standalone/xx-sourcepackage-export.txt: 
31:    >>> from mechanize import LinkNotFoundError
./lib/lp/soyuz/stories/ppa/xx-ppa-files.txt:127:    >>> from mechanize  
import LinkNotFoundError

>
>>>         'mocker',
>>>         'oauth',
>>>         'python-openid',
>>         'pytz',
>> +        # This appears to be a broken indirect dependency from
> zope.security:
>> +        'RestrictedPython',
>>         'setuptools',
>>         'sourcecodegen',
>>         'storm',
>> -        'chameleon.core',
>> -        'chameleon.zpt',
>> +        'transaction',
>> +        'wadllib',
>>         'z3c.pt',
>>         'z3c.ptcompat',
>> -        'wadllib',
>> +        'zc.zservertracelog',
>> +        'zope.app.appsetup',
>> +        'zope.app.component',
>> +        'zope.app.dav', # ./package-includes/dav-configure.zcml
>
> Really!?!

Yeah, see that dav-configure.zcml thing.  That's why the comment is  
there.  Do you want me to try removing that file and then removing the  
dependency?

Unless you are sure we can get rid of that zcml file, I think I'd  
prefer to make a bug for this, asking to investigate, and just leave  
things as they are.

>
>> +        'zope.app.error',
>> +        'zope.app.exception',
>> +        'zope.app.file',
>> +        'zope.app.form',
>> +        'zope.app.pagetemplate',
>> +        'zope.app.pluggableauth',
>> +        'zope.app.publication',
>> +        'zope.app.publisher',
>> +        'zope.app.security',
>> +        'zope.app.securitypolicy',
>> +        'zope.app.server',
>> +        'zope.app.session',
>> +        'zope.app.testing',
>> +        'zope.app.wsgi',
>> +        'zope.app.zapi',
>> +        'zope.contenttype',
>
> We use that directly?

Yeah, I tried to be pretty careful with these.

lib/canonical/launchpad/scripts/sftracker.py:45:from zope.contenttype  
import guess_content_type
lib/canonical/lazr/folder.py:17:from zope.contenttype import  
guess_content_type
lib/canonical/widgets/image.py:10:from zope.contenttype import  
guess_content_type

>
>> +        'zope.component[zcml]',
>> +        'zope.datetime',
>> +        'zope.thread',
>
> And this?

lib/canonical/launchpad/webapp/launchbag.py:16:from zope import thread

>> +        'zope.error',
>> +        'zope.event',
>> +        'zope.exceptions',
>> +        'zope.formlib',
>> +        'zope.i18n',
>> +        'zope.interface',
>> +        'zope.hookable', # indirect, via zope.app.component
>
>> +        'zope.lifecycleevent',
>> +        'zope.location',
>> +        'zope.pagetemplate',
>> +        'zope.publisher',
>> +        'zope.proxy',
>> +        'zope.schema',
>> +        'zope.security',
>> +        'zope.sendmail',
>> +        'zope.server',
>> +        'zope.session',
>> +        'zope.tal',
>> +        'zope.tales',
>> +        'zope.testbrowser',
>> +        'zope.testing',
>> +        'zope.traversing',
>> +        'zope.viewlet', # only fixing a broken dependency
>
> Isn't possible to trim that list a little, to contain only what we  
> actually
> requires and let the rest be pulled in automatically?

Like I said, unless there's a comment about broken dependencies, we're  
using it directly AFAIK.  I didn't review the list again, but I was  
careful at the time.

>> === modified file 'versions.cfg'
>> --- versions.cfg	2009-08-12 12:35:51 +0000
>> +++ versions.cfg	2009-08-18 21:42:18 +0000
>> @@ -2,17 +2,20 @@
>> versions = versions
>>
>> [versions]
>> -# Alphabetical, please! :-)
>> +# Alphabetical, case-insensitive, please! :-)
>> bzr = 1.17
>> chameleon.core = 1.0b35
>> chameleon.zpt = 1.0b17
>> +ClientForm = 0.2.10
>> # Required by Windmill to run on 2.4
>> ctypes = 1.0.2
>> docutils = 0.5
>> elementtree = 1.2.6-20050316
>> -# We use a version of feedvalidator that has been changed to not
>> -# change the default socket timeout on import.
>> -feedvalidator = 0.0.0DEV-r1049-hacked
>> +feedvalidator = 0.0.0DEV-r1049
>> +# We could use a locally hacked version of feedvalidator that has  
>> been
> changed
>> +# to not change the default socket timeout on import.  We are  
>> currently
> handling
>> +# that problem elsewhere.
>> +# feedvalidator = 0.0.0DEV-r1049-hacked
>
> Why not simply remove the comments and line altogether?

'Cause I didn't know if what I had done was a good idea. :-) I removed  
the comments now.

Here's the incremental diff so far.  Please let me know what you think  
about my replies to the other stuff, particularly the test_mockdb bits.

Thank you!

Gary

=== modified file 'buildout-templates/_pythonpath.py.in'
--- buildout-templates/_pythonpath.py.in	2009-08-07 13:16:30 +0000
+++ buildout-templates/_pythonpath.py.in	2009-08-21 17:55:58 +0000
@@ -12,9 +12,9 @@
  # Enable Storm's C extensions
  os.environ['STORM_CEXTENSIONS'] = '1'

-# we don't want to bother tests or logs with these.
+# We don't want to bother tests or logs with these.
  import warnings
  warnings.filterwarnings(
      'ignore',
      'Module .+ was already imported from .+, but .+ is being added.*',
-    UserWarning)
\ No newline at end of file
+    UserWarning)

=== modified file 'buildout.cfg'
--- buildout.cfg	2009-08-05 19:05:57 +0000
+++ buildout.cfg	2009-08-21 18:20:33 +0000
@@ -56,7 +56,8 @@
  initialization = import os
                   os.environ['STORM_CEXTENSIONS'] = '1'
                   os.environ.setdefault('LPCONFIG', '$ 
{configuration:instance_name}')
-                 # this can hopefully be removed when Twisted is used  
as an egg.
+                 # XXX 2009-08-21 gary bug 417077
+                 # This can hopefully be removed when Twisted is used  
as an egg.
                   import warnings
                   warnings.filterwarnings(
                   'ignore',

=== modified file 'lib/canonical/config/__init__.py'
--- lib/canonical/config/__init__.py	2009-08-05 18:52:52 +0000
+++ lib/canonical/config/__init__.py	2009-08-21 18:22:07 +0000
@@ -195,8 +195,6 @@

def _setZConfig(self):
          """Modify the config, adding automatically generated  
settings"""
-        self.root = TREE_ROOT
-
          schemafile = pkg_resources.resource_filename(
              'zope.app.server', 'schema.xml')
          schema = ZConfig.loadSchema(schemafile)

=== modified file 'versions.cfg'
--- versions.cfg	2009-08-21 17:55:25 +0000
+++ versions.cfg	2009-08-21 18:42:03 +0000
@@ -16,10 +12,6 @@
  docutils = 0.5
  elementtree = 1.2.6-20050316
  feedvalidator = 0.0.0DEV-r1049
-# We could use a locally hacked version of feedvalidator that has  
been changed
-# to not change the default socket timeout on import.  We are  
currently handling
-# that problem elsewhere.
-# feedvalidator = 0.0.0DEV-r1049-hacked
  functest = 0.8.7
  funkload = 1.10.0
  httplib2 = 0.4.0

Revision history for this message

Francis J. Lacoste (flacoste) wrote on 2009-08-21:

Download full text (3.4 KiB)

On August 21, 2009, Gary Poster wrote:
> On Aug 18, 2009, at 5:57 PM, Francis J. Lacoste wrote:

>
> >> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'
> >
> > Instead of commenting everything here, why didn't you simply comment
> > out the
> > line adding the MockDBTestCase to the suite in suite() ? I think
> > this would
> > basically have the same effect. (With a comment explaining why this is
> > disabled.
>
> The reason why was that there were many tests that were still able to
> run. All of the ones that do not rely on the retry stuff still are
> registered. Is that not worthwhile for some reason?

It is. So that's fine.

>
> >> === modified file 'setup.py'
> >> --- setup.py 2009-07-24 11:14:47 +0000
> >> +++ setup.py 2009-08-18 21:42:18 +0000
> >> @@ -20,26 +20,78 @@
> >> maintainer='Launchpad Developers',
> >> description=('A unique collaboration and Bazaar code hosting
> >> platform '
> >> 'for software projects.'),
> >> - license='LGPL v3',
> >> + license='Affero GPL v3',
> >> + # this list should only contain direct dependencies--things
> >> imported or
> >> + # used in zcml.
> >> install_requires=[
> >> 'bzr',
> >> + 'chameleon.core',
> >> + 'chameleon.zpt',
> >> 'feedvalidator',
> >> 'funkload',
> >> 'launchpadlib',
> >> 'lazr.smtptest',
> >> 'lazr.uri',
> >> + 'mechanize',
> >
> > Do we require mechanize? Isn't this an indirect dependency of
> > zope.testbrowser?
>
> We have it here because of testbrowser, but we import it directly.
>
> Here and in the other usage lists below I show you the uses I found
> quickly today, reviewing your questions. There may be more.
>
> lib/lp/code/browser/tests/test_product.py:11:from mechanize import
> LinkNotFoundError
> ./lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-
> redirection.txt:137: >>> from mechanize._clientcookie import Cookie
> ./lib/lp/translations/stories/standalone/xx-sourcepackage-export.txt:
> 31: >>> from mechanize import LinkNotFoundError
> ./lib/lp/soyuz/stories/ppa/xx-ppa-files.txt:127: >>> from mechanize
> import LinkNotFoundError
>
> >>> 'mocker',
> >>> 'oauth',
> >>> 'python-openid',
> >>
> >> 'pytz',
> >> + # This appears to be a broken indirect dependency from
> >
> > zope.security:
> >> + 'RestrictedPython',
> >> 'setuptools',
> >> 'sourcecodegen',
> >> 'storm',
> >> - 'chameleon.core',
> >> - 'chameleon.zpt',
> >> + 'transaction',
> >> + 'wadllib',
> >> 'z3c.pt',
> >> 'z3c.ptcompat',
> >> - 'wadllib',
> >> + 'zc.zservertracelog',
> >> + 'zope.app.appsetup',
> >> + 'zope.app.component',
> >> + 'zope.app.dav', # ./package-includes/dav-configure.zcml
> >
> > Really!?!
>
> Yeah, see that dav-configure.zcml thing. That's why the comment is
> there. Do you want me to try removing that file and then removing the
> dependency?
>
> Unless you are sure we can get rid of that zcml file, I think I'd
> prefer to make a bug for this, asking to investigate, and just leave
> things as...

On August 21, 2009, Gary Poster wrote:
> On Aug 18, 2009, at 5:57 PM, Francis J. Lacoste wrote:

>
> >> === modified file 'lib/canonical/testing/ftests/test_mockdb.py'
> >
> > Instead of commenting everything here, why didn't you simply comment
> > out the
> > line adding the MockDBTestCase to the suite in suite() ? I think
> > this would
> > basically have the same effect. (With a comment explaining why this is
> > disabled.
>
> The reason why was that there were many tests that were still able to
> run.  All of the ones that do not rely on the retry stuff still are
> registered.  Is that not worthwhile for some reason?

It is. So that's fine.

>
> >> === modified file 'setup.py'
> >> --- setup.py	2009-07-24 11:14:47 +0000
> >> +++ setup.py	2009-08-18 21:42:18 +0000
> >> @@ -20,26 +20,78 @@
> >>     maintainer='Launchpad Developers',
> >>     description=('A unique collaboration and Bazaar code hosting
> >> platform '
> >>                  'for software projects.'),
> >> -    license='LGPL v3',
> >> +    license='Affero GPL v3',
> >> +    # this list should only contain direct dependencies--things
> >> imported or
> >> +    # used in zcml.
> >>     install_requires=[
> >>         'bzr',
> >> +        'chameleon.core',
> >> +        'chameleon.zpt',
> >>         'feedvalidator',
> >>         'funkload',
> >>         'launchpadlib',
> >>         'lazr.smtptest',
> >>         'lazr.uri',
> >> +        'mechanize',
> >
> > Do we require mechanize? Isn't this an indirect dependency of
> > zope.testbrowser?
>
> We have it here because of testbrowser, but we import it directly.
>
> Here and in the other usage lists below I show you the uses I found
> quickly today, reviewing your questions.  There may be more.
>
> lib/lp/code/browser/tests/test_product.py:11:from mechanize import
> LinkNotFoundError
> ./lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-
> redirection.txt:137:    >>> from mechanize._clientcookie import Cookie
> ./lib/lp/translations/stories/standalone/xx-sourcepackage-export.txt:
> 31:    >>> from mechanize import LinkNotFoundError
> ./lib/lp/soyuz/stories/ppa/xx-ppa-files.txt:127:    >>> from mechanize
> import LinkNotFoundError
>
> >>>         'mocker',
> >>>         'oauth',
> >>>         'python-openid',
> >>
> >>         'pytz',
> >> +        # This appears to be a broken indirect dependency from
> >
> > zope.security:
> >> +        'RestrictedPython',
> >>         'setuptools',
> >>         'sourcecodegen',
> >>         'storm',
> >> -        'chameleon.core',
> >> -        'chameleon.zpt',
> >> +        'transaction',
> >> +        'wadllib',
> >>         'z3c.pt',
> >>         'z3c.ptcompat',
> >> -        'wadllib',
> >> +        'zc.zservertracelog',
> >> +        'zope.app.appsetup',
> >> +        'zope.app.component',
> >> +        'zope.app.dav', # ./package-includes/dav-configure.zcml
> >
> > Really!?!
>
> Yeah, see that dav-configure.zcml thing.  That's why the comment is
> there.  Do you want me to try removing that file and then removing the
> dependency?
>
> Unless you are sure we can get rid of that zcml file, I think I'd
> prefer to make a bug for this, asking to investigate, and just leave
> things as they are.
>

That's fine.

>
> Here's the incremental diff so far.  Please let me know what you think
> about my replies to the other stuff, particularly the test_mockdb bits.
>

I think you are good to go!

-- 
Francis J. Lacoste
francis.lacoste@canonical.com

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Celso Providelo

Colin Watson

Gary Poster

Launchpad code reviewers from Canonical

Mantavya Gajjar (Open ERP)

Stuart Bishop

William Grant

to status/vote changes:

Tomi Karjalainen

Launchpad itself

Merge lp:~gary/launchpad/zbuildout into lp:launchpad/db-devel

Commit message

Description of the change

Preview Diff

Subscribers

 === modified file 'buildout-templates/_pythonpath.py.in'
 --- buildout-templates/_pythonpath.py.in	2009-06-24 20:22:29 +0000
 +++ buildout-templates/_pythonpath.py.in	2009-08-07 13:16:30 +0000
@@ -11,3 +11,10 @@
  sys.path[0:0] = [${string-paths}]
  # Enable Storm's C extensions
  os.environ['STORM_CEXTENSIONS'] = '1'
++
++# we don't want to bother tests or logs with these.
++import warnings
++warnings.filterwarnings(
++    'ignore',
++    'Module .+ was already imported from .+, but .+ is being added.*',
++    UserWarning)
 \ No newline at end of file
 === modified file 'buildout-templates/bin/test.in'
 --- buildout-templates/bin/test.in	2009-07-30 13:26:13 +0000
 +++ buildout-templates/bin/test.in	2009-08-10 22:08:05 +0000
@@ -132,6 +132,7 @@
  pgsql.installFakeConnect()
  from zope.testing import testrunner
++from zope.testing.testrunner import options
  defaults = [
      # Find tests in the tests and ftests directories
@@ -188,24 +189,24 @@
      def load_list(option, opt_str, list_name, parser):
          patch_find_tests(filter_tests(list_name))
--    testrunner.parser.add_option(
++    options.parser.add_option(
          '--load-list', type=str, action='callback', callback=load_list)
      def list_test_option(option, opt, value, parser):
          patch_find_tests(list_tests)
--    testrunner.parser.add_option(
++    options.parser.add_option(
          '--list', action='callback', callback=list_test_option)
      def use_subunit(option, opt, value, parser):
          patch_zope_testresult(TestProtocolClient(sys.stdout))
--    testrunner.parser.add_option(
++    options.parser.add_option(
          '--subunit', action='callback', callback=use_subunit)
--    options = testrunner.get_options(args=args, defaults=defaults)
++    local_options = options.get_options(args=args, defaults=defaults)
      # Turn on Layer profiling if requested.
      from canonical.testing import profiled
--    if options.verbose >= 3 and main_process:
++    if local_options.verbose >= 3 and main_process:
          profiled.setup_profiling()
      # The working directory change is just so that the test script
@@ -222,6 +223,6 @@
      logging.disable(999999999)
      # Print Layer profiling report if requested.
--    if main_process and options.verbose >= 3:
++    if main_process and local_options.verbose >= 3:
          profiled.report_profile_stats()
      sys.exit(result)
 === modified file 'buildout.cfg'
 --- buildout.cfg	2009-08-05 03:30:57 +0000
 +++ buildout.cfg	2009-08-05 19:05:57 +0000
@@ -56,6 +56,12 @@
  initialization = import os
                   os.environ['STORM_CEXTENSIONS'] = '1'
                   os.environ.setdefault('LPCONFIG', '${configuration:instance_name}')
++                 # this can hopefully be removed when Twisted is used as an egg.
++                 import warnings
++                 warnings.filterwarnings(
++                 'ignore',
++                 'Module .+ was already imported from .+, but .+ is being added.*',
++                 UserWarning)
  entry-points = stxdocs=zope.configuration.stxdocs:main
      googletestservice=canonical.launchpad.testing.googletestservice:main
      windmill=windmill.bin.windmill_bin:main
 === removed symlink 'lib/BTrees'
 === target was u'../sourcecode/zope/src/BTrees/'
 === removed symlink 'lib/ClientForm'
 === target was u'../sourcecode/zope/src/ClientForm'
 === removed symlink 'lib/RestrictedPython'
 === target was u'../sourcecode/zope/src/RestrictedPython'
 === removed symlink 'lib/ZConfig'
 === target was u'../sourcecode/zope/src/ZConfig/'
 === removed symlink 'lib/ZODB'
 === target was u'../sourcecode/zope/src/ZODB/'
 === modified file 'lib/canonical/config/__init__.py'
 --- lib/canonical/config/__init__.py	2009-07-19 04:41:14 +0000
 +++ lib/canonical/config/__init__.py	2009-08-05 18:52:52 +0000
@@ -16,6 +16,7 @@
  import sys
  from urlparse import urlparse, urlunparse
++import pkg_resources
  import ZConfig
  from lazr.config import ImplicitTypeSchema
@@ -194,8 +195,10 @@
      def _setZConfig(self):
          """Modify the config, adding automatically generated settings"""
--        schemafile = os.path.join(
--            self.root, 'lib/zope/app/server/schema.xml')
++        self.root = TREE_ROOT
++
++        schemafile = pkg_resources.resource_filename(
++            'zope.app.server', 'schema.xml')
          schema = ZConfig.loadSchema(schemafile)
          root_options, handlers = ZConfig.loadConfig(
              schema, self.zope_config_file)
 === modified file 'lib/canonical/config/tests/test_config.py'
 --- lib/canonical/config/tests/test_config.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/config/tests/test_config.py	2009-08-05 18:52:52 +0000
@@ -11,6 +11,7 @@
  import ZConfig
  import os
++import pkg_resources
  import unittest
  from zope.testing.doctest import DocTestSuite, NORMALIZE_WHITESPACE, ELLIPSIS
@@ -20,11 +21,10 @@
  # Calculate some landmark paths.
  import canonical.config
++schema_file = pkg_resources.resource_filename('zope.app.server', 'schema.xml')
++schema = ZConfig.loadSchema(schema_file)
++
  here = os.path.dirname(canonical.config.__file__)
--schema_file = os.path.join(
--    here, os.pardir, os.pardir, 'zope/app/server/schema.xml')
--schema = ZConfig.loadSchema(schema_file)
--
  lazr_schema_file = os.path.join(here, 'schema-lazr.conf')
 === modified file 'lib/canonical/configure.zcml'
 --- lib/canonical/configure.zcml	2009-07-23 13:44:13 +0000
 +++ lib/canonical/configure.zcml	2009-08-13 15:22:00 +0000
@@ -83,7 +83,7 @@
           virtual host layers.  The directives come in pairs.
           The separate registration for the resources namespace (@@) is needed
           because otherwise the lookup for /@@/ will fail because the
--         layer-specifif defaultView directive also registers the
++         layer-specific defaultView directive also registers the
           default view name as an unnamed adapter.
           -->
        -->
 === modified file 'lib/canonical/launchpad/browser/tests/registration.py'
 --- lib/canonical/launchpad/browser/tests/registration.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/launchpad/browser/tests/registration.py	2009-08-05 18:52:52 +0000
@@ -29,7 +29,7 @@
      Return the Browser object after the registration is finished.
      """
--    token_url = canonical_url(token)
++    token_url = canonical_url(token).encode('utf8')
      logout()
      browser = setupBrowser()
      browser.open(token_url)
 === modified file 'lib/canonical/launchpad/doc/announcement-date-widget.txt'
 --- lib/canonical/launchpad/doc/announcement-date-widget.txt	2009-02-28 03:36:29 +0000
 +++ lib/canonical/launchpad/doc/announcement-date-widget.txt	2009-08-13 19:36:01 +0000
@@ -59,7 +59,9 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.foo', u'Foo',
--    Please do not provide a date if you want to publish immediately.)
++                       LaunchpadValidationError(u'Please do not provide a date
++                                                  if you want to publish
++                                                  immediately.'))
  If you choose to publish at a specific date in the future, the date field
  must be filled.
@@ -70,4 +72,4 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.foo', u'Foo',
--    Please provide a publication date.)
++    LaunchpadValidationError(u'Please provide a publication date.'))
 === modified file 'lib/canonical/launchpad/doc/batch_navigation.txt'
 --- lib/canonical/launchpad/doc/batch_navigation.txt	2009-03-24 12:43:49 +0000
 +++ lib/canonical/launchpad/doc/batch_navigation.txt	2009-08-06 11:02:58 +0000
@@ -49,10 +49,13 @@
  This section demonstrates that batching generates sensible SQL queries when used
  with SQLObject, i.e. that it puts appropriate LIMIT clauses on queries.
--Imports:
++Imports and initialization:
    >>> from canonical.launchpad.database import EmailAddress
    >>> from canonical.ftests.pgsql import CursorWrapper
++  >>> from canonical.launchpad.interfaces import IStore
++  >>> ignore = IStore(EmailAddress) # Prime the database connection.
++  ... # (Priming is important if this test is run in isolation).
    >>> CursorWrapper.record_sql = True
  Prepare a query, and create a batch of the results:
 === modified file 'lib/canonical/launchpad/doc/checkbox-matrix-widget.txt'
 --- lib/canonical/launchpad/doc/checkbox-matrix-widget.txt	2008-09-22 22:09:40 +0000
 +++ lib/canonical/launchpad/doc/checkbox-matrix-widget.txt	2009-08-13 19:36:01 +0000
@@ -24,16 +24,16 @@
      >>> matrix_widget._getFormValue()
      []
--If we pass in a value via the request, we'll be able to get the licenses
++If we pass in a value via the request, we'll be able to get the licenses
  as a set from _getFormValue() or getInputValue():
      >>> request = LaunchpadTestRequest(
      ...     form={'field.licenses': ['GNU_LGPL_V2_1', 'GNU_GPL_V2']})
      >>> matrix_widget = CheckBoxMatrixWidget(licenses_field, vtype, request)
      >>> matrix_widget._getFormValue()
--    Set([<...License.GNU_GPL_V2...>, <...License.GNU_LGPL_V2_1...>])
++    set([<...License.GNU_GPL_V2...>, <...License.GNU_LGPL_V2_1...>])
      >>> matrix_widget.getInputValue()
--    Set([<...License.GNU_GPL_V2...>, <...License.GNU_LGPL_V2_1...>])
++    set([<...License.GNU_GPL_V2...>, <...License.GNU_LGPL_V2_1...>])
  It should render as many rows as are specified by the column_count attribute.
 === modified file 'lib/canonical/launchpad/doc/feeds.txt'
 --- lib/canonical/launchpad/doc/feeds.txt	2009-08-05 14:48:06 +0000
 +++ lib/canonical/launchpad/doc/feeds.txt	2009-08-05 18:52:52 +0000
@@ -26,8 +26,8 @@
      >>> from zope.configuration import xmlconfig
      >>> zcmlcontext = xmlconfig.string("""
      ... <configure xmlns:browser="http://namespaces.zope.org/browser">
--    ...   <include file="lib/canonical/launchpad/webapp/meta.zcml" />
--    ...   <include file="lib/zope/app/zcmlfiles/meta.zcml" />
++    ...   <include package="canonical.launchpad.webapp" file="meta.zcml" />
++    ...   <include package="zope.app.zcmlfiles" file="meta.zcml" />
      ...   <browser:feeds
      ...       module="canonical.launchpad.ftests.feeds_helper"
      ...       classes="ThingFeedView"
 === modified file 'lib/canonical/launchpad/doc/image-widget.txt'
 --- lib/canonical/launchpad/doc/image-widget.txt	2008-06-04 07:51:28 +0000
 +++ lib/canonical/launchpad/doc/image-widget.txt	2009-08-13 19:36:01 +0000
@@ -2,8 +2,8 @@
  In Launchpad we have images associated with people, products, distributions,
  etc, and we want to allow people to have full control over their images. That
--is, they must be able to upload a new image and delete (or keep) an existing
--one. For this we created this widget, which can be embeded into any form we
++is, they must be able to upload a new image and delete (or keep) an existing
++one. For this we created this widget, which can be embeded into any form we
  want, which doesn't require us to add any submit buttons to indicate that the
  image should be kept, deleted or changed.
@@ -170,7 +170,7 @@
  #=== The LogoTiedWithMugshotWidget ===
+ #
  #This is a specialized version of the ImageChangeWidget which also returns a
--#resized version of the uploaded image, in case one was uploaded.
++#resized version of the uploaded image, in case one was uploaded.
+ #
  #    >>> from canonical.widgets.image import LogoTiedWithMugshotWidget
  #    >>> mugshot_file_name = os.path.join(
@@ -385,7 +385,8 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       This image is not exactly 192x192 pixels in size.)
++                       LaunchpadValidationError(u'\nThis image is not exactly
++                                                  192x192\npixels in size.'))
  This is what we see when the image is the correct dimensions, and within the
@@ -429,7 +430,9 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       This image exceeds the maximum allowed size in bytes.)
++                       LaunchpadValidationError(u'\nThis image exceeds the
++                                                  maximum allowed size in
++                                                  bytes.'))
  A similar error will be raised if the image's dimensions are bigger than the
  maximum we allow.
@@ -443,7 +446,8 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       This image is not exactly 191x193 pixels in size.)
++                       LaunchpadValidationError(u'\nThis image is not exactly
++                                                  191x193\npixels in size.'))
      >>> person_mugshot.dimensions = (image.size[0] + 1, image.size[1] - 1)
      >>> mugshot.seek(0)
@@ -453,7 +457,8 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       This image is not exactly 193x191 pixels in size.)
++                       LaunchpadValidationError(u'\nThis image is not exactly
++                                                  193x191\npixels in size.'))
  We also won't accept anything that is not an image; that is, we only accept
  what can be parsed by the Python Imaging Library module.
@@ -468,8 +473,10 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       The file uploaded was not recognized as an image;
--                       please check it and retry.)
++                       LaunchpadValidationError(u'\nThe file uploaded was not
++                                                  recognized as an image;
++                                                  please\ncheck it and
++                                                  retry.'))
  Finally, if the user specifies the 'change' action he must also provide a file
  to be uploaded.
@@ -481,7 +488,8 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       Please specify the image you want to use.)
++                       LaunchpadValidationError(u'Please specify the image you
++                                                  want to use.'))
  == Non-exact Image Dimensions ==
@@ -503,7 +511,8 @@
      Traceback (most recent call last):
      ...
      WidgetInputError: ('field.mugshot', u'Mugshot',
--                       This image is larger than 64x64 pixels in size.)
++                       LaunchpadValidationError(u'\nThis image is larger than
++                                                  64x64\npixels in size.'))
  If the image is smaller than the dimensions, the input validates:
@@ -524,4 +533,3 @@
      >>> fileupload = widget.getInputValue()
      >>> fileupload.filename
      u'mugshot.png'
--
 === modified file 'lib/canonical/launchpad/doc/launchpadlib.txt'
 --- lib/canonical/launchpad/doc/launchpadlib.txt	2008-05-22 19:11:05 +0000
 +++ lib/canonical/launchpad/doc/launchpadlib.txt	2009-06-11 01:28:55 +0000
@@ -11,7 +11,7 @@
      >>> browser.open('http://launchpad.dev:8085/~stimpy')
      Traceback (most recent call last):
      ...
--    HTTPError: HTTP Error 404: Not Found
++    httperror_seek_wrapper: HTTP Error 404: Not Found
  ...and when he doesn't, create him.
 === modified file 'lib/canonical/launchpad/doc/loginstatus-pages.txt'
 --- lib/canonical/launchpad/doc/loginstatus-pages.txt	2009-04-17 10:32:16 +0000
 +++ lib/canonical/launchpad/doc/loginstatus-pages.txt	2009-08-13 13:33:26 +0000
@@ -112,7 +112,7 @@
      >>> print view()
      <...
--    ...<a href="http://launchpad.dev/~name16"...>...
++    ...<a href="/~name16"...>...
  The link is for the user's Profile page in the 'overview' facet, even if
  we are viewing the 'bugs' facet.
 === modified file 'lib/canonical/launchpad/doc/navigation.txt'
 --- lib/canonical/launchpad/doc/navigation.txt	2008-09-30 22:25:04 +0000
 +++ lib/canonical/launchpad/doc/navigation.txt	2009-06-11 01:28:55 +0000
@@ -249,8 +249,8 @@
      >>> from zope.configuration import xmlconfig
      >>> zcmlcontext = xmlconfig.string("""
      ... <configure xmlns:browser="http://namespaces.zope.org/browser">
--    ...   <include file="lib/canonical/launchpad/webapp/meta.zcml" />
--    ...   <include file="lib/zope/app/zcmlfiles/meta.zcml" />
++    ...   <include package="canonical.launchpad.webapp" file="meta.zcml" />
++    ...   <include package="zope.app.zcmlfiles" file="meta.zcml" />
      ...   <browser:navigation
      ...       module="canonical.launchpad.ftests"
      ...       classes="ThingSetNavigation"
 === modified file 'lib/canonical/launchpad/doc/renamed-view.txt'
 --- lib/canonical/launchpad/doc/renamed-view.txt	2008-09-30 22:25:04 +0000
 +++ lib/canonical/launchpad/doc/renamed-view.txt	2009-06-11 01:28:55 +0000
@@ -100,8 +100,8 @@
      >>> from zope.configuration import xmlconfig
      >>> zcmlcontext = xmlconfig.string("""
      ... <configure xmlns:browser="http://namespaces.zope.org/browser">
--    ...   <include file="lib/canonical/launchpad/webapp/meta.zcml" />
--    ...   <include file="lib/zope/app/zcmlfiles/meta.zcml" />
++    ...   <include package="canonical.launchpad.webapp" file="meta.zcml" />
++    ...   <include package="zope.app.zcmlfiles" file="meta.zcml" />
      ...   <browser:renamed-page
      ...       for="canonical.launchpad.interfaces.IQuestionTarget"
      ...       name="+old_tickets_page"
 === modified file 'lib/canonical/launchpad/doc/storm.txt'
 --- lib/canonical/launchpad/doc/storm.txt	2009-07-24 12:55:03 +0000
 +++ lib/canonical/launchpad/doc/storm.txt	2009-08-05 18:52:52 +0000
@@ -83,6 +83,7 @@
      ...
      InternalError: transaction is read-only
++    >>> transaction.abort()
      >>> t = transaction.begin()
      >>> account = auth_slave.find(
      ...     Account, openid_identifier='sabdfl_oid').one()
 === modified file 'lib/canonical/launchpad/doc/stripped-text-widget.txt'
 --- lib/canonical/launchpad/doc/stripped-text-widget.txt	2007-06-17 11:57:02 +0000
 +++ lib/canonical/launchpad/doc/stripped-text-widget.txt	2009-08-13 19:36:01 +0000
@@ -38,4 +38,4 @@
    >>> widget.getInputValue()
    Traceback (most recent call last):
    ...
--  WidgetInputError: ('field', u'Title', )
++  WidgetInputError: ('field', u'Title', RequiredMissing())
 === modified file 'lib/canonical/launchpad/doc/tales.txt'
 --- lib/canonical/launchpad/doc/tales.txt	2009-08-11 07:20:12 +0000
 +++ lib/canonical/launchpad/doc/tales.txt	2009-08-13 21:32:59 +0000
@@ -268,11 +268,11 @@
      >>> print test_tales("bug/fmt:url/+text", bug=bug)
      http://bugs.launchpad.dev/bugs/1/+text
--    # Undo the custom setup we did above.
--
--    >>> login(ANONYMOUS)
--
--Persons and teams can also specify an alternate rootsite to fmt:url.
++
++fmt:url accepts an rootsite extension to make URLs to a specific application.
++
++    >>> login(ANONYMOUS,
++    ...     LaunchpadTestRequest(SERVER_URL='http://code.launchpad.net'))
      >>> print test_tales("person/fmt:url:bugs", person=sabdfl)
      http://bugs.launchpad.dev/~sabdfl
@@ -280,6 +280,14 @@
      >>> print test_tales("person/fmt:url:feeds", person=sabdfl)
      http://feeds.launchpad.dev/~sabdfl
++    >>> print test_tales("pillar/fmt:url:answers", pillar=ubuntu)
++    http://answers.launchpad.dev/ubuntu
++
++    >>> print test_tales("bug/fmt:url:mainsite", bug=bug)
++    http://launchpad.dev/bugs/1
++
++    >>> login(ANONYMOUS)
++
  The fmt: namespace to get links
  -------------------------------
@@ -319,6 +327,34 @@
      >>> test_tales("person/fmt:link", person=ubuntu_team)
      u'<a href=".../~ubuntu-team" class="sprite team">Ubuntu Team</a>'
++The link can make the URL go to a specific app.
++
++    >>> login(ANONYMOUS,
++    ...     LaunchpadTestRequest(SERVER_URL='http://code.launchpad.net'))
++
++    >>> print test_tales("pillar/fmt:link:translations", pillar=ubuntu)
++    <a ...http://translations.launchpad.dev/ubuntu...
++
++    >>> print test_tales("person/fmt:url:feeds", person=sabdfl)
++    http://feeds.launchpad.dev/~sabdfl
++
++    >>> print test_tales("bug/fmt:url:mainsite", bug=bug)
++    http://launchpad.dev/bugs/1
++
++The default behaviour for pillars, persons, and teams is to link to
++the mainsite.
++
++    >>> print test_tales("pillar/fmt:link", pillar=ubuntu)
++    <a ...http://launchpad.dev/ubuntu...
++
++    >>> print test_tales("person/fmt:link", person=sabdfl)
++    <a ...http://launchpad.dev/~sabdfl...
++
++    >>> print test_tales("team/fmt:link", team=ubuntu_team)
++    <a ...http://launchpad.dev/~ubuntu-team...
++
++    >>> login(ANONYMOUS)
++
  The person's displayname is escaped to prevent markup from being
  interpreted by the browser. For example, a script added to Sample
  Person's displayname will be escaped; averting a XSS vulnerability.
 === modified file 'lib/canonical/launchpad/ftests/feeds_helper.py'
 --- lib/canonical/launchpad/ftests/feeds_helper.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/launchpad/ftests/feeds_helper.py	2009-08-05 18:52:52 +0000
@@ -15,9 +15,13 @@
+     ]
++import socket
++original_timeout = socket.getdefaulttimeout()
  import feedvalidator
++if socket.getdefaulttimeout() != original_timeout:
++    # feedvalidator's __init__ uses setdefaulttimeout promiscuously
++    socket.setdefaulttimeout(original_timeout)
  from cStringIO import StringIO
--import socket
  from textwrap import wrap
  from zope.interface import implements, Interface, Attribute
 === modified file 'lib/canonical/launchpad/icing/style-3-0.css'
 --- lib/canonical/launchpad/icing/style-3-0.css	2009-08-07 14:01:57 +0000
 +++ lib/canonical/launchpad/icing/style-3-0.css	2009-08-13 02:24:56 +0000
@@ -179,7 +179,16 @@
  #colophon {
      float: none;
+     }
++#globalsearch {
++  float: none;
++  display: auto;
++  font-weight: inherit;
++  margin-right: auto;
++}
++#globalsearch input {
++  font-size: inherit;
++}
  /* Page layout */
  .yui-d0 {
@@ -195,9 +204,25 @@
  .footer {
      clear: both;
      margin-top: 2em;
--    border-top: 1px solid #cdd3dd;
      padding-top: 0.5em;
+     }
++.footer .lp-arcana {
++    background: url(/@@/footer-background.png) top left repeat-x;
++    padding: 0.8em;
++    -moz-border-radius: 3px 3px 0 0;
++    -webkit-border-radius: 3px 3px 0 0;
++    border-radius: 3px 3px 0 0;
++    }
++.footer .lp-arcana img {
++    vertical-align: middle;
++    }
++.footer .lp-arcana .search-icon {
++    background: url(icon-sprites) 100% -191px no-repeat;
++    }
++.footer .colophon {
++    margin: 3em 3em 1em 3em;
++    text-align: center;
++    }
  .portlet, .aside {
      clear: both;
      border-top: 1px solid #d6d6d6;
@@ -424,7 +449,7 @@
  /* The text of the first tab should align with the heading directly
     above it */
  .location-portlet li:first-child a, .location-portlet li:first-child span {
--    margin-left: -0.5em;
++    margin-left: -0.5em;
+ }
  .location-portlet li a:link, .location-portlet li a:visited {
      color: #000;
 === modified file 'lib/canonical/launchpad/icing/style.css'
 --- lib/canonical/launchpad/icing/style.css	2009-08-05 08:15:28 +0000
 +++ lib/canonical/launchpad/icing/style.css	2009-08-12 21:08:05 +0000
@@ -71,6 +71,7 @@
  .info {background:url(icon-sprites) 0 -96px no-repeat;}
  .question {background:url(icon-sprites) 0 -128px no-repeat;}
  .download-icon {background:url(icon-sprites) 0 -160px no-repeat;}
++.download {background:url(icon-sprites) 0 -160px no-repeat;}
  .search-icon {background:url(icon-sprites) 0 -187px no-repeat;}
  .no {background:url(icon-sprites) 0 -224px no-repeat;}
  .yes {background:url(icon-sprites) 0 -256px no-repeat;}
 === added file 'lib/canonical/launchpad/images/footer-background.png'
 Binary files lib/canonical/launchpad/images/footer-background.png	1970-01-01 00:00:00 +0000 and lib/canonical/launchpad/images/footer-background.png	2009-08-13 01:01:39 +0000 differ
 === modified file 'lib/canonical/launchpad/pagetests/basics/max-batch-size.txt'
 --- lib/canonical/launchpad/pagetests/basics/max-batch-size.txt	2009-07-17 14:00:36 +0000
 +++ lib/canonical/launchpad/pagetests/basics/max-batch-size.txt	2009-08-05 18:52:52 +0000
@@ -10,7 +10,7 @@
      >>> anon_browser.open('http://launchpad.dev/projects/+all?start=0&batch=1000')
      Traceback (most recent call last):
      ...
--    HTTPError: HTTP Error 400: Bad Request
++    httperror_seek_wrapper: HTTP Error 400: Bad Request
      >>> print extract_text(find_main_content(anon_browser.contents))
      Invalid Batch Size
 === modified file 'lib/canonical/launchpad/pagetests/feeds/xx-authentication.txt'
 --- lib/canonical/launchpad/pagetests/feeds/xx-authentication.txt	2008-01-31 23:21:01 +0000
 +++ lib/canonical/launchpad/pagetests/feeds/xx-authentication.txt	2009-07-16 13:28:25 +0000
@@ -23,4 +23,4 @@
      >>> browser.open('http://feeds.launchpad.dev/bugs/14/bug.atom')
      Traceback (most recent call last):
        ...
--    HTTPError: HTTP Error 403: Forbidden
++    httperror_seek_wrapper: HTTP Error 403: Forbidden
 === modified file 'lib/canonical/launchpad/pagetests/oauth/access-token.txt'
 --- lib/canonical/launchpad/pagetests/oauth/access-token.txt	2008-07-22 20:34:14 +0000
 +++ lib/canonical/launchpad/pagetests/oauth/access-token.txt	2009-07-16 13:28:25 +0000
@@ -40,7 +40,7 @@
      ...     'http://launchpad.dev/+access-token', data=urlencode(data))
      Traceback (most recent call last):
      ...
--    HTTPError: HTTP Error 401: Unauthorized
++    httperror_seek_wrapper: HTTP Error 401: Unauthorized
  The token's context, when not None, is sent to the consumer together
  with the token's key and secret.
 === modified file 'lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-redirection.txt'
 --- lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-redirection.txt	2009-05-12 16:29:13 +0000
 +++ lib/canonical/launchpad/pagetests/standalone/xx-beta-testers-redirection.txt	2009-08-06 11:35:34 +0000
@@ -27,7 +27,7 @@
      ...         function(*args, **kwargs)
      ...     except HTTPError, exc:
      ...         print str(exc)
--    ...         location = exc.headers.getheader('Location')
++    ...         location = exc.hdrs.getheader('Location')
      ...         if location is not None:
      ...             print 'Location:', location
 === modified file 'lib/canonical/launchpad/scripts/ftests/test_oops_prune.py'
 --- lib/canonical/launchpad/scripts/ftests/test_oops_prune.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/launchpad/scripts/ftests/test_oops_prune.py	2009-08-07 12:46:37 +0000
@@ -58,9 +58,12 @@
              INSERT INTO MessageChunk(message, sequence, content)
              VALUES (1, 99, 'OOPS-%s')
              """ % self.referenced_oops_code)
++        if not os.path.exists(config.error_reports.error_dir):
++            os.mkdir(config.error_reports.error_dir)
      def tearDown(self):
          shutil.rmtree(self.oops_dir)
++        shutil.rmtree(config.error_reports.error_dir)
      def test_referenced_oops(self):
          self.failUnlessEqual(
 === modified file 'lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt'
 --- lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt	2009-04-02 16:39:20 +0000
 +++ lib/canonical/launchpad/webapp/ftests/test_adapter_permissions.txt	2009-06-11 01:28:55 +0000
@@ -28,6 +28,7 @@
  Test this once more to ensure the settings stick across transactions.
++    >>> transaction.abort()
      >>> t = transaction.begin()
      >>> main_slave.find(Person, name='janitor').one().displayname = 'BenD'
      >>> transaction.commit()
 === modified file 'lib/canonical/launchpad/webapp/tales.py'
 --- lib/canonical/launchpad/webapp/tales.py	2009-08-13 00:51:50 +0000
 +++ lib/canonical/launchpad/webapp/tales.py	2009-08-14 12:56:19 +0000
@@ -458,7 +458,25 @@
          return self.url()
      def traverse(self, name, furtherPath):
--        if name in self.traversable_names:
++        if name.startswith('link:') or name.startswith('url:'):
++            rootsite = name.split(':')[1]
++            extra_path = None
++            if len(furtherPath) > 0:
++                extra_path = '/'.join(reversed(furtherPath))
++            # Remove remaining entries in furtherPath so that traversal
++            # stops here.
++            del furtherPath[:]
++            if name.startswith('link:'):
++                if rootsite is None:
++                    return self.link(extra_path)
++                else:
++                    return self.link(extra_path, rootsite=rootsite)
++            else:
++                if rootsite is None:
++                    self.url(extra_path)
++                else:
++                    return self.url(extra_path, rootsite=rootsite)
++        elif name in self.traversable_names:
              if len(furtherPath) >= 1:
                  extra_path = '/'.join(reversed(furtherPath))
                  del furtherPath[:]
@@ -472,13 +490,16 @@
          else:
              raise TraversalError, name
--    def link(self, view_name):
++    def link(self, view_name, rootsite=None):
          """Return an HTML link to the object's page.
          The link consists of an icon followed by the object's name.
          :param view_name: If not None, the link will point to the page with
              that name on this object.
++        :param rootsite: If not None, return the URL to the page on the
++            specified rootsite.  Note this is available only for subclasses
++            that allow specifying the rootsite.
          """
          raise NotImplementedError(
              "No link implementation for %r, IPathAdapter implementation "
@@ -958,23 +979,6 @@
      final_traversable_names = {'local-time': 'local_time'}
      final_traversable_names.update(ObjectFormatterAPI.final_traversable_names)
--    def traverse(self, name, furtherPath):
--        """Special-case traversal for links with an optional rootsite."""
--        if name.startswith('link:') or name.startswith('url:'):
--            rootsite = name.split(':')[1]
--            extra_path = None
--            if len(furtherPath) > 0:
--                extra_path = '/'.join(reversed(furtherPath))
--            # Remove remaining entries in furtherPath so that traversal
--            # stops here.
--            del furtherPath[:]
--            if name.startswith('link:'):
--                return self.link(extra_path, rootsite=rootsite)
--            else:
--                return self.url(extra_path, rootsite=rootsite)
--        else:
--            return super(PersonFormatterAPI, self).traverse(name, furtherPath)
--
      def local_time(self):
          """Return the local time for this person."""
          time_zone = 'UTC'
@@ -982,12 +986,22 @@
              time_zone = self._context.time_zone
          return datetime.now(pytz.timezone(time_zone)).strftime('%T %Z')
--    def link(self, view_name, rootsite=None):
--        """Return an HTML link to the person's page containing an icon
--        followed by the person's name.
++    def url(self, view_name=None, rootsite='mainsite'):
++        """See `ObjectFormatterAPI`.
++
++        The default URL for a person is to the mainsite.
++        """
++        return super(PersonFormatterAPI, self).url(view_name, rootsite)
++
++    def link(self, view_name, rootsite='mainsite'):
++        """See `ObjectFormatterAPI`.
++
++        Return an HTML link to the person's page containing an icon
++        followed by the person's name. The default URL for a person is to
++        the mainsite.
          """
          person = self._context
--        url = canonical_url(person, rootsite=rootsite, view_name=view_name)
++        url = self.url(view_name, rootsite)
          custom_icon = ObjectImageDisplayAPI(person)._get_custom_icon_url()
          if custom_icon is None:
              css_class = ObjectImageDisplayAPI(person).sprite_css()
@@ -1024,12 +1038,16 @@
      hidden = u'<hidden>'
--    def url(self, view_name=None):
--        """See `ObjectFormatterAPI`."""
++    def url(self, view_name=None, rootsite='mainsite'):
++        """See `ObjectFormatterAPI`.
++
++        The default URL for a team is to the mainsite. None is returned
++        when the user does not have permission to review the team.
++        """
          if not check_permission('launchpad.View', self._context):
              # This person has no permission to view the team details.
              return None
--        return super(TeamFormatterAPI, self).url(view_name)
++        return super(TeamFormatterAPI, self).url(view_name, rootsite)
      def api_url(self, context):
          """See `ObjectFormatterAPI`."""
@@ -1038,8 +1056,12 @@
              return None
          return super(TeamFormatterAPI, self).api_url(context)
--    def link(self, view_name, rootsite=None):
--        """See `ObjectFormatterAPI`."""
++    def link(self, view_name, rootsite='mainsite'):
++        """See `ObjectFormatterAPI`.
++
++        The default URL for a team is to the mainsite. None is returned
++        when the user does not have permission to review the team.
++        """
          person = self._context
          if not check_permission('launchpad.View', person):
              # This person has no permission to view the team details.
@@ -1116,7 +1138,7 @@
          """
          return queryAdapter(self._context, IPathAdapter, 'image').sprite_css()
--    def link(self, view_name):
++    def link(self, view_name, rootsite=None):
          """Return html including a link, description and icon.
          Icon and link are optional, depending on type and permissions.
@@ -1132,7 +1154,7 @@
          summary = self._make_link_summary()
          if check_permission(self._link_permission, self._context):
--            url = self.url(view_name)
++            url = self.url(view_name, rootsite)
          else:
              url = ''
          if url:
@@ -1152,33 +1174,40 @@
          displayname = self._context.displayname
          return {'displayname': displayname}
--    def link(self, view_name):
++    def url(self, view_name=None, rootsite=None):
++        """See `ObjectFormatterAPI`.
++
++        The default URL for a pillar is to the mainsite.
++        """
++        return super(PillarFormatterAPI, self).url(view_name, rootsite)
++
++    def link(self, view_name, rootsite='mainsite'):
          """The html to show a link to a Product, Project or distribution.
          In the case of Products or Project groups we display the custom
--        icon, if one exists."""
++        icon, if one exists. The default URL for a pillar is to the mainsite.
++        """
          html = super(PillarFormatterAPI, self).link(view_name)
          context = self._context
--        if IProduct.providedBy(context) or IProject.providedBy(context):
--            custom_icon = ObjectImageDisplayAPI(
--                context)._get_custom_icon_url()
--            url = canonical_url(context, view_name=view_name)
--            summary = self._make_link_summary()
--            if custom_icon is None:
--                css_class = ObjectImageDisplayAPI(context).sprite_css()
--                html = (u'<a href="%s" class="%s">%s</a>') % (
--                    url, css_class, summary)
--            else:
--                html = (u'<a href="%s" class="bg-image" '
--                         'style="background-image: url(%s)">%s</a>') % (
--                    url, custom_icon, summary)
--            if IProduct.providedBy(context):
--                license_status = context.license_status
--                if license_status != LicenseStatus.OPEN_SOURCE:
--                    html = '<span title="%s">%s (%s)</span>' % (
--                            license_status.description, html,
--                            license_status.title)
++        custom_icon = ObjectImageDisplayAPI(
++            context)._get_custom_icon_url()
++        url = self.url(view_name, rootsite)
++        summary = self._make_link_summary()
++        if custom_icon is None:
++            css_class = ObjectImageDisplayAPI(context).sprite_css()
++            html = (u'<a href="%s" class="%s">%s</a>') % (
++                url, css_class, summary)
++        else:
++            html = (u'<a href="%s" class="bg-image" '
++                     'style="background-image: url(%s)">%s</a>') % (
++                url, custom_icon, summary)
++        if IProduct.providedBy(context):
++            license_status = context.license_status
++            if license_status != LicenseStatus.OPEN_SOURCE:
++                html = '<span title="%s">%s (%s)</span>' % (
++                        license_status.description, html,
++                        license_status.title)
          return html
@@ -1231,7 +1260,7 @@
              html += ')'
          return html % replacements
--    def url(self, view_name):
++    def url(self, view_name, rootsite=None):
          """Return the URL to download the file."""
          return self._getDownloadURL(self._context.libraryfile)
@@ -1294,7 +1323,7 @@
  class PreviewDiffFormatterAPI(ObjectFormatterAPI):
      """Formatter for preview diffs."""
--    def url(self, view_name=None):
++    def url(self, view_name=None, rootsite=None):
          """Use the url of the librarian file containing the diff.
          """
          librarian_alias = self._context.diff_text
@@ -1423,7 +1452,7 @@
                  'branch': self._context.branch.bzr_identity,
+                }
--    def url(self, view_name=None):
++    def url(self, view_name=None, rootsite=None):
          """See `ObjectFormatterAPI`."""
          # The url of a code import is the associated branch.
          # This is still here primarily for supporting branch deletion,
@@ -2064,7 +2093,7 @@
          """Return the default representation of the link."""
          return self._context.render()
--    def url(self, view_name=None):
++    def url(self, view_name=None, rootsite=None):
          """Return the URL representation of the link."""
          if self._context.enabled:
              return self._context.url
 === modified file 'lib/canonical/testing/__init__.py'
 --- lib/canonical/testing/__init__.py	2009-07-17 18:46:25 +0000
 +++ lib/canonical/testing/__init__.py	2009-08-10 22:08:05 +0000
@@ -63,8 +63,9 @@
      logging._handlers.clear()
      # Reset the setup
--    import zope.testing.testrunner
--    zope.testing.testrunner.configure_logging()
++    from zope.testing.testrunner.runner import Runner
++    from zope.testing.testrunner.logsupport import Logging
++    Logging(Runner()).global_setup()
  # This import registers the 'doctest' Unicode codec.
 === modified file 'lib/canonical/testing/ftests/test_mockdb.py'
 --- lib/canonical/testing/ftests/test_mockdb.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/testing/ftests/test_mockdb.py	2009-08-05 18:52:52 +0000
@@ -11,7 +11,7 @@
  import unittest
  import psycopg2
--from zope.testing.testrunner import dont_retry, RetryTest
++# from zope.testing.testrunner import dont_retry, RetryTest
  from canonical.config import config
  from canonical.testing import mockdb, DatabaseLayer
@@ -101,63 +101,63 @@
          self.connections.append(con)
          return con
--    @dont_retry
--    def testIncorrectReplay(self):
--        # Record nothing but a close on a single connection.
--        con = self.connect()
--        con.close()
--        self.script.store()
--
--        # Replay correctly.
--        self.switchToReplayMode()
--        con = self.connect()
--        con.close()
--
--        # Replay incorrectly.
--        self.switchToReplayMode()
--        con = self.connect()
--        self.assertRaises(RetryTest, con.rollback)
--
--    @dont_retry
--    def testMultipleConnections(self):
++    # @dont_retry
++    #def testIncorrectReplay(self):
++    #    # Record nothing but a close on a single connection.
++    #    con = self.connect()
++    #    con.close()
++    #    self.script.store()
++
++    #    # Replay correctly.
++    #    self.switchToReplayMode()
++    #    con = self.connect()
++    #    con.close()
++
++    #    # Replay incorrectly.
++    #    self.switchToReplayMode()
++    #    con = self.connect()
++    #    self.assertRaises(RetryTest, con.rollback)
++
++    # @dont_retry
++    #def testMultipleConnections(self):
          # Ensure that commands issued via different connections
          # maintain their global order.
--        con1 = self.connect()
--        con2 = self.connect()
--        con1.close()
--        con2.close()
--        self.script.store()
--
--        # Replay correctly.
--        self.switchToReplayMode()
--        con1 = self.connect()
--        con2 = self.connect()
--        con1.close()
--        con2.close()
--
--        # Replay in the wrong order.
--        self.switchToReplayMode()
--        con1 = self.connect()
--        con2 = self.connect()
--        self.assertRaises(RetryTest, con2.close)
--
--    @dont_retry
--    def testConnectionParams(self):
++    #    con1 = self.connect()
++    #    con2 = self.connect()
++    #    con1.close()
++    #    con2.close()
++    #    self.script.store()
++
++    #    # Replay correctly.
++    #    self.switchToReplayMode()
++    #    con1 = self.connect()
++    #    con2 = self.connect()
++    #    con1.close()
++    #    con2.close()
++
++    #    # Replay in the wrong order.
++    #    self.switchToReplayMode()
++    #    con1 = self.connect()
++    #    con2 = self.connect()
++    #    self.assertRaises(RetryTest, con2.close)
++
++    # @dont_retry
++    #def testConnectionParams(self):
          # Make sure we can correctly connect with different connection parms.
--        for mode in self.modes():
--            for dbuser in ['launchpad', 'testadmin']:
--                connection_string = "%s user=%s" % (
--                        config.database.main_master, dbuser)
--                con = self.connect(connection_string)
--                cur = con.cursor()
--                cur.execute("SHOW session authorization")
--                self.failUnlessEqual(cur.fetchone()[0], dbuser)
++    #    for mode in self.modes():
++    #        for dbuser in ['launchpad', 'testadmin']:
++    #            connection_string = "%s user=%s" % (
++    #                    config.database.main_master, dbuser)
++    #            con = self.connect(connection_string)
++    #            cur = con.cursor()
++    #            cur.execute("SHOW session authorization")
++    #            self.failUnlessEqual(cur.fetchone()[0], dbuser)
          # Confirm that unexpected connection parameters raises a RetryTest.
--        self.switchToReplayMode()
--        self.assertRaises(RetryTest, self.connect, "whoops")
++    #    self.switchToReplayMode()
++    #    self.assertRaises(RetryTest, self.connect, "whoops")
--    @dont_retry
++    # @dont_retry
      def testFailedConnection(self):
          # Ensure failed database connections are reproducable.
          for mode in self.modes():
@@ -168,13 +168,13 @@
                      psycopg2.OperationalError, self.connect, connection_string
+                     )
--    @dont_retry
++    # @dont_retry
      def testNoopSession(self):
          # Minimal do-nothing case.
          for mode in self.modes():
              con = self.connect()
--    @dont_retry
++    # @dont_retry
      def testSimpleQuery(self):
          # Ensure that we can script and replay a simple query.
          for mode in self.modes():
@@ -199,7 +199,7 @@
              name = cur.fetchone()[0]
              self.assertEqual(name, 'carlos')
--    @dont_retry
++    # @dont_retry
      def testExceptions(self):
          # Confirm that expected exceptions are raised correctly.
          for mode in self.modes():
@@ -210,38 +210,38 @@
                      cur.execute, "SELECT blood FROM Stone"
+                     )
--    @dont_retry
--    def testUnexpectedQuery(self):
--        for mode in self.modes():
--            con = self.connect()
--            cur = con.cursor()
--            if mode != 'replay':
--                cur.execute("SELECT name FROM Person WHERE name='sabdfl'")
--            else:
++    # @dont_retry
++    #def testUnexpectedQuery(self):
++    #    for mode in self.modes():
++    #        con = self.connect()
++    #        cur = con.cursor()
++    #        if mode != 'replay':
++    #            cur.execute("SELECT name FROM Person WHERE name='sabdfl'")
++    #        else:
                  # Issue an unexpected query in replay mode. A RetryTest
                  # exception should be raised.
--                self.assertRaises(
--                        RetryTest, cur.execute,
--                        "SELECT name FROM Person WHERE name='stub'"
--                        )
++    #            self.assertRaises(
++    #                    RetryTest, cur.execute,
++    #                    "SELECT name FROM Person WHERE name='stub'"
++    #                    )
--    @dont_retry
--    def testUnexpectedQueryParameters(self):
--        for mode in self.modes():
--            con = self.connect()
--            cur = con.cursor()
--            query = "SELECT name FROM Person WHERE name=%s"
--            if mode != 'replay':
--                cur.execute(query, ('sabdfl',))
--            else:
++    # @dont_retry
++    #def testUnexpectedQueryParameters(self):
++    #    for mode in self.modes():
++    #        con = self.connect()
++    #        cur = con.cursor()
++    #        query = "SELECT name FROM Person WHERE name=%s"
++    #        if mode != 'replay':
++    #            cur.execute(query, ('sabdfl',))
++    #        else:
                  # Issue a query with unexpected bound parameters in replay
                  # mode. A RetryTest should be raised.
--                self.assertRaises(
--                        RetryTest, cur.execute,
--                        query, ('stub',)
--                        )
++    #            self.assertRaises(
++    #                    RetryTest, cur.execute,
++    #                    query, ('stub',)
++    #                    )
--    @dont_retry
++    # @dont_retry
      def testCommit(self):
          # Confirm commit behavior.
          for mode in self.modes():
@@ -283,7 +283,7 @@
                  """)
              con.commit()
--    @dont_retry
++    # @dont_retry
      def testRollback(self):
          # Confirm rollback behavior.
          for mode in self.modes():
@@ -323,7 +323,7 @@
                      "Rollback did not roll back changes."
+                     )
--    @dont_retry
++    # @dont_retry
      def testFailedCommit(self):
          # Confirm exeptions raised on commit are recorded and replayed.
          for mode in self.modes():
@@ -344,7 +344,7 @@
              else:
                  self.assertRaises(psycopg2.InterfaceError, con.rollback)
--    @dont_retry
++    # @dont_retry
      def testFailedSetIsolationLevel(self):
          # Confirm exeptions raised on commit are recorded and replayed.
          for mode in self.modes():
@@ -354,7 +354,7 @@
                      psycopg2.InterfaceError, con.set_isolation_level, 666
+                     )
--    @dont_retry
++    # @dont_retry
      def testClose(self):
          # Confirm and record close behavior.
          for mode in self.modes():
@@ -376,7 +376,7 @@
                  self.fail(
                          "Connection.close() now DB-API compliant. Fix test.")
--    @dont_retry
++    # @dont_retry
      def testCursorDescription(self):
          # Confirm cursor.description behavior.
          for mode in self.modes():
@@ -402,7 +402,7 @@
              else:
                  self.failUnlessEqual(direct_description, cur.description)
--    @dont_retry
++    # @dont_retry
      def testCursorRowcount(self):
          # Confirm and record cursor.rowcount behavior.
          for mode in self.modes():
@@ -443,7 +443,7 @@
              cur.execute("DELETE FROM WikiName WHERE person=1")
              self.failUnlessEqual(cur.rowcount, 1)
--    @dont_retry
++    # @dont_retry
      def testCursorClose(self):
          # Confirm and record cursor.close behavior.
          for mode in self.modes():
@@ -457,7 +457,7 @@
              cur = con.cursor()
              cur.execute("SELECT name FROM Person WHERE name='stub'")
--    @dont_retry
++    # @dont_retry
      def testFetchOne(self):
          for mode in self.modes():
              con = self.connect()
@@ -491,7 +491,7 @@
                  self.failUnlessEqual(row[0], i, "Bad result %s" % repr(row))
              self.failUnless(cur.fetchone() is None, "Too many results")
--    @dont_retry
++    # @dont_retry
      def testCursorIteration(self):
          # psycopg1 does not support this extension.
          for mode in self.modes():
@@ -516,7 +516,7 @@
          ##          )
          ##      self.failUnlessEqual(row[0], 1, "Bad result %s" % repr(row))
--    @dont_retry
++    # @dont_retry
      def testFetchAll(self):
          for mode in self.modes():
              con = self.connect()
 === modified file 'lib/canonical/testing/mockdb.py'
 --- lib/canonical/testing/mockdb.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/testing/mockdb.py	2009-08-05 18:52:52 +0000
@@ -25,7 +25,7 @@
  import urllib
  import psycopg2
--from zope.testing.testrunner import RetryTest
++# from zope.testing.testrunner import RetryTest
  from canonical.config import config
@@ -386,7 +386,8 @@
          if os.path.exists(self.script_filename):
              os.unlink(self.script_filename)
          self.invalid = True
--        raise RetryTest(reason)
++        raise RetryTest(reason) # Leaving this as a name error: this should
++        # not be called unless we reinstate the retry behavior in zope.testing.
  class MockDbConnection:
 === modified file 'lib/canonical/testing/tests/test_mockdb.py'
 --- lib/canonical/testing/tests/test_mockdb.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/testing/tests/test_mockdb.py	2009-08-05 18:52:52 +0000
@@ -11,7 +11,7 @@
  import unittest
  from zope.testing.doctestunit import DocTestSuite
--from zope.testing.testrunner import dont_retry, RetryTest
++#from zope.testing.testrunner import dont_retry, RetryTest
  from canonical.testing import mockdb
@@ -24,7 +24,7 @@
          if os.path.exists(self.script_filename):
              os.unlink(self.script_filename)
--    @dont_retry
++    #@dont_retry
      def testSerialize(self):
          # Ensure the scripts can store and retrieve their logs
          recorder = mockdb.ScriptRecorder(self.script_filename)
@@ -34,29 +34,29 @@
          replayer = mockdb.ScriptPlayer(self.script_filename)
          self.failUnlessEqual(replayer.log, ['Arbitrary Data'])
--    @dont_retry
--    def testHandleInvalidScript(self):
++    #@dont_retry
++    #def testHandleInvalidScript(self):
          # Ensure a RetryTest exception is raised and the invalid script
          # file removed when handleInvalidScript() is called
--        recorder = mockdb.ScriptRecorder(self.script_filename)
--        recorder.store()
--
--        replayer = mockdb.ScriptPlayer(self.script_filename)
--
--        self.assertRaises(
--                RetryTest, replayer.handleInvalidScript, 'Reason')
--        self.failIf(os.path.exists(self.script_filename))
--
--    @dont_retry
--    def testShortScript(self):
++    #    recorder = mockdb.ScriptRecorder(self.script_filename)
++    #    recorder.store()
++
++    #    replayer = mockdb.ScriptPlayer(self.script_filename)
++
++    #    self.assertRaises(
++    #            RetryTest, replayer.handleInvalidScript, 'Reason')
++    #    self.failIf(os.path.exists(self.script_filename))
++
++    #@dont_retry
++    #def testShortScript(self):
          # Ensure a RetryTest exception is raised if an attempt is made
          # to pull results from an exhausted script.
--        recorder = mockdb.ScriptRecorder(self.script_filename)
--        recorder.store()
--        replayer = mockdb.ScriptPlayer(self.script_filename)
--        self.assertRaises(RetryTest, replayer.getNextEntry, None, None)
++    #    recorder = mockdb.ScriptRecorder(self.script_filename)
++    #    recorder.store()
++    #    replayer = mockdb.ScriptPlayer(self.script_filename)
++    #    self.assertRaises(RetryTest, replayer.getNextEntry, None, None)
--    @dont_retry
++    #@dont_retry
      def testScriptFilename(self):
          # Ensure evil characters in the key don't mess up the script_filename
          # results. Only '/' is really evil - other chars should all work
@@ -86,7 +86,7 @@
      # This test does not use @dont_retry.
      # It needs to leak RetryTest exeptions as it tests that the
      # test runner is handling them correctly.
--    def testRetryTestRetriesTest(self):
++    #def testRetryTestRetriesTest(self):
          # The first time this test is run it raises a RetryTest exception.
          # The second time it is run it succeeds. This means that this
          # test will fail if RetryTest handling is not being done correctly
@@ -97,106 +97,106 @@
          # version of zope.testing is in use and to minimize the zope.testing
          # patch until we decide if RetryTest handling is to be pushed
          # upstream or not.
--        MockDbTestCase._retry_count += 1
--        if MockDbTestCase._retry_count % 2 == 1:
--            raise RetryTest(
--                "Testing RetryTest behavior. This exception will be raised "
--                "but the test runner doesn't consider it a failure")
++    #    MockDbTestCase._retry_count += 1
++    #    if MockDbTestCase._retry_count % 2 == 1:
++    #        raise RetryTest(
++    #            "Testing RetryTest behavior. This exception will be raised "
++    #            "but the test runner doesn't consider it a failure")
  _doctest_retry_count = 0
--def retry_on_odd_numbered_calls():
--    """Helper for doctest RetryTest test.
++#def retry_on_odd_numbered_calls():
++#    """Helper for doctest RetryTest test.
--    This helper raises a RetryTest exception on odd numbered calls,
--    and prints 'Retry not raised' on even numbered calls.
++#    This helper raises a RetryTest exception on odd numbered calls,
++#    and prints 'Retry not raised' on even numbered calls.
--    >>> try:
--    ...     retry_on_odd_numbered_calls()
--    ... except RetryTest:
--    ...     print "Caught RetryTest."
--    ...
--    Retry raised.
--    Caught RetryTest.
--    >>> try:
--    ...     retry_on_odd_numbered_calls()
--    ... except RetryTest:
--    ...     print "Caught RetryTest."
--    ...
--    Retry not raised.
--    """
--    global _doctest_retry_count
--    _doctest_retry_count += 1
--    if _doctest_retry_count % 2 == 1:
--        print "Retry raised."
--        raise RetryTest
--    print "Retry not raised."
--
--
--def testRetryTestInDoctest_will_raise_but_testrunner_ignores_it():
--    """Test a RetryTest exception in a doctest works as expected.
--
--    This doctest raises a RetryTest exception the first time it is run.
--    On the second run, it succeeds.
--
--    If the testrunner is correctly handling RetryTest exceptions raised
--    by doctests, then the RetryTest exception will not be reported as
--    a failure. This test will then be rerun and succeed.
--
--    If the testrunner is not correctly handling RetryTest exceptions,
--    then the RetryTesst exception will be flagged as an error.
--
--    This test confirms that a RetryException raised where no exception
--    was expected works.
--
--    >>> retry_on_odd_numbered_calls()
--    Retry not raised.
--    """
--
--
--def retry_on_odd_numbered_calls2():
--    """Helper for doctest RetryTest test.
--
--    This helper raises a RetryTest exception on odd numbered calls,
--    and a RuntimeError on even numbered calls.
--
--    >>> try:
--    ...     retry_on_odd_numbered_calls2()
--    ... except RetryTest:
--    ...     print "Caught RetryTest."
--    ...
--    Retry raised.
--    Caught RetryTest.
--    >>> try:
--    ...     retry_on_odd_numbered_calls2()
--    ... except RetryTest:
--    ...     print "Caught RetryTest."
--    ...
--    Traceback (most recent call last):
--    ...
--    RuntimeError: Retry not raised.
--    """
--    global _doctest_retry_count
--    _doctest_retry_count += 1
--    if _doctest_retry_count % 2 == 1:
--        print "Retry raised."
--        raise RetryTest
--    raise RuntimeError("Retry not raised.")
--
--
--def testRetryTestInDoctest2():
--    """Test a RetryTest exception in a doctest works as expected.
--
--    This test is the same as testRetryTestInDoctest, except it confirms
--    that a RetryException raised where a different exception was expected
--    works.
--
--    >>> retry_on_odd_numbered_calls2()
--    Traceback (most recent call last):
--    ...
--    RuntimeError: Retry not raised.
--    """
++#    >>> try:
++#    ...     retry_on_odd_numbered_calls()
++#    ... except RetryTest:
++#    ...     print "Caught RetryTest."
++#    ...
++#    Retry raised.
++#    Caught RetryTest.
++#    >>> try:
++#    ...     retry_on_odd_numbered_calls()
++#    ... except RetryTest:
++#    ...     print "Caught RetryTest."
++#    ...
++#    Retry not raised.
++#    """
++#    global _doctest_retry_count
++#    _doctest_retry_count += 1
++#    if _doctest_retry_count % 2 == 1:
++#        print "Retry raised."
++#        raise RetryTest
++#    print "Retry not raised."
++
++
++#def testRetryTestInDoctest_will_raise_but_testrunner_ignores_it():
++#    """Test a RetryTest exception in a doctest works as expected.
++
++#    This doctest raises a RetryTest exception the first time it is run.
++#    On the second run, it succeeds.
++
++#    If the testrunner is correctly handling RetryTest exceptions raised
++#    by doctests, then the RetryTest exception will not be reported as
++#    a failure. This test will then be rerun and succeed.
++
++#    If the testrunner is not correctly handling RetryTest exceptions,
++#    then the RetryTesst exception will be flagged as an error.
++
++#    This test confirms that a RetryException raised where no exception
++#    was expected works.
++
++#    >>> retry_on_odd_numbered_calls()
++#    Retry not raised.
++#    """
++
++
++#def retry_on_odd_numbered_calls2():
++#    """Helper for doctest RetryTest test.
++
++#    This helper raises a RetryTest exception on odd numbered calls,
++#    and a RuntimeError on even numbered calls.
++
++#    >>> try:
++#    ...     retry_on_odd_numbered_calls2()
++#    ... except RetryTest:
++#    ...     print "Caught RetryTest."
++#    ...
++#    Retry raised.
++#    Caught RetryTest.
++#    >>> try:
++#    ...     retry_on_odd_numbered_calls2()
++#    ... except RetryTest:
++#    ...     print "Caught RetryTest."
++#    ...
++#    Traceback (most recent call last):
++#    ...
++#    RuntimeError: Retry not raised.
++#    """
++#    global _doctest_retry_count
++#    _doctest_retry_count += 1
++#    if _doctest_retry_count % 2 == 1:
++#        print "Retry raised."
++#        raise RetryTest
++#    raise RuntimeError("Retry not raised.")
++
++
++#def testRetryTestInDoctest2():
++#    """Test a RetryTest exception in a doctest works as expected.
++
++#    This test is the same as testRetryTestInDoctest, except it confirms
++#    that a RetryException raised where a different exception was expected
++#    works.
++
++#    >>> retry_on_odd_numbered_calls2()
++#    Traceback (most recent call last):
++#    ...
++#    RuntimeError: Retry not raised.
++#    """
 === modified file 'lib/canonical/widgets/date.py'
 --- lib/canonical/widgets/date.py	2009-06-25 05:30:52 +0000
 +++ lib/canonical/widgets/date.py	2009-08-13 19:36:01 +0000
@@ -80,7 +80,7 @@
        >>> print widget.getInputValue()  #doctest: +ELLIPSIS
        Traceback (most recent call last):
        ...
--      WidgetInputError: (... Please pick a date after 2006-05-22 17:00:00)
++      WidgetInputError: (...Please pick a date after 2006-05-22 17:00:00...)
      If the date provided is greater than from_date then the widget works as
      expected.
@@ -96,7 +96,7 @@
        >>> print widget.getInputValue()  #doctest: +ELLIPSIS
        Traceback (most recent call last):
        ...
--      WidgetInputError: (... Please pick a date before 2008-01-25 16:00:00)
++      WidgetInputError: (...Please pick a date before 2008-01-25 16:00:00...)
      A datetime picker can be disabled initially:
@@ -288,7 +288,7 @@
          value = super(DateTimeWidget, self).getInputValue()
          if value is None:
              return None
--        # Establish if the value is within the date range.
++        # Establish if the value is within the date range.
          self._align_date_constraints_with_time_zone()
          if self.from_date is not None and value < self.from_date:
              limit = self.from_date.strftime(self.timeformat)
@@ -570,4 +570,3 @@
              return u""
          value = value.astimezone(time_zone)
          return escape(value.strftime("%Y-%m-%d %H:%M:%S %Z"))
--
 === modified file 'lib/lp/answers/interfaces/questiontarget.py'
 --- lib/lp/answers/interfaces/questiontarget.py	2009-06-24 23:10:46 +0000
 +++ lib/lp/answers/interfaces/questiontarget.py	2009-08-13 15:22:00 +0000
@@ -13,8 +13,6 @@
      'ISearchQuestionsForm',
+     ]
--import sets
--
  from zope.interface import Interface
  from zope.schema import Choice, List, Set, TextLine
@@ -156,7 +154,7 @@
      status = Set(title=_('Status'), required=False,
                   value_type=Choice(vocabulary=QuestionStatus),
--                 default=sets.Set(QUESTION_STATUS_DEFAULT_SEARCH))
++                 default=set(QUESTION_STATUS_DEFAULT_SEARCH))
  class IAnswersFrontPageSearchForm(ISearchQuestionsForm):
 === modified file 'lib/lp/answers/stories/answer-contact-report.txt'
 --- lib/lp/answers/stories/answer-contact-report.txt	2009-06-12 16:36:02 +0000
 +++ lib/lp/answers/stories/answer-contact-report.txt	2009-08-13 13:33:26 +0000
@@ -1,6 +1,6 @@
  = Answer Contact Report =
--To view the answer contact report for a given person, the user chooses
++To view the answer contact report for a given person, the user chooses
  the 'Answer Contact For' link from the actions portlet while viewing
  the Person's page.
@@ -11,7 +11,7 @@
  Since No Privileges Person is not an answer contact, the report states
  that.
--
++
      >>> content = find_main_content(anon_browser.contents)
      >>> print content.find('p').renderContents()
      No Privileges Person is not an answer contact for any project.
@@ -23,7 +23,7 @@
      >>> anon_browser.getLink('Answer contact for').click()
      >>> print anon_browser.title
      Projects for which Foo Bar is an answer contact
--
++
      >>> content = find_tag_by_id(
      ...     anon_browser.contents, "direct-answer-contacts-for-list")
      >>> print extract_text(content).encode('ascii', 'backslashreplace')
@@ -35,7 +35,7 @@
      >>> print extract_text(content)
      Gnome Baker
      The Gnome Panel Applets
--
++
  Clicking on the name of the project will show the project answers.
      >>> anon_browser.getLink('Gnome Baker').click()
@@ -51,18 +51,18 @@
      >>> browser.getLink('Answer contact for').click()
      >>> print browser.title
      Projects for which Sample Person is an answer contact
--
++
      >>> content = find_tag_by_id(
      ... browser.contents, "team-answer-contacts-for-list")
      >>> print extract_text(content)
      Gnome Baker &mdash; Remove team
      The Gnome Panel Applets &mdash; Remove team
--
++
      >>> browser.getLink(id="gnomebaker-setteamanswercontact").click()
      >>> print browser.title
      Answer contact for Gnome Baker
--
--The Remove yourself/team links only appears in his profile. He cannot
++
++The Remove yourself/team links only appears in his profile. He cannot
  see the link for other users
      >>> browser.open(
@@ -70,7 +70,7 @@
      >>> browser.getLink('Answer contact for').click()
      >>> print browser.title
      Projects for which Foo Bar is an answer contact
--
++
      >>> content = find_tag_by_id(browser.contents, "direct-answer-contacts-for-list")
      >>> print extract_text(content).encode('ascii', 'backslashreplace')
      Gnome Baker
 === modified file 'lib/lp/answers/stories/questions-index.txt'
 --- lib/lp/answers/stories/questions-index.txt	2009-03-24 12:43:49 +0000
 +++ lib/lp/answers/stories/questions-index.txt	2009-08-13 21:32:59 +0000
@@ -60,7 +60,7 @@
      ...     anon_browser.contents, 'most-active-projects'))
      Most active projects
      Ubuntu
--    Mozilla Firefox
++    Mozilla Firefox ...
  Clicking on these project links will bring the user to the project
  Answers front page:
 === modified file 'lib/lp/answers/templates/person-answer-contact-for.pt'
 --- lib/lp/answers/templates/person-answer-contact-for.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/answers/templates/person-answer-contact-for.pt	2009-08-13 13:33:26 +0000
@@ -23,8 +23,8 @@
      <ul class="listing" id="direct-answer-contacts-for-list">
        <li tal:repeat="question_target view/direct_question_targets">
--        <a href="#"
--          tal:attributes="href string:${question_target/fmt:url}"
++        <a
++          tal:attributes="href question_target/fmt:url:answers"
            tal:content="question_target/title">Project Title</a>
          <tal:link condition="view/showRemoveYourselfLink">
            &mdash;
@@ -42,8 +42,8 @@
      <ul class="listing" id="team-answer-contacts-for-list">
        <li tal:repeat="question_target view/team_question_targets">
--        <a href="#"
--          tal:attributes="href string:${question_target/fmt:url}"
++        <a
++          tal:attributes="href question_target/fmt:url:answers"
            tal:content="question_target/title">Project Title</a>
          <tal:link condition="view/showRemoveYourselfLink">
            &mdash;
 === modified file 'lib/lp/answers/templates/questions-index.pt'
 --- lib/lp/answers/templates/questions-index.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/answers/templates/questions-index.pt	2009-08-13 21:32:59 +0000
@@ -81,11 +81,7 @@
                tal:condition="projects"
                style="list-style: none;">
              <li tal:repeat="project projects">
--              <a href="#"
--                 tal:attributes="href project/fmt:url">
--                <img alt="" tal:replace="structure project/image:icon" />
--                <span tal:replace="project/displayname" />
--              </a>
++              <a tal:replace="structure project/fmt:link:answers"/>
              </li>
            </ul>
          </div>
@@ -110,28 +106,29 @@
            </table>
          </div>
        </div>
--                        <div id="application-footer">
--                          <div>
--                            <strong
--                              tal:content="view/answered_question_count"
--                            >52</strong>
--                            questions answered and
--                            <strong
--                              tal:content="view/solved_question_count"
--                            >52</strong>
--                            questions solved out of
--                          </div>
--                          <div>
--                            <strong
--                              tal:content="view/question_count"
--                            >318</strong>
--                            questions asked across
--                            <strong
--                              tal:content="view/projects_with_questions_count"
--                            >28</strong>
--                            projects
--                          </div>
--                        </div>
++
++      <div id="application-footer">
++        <div>
++          <strong
++            tal:content="view/answered_question_count"
++          >52</strong>
++          questions answered and
++          <strong
++            tal:content="view/solved_question_count"
++          >52</strong>
++          questions solved out of
++        </div>
++        <div>
++          <strong
++            tal:content="view/question_count"
++          >318</strong>
++          questions asked across
++          <strong
++            tal:content="view/projects_with_questions_count"
++          >28</strong>
++          projects
++        </div>
++      </div>
      </div><!--main-->
 === modified file 'lib/lp/app/browser/tests/base-layout.txt'
 --- lib/lp/app/browser/tests/base-layout.txt	2009-08-06 11:07:45 +0000
 +++ lib/lp/app/browser/tests/base-layout.txt	2009-08-13 02:24:56 +0000
@@ -44,7 +44,6 @@
        <body id="document" class="tab-overview main_side public yui-skin-sam">
          <div class="yui-d0">
            <div id="locationbar"> ...
--             <form id="globalsearch" ...
               <div id="lp-hierarchy" class="home">...
            <div class="location-portlet top-portlet">
              <img ...
@@ -77,7 +76,8 @@
              </div><!-- yui-b side -->
            </div><!-- yui-t4 -->
            <div id="footer" class="footer"> ...
--          </div><!-- footer-->
++            <form id="globalsearch" ...
++          </div>
          </div><!-- yui-d0-->
          <script>LP.client.cache['context'] ...
        </body>
@@ -105,7 +105,6 @@
        <body id="document" class="tab-overview main_only public yui-skin-sam">
          <div class="yui-d0">
            <div id="locationbar"> ...
--             <form id="globalsearch" ...
               <div id="lp-hierarchy" class="home">...
            <div class="location-portlet top-portlet">
              <img ...
@@ -134,7 +133,8 @@
              <!-- yui-b side -->
              <!-- yui-t4 -->
            <div id="footer" class="footer"> ...
--          </div><!-- footer-->
++            <form id="globalsearch" ...
++          </div>
          </div><!-- yui-d0-->
          <script>LP.client.cache['context'] ...
        </body>
@@ -191,7 +191,7 @@
              <!-- yui-b side -->
              <!-- yui-t4 -->
            <div id="footer" class="footer"> ...
--          </div><!-- footer-->
++          </div>
          </div><!-- yui-d0-->
          <script>LP.client.cache['context'] ...
        </body>
@@ -241,7 +241,7 @@
              <!-- yui-b side -->
              <!-- yui-t4 -->
            <div id="footer" class="footer"> ...
--          </div><!-- footer-->
++          </div>
          </div><!-- yui-d0-->
          <script>LP.client.cache['context'] ...
        </body>
 === modified file 'lib/lp/app/templates/base-layout-macros.pt'
 --- lib/lp/app/templates/base-layout-macros.pt	2009-08-06 11:07:45 +0000
 +++ lib/lp/app/templates/base-layout-macros.pt	2009-08-13 02:24:56 +0000
@@ -330,4 +330,41 @@
    </ul>
  </metal:location-tabs>
++
++<metal:footer define-macro="footer">
++  <div id="footer" class="footer">
++    <div class="lp-arcana">
++      <img src="/@@/launchpad-logo-and-name-hierarchy.png" alt="Launchpad"/>
++      &nbsp;&bull;&nbsp;
++      <a href="/+tour">Take the tour</a>
++      &nbsp;&bull;&nbsp;
++      <a href="https://help.launchpad.net/">Read the guide</a>
++      &nbsp;
++      <form id="globalsearch" method="get" accept-charset="UTF-8"
++        class="sprite-after search-icon"
++        tal:condition="view/macro:pagehas/globalsearch"
++        tal:attributes="action string:${rooturl}+search">
++        <input type="search" id="search-text" name="field.text" />
++      </form>
++    </div>
++
++    <div class="colophon">
++      &copy; 2004-2009
++      <a href="http://canonical.com/">Canonical&nbsp;Ltd.</a>
++      &nbsp;&bull;&nbsp;
++      <a href="/legal">Terms of use</a>
++      &nbsp;&bull;&nbsp;
++      <a href="/feedback"
++        tal:condition="request/lp:person">Contact us</a>
++      <span id="lp-version" tal:condition="not:is_lpnet">
++      &nbsp;&bull;&nbsp;
++        <a href="https://help.launchpad.net/LaunchpadReleases"
++          tal:content="version"/>
++        <tal:devmode condition="devmode">devmode</tal:devmode>
++        <tal:demo condition="is_demo">demo site</tal:demo>
++        <tal:edge condition="is_edge">beta site</tal:edge>
++      </span>
++    </div>
++  </div>
++</metal:footer>
  </macros>
 === modified file 'lib/lp/app/templates/base-layout.pt'
 --- lib/lp/app/templates/base-layout.pt	2009-08-11 04:35:26 +0000
 +++ lib/lp/app/templates/base-layout.pt	2009-08-13 02:24:56 +0000
@@ -81,12 +81,6 @@
      <div class="yui-d0">
        <div id="locationbar">
          <tal:login replace="structure context/@@login_status" />
--        <form id="globalsearch" method="get" accept-charset="UTF-8"
--          class="sprite search-icon"
--          tal:condition="view/macro:pagehas/globalsearch"
--          tal:attributes="action string:${rooturl}+search">
--          <input type="search" id="search-text" name="field.text" />
--        </form>
          <tal:hierarchy replace="structure context/@@+hierarchy" />
        </div><!--id="locationbar"-->
@@ -135,28 +129,8 @@
          </div><!-- yui-b side -->
        </div><!-- yui-t4 -->
--      <div id="footer" class="footer">
--        <div id="lp-arcana">
--          &copy;&nbsp;2004-2009&nbsp;<a
--            href="http://canonical.com/">Canonical&nbsp;Ltd.</a> |
--          <a href="/legal">Terms of use</a>
--          <span id="lp-version" tal:condition="not:is_lpnet"> |
--            <a href="https://help.launchpad.net/LaunchpadReleases"
--              tal:content="version"/>
--            <tal:devmode condition="devmode">devmode</tal:devmode>
--            <tal:demo condition="is_demo">demo site</tal:demo>
--            <tal:edge condition="is_edge">beta site</tal:edge>
--          </span>
--        </div>
--
--        <div id="colophon">
--          <a href="/+tour"
--            tal:condition="not:request/lp:person">What is Launchpad?</a>
--          <a href="/feedback"
--            tal:condition="request/lp:person">Contact us</a> |
--          <a href="https://help.launchpad.net/">Get help with Launchpad</a>
--        </div>
--      </div><!-- footer-->
++      <metal:footer
++        use-macro="context/@@+base-layout-macros/footer"/>
      </div><!-- yui-d0-->
      <metal:lp-client-cache
 === modified file 'lib/lp/bugs/stories/bugtracker/xx-bugtracker-handshake-tokens.txt'
 --- lib/lp/bugs/stories/bugtracker/xx-bugtracker-handshake-tokens.txt	2009-06-12 16:36:02 +0000
 +++ lib/lp/bugs/stories/bugtracker/xx-bugtracker-handshake-tokens.txt	2009-07-16 13:28:25 +0000
@@ -24,4 +24,4 @@
      >>> anon_browser.open(token_url)
      Traceback (most recent call last):
        ...
--    HTTPError: HTTP Error 405: Method Not Allowed
++    httperror_seek_wrapper: HTTP Error 405: Method Not Allowed
 === modified file 'lib/lp/bugs/stories/bugtracker/xx-bugtracker-remote-bug.txt'
 --- lib/lp/bugs/stories/bugtracker/xx-bugtracker-remote-bug.txt	2009-08-11 06:29:52 +0000
 +++ lib/lp/bugs/stories/bugtracker/xx-bugtracker-remote-bug.txt	2009-08-12 13:34:48 +0000
@@ -38,7 +38,7 @@
    >>> browser.open('http://launchpad.dev/bugs/bugtrackers/mozilla.org/99999')
    Traceback (most recent call last):
    ...
--  HTTPError: HTTP Error 404: Not Found
++  httperror_seek_wrapper: HTTP Error 404: Not Found
    >>> browser.handleErrors = False
 === modified file 'lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt'
 --- lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt	2009-07-01 13:16:44 +0000
 +++ lib/lp/bugs/stories/bugwatches/xx-bugwatch-comments.txt	2009-08-13 21:32:59 +0000
@@ -83,7 +83,7 @@
  Launchpad Beta Testers team such as sabdfl:
      >>> sabdfl_browser.open(
--    ...     'http://launchpad.dev/redfish/+bug/15/comments/1')
++    ...     'http://bugs.launchpad.dev/redfish/+bug/15/comments/1')
      >>> print_comments(sabdfl_browser.contents)
      <p>Package: gnome-volume-<wbr></wbr>manager<br />
      Version: 1.2.0-1<br />
@@ -125,11 +125,12 @@
      >>> logout()
      >>> sabdfl_browser.open(
--    ...     'http://launchpad.dev/redfish/+bug/15/comments/1')
++    ...     'http://bugs.launchpad.dev/redfish/+bug/15/comments/1')
      >>> print_comments(sabdfl_browser.contents)
  Anonymous users can't see the comment either.
--    >>> anon_browser.open('http://launchpad.dev/redfish/+bug/15/comments/1')
++    >>> anon_browser.open(
++    ...     'http://bugs.launchpad.dev/redfish/+bug/15/comments/1')
      >>> print_comments(anon_browser.contents)
 === modified file 'lib/lp/code/stories/branches/xx-upload-directions.txt'
 --- lib/lp/code/stories/branches/xx-upload-directions.txt	2009-04-17 10:32:16 +0000
 +++ lib/lp/code/stories/branches/xx-upload-directions.txt	2009-08-13 21:32:59 +0000
@@ -45,7 +45,7 @@
      >>> instructions = find_tag_by_id(content, 'upload-directions')
      >>> print instructions.renderContents()
      Only
--    <a href="/~name12">Sample Person</a>
++    <a href="http://launchpad.dev/~name12">Sample Person</a>
      can upload to this branch. If you are Sample Person please
      <a href="+login">log in</a> for upload directions.
@@ -57,7 +57,7 @@
      >>> instructions = find_tag_by_id(content, 'upload-directions')
      >>> print instructions.renderContents()
      You cannot upload to this branch. Only
--    <a href="/~name12">Sample Person</a>
++    <a href="http://launchpad.dev/~name12">Sample Person</a>
      can upload to this branch.
  The user is the owner of the branch and logs in. The page gives the full upload
@@ -90,7 +90,7 @@
      >>> instructions = find_tag_by_id(content, 'ssh-key-directions')
      >>> print instructions.renderContents()
      To authenticate with the Launchpad branch upload service, you need to
--    <a href="/~name12/+editsshkeys">
++    <a href="http://launchpad.dev/~name12/+editsshkeys">
      register a SSH key </a>.
  Click the link and register a key.
@@ -149,7 +149,7 @@
      >>> instructions = find_tag_by_id(content, 'upload-directions')
      >>> print instructions.renderContents()
      Members of <a
--    href="/~landscape-developers">Landscape
++    href="http://launchpad.dev/~landscape-developers">Landscape
      Developers</a> can upload to this branch. <a href="+login">Log in</a> for
      directions.
@@ -161,7 +161,7 @@
      >>> instructions = find_tag_by_id(content, 'upload-directions')
      >>> print instructions.renderContents()
      You cannot upload to this branch. Members of <a
--    href="/~landscape-developers">Landscape
++    href="http://launchpad.dev/~landscape-developers">Landscape
      Developers</a> can upload to this branch.
  Finally, if the user is a member of the team, we display the same "you can
 === modified file 'lib/lp/code/templates/project-branches.pt'
 --- lib/lp/code/templates/project-branches.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/code/templates/project-branches.pt	2009-08-10 19:04:22 +0000
@@ -6,24 +6,22 @@
    xml:lang="en"
    lang="en"
    dir="ltr"
--  metal:use-macro="context/@@main_template/master"
++  metal:use-macro="view/macro:page/onecolumn"
    i18n:domain="launchpad"
+ >
  <body>
--<metal:leftportlets fill-slot="portlets_one">
--</metal:leftportlets>
--
--<metal:rightportlets fill-slot="portlets_two">
--</metal:rightportlets>
--
--<metal:heading fill-slot="pageheading">
++<div metal:fill-slot="main" tal:define="branches view/branches">
++
++  <div style="float:right" id="floating-links"
++       tal:define="menu context/menu:overview">
++    <div tal:define="link menu/branch_visibility"
++         tal:condition="link/enabled"
++         tal:content="structure link/render" />
++  </div>
++
    <h1>Bazaar branches for <tal:project-name replace="context/displayname"/></h1>
--</metal:heading>
--
--<div metal:fill-slot="main"
--     tal:define="branches view/branches">
    <tal:branchlisting content="structure branches/@@+branch-listing" />
 === modified file 'lib/lp/registry/browser/configure.zcml'
 --- lib/lp/registry/browser/configure.zcml	2009-08-12 13:20:10 +0000
 +++ lib/lp/registry/browser/configure.zcml	2009-08-14 00:52:28 +0000
@@ -480,8 +480,11 @@
      <browser:menus
          classes="
              ProjectFacets
++            ProjectActionMenu
++            ProjectEditNavigationMenu
              ProjectOverviewMenu
              ProjectBountiesMenu
++            ProjectBugsMenu
              ProjectSeriesSpecificationsMenu
              ProjectSpecificationsMenu
              ProjectSetContextMenu
 === modified file 'lib/lp/registry/browser/project.py'
 --- lib/lp/registry/browser/project.py	2009-08-05 01:49:41 +0000
 +++ lib/lp/registry/browser/project.py	2009-08-13 18:10:15 +0000
@@ -13,12 +13,14 @@
      'ProjectBountiesMenu',
      'ProjectBrandingView',
      'ProjectBreadcrumbBuilder',
++    'ProjectBugsMenu',
      'ProjectEditView',
      'ProjectFacets',
      'ProjectMaintainerReassignmentView',
      'ProjectNavigation',
      'ProjectRdfView',
      'ProjectReviewView',
++    'ProjectActionMenu',
      'ProjectOverviewMenu',
      'ProjectSeriesSpecificationsMenu',
      'ProjectSetBreadcrumbBuilder',
@@ -34,6 +36,7 @@
  from zope.component import getUtility
  from zope.event import notify
  from zope.formlib import form
++from zope.interface import implements, Interface
  from zope.schema import Choice
  from z3c.ptcompat import ViewPageTemplateFile
@@ -41,6 +44,7 @@
  from canonical.cachedproperty import cachedproperty
  from canonical.launchpad import _
  from canonical.launchpad.webapp.interfaces import NotFoundError
++from canonical.launchpad.webapp.menu import NavigationMenu
  from lp.registry.interfaces.product import IProductSet
  from lp.registry.interfaces.project import (
      IProject, IProjectSeries, IProjectSet)
@@ -137,41 +141,33 @@
      def bugs(self):
          site = 'bugs'
          text = 'Bugs'
--
          return Link('', text, enabled=self.context.hasProducts(), site=site)
      def answers(self):
          site = 'answers'
          text = 'Answers'
--
          return Link('', text, enabled=self.context.hasProducts(), site=site)
      def specifications(self):
          site = 'blueprints'
          text = 'Blueprints'
--
          return Link('', text, enabled=self.context.hasProducts(), site=site)
      def translations(self):
          site = 'translations'
          text = 'Translations'
--
          return Link('', text, enabled=self.context.hasProducts(), site=site)
--class ProjectOverviewMenu(ApplicationMenu):
--
--    usedfor = IProject
--    facet = 'overview'
--    links = [
--        'edit', 'branding', 'driver', 'reassign', 'top_contributors',
--        'mentorship', 'announce', 'announcements', 'administer',
--        'branch_visibility', 'rdf', 'subscribe']
--
--    @enabled_with_permission('launchpad.Edit')
--    def edit(self):
--        text = 'Change details'
--        return Link('+edit', text, icon='edit')
++class ProjectAdminMenuMixin:
++
++    @enabled_with_permission('launchpad.Admin')
++    def administer(self):
++        text = 'Administer'
++        return Link('+review', text, icon='edit')
++
++
++class ProjectEditMenuMixin(ProjectAdminMenuMixin):
      @enabled_with_permission('launchpad.Edit')
      def branding(self):
@@ -181,14 +177,30 @@
      @enabled_with_permission('launchpad.Edit')
      def reassign(self):
          text = 'Change maintainer'
--        return Link('+reassign', text, icon='edit')
++        summary = 'Change the maintainer of this project group'
++        return Link('+reassign', text, summary, icon='edit')
      @enabled_with_permission('launchpad.Edit')
      def driver(self):
          text = 'Appoint driver'
--        summary = 'Someone with permission to set goals for all projects'
++        summary = 'Appoint the driver of this project group'
          return Link('+driver', text, summary, icon='edit')
++
++class ProjectOverviewMenu(ProjectEditMenuMixin, ApplicationMenu):
++
++    usedfor = IProject
++    facet = 'overview'
++    links = [
++        'branding', 'driver', 'reassign', 'top_contributors', 'mentorship',
++        'announce', 'announcements', 'branch_visibility', 'rdf',
++        'new_product', 'administer', 'milestones']
++
++    @enabled_with_permission('launchpad.Edit')
++    def new_product(self):
++        text = 'Register another project in %s' % self.context.displayname
++        return Link('+newproduct', text, icon='edit')
++
      def top_contributors(self):
          text = 'More contributors'
          return Link('+topcontributors', text, icon='info')
@@ -213,26 +225,59 @@
          enabled = bool(self.context.getAnnouncements())
          return Link('+announcements', text, enabled=enabled)
++    def milestones(self):
++        text = 'See all milestones'
++        return Link('+milestones', text)
++
      def rdf(self):
          text = structured(
              'Download <abbr title="Resource Description Framework">'
              'RDF</abbr> metadata')
--        return Link('+rdf', text, icon='download')
--
--    @enabled_with_permission('launchpad.Admin')
--    def administer(self):
--        text = 'Administer'
--        return Link('+review', text, icon='edit')
++        return Link('+rdf', text, icon='download-icon')
      @enabled_with_permission('launchpad.Admin')
      def branch_visibility(self):
          text = 'Define branch visibility'
          return Link('+branchvisibility', text, icon='edit', site='mainsite')
++
++class IProjectActionMenu(Interface):
++    """Marker interface for views that use ProjectActionMenu."""
++
++
++class ProjectActionMenu(ProjectAdminMenuMixin, NavigationMenu):
++
++    usedfor = IProjectActionMenu
++    facet = 'overview'
++    title = 'Action menu'
++    links = ('subscribe', 'edit', 'administer')
++
++    # XXX: salgado, bug=412178, 2009-08-10: This should be shown in the +index
++    # page of the project's bugs facet, but that would require too much work
++    # and I just want to convert this page to 3.0, so I'll leave it here for
++    # now.
      def subscribe(self):
          text = 'Subscribe to bug mail'
          return Link('+subscribe', text, icon='edit')
++    @enabled_with_permission('launchpad.Edit')
++    def edit(self):
++        text = 'Change details'
++        return Link('+edit', text, icon='edit')
++
++
++class IProjectEditMenu(Interface):
++    """A marker interface for the 'Change details' navigation menu."""
++
++
++class ProjectEditNavigationMenu(NavigationMenu, ProjectEditMenuMixin):
++    """A sub-menu for different aspects of editing a Project's details."""
++
++    usedfor = IProjectEditMenu
++    facet = 'overview'
++    title = 'Change project group'
++    links = ('branding', 'reassign', 'driver', 'administer')
++
  class ProjectBountiesMenu(ApplicationMenu):
@@ -286,12 +331,24 @@
          return Link('+addquestion', text, icon='add')
++class ProjectBugsMenu(ApplicationMenu):
++
++    usedfor = IProject
++    facet = 'bugs'
++    links = ['new']
++
++    def new(self):
++        text = 'Report a Bug'
++        return Link('+filebug', text, icon='add')
++
++
  class ProjectView(HasAnnouncementsView, FeedsMixin):
--    pass
++    implements(IProjectActionMenu)
  class ProjectEditView(LaunchpadEditFormView):
      """View class that lets you edit a Project object."""
++    implements(IProjectEditMenu)
      label = "Change project group details"
      schema = IProject
 === modified file 'lib/lp/registry/browser/tests/project-add-views.txt'
 --- lib/lp/registry/browser/tests/project-add-views.txt	2009-05-12 08:11:06 +0000
 +++ lib/lp/registry/browser/tests/project-add-views.txt	2009-08-13 19:36:01 +0000
@@ -310,4 +310,5 @@
      >>> for error in view.errors:
      ...     print error
--    ('name', 'URL', badger is already used by another project)
++    ('name', 'URL',
++     LaunchpadValidationError(u'badger is already used by another project'))
 === modified file 'lib/lp/registry/doc/product-widgets.txt'
 --- lib/lp/registry/doc/product-widgets.txt	2009-07-01 13:16:44 +0000
 +++ lib/lp/registry/doc/product-widgets.txt	2009-08-13 19:36:01 +0000
@@ -301,7 +301,7 @@
      ...             print
      >>> license_widget.getInputValue()
--    Set([<DBItem License.GNU_GPL_V2, (130) ...>])
++    set([<DBItem License.GNU_GPL_V2, (130) ...>])
      >>> print_checked_items(license_widget())
      [ ] Apache License ...
 === modified file 'lib/lp/registry/interfaces/product.py'
 --- lib/lp/registry/interfaces/product.py	2009-08-10 17:08:27 +0000
 +++ lib/lp/registry/interfaces/product.py	2009-08-13 15:22:00 +0000
@@ -24,7 +24,6 @@
  import re
--import sets
  from textwrap import dedent
@@ -955,8 +954,7 @@
          title=_('Licenses'),
          value_type=Choice(vocabulary=License),
          required=False,
--        # Zope requires sets.Set() instead of the builtin set().
--        default=sets.Set(
++        default=set(
              [License.OTHER_PROPRIETARY, License.OTHER_OPEN_SOURCE]))
      has_zero_licenses = Choice(
 === modified file 'lib/lp/registry/stories/announcements/xx-announcements.txt'
 --- lib/lp/registry/stories/announcements/xx-announcements.txt	2009-08-11 17:32:21 +0000
 +++ lib/lp/registry/stories/announcements/xx-announcements.txt	2009-08-13 18:10:15 +0000
@@ -265,7 +265,7 @@
      >>> anon_browser.open('http://launchpad.dev/apache')
      >>> count_show_links(anon_browser.contents)
--    2
++    1
      >>> anon_browser.open('http://launchpad.dev/tomcat')
      >>> count_show_links(anon_browser.contents)
 === modified file 'lib/lp/registry/stories/distribution/xx-distribution-countrymirrors.txt'
 --- lib/lp/registry/stories/distribution/xx-distribution-countrymirrors.txt	2009-07-23 13:44:13 +0000
 +++ lib/lp/registry/stories/distribution/xx-distribution-countrymirrors.txt	2009-08-05 18:52:52 +0000
@@ -60,5 +60,5 @@
      >>> browser.open('http://launchpad.dev/debian/+countrymirrors-archive')
      Traceback (most recent call last):
      ...
--    HTTPError: HTTP Error 404: Not Found
++    httperror_seek_wrapper: HTTP Error 404: Not Found
 === modified file 'lib/lp/registry/stories/gpg-coc/98-cocacknowledge.txt'
 --- lib/lp/registry/stories/gpg-coc/98-cocacknowledge.txt	2009-05-12 01:39:29 +0000
 +++ lib/lp/registry/stories/gpg-coc/98-cocacknowledge.txt	2009-08-13 21:32:59 +0000
@@ -24,12 +24,8 @@
    >>> admin_browser.getControl(name='searchfor').value = ["all"]
    >>> admin_browser.getControl(name='name').value = "mark"
    >>> admin_browser.getControl(name='search').click()
--  >>> 'paper submission accepted by Foo Bar' in admin_browser.contents
--  True
--  >>> '<a href="http://launchpad.dev/~sabdfl"' in admin_browser.contents
--  True
--  >>> '[ACTIVE]' in admin_browser.contents
--  True
++  >>> print extract_text(find_tag_by_id(admin_browser.contents, 'matches'))
++  Mark ... paper submission accepted by Foo Bar [ACTIVE]
  Test if the advertisement email was sent:
 === modified file 'lib/lp/registry/stories/mailinglists/lifecycle.txt'
 --- lib/lp/registry/stories/mailinglists/lifecycle.txt	2009-07-23 13:44:13 +0000
 +++ lib/lp/registry/stories/mailinglists/lifecycle.txt	2009-08-06 19:11:48 +0000
@@ -580,7 +580,7 @@
      >>> admin_browser.getControl(name='field.aardvarks').value = ['approve']
      >>> admin_browser.getControl('Submit').click()
      >>> act()
--    >>> browser.reload()
++    >>> browser.open(browser.url) # A `reload` would resubmit.
      >>> browser.getLink('Mailing list archive')
      <Link text='Mailing list archive'
            url='http://lists.launchpad.dev/aardvarks'>
@@ -618,7 +618,7 @@
      >>> admin_browser.getControl(name='field.antelopes').value = ['approve']
      >>> admin_browser.getControl('Submit').click()
      >>> act()
--    >>> user_browser.reload()
++    >>> user_browser.open(user_browser.url) # A `reload` would resubmit.
      >>> user_browser.getLink('Mailing list archive')
      <Link text='Mailing list archive'
            url='http://lists.launchpad.dev/antelopes'>
 === modified file 'lib/lp/registry/stories/milestone/object-milestones.txt'
 --- lib/lp/registry/stories/milestone/object-milestones.txt	2009-08-12 01:12:12 +0000
 +++ lib/lp/registry/stories/milestone/object-milestones.txt	2009-08-14 00:52:28 +0000
@@ -88,7 +88,9 @@
  === Projects ===
--The main project page has a portlet "Active milestones":
++The project "All milestones" page lists all milestones for all products and
++series, including the inactive ones. They do not include the bug and blueprint
++counts (because they are costly to retrieve).
      >>> from canonical.launchpad.ftests import login, logout
      >>> from lp.registry.tests.test_project_milestone import (
@@ -98,15 +100,6 @@
      >>> test_helper.setUpProjectMilestoneTests()
      >>> logout()
      >>> anon_browser.open('http://launchpad.dev/gnome')
--    >>> print milestones_in_portlet(anon_browser)
--    2011-04-01  1.2
--    2010-04-02  1.1.
--    2010-04-01  1.1
--
--The project "All milestones" page lists all milestones for all products and
--series, including the inactive ones. They do not include the bug and blueprint
--counts (because they are costly to retrieve).
--
      >>> anon_browser.getLink('See all milestones').click()
      >>> print all_milestones(anon_browser)
      GNOME 2.1.6  None        This is an inactive milestone
 === modified file 'lib/lp/registry/stories/product/xx-launchpad-project-search.txt'
 --- lib/lp/registry/stories/product/xx-launchpad-project-search.txt	2009-06-11 19:21:46 +0000
 +++ lib/lp/registry/stories/product/xx-launchpad-project-search.txt	2009-08-14 03:21:52 +0000
@@ -118,7 +118,7 @@
      >>> anon_browser.open('http://launchpad.dev/projects?text=ubuntu')
      >>> print_search_results(anon_browser)
--    sprite distribution Ubuntu
++    bg-image Ubuntu
      sprite distribution ubuntutest
      sprite product Evolution
      sprite product Tomcat
 === modified file 'lib/lp/registry/stories/project/xx-project-driver.txt'
 --- lib/lp/registry/stories/project/xx-project-driver.txt	2008-07-02 12:51:24 +0000
 +++ lib/lp/registry/stories/project/xx-project-driver.txt	2009-08-13 18:10:15 +0000
@@ -26,6 +26,6 @@
  Sample Person is listed as the driver of the project.
--  >>> for tag in find_tags_by_class(browser.contents, 'summary'):
--  ...     tag.find(text='Driver:').findNext('a').string
--  u'Sample Person'
++  >>> print extract_text(find_tag_by_id(browser.contents, 'driver'))
++  Driver: Sample Person
++  Appoint driver
 === modified file 'lib/lp/registry/stories/project/xx-project-edit.txt'
 --- lib/lp/registry/stories/project/xx-project-edit.txt	2009-08-01 00:02:55 +0000
 +++ lib/lp/registry/stories/project/xx-project-edit.txt	2009-08-13 18:10:15 +0000
@@ -53,21 +53,20 @@
  The project summary shows the status as reviewed for admins only.
--    >>> for tag in find_tags_by_class(admin_browser.contents, 'summary'):
--    ...       print extract_text(
--    ...           tag.find(text='Project group status:').findParent('tr'))
--    Project group status:
--    Active
--    Reviewed
++    >>> print extract_text(find_tag_by_id(admin_browser.contents, 'status'))
++    Status: Active Reviewed
  Other users cannot see the Project group status in the details portlet.
      >>> anon_browser.open('http://launchpad.dev/new-name')
      >>> print extract_text(
      ...     find_tag_by_id(anon_browser.contents, 'portlet-details'))
++    Project group information
      Maintainer: Sample Person
      Driver: Not yet appointed
      Bug tracker: The Mozilla.org Bug Tracker
++    Registered ... by Sample Person
++    Download RDF metadata
  Administrators can also change the maintainer and registrant independent
  of each other, as well as adding aliases to the project group.
@@ -90,17 +89,14 @@
  The project maintainer and registrant are now updated.
--    >>> for tag in find_tags_by_class(admin_browser.contents, 'summary'):
--    ...       print extract_text(
--    ...           tag.find(text='Maintainer:').findParent('tr'))
--    Maintainer:
--    Celso Providelo
++    >>> print extract_text(
++    ...     find_tag_by_id(admin_browser.contents, 'maintainer'))
++    Maintainer: Celso Providelo
++    Change maintainer
--    >>> tag = find_tag_by_id(admin_browser.contents, 'portlet-lifecycle')
      >>> print extract_text(
--    ...       tag.find(text='Registered by:').findParent('tr'))
--    Registered by:
--    David Allouche
++    ...     find_tag_by_id(admin_browser.contents, 'registrant'))
++    Registered ... by David Allouche
  The registrant really should only be a person, not a team, but that
  constraint has to be relaxed to account for old data where we do have
@@ -111,15 +107,6 @@
      >>> admin_browser.getControl('Registrant').value = 'registry'
      >>> admin_browser.getControl('Change').click()
--    >>> for tag in find_tags_by_class(admin_browser.contents, 'summary'):
--    ...       print extract_text(
--    ...           tag.find(text='Maintainer:').findParent('tr'))
--    Maintainer:
--    Celso Providelo
--
--    >>> tag = find_tag_by_id(admin_browser.contents, 'portlet-lifecycle')
      >>> print extract_text(
--    ...       tag.find(text='Registered by:').findParent('tr'))
--    Registered by:
--    Registry Administrators
--
++    ...     find_tag_by_id(admin_browser.contents, 'registrant'))
++    Registered ... by Registry Administrators
 === modified file 'lib/lp/registry/stories/project/xx-project-index.txt'
 --- lib/lp/registry/stories/project/xx-project-index.txt	2009-08-11 17:32:21 +0000
 +++ lib/lp/registry/stories/project/xx-project-index.txt	2009-08-13 18:10:15 +0000
@@ -23,28 +23,28 @@
      >>> anon_browser.title
      'The GNOME Project in Launchpad'
--The page lists the member projects.
++The page lists the member projects, together with the releases/milestones of
++its development focus.
      >>> print extract_text(find_tag_by_id(anon_browser.contents, 'products'))
      Projects
      Gnome Applets ...
      Evolution ...
--      Releases: 2.1.6
++    trunk series: 2.1.6,
      GNOME Terminal ...
--      Milestones: 2.30.0
      gnomebaker ...
--      Releases: 2.1.7
--      Milestones: 2.30.1, 2.1.7
++    trunk series: 2.1.7, 2.1.7
      NetApplet ...
--      Releases: 1.0
++    trunk series: 1.0,
++    ...
  And the projects, milestone, and releases are linked.
      >>> print anon_browser.getLink('gnomebaker').url
      http://launchpad.dev/gnomebaker
--    >>> print anon_browser.getLink('2.30.1').url
--    http://launchpad.dev/gnomebaker/+milestone/2.30.1
++    >>> print anon_browser.getLink('1.0').url
++    http://launchpad.dev/netapplet/trunk/1.0
      >>> print anon_browser.getLink('2.1.7').url
      http://launchpad.dev/gnomebaker/trunk/2.1.7
 === modified file 'lib/lp/registry/templates/codeofconduct-admin.pt'
 --- lib/lp/registry/templates/codeofconduct-admin.pt	2009-07-22 14:19:22 +0000
 +++ lib/lp/registry/templates/codeofconduct-admin.pt	2009-08-13 21:32:59 +0000
@@ -71,7 +71,7 @@
        <tal:results condition="view/search">
          <h2>Results</h2>
--     <ul tal:condition="view/results">
++     <ul id="matches" tal:condition="view/results">
         <li tal:repeat="code view/results">
             <a tal:replace="structure code/owner/fmt:link">OWNER</a>
             <a tal:attributes="href code/id"
 === modified file 'lib/lp/registry/templates/project-details.pt'
 --- lib/lp/registry/templates/project-details.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/registry/templates/project-details.pt	2009-08-13 18:10:15 +0000
@@ -4,53 +4,62 @@
    xmlns:i18n="http://xml.zope.org/namespaces/i18n"
    omit-tag="">
--<div class="portlet" id="portlet-details">
--  <div class="portletBody portletContent">
--    <table class="summary">
--      <tbody>
--        <tr tal:condition="context/required:launchpad.Admin">
--          <th>Project group status:</th>
--          <td>
--            <tal:block condition="context/active">Active</tal:block>
--            <tal:block condition="not: context/active">Disabled</tal:block>
--            <tal:block condition="context/reviewed"><br />Reviewed</tal:block>
--            <tal:block condition="context/icon"><br />Branded
--              <tal:icon replace="structure context/image:icon" />
--            </tal:block>
--          </td>
--        </tr>
--        <tr>
--          <th>Maintainer:</th>
--          <td>
--            <a
--              tal:content="context/owner/displayname"
--              tal:attributes="href context/owner/fmt:url"
--            >ownername</a>
--          </td>
--        </tr>
--        <tr>
--          <th>Driver:</th>
--          <td tal:condition="not:context/driver">Not yet appointed</td>
--          <td tal:condition="context/driver">
--            <a
--              tal:content="context/driver/displayname"
--              tal:attributes="href context/driver/fmt:url"
--            >Carlos Perello</a>
--          </td>
--        </tr>
--        <tr>
--          <th>Bug tracker:</th>
--          <td>
--            <a
--              tal:condition="context/bugtracker"
--              tal:content="context/bugtracker/title"
--              tal:attributes="href context/bugtracker/fmt:url"
--            >tracker title</a>
--            <tal:none condition="not:context/bugtracker">None specified</tal:none>
--          </td>
--        </tr>
--      </tbody>
--    </table>
++<div id="portlet-details" class="portlet">
++  <h2>Project group information</h2>
++  <div class="two-column-list" tal:define="overview_menu context/menu:overview">
++    <dl tal:condition="context/required:launchpad.Admin" id="status">
++      <dt>Status:</dt>
++      <dd>
++        <tal:block condition="context/active">Active</tal:block>
++        <tal:block condition="not: context/active">Disabled</tal:block>
++        <tal:block condition="context/reviewed"><br />Reviewed</tal:block>
++        <tal:block condition="context/icon"><br />Branded
++          <tal:icon replace="structure context/image:icon" />
++        </tal:block>
++      </dd>
++    </dl>
++    <dl id="maintainer">
++      <dt>Maintainer:</dt>
++      <dd>
++        <a tal:replace="structure context/owner/fmt:link" />
++        <tal:edit-maintainer
++          content="structure overview_menu/reassign/fmt:icon" />
++      </dd>
++    </dl>
++    <dl id="driver">
++      <dt>Driver:</dt>
++      <dd>
++        <tal:no-driver condition="not:context/driver">
++          Not yet appointed
++        </tal:no-driver>
++        <tal:has-driver condition="context/driver">
++          <a tal:replace="structure context/driver/fmt:link">Driver</a>
++        </tal:has-driver>
++        <tal:edit-maintainer
++          content="structure overview_menu/driver/fmt:icon" />
++      </dd>
++    </dl>
++    <dl id="bug-tracker">
++      <dt>Bug tracker:</dt>
++      <dd>
++        <a
++          tal:condition="context/bugtracker"
++          tal:content="context/bugtracker/title"
++          tal:attributes="href context/bugtracker/fmt:url"
++        >tracker title</a>
++        <tal:none condition="not:context/bugtracker">None specified</tal:none>
++      </dd>
++    </dl>
    </div>
++  <p id="registrant">
++    Registered
++    <tal:created replace="context/datecreated/fmt:approximatedate" />
++    by <tal:registrant replace="structure context/registrant/fmt:link" />
++  </p>
++  <ul class="horizontal">
++    <li>
++      <a tal:replace="structure context/menu:overview/rdf/fmt:link-icon" />
++    </li>
++  </ul>
  </div>
  </tal:root>
 === modified file 'lib/lp/registry/templates/project-edit.pt'
 --- lib/lp/registry/templates/project-edit.pt	2009-07-18 00:05:49 +0000
 +++ lib/lp/registry/templates/project-edit.pt	2009-08-05 19:26:37 +0000
@@ -3,20 +3,14 @@
    xmlns:tal="http://xml.zope.org/namespaces/tal"
    xmlns:metal="http://xml.zope.org/namespaces/metal"
    xmlns:i18n="http://xml.zope.org/namespaces/i18n"
--  xml:lang="en"
--  lang="en"
--  dir="ltr"
--  metal:use-macro="context/@@main_template/master"
++  metal:use-macro="view/macro:page/main_only"
    i18n:domain="launchpad"
+ >
    <body>
--    <metal:heading fill-slot="heading">
--      <h1>Change project group details</h1>
--    </metal:heading>
--
      <div metal:fill-slot="main">
--      <div metal:use-macro="context/@@launchpad_form/form">
++      <div class="top-portlet"
++           metal:use-macro="context/@@launchpad_form/form">
            <p metal:fill-slot="extra_info">
              Avoid changing the Name,
@@ -28,6 +22,7 @@
        </div>
++      <tal:menu replace="structure view/@@+related-pages" />
      </div>
    </body>
 === modified file 'lib/lp/registry/templates/project-index.pt'
 --- lib/lp/registry/templates/project-index.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/registry/templates/project-index.pt	2009-08-13 18:10:15 +0000
@@ -3,43 +3,37 @@
    xmlns:tal="http://xml.zope.org/namespaces/tal"
    xmlns:metal="http://xml.zope.org/namespaces/metal"
    xmlns:i18n="http://xml.zope.org/namespaces/i18n"
--  xml:lang="en"
--  lang="en"
--  dir="ltr"
--  metal:use-macro="view/macro:page/default"
++  metal:use-macro="view/macro:page/main_side"
    i18n:domain="launchpad"
+ >
    <body>
--    <metal:portlets fill-slot="portlets">
--      <tal:portlet tal:replace="structure context/@@+portlet-lifecycle" />
--      <tal:portlet tal:replace="structure context/@@+portlet-milestones" />
--    </metal:portlets>
--
--    <div metal:fill-slot="main">
--
--    <p id="project-inactive" tal:condition="not: context/active" class="warning message">
--      This project is currently inactive <a href="+review">(change this)</a>
--    </p>
--
--    <tal:block condition="view/required:launchpad.Edit">
--      <p tal:condition="not: context/products" class="warning message">
--        There are no projects registered for
--        <span tal:replace="context/displayname">project displayname</span>.
--        <br />
--        You need to <a href="+newproduct">register another project that is
--        part of <tal:project replace="context/displayname" /></a> or associate
--        an existing project with it.
++    <tal:heading metal:fill-slot="heading">
++      <h1 tal:content="context/title">Mozilla</h1>
++    </tal:heading>
++
++    <tal:main metal:fill-slot="main">
++
++    <div class="top-portlet">
++      <p id="project-inactive" class="warning message"
++         tal:condition="not: context/active">
++        This project is currently inactive
++        <a tal:attributes="href context/menu:overview/administer/url"
++          >(change this)</a>
        </p>
--    </tal:block>
--
--      <div
--        style="width: 200px; height: 200px; float: right;"
--        tal:content="structure context/image:mugshot"
--      />
--      <h1 tal:content="context/title">Mozilla</h1>
--
--      <div class="description" tal:content="context/summary">
++
++      <tal:block condition="view/required:launchpad.Edit">
++        <p tal:condition="not: context/products" class="warning message">
++          There are no projects registered for
++          <span tal:replace="context/displayname">project displayname</span>.
++          <br />
++          You need to <a href="+newproduct">register another project that is
++          part of <tal:project replace="context/displayname" /></a> or associate
++          an existing project with it.
++        </p>
++      </tal:block>
++
++      <div class="summary" tal:content="context/summary">
          This is the project group Summary, which should be a single paragraph
          summarising the project group's purpose.
        </div>
@@ -58,92 +52,97 @@
          Apache Server, and mention other projects such as APR.
        </div>
--      <ul tal:condition="context/homepageurl">
--        <li>
--          <a rel="nofollow"
--            tal:attributes="href context/homepageurl;
--                            class string:sprite external-link">
--            <strong>Visit the <tal:name replace="context/title" />
--            home page &raquo;</strong>
--          </a>
--        </li>
--      </ul>
--      <ul class="buttons" tal:condition="context/products">
--        <li>
--          <a href="+filebug">
--            <img alt="Report a bug" src="/+icing/but-sml-reportabug.gif" />
--          </a>
--        </li>
--        <li tal:content="structure context/@@+ask-a-question-button" />
--        <li tal:content="structure context/@@+help-translate-button" />
--        <li tal:define="has_mentoring context/mentoring_offers/count">
--          <a href="+mentoring" tal:condition="has_mentoring">
--            <img alt="Mentoring available"
--                 src="/+icing/but-sml-mentoring.gif"/>
--          </a>
--          <a href="+mentoring" tal:condition="not: has_mentoring">
--            <img alt="No mentoring available"
--                 src="/+icing/but-sml-mentoring-off.gif" />
--          </a>
--        </li>
--      </ul>
--      <div class="left">
--        <div
--          class="section"
--          tal:content="structure context/@@+portlet-coming-sprints"
--        />
--        <div
--          class="section"
--          tal:content="structure context/@@+portlet-latestannouncements"
--        />
--        <div class="portlet" id="products">
++      <ul id="external-links" class="horizontal"
++          tal:condition="context/homepageurl">
++        <li>
++          <a rel="nofollow" class="sprite external-link"
++             tal:attributes="href context/homepageurl">Home page</a>
++        </li>
++      </ul>
++    </div>
++
++    <div class="yui-g">
++      <div class="yui-u first">
++        <tal:details replace="structure context/@@+details" />
++      </div>
++      <div class="yui-u" id="products">
++        <div class="portlet">
            <h2>Projects</h2>
--          <p tal:condition="not: context/products">
++          <p tal:condition="not: context/products">
              There are no projects registered for
              <span tal:replace="context/displayname">project displayname</span>.
            </p>
--          <div
--            tal:condition="context/products"
--            tal:repeat="product context/products">
--            <tal:link replace="structure product/fmt:link" />
--            <div tal:condition="product/releases/count">
--              Releases:
--              <tal:release repeat="release product/releases">
--                <a
--                  tal:attributes="href release/fmt:url"
--                  tal:content="release/version"
--                >version</a><tal:comma
--                condition="not:repeat/release/end">,</tal:comma>
--              </tal:release>
--            </div>
--            <div tal:condition="product/milestones">
--              Milestones:
--              <tal:milestone repeat="milestone product/milestones">
--                <a
--                  tal:attributes="href milestone/fmt:url"
--                  tal:content="milestone/name"
--                >name</a><tal:comma
--                condition="not:repeat/milestone/end">,</tal:comma>
--              </tal:milestone>
--            </div>
--          </div>
--          <tal:block condition="context/required:launchpad.Edit">
--            <a href="+newproduct">&raquo; Register another project that is
--            part of <tal:project replace="context/displayname" /></a>
--          </tal:block>
++          <dl tal:condition="context/products"
++              tal:repeat="product context/products">
++            <dt><a tal:replace="structure product/fmt:link">product</a></dt>
++            <dd tal:define="dev_focus product/development_focus">
++              <a tal:attributes="href dev_focus/fmt:url"
++                 tal:content="dev_focus/name">trunk</a>
++              series:
++              <tal:releases repeat="release dev_focus/releases">
++                <strong>
++                  <a tal:attributes="href release/fmt:url"
++                     tal:content="release/version">release
++                </a></strong>,
++              </tal:releases>
++              <tal:releases repeat="milestone dev_focus/milestones">
++                <strong>
++                  <a tal:attributes="href milestone/fmt:url"
++                     tal:content="milestone/name">milestone
++                  </a></strong><tal:comma
++                    condition="not:repeat/milestone/end">,</tal:comma>
++              </tal:releases>
++            </dd>
++          </dl>
++          <ul>
++            <li>
++              <a tal:replace="structure context/menu:overview/milestones/fmt:link" />
++            </li>
++            <li tal:condition="context/menu:overview/new_product/enabled">
++              <a tal:replace="structure context/menu:overview/new_product/fmt:link" />
++            </li>
++          </ul>
          </div>
--        <div tal:replace="structure context/@@+portlet-latestspecs" />
--      </div>
--      <div class="right">
--        <div tal:replace="structure context/@@+details" />
--        <div tal:replace="structure context/@@+portlet-top-contributors" />
--        <div tal:replace="structure context/@@+portlet-latestbugs" />
--        <div
--          class="section"
--          tal:content="structure context/@@+portlet-latestquestions"
--        />
--      </div>
--    </div>
++      </div>
++    </div>
++    <div class="yui-g">
++      <div class="yui-u first">
++        <tal:bugs content="structure context/@@+portlet-latestbugs" />
++        <tal:specs content="structure context/@@+portlet-latestspecs" />
++        <tal:sprints content="structure context/@@+portlet-coming-sprints" />
++      </div>
++      <div class="yui-u">
++        <tal:questions content="structure context/@@+portlet-latestquestions" />
++        <tal:contributors content="structure context/@@+portlet-top-contributors" />
++      </div>
++    </div>
++    </tal:main>
++
++    <tal:side metal:fill-slot="side">
++      <div id="object-actions" class="top-portlet">
++        <tal:menu replace="structure view/@@+global-actions" />
++      </div>
++      <div id="involvement" class="portlet involvement"
++           tal:condition="context/products">
++        <h2>Get Involved</h2>
++        <ul>
++          <li>
++            <a class="bugs"
++              tal:attributes="href context/menu:bugs/new/url">Report a Bug</a>
++          </li>
++          <li>
++            <a class="question"
++              tal:attributes="href context/menu:answers/new/url">Ask a question</a>
++          </li>
++          <li>
++            <a class="translate"
++              tal:attributes="href context/menu:translations/overview/url">Help translate</a>
++          </li>
++        </ul>
++      </div>
++      <tal:portlet tal:replace="structure context/@@+portlet-latestannouncements" />
++    </tal:side>
++
    </body>
  </html>
 === modified file 'lib/lp/soyuz/browser/tests/archive-views.txt'
 --- lib/lp/soyuz/browser/tests/archive-views.txt	2009-07-30 01:24:18 +0000
 +++ lib/lp/soyuz/browser/tests/archive-views.txt	2009-08-13 19:36:01 +0000
@@ -932,7 +932,9 @@
      >>> print archive_widget.getInputValue()
      Traceback (most recent call last):
      ...
--    WidgetInputError: ('destination_archive', u'Destination PPA', )
++    WidgetInputError: ('destination_archive',
++                       u'Destination PPA',
++                       RequiredMissing())
  === Copy private files to public archives ===
 === modified file 'lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt'
 --- lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt	2009-08-13 19:20:04 +0000
 +++ lib/lp/soyuz/browser/tests/binarypackagerelease-views.txt	2009-08-13 19:51:29 +0000
@@ -1,6 +1,6 @@
  = BinaryPackageRelease Pages =
--  >>> from zope.component import queryMultiAdapter
++  >>> from zope.component import getMultiAdapter
    >>> from zope.publisher.browser import TestRequest
    >>> from canonical.launchpad.database import BinaryPackageRelease
@@ -16,7 +16,7 @@
  Let's instantiate the view for +portlet-details:
--  >>> pmount_view = queryMultiAdapter(
++  >>> pmount_view = getMultiAdapter(
    ...     (pmount_bin, request), name="+portlet-details")
  Main functionality of this class is to provide abstracted model of the
@@ -52,4 +52,3 @@
    ('tramp-package', None, '', None)
  Other relationship groups use the same mechanism.
--
 === modified file 'lib/lp/soyuz/stories/ppa/xx-edit-dependencies.txt'
 --- lib/lp/soyuz/stories/ppa/xx-edit-dependencies.txt	2009-06-19 16:15:02 +0000
 +++ lib/lp/soyuz/stories/ppa/xx-edit-dependencies.txt	2009-08-06 20:16:29 +0000
@@ -142,7 +142,7 @@
  Now Celso's PPA will list Mark's and No-Priv's PPA as its dependencies.
--    >>> admin_browser.reload()
++    >>> admin_browser.open(admin_browser.url) # Reload will set old form values.
      >>> print_ppa_dependencies(admin_browser.contents)
      PPA for Mark Shuttleworth
      PPA for No Privileges Person
 === modified file 'lib/lp/translations/browser/project.py'
 --- lib/lp/translations/browser/project.py	2009-07-17 02:25:09 +0000
 +++ lib/lp/translations/browser/project.py	2009-08-05 19:26:37 +0000
@@ -11,7 +11,7 @@
+     ]
  from canonical.launchpad.webapp import (
--    ApplicationMenu, enabled_with_permission, Link, LaunchpadView)
++    canonical_url, enabled_with_permission, Link, LaunchpadView)
  from canonical.launchpad.webapp.menu import NavigationMenu
  from lp.registry.interfaces.project import IProject
@@ -20,7 +20,7 @@
      usedfor = IProject
      facet = 'translations'
--    links = ['products', 'changetranslators']
++    links = ['products', 'changetranslators', 'overview']
      @enabled_with_permission('launchpad.Edit')
      def changetranslators(self):
@@ -31,6 +31,11 @@
          text = 'Products'
          return Link('', text)
++    def overview(self):
++        text = 'Overview'
++        link = canonical_url(self.context, rootsite='translations')
++        return Link(link, text, icon='translation')
++
  class ProjectView(LaunchpadView):
      pass
 === modified file 'lib/lp/translations/stories/distroseries/xx-distroseries-translations.txt'
 --- lib/lp/translations/stories/distroseries/xx-distroseries-translations.txt	2009-07-01 20:45:39 +0000
 +++ lib/lp/translations/stories/distroseries/xx-distroseries-translations.txt	2009-07-16 13:28:25 +0000
@@ -101,7 +101,7 @@
      ...     'http://translations.launchpad.dev/ubuntu/hoary/+lang/es')
      Traceback (most recent call last):
      ...
--    HTTPError: HTTP Error 503: Service Unavailable
++    httperror_seek_wrapper: HTTP Error 503: Service Unavailable
      >>> main_content = find_main_content(user_browser.contents)
      >>> print main_content.findNext('p').renderContents()
      Translations for this release series are not available yet.
 === modified file 'lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape.txt'
 --- lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape.txt	2009-07-01 20:45:39 +0000
 +++ lib/lp/translations/stories/standalone/xx-pofile-translate-html-tags-escape.txt	2009-08-06 11:42:11 +0000
@@ -31,6 +31,5 @@
    >>> text = find_tag_by_id(
    ...     user_browser.contents, 'msgset_67_hr_translation_0')
    >>> print extract_text(text.renderContents())
--  Upotreba:
--
--  %s [opcije] &lt;foo&gt; [&lt;etiketa&gt;]%s%s%s
++  Upotreba:
++  %s [opcije] &lt;foo&gt; [&lt;etiketa&gt;]%s%s%s
 === modified file 'lib/lp/translations/stories/translationgroups/30-show-group-translation-targets.txt'
 --- lib/lp/translations/stories/translationgroups/30-show-group-translation-targets.txt	2009-07-01 13:16:44 +0000
 +++ lib/lp/translations/stories/translationgroups/30-show-group-translation-targets.txt	2009-08-13 21:32:59 +0000
@@ -16,7 +16,7 @@
      >>> for link in portlet.findAll('a'):
      ...     print '%s: %s' % (link.find(text=True), link['href'])
      Ubuntu: /ubuntu
--    NetApplet: http://translations.launchpad.dev/netapplet
++    NetApplet: /netapplet
      GNOME: /gnome
  If we disable some of these projects...
 === modified file 'lib/lp/translations/templates/translationgroup-portlet-relateds.pt'
 --- lib/lp/translations/templates/translationgroup-portlet-relateds.pt	2009-07-17 17:59:07 +0000
 +++ lib/lp/translations/templates/translationgroup-portlet-relateds.pt	2009-08-13 21:32:59 +0000
@@ -25,7 +25,8 @@
          <h3>Projects:</h3>
          <ul>
            <li tal:repeat="product context/products">
--            <tal:link replace="structure product/fmt:link" />
++            <tal:link replace="structure product/fmt:link:translations" />
++            <tal:link replace="structure product/fmt:url:translations" />
            </li>
          </ul>
        </tal:products>
 === removed symlink 'lib/mechanize'
 === target was u'../sourcecode/zope/src/mechanize'
 === removed symlink 'lib/persistent'
 === target was u'../sourcecode/zope/src/persistent/'
 === removed symlink 'lib/transaction'
 === target was u'../sourcecode/zope/src/transaction/'
 === removed symlink 'lib/twisted'
 === target was u'../sourcecode/twisted/twisted/'
 === added symlink 'lib/twisted'
 === target is u'../sourcecode/twisted/twisted/'
 === removed directory 'lib/zc'
 === removed file 'lib/zc/__init__.py'
 --- lib/zc/__init__.py	2009-01-03 19:32:48 +0000
 +++ lib/zc/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,2 +0,0 @@
--# This is a namespace package for zc.* packages.  We should absolutely get rid
--# of this once we are using eggs.
 === removed symlink 'lib/zc/zservertracelog'
 === target was u'../../sourcecode/zc.zservertracelog/src/zc/zservertracelog'
 === removed symlink 'lib/zdaemon'
 === target was u'../sourcecode/zope/src/zdaemon/'
 === removed symlink 'lib/zodbcode'
 === target was u'../sourcecode/zope/src/zodbcode/'
 === removed symlink 'lib/zope'
 === target was u'../sourcecode/zope/src/zope/'
 === modified file 'setup.py'
 --- setup.py	2009-07-24 11:14:47 +0000
 +++ setup.py	2009-08-05 18:52:52 +0000
@@ -20,26 +20,78 @@
      maintainer='Launchpad Developers',
      description=('A unique collaboration and Bazaar code hosting platform '
                   'for software projects.'),
--    license='LGPL v3',
++    license='Affero GPL v3',
++    # this list should only contain direct dependencies--things imported or
++    # used in zcml.
      install_requires=[
          'bzr',
++        'chameleon.core',
++        'chameleon.zpt',
          'feedvalidator',
          'funkload',
          'launchpadlib',
          'lazr.smtptest',
          'lazr.uri',
++        'mechanize',
          'mocker',
          'oauth',
          'python-openid',
          'pytz',
++        # This appears to be a broken indirect dependency from zope.security:
++        'RestrictedPython',
          'setuptools',
          'sourcecodegen',
          'storm',
--        'chameleon.core',
--        'chameleon.zpt',
++        'transaction',
++        'wadllib',
          'z3c.pt',
          'z3c.ptcompat',
--        'wadllib',
++        'zc.zservertracelog',
++        'zope.app.appsetup',
++        'zope.app.component',
++        'zope.app.dav', # ./package-includes/dav-configure.zcml
++        'zope.app.error',
++        'zope.app.exception',
++        'zope.app.file',
++        'zope.app.form',
++        'zope.app.pagetemplate',
++        'zope.app.pluggableauth',
++        'zope.app.publication',
++        'zope.app.publisher',
++        'zope.app.security',
++        'zope.app.securitypolicy',
++        'zope.app.server',
++        'zope.app.session',
++        'zope.app.testing',
++        'zope.app.wsgi',
++        'zope.app.zapi',
++        'zope.contenttype',
++        'zope.component[zcml]',
++        'zope.datetime',
++        'zope.thread',
++        'zope.error',
++        'zope.event',
++        'zope.exceptions',
++        'zope.formlib',
++        'zope.i18n',
++        'zope.interface',
++        'zope.hookable', # indirect, via zope.app.component
++        'zope.lifecycleevent',
++        'zope.location',
++        'zope.pagetemplate',
++        'zope.publisher',
++        'zope.proxy',
++        'zope.schema',
++        'zope.security',
++        'zope.sendmail',
++        'zope.server',
++        'zope.session',
++        'zope.tal',
++        'zope.tales',
++        'zope.testbrowser',
++        'zope.testing',
++        'zope.traversing',
++        'zope.viewlet', # only fixing a broken dependency
          # Loggerhead dependencies. These should be removed once
          # bug 383360 is fixed and we include it as a source dist.
          'Paste',
 === modified file 'utilities/sourcedeps.conf'
 --- utilities/sourcedeps.conf	2009-07-21 07:55:23 +0000
 +++ utilities/sourcedeps.conf	2009-08-14 17:26:21 +0000
@@ -18,10 +18,8 @@
  testresources=lp:~launchpad-pqm/testresources/dev/
  testtools=lp:~launchpad-pqm/testtools/trunk/
  twisted=lp:~launchpad-pqm/twisted/trunk/
--zc.zservertracelog=lp:~launchpad-pqm/zc.zservertracelog/trunk/
  cscvs=lp:~launchpad-pqm/launchpad-cscvs/devel/
  lpreview=lp:~launchpad-pqm/bzr-lpreview/devel/
--zope=lp:~launchpad-pqm/zope3/3.4/
  pygettextpo=lp:~launchpad/pygettextpo/trunk/
  old_xmlplus=lp:~launchpad/dtdparser/trunk/
  lsprof=lp:~launchpad/lsprof/trunk/
 === modified file 'versions.cfg'
 --- versions.cfg	2009-08-12 12:35:51 +0000
 +++ versions.cfg	2009-08-14 18:08:23 +0000
@@ -2,17 +2,20 @@
  versions = versions
  [versions]
--# Alphabetical, please! :-)
++# Alphabetical, case-insensitive, please! :-)
  bzr = 1.17
  chameleon.core = 1.0b35
  chameleon.zpt = 1.0b17
++ClientForm = 0.2.10
  # Required by Windmill to run on 2.4
  ctypes = 1.0.2
  docutils = 0.5
  elementtree = 1.2.6-20050316
--# We use a version of feedvalidator that has been changed to not
--# change the default socket timeout on import.
--feedvalidator = 0.0.0DEV-r1049-hacked
++feedvalidator = 0.0.0DEV-r1049
++# We could use a locally hacked version of feedvalidator that has been changed
++# to not change the default socket timeout on import.  We are currently handling
++# that problem elsewhere.
++# feedvalidator = 0.0.0DEV-r1049-hacked
  functest = 0.8.7
  funkload = 1.10.0
  httplib2 = 0.4.0
@@ -20,6 +23,7 @@
  launchpadlib = 1.0.2
  lazr.smtptest = 1.1
  lazr.uri = 1.0.1
++mechanize = 0.1.7b
  mocker = 0.10.1
  mozrunner = 1.3.4
  oauth = 1.0
@@ -27,12 +31,14 @@
  PasteDeploy = 1.3.3
  python-openid = 2.2.1
  pytz = 2009j
++RestrictedPython = 3.4.2
  setuptools = 0.6c9
  simplejson = 2.0.9
  simplesettings = 0.4
  SimpleTal = 4.1
  sourcecodegen = 0.6.9
  storm = 0.15
++transaction = 1.0a1
  uuid = 1.30
  wadllib = 0.1
  webunit = 1.3.8
@@ -48,27 +54,90 @@
  zc.buildout = 1.4.0dev-gary-r102684
  zc.lockfile = 1.0.0
  zc.recipe.egg = 1.3.0dev-gary-r102684
--zc.recipe.testrunner = 1.1.0
++zc.zservertracelog = 1.1.5
  ZConfig = 2.6.1
++zdaemon = 2.0.2
++ZODB3 = 3.8.1
++zodbcode = 3.4.0
++zope.annotation = 3.4.1
++zope.app.applicationcontrol = 3.4.3
++zope.app.appsetup = 3.4.1
++zope.app.authentication = 3.4.4
++zope.app.basicskin = 3.4.0
++zope.app.broken = 3.4.0
++zope.app.component = 3.4.1
++zope.app.container = 3.5.6
++zope.app.content = 3.4.0
++zope.app.dav = 3.4.1
++zope.app.debug = 3.4.1
++zope.app.dependable = 3.4.0
++zope.app.error = 3.5.1
++zope.app.exception = 3.4.1
++zope.app.file = 3.4.4
++zope.app.folder = 3.4.0
++zope.app.form = 3.4.1
++zope.app.generations = 3.4.1
++zope.app.http = 3.4.1
++zope.app.i18n = 3.4.4
++zope.app.interface = 3.4.0
  zope.app.locales = 3.5.1
++zope.app.pagetemplate = 3.4.1
++zope.app.pluggableauth = 3.4.0
++zope.app.principalannotation = 3.4.0
++zope.app.publication = 3.4.4
++zope.app.publisher = 3.4.1
++zope.app.renderer = 3.4.0
++zope.app.rotterdam = 3.4.1
++zope.app.schema = 3.4.0
++zope.app.security = 3.5.2
++zope.app.securitypolicy = 3.4.6
++zope.app.server = 3.4.2
++zope.app.session = 3.5.1
++zope.app.testing = 3.4.3
++zope.app.wsgi = 3.4.1
++zope.app.zapi = 3.4.0
++zope.app.zcmlfiles = 3.4.3
++zope.app.zopeappgenerations = 3.4.0
++zope.cachedescriptors = 3.4.1
  zope.component = 3.5.1
--zope.configuration = 3.6.0
++zope.configuration = 3.5.0
  zope.contentprovider = 3.5.0
++zope.contenttype = 3.4.0
++zope.copypastemove = 3.4.0
++zope.datetime = 3.4.0
  zope.deferredimport = 3.4.0
  zope.deprecation = 3.4.0
  zope.dottedname = 3.4.5
++zope.dublincore = 3.4.0
++zope.error = 3.5.1
  zope.event = 3.4.1
  zope.exceptions = 3.5.2
++zope.filerepresentation = 3.4.0
++zope.formlib = 3.4.0
  zope.hookable = 3.4.0
  zope.i18n = 3.6.0
  zope.i18nmessageid = 3.4.3
  zope.interface = 3.5.1
++zope.lifecycleevent = 3.4.0
  zope.location = 3.5.2
++zope.minmax = 1.1.0
++zope.modulealias = 3.4.0
++zope.pagetemplate = 3.4.0
  zope.proxy = 3.5.0
  zope.publisher = 3.5.6
  zope.schema = 3.5.4
--zope.security = 3.6.0
--zope.tal = 3.4.0
++zope.security = 3.7.1
++zope.securitypolicy = 3.4.1
++zope.sendmail = 3.4.0
++zope.server = 3.4.3
++zope.session = 3.4.1
++zope.size = 3.4.0
++zope.structuredtext = 3.4.0
++zope.tal = 3.4.1
  zope.tales = 3.4.0
--zope.testing = 3.7.4
--zope.traversing = 3.5.2
++zope.testbrowser = 3.4.2
++zope.testing = 3.8.1
++zope.thread = 3.4
++# zope.traversing = 3.5.2
++zope.traversing = 3.4.1
++zope.viewlet = 3.4.2