Merge lp:~jameinel/bzr/2.1b1-pack-on-the-fly into lp:bzr

[John A Meinel]

[John A Meinel (jameinel)]

This adds 'pack-on-the-fly' support for gc streaming.

1) It restores 'groupcompress' sorting for the requested inventories and texts.
2) It uses a heuristic that is approximately:
  if a given block is less than 75% the size of a 'fully utilized' block, then don't re-use the
  content directly, but schedule it to be packed into a new block.
  The specifics are in '_LazyGroupContentManager.check_is_well_utilized()'
3) I did some real-world testing, and the results seem pretty good.
   To start with, the copy of bzr.dev on Launchpad is currently very poorly packed, taking up >90MB of disk space for a single pack file. After branching that using bzr.dev, I get a 101MB repository locally. If I 'bzr pack', I end up with 39MB (30MB in .pack, and 8.8MB in indices)

101MB poorly-packed-from-lp
101MB post 'bzr.dev branch new-repo' (takes 1m0s locally)
 39MB post 'bzr pack' (takes 2m0s locally)

I then tested the results of using the pack-on-the-fly
 41MB post 'bzr-pack branch new-repo' (takes 1m43s locally)
 41MB post 'bzr-pack branch new-repo new-repo2) (takes 1m0s)

Which means that pack-on-the-fly is working as we hoped it would. It
 a) Gives almost as good of pack results as if we had issued 'bzr pack'
 b) Takes a bit of extra time when the source is poorly packed (1m => 1m45s)
 c) Takes no extra time when the source is already properly packed (1m => 1m)

4) Unfortunately this was built on top of bzr.dev, but we can land it there, and then cherrypick it back to 2.0. I'll still submit a merge request for 2.0.

[Robert Collins (lifeless)]

Conceptually great; I'm looking now.

The review merge is bong; I'm going to pull locally, sync up and get a clean diff.

=== modified file 'Makefile'
--- Makefile	2009-08-03 20:38:39 +0000
+++ Makefile	2009-08-27 00:53:27 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005, 2006, 2007, 2008 Canonical Ltd
+# Copyright (C) 2005, 2006, 2007, 2008, 2009 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -40,8 +40,6 @@
 
 check-nodocs: extensions
 	$(PYTHON) -Werror -O ./bzr selftest -1v $(tests)
-	@echo "Running all tests with no locale."
-	LC_CTYPE= LANG=C LC_ALL= ./bzr selftest -1v $(tests) 2>&1 | sed -e 's/^/[ascii] /'
 
 # Run Python style checker (apt-get install pyflakes)
 #

=== modified file 'NEWS'
--- NEWS	2009-08-30 22:02:45 +0000
+++ NEWS	2009-09-03 21:04:22 +0000
@@ -6,6 +6,55 @@
 .. contents:: List of Releases
    :depth: 1
 
+In Development
+##############
+
+Compatibility Breaks
+********************
+
+New Features
+************
+
+Bug Fixes
+*********
+
+* ``bzr check`` in pack-0.92, 1.6 and 1.9 format repositories will no
+  longer report incorrect errors about ``Missing inventory ('TREE_ROOT', ...)``
+  (Robert Collins, #416732)
+
+* Don't restrict the command name used to run the test suite.
+  (Vincent Ladeuil, #419950)
+
+Improvements
+************
+
+Documentation
+*************
+
+API Changes
+***********
+
+* ``bzrlib.tests`` now uses ``stopTestRun`` for its ``TestResult``
+  subclasses - the same as python's unittest module. (Robert Collins)
+
+Internals
+*********
+
+* The ``bzrlib.lsprof`` module has a new class ``BzrProfiler`` which makes
+  profiling in some situations like callbacks and generators easier.
+  (Robert Collins)
+
+Testing
+*******
+
+* Passing ``--lsprof-tests -v`` to bzr selftest will cause lsprof output to
+  be output for every test. Note that this is very verbose! (Robert Collins)
+
+* Test parameterisation now does a shallow copy, not a deep copy of the test
+  to be parameterised. This is not expected to break external use of test
+  parameterisation, and is substantially faster. (Robert Collins)
+
+
 bzr 2.0rc2
 ##########
 
@@ -20,10 +69,34 @@
   revisions that are in the fallback repository. (Regressed in 2.0rc1).
   (John Arbash Meinel, #419241)
 
+* Fetches from 2a to 2a are now again requested in 'groupcompress' order.
+  Groups that are seen as 'underutilized' will be repacked on-the-fly.
+  This means that when the source is fully packed, there is minimal
+  overhead during the fetch, but if the source is poorly packed the result
+  is a fairly well packed repository (not as good as 'bzr pack' but
+  good-enough.) (Robert Collins, John Arbash Meinel, #402652)
+
 * Fix a segmentation fault when computing the ``merge_sort`` of a graph
   that has a ghost in the mainline ancestry.
   (John Arbash Meinel, #419241)
 
+* ``groupcompress`` sort order is now more stable, rather than relying on
+  ``topo_sort`` ordering. The implementation is now
+  ``KnownGraph.gc_sort``. (John Arbash Meinel)
+
+* Local data conversion will generate correct deltas. This is a critical
+  bugfix vs 2.0rc1, and all 2.0rc1 users should upgrade to 2.0rc2 before
+  converting repositories. (Robert Collins, #422849)
+
+* Network streams now decode adjacent records of the same type into a
+  single stream, reducing layering churn. (Robert Collins)
+
+Documentation
+*************
+
+* The main table of contents now provides links to the new Migration Docs
+  and Plugins Guide. (Ian Clatworthy)
+
 
 bzr 2.0rc1
 ##########
@@ -64,6 +137,9 @@
 Bug Fixes
 *********
 
+* Further tweaks to handling of ``bzr add`` messages about ignored files.
+  (Jason Spashett, #76616)
+
 * Fetches were being requested in 'groupcompress' order, but weren't
   recombining the groups. Thus they would 'fragment' to get the correct
   order, but not 'recombine' to actually benefit from it. Until we get
@@ -133,9 +209,6 @@
   classes changed to manage lock lifetime of the trees they open in a way
   consistent with reader-exclusive locks. (Robert Collins, #305006)
 
-Internals
-*********
-
 Testing
 *******
 
@@ -149,13 +222,29 @@
   conversion will commit too many copies a file.
   (Martin Pool, #415508)
 
+Improvements
+************
+
+* ``bzr push`` locally on windows will no longer give a locking error with
+  dirstate based formats. (Robert Collins)
+
+* ``bzr shelve`` and ``bzr unshelve`` now work on windows.
+  (Robert Collins, #305006)
+
 API Changes
 ***********
 
+* ``bzrlib.shelf_ui`` has had the ``from_args`` convenience methods of its
+  classes changed to manage lock lifetime of the trees they open in a way
+  consistent with reader-exclusive locks. (Robert Collins, #305006)
+
 * ``Tree.path_content_summary`` may return a size of None, when called on
   a tree with content filtering where the size of the canonical form
   cannot be cheaply determined.  (Martin Pool)
 
+* When manually creating transport servers in test cases, a new helper
+  ``TestCase.start_server`` that registers a cleanup and starts the server
+  should be used. (Robert Collins)
 
 bzr 1.18
 ########
@@ -493,6 +582,17 @@
   ``countTestsCases``. (Robert Collins)
 
 
+bzr 1.17.1 (unreleased)
+#######################
+
+Bug Fixes
+*********
+
+* The optional ``_knit_load_data_pyx`` C extension was never being
+  imported.  This caused significant slowdowns when reading data from
+  knit format repositories.  (Andrew Bennetts, #405653)
+  
+
 bzr 1.17 "So late it's brunch" 2009-07-20
 #########################################
 :Codename: so-late-its-brunch
@@ -991,6 +1091,9 @@
 Testing
 *******
 
+* ``make check`` no longer repeats the test run in ``LANG=C``.
+  (Martin Pool, #386180)
+
 * The number of cores is now correctly detected on OSX. (John Szakmeister)
 
 * The number of cores is also detected on Solaris and win32. (Vincent Ladeuil)
@@ -4971,7 +5074,7 @@
   checkouts.  (Aaron Bentley, #182040)
 
 * Stop polluting /tmp when running selftest.
-  (Vincent Ladeuil, #123623)
+  (Vincent Ladeuil, #123363)
 
 * Switch from NFKC => NFC for normalization checks. NFC allows a few
   more characters which should be considered valid.

=== modified file 'bzr'
--- bzr	2009-08-11 03:02:56 +0000
+++ bzr	2009-08-28 05:11:10 +0000
@@ -23,7 +23,7 @@
 import warnings
 
 # update this on each release
-_script_version = (2, 0, 0)
+_script_version = (2, 1, 0)
 
 if __doc__ is None:
     print "bzr does not support python -OO."

=== modified file 'bzrlib/__init__.py'
--- bzrlib/__init__.py	2009-08-27 07:49:53 +0000
+++ bzrlib/__init__.py	2009-08-30 21:34:42 +0000
@@ -50,7 +50,7 @@
 # Python version 2.0 is (2, 0, 0, 'final', 0)."  Additionally we use a
 # releaselevel of 'dev' for unreleased under-development code.
 
-version_info = (2, 0, 0, 'candidate', 1)
+version_info = (2, 1, 0, 'dev', 0)
 
 # API compatibility version: bzrlib is currently API compatible with 1.15.
 api_minimum_version = (1, 17, 0)

=== modified file 'bzrlib/_known_graph_py.py'
--- bzrlib/_known_graph_py.py	2009-08-17 20:41:26 +0000
+++ bzrlib/_known_graph_py.py	2009-08-25 18:45:40 +0000
@@ -97,6 +97,10 @@
         return [node for node in self._nodes.itervalues()
                 if not node.parent_keys]
 
+    def _find_tips(self):
+        return [node for node in self._nodes.itervalues()
+                      if not node.child_keys]
+
     def _find_gdfo(self):
         nodes = self._nodes
         known_parent_gdfos = {}
@@ -218,6 +222,51 @@
         # We started from the parents, so we don't need to do anymore work
         return topo_order
 
+    def gc_sort(self):
+        """Return a reverse topological ordering which is 'stable'.
+
+        There are a few constraints:
+          1) Reverse topological (all children before all parents)
+          2) Grouped by prefix
+          3) 'stable' sorting, so that we get the same result, independent of
+             machine, or extra data.
+        To do this, we use the same basic algorithm as topo_sort, but when we
+        aren't sure what node to access next, we sort them lexicographically.
+        """
+        tips = self._find_tips()
+        # Split the tips based on prefix
+        prefix_tips = {}
+        for node in tips:
+            if node.key.__class__ is str or len(node.key) == 1:
+                prefix = ''
+            else:
+                prefix = node.key[0]
+            prefix_tips.setdefault(prefix, []).append(node)
+
+        num_seen_children = dict.fromkeys(self._nodes, 0)
+
+        result = []
+        for prefix in sorted(prefix_tips):
+            pending = sorted(prefix_tips[prefix], key=lambda n:n.key,
+                             reverse=True)
+            while pending:
+                node = pending.pop()
+                if node.parent_keys is None:
+                    # Ghost node, skip it
+                    continue
+                result.append(node.key)
+                for parent_key in sorted(node.parent_keys, reverse=True):
+                    parent_node = self._nodes[parent_key]
+                    seen_children = num_seen_children[parent_key] + 1
+                    if seen_children == len(parent_node.child_keys):
+                        # All children have been processed, enqueue this parent
+                        pending.append(parent_node)
+                        # This has been queued up, stop tracking it
+                        del num_seen_children[parent_key]
+                    else:
+                        num_seen_children[parent_key] = seen_children
+        return result
+
     def merge_sort(self, tip_key):
         """Compute the merge sorted graph output."""
         from bzrlib import tsort

=== modified file 'bzrlib/_known_graph_pyx.pyx'
--- bzrlib/_known_graph_pyx.pyx	2009-08-26 16:03:59 +0000
+++ bzrlib/_known_graph_pyx.pyx	2009-09-02 13:32:52 +0000
@@ -25,11 +25,18 @@
     ctypedef struct PyObject:
         pass
 
+    int PyString_CheckExact(object)
+
+    int PyObject_RichCompareBool(object, object, int)
+    int Py_LT
+
+    int PyTuple_CheckExact(object)
     object PyTuple_New(Py_ssize_t n)
     Py_ssize_t PyTuple_GET_SIZE(object t)
     PyObject * PyTuple_GET_ITEM(object t, Py_ssize_t o)
     void PyTuple_SET_ITEM(object t, Py_ssize_t o, object v)
 
+    int PyList_CheckExact(object)
     Py_ssize_t PyList_GET_SIZE(object l)
     PyObject * PyList_GET_ITEM(object l, Py_ssize_t o)
     int PyList_SetItem(object l, Py_ssize_t o, object l) except -1
@@ -108,14 +115,65 @@
     return <_KnownGraphNode>temp_node
 
 
-cdef _KnownGraphNode _get_parent(parents, Py_ssize_t pos):
+cdef _KnownGraphNode _get_tuple_node(tpl, Py_ssize_t pos):
     cdef PyObject *temp_node
-    cdef _KnownGraphNode node
 
-    temp_node = PyTuple_GET_ITEM(parents, pos)
+    temp_node = PyTuple_GET_ITEM(tpl, pos)
     return <_KnownGraphNode>temp_node
 
 
+def get_key(node):
+    cdef _KnownGraphNode real_node
+    real_node = node
+    return real_node.key
+
+
+cdef object _sort_list_nodes(object lst_or_tpl, int reverse):
+    """Sort a list of _KnownGraphNode objects.
+
+    If lst_or_tpl is a list, it is allowed to mutate in place. It may also
+    just return the input list if everything is already sorted.
+    """
+    cdef _KnownGraphNode node1, node2
+    cdef int do_swap, is_tuple
+    cdef Py_ssize_t length
+
+    is_tuple = PyTuple_CheckExact(lst_or_tpl)
+    if not (is_tuple or PyList_CheckExact(lst_or_tpl)):
+        raise TypeError('lst_or_tpl must be a list or tuple.')
+    length = len(lst_or_tpl)
+    if length == 0 or length == 1:
+        return lst_or_tpl
+    if length == 2:
+        if is_tuple:
+            node1 = _get_tuple_node(lst_or_tpl, 0)
+            node2 = _get_tuple_node(lst_or_tpl, 1)
+        else:
+            node1 = _get_list_node(lst_or_tpl, 0)
+            node2 = _get_list_node(lst_or_tpl, 1)
+        if reverse:
+            do_swap = PyObject_RichCompareBool(node1.key, node2.key, Py_LT)
+        else:
+            do_swap = PyObject_RichCompareBool(node2.key, node1.key, Py_LT)
+        if not do_swap:
+            return lst_or_tpl
+        if is_tuple:
+            return (node2, node1)
+        else:
+            # Swap 'in-place', since lists are mutable
+            Py_INCREF(node1)
+            PyList_SetItem(lst_or_tpl, 1, node1)
+            Py_INCREF(node2)
+            PyList_SetItem(lst_or_tpl, 0, node2)
+            return lst_or_tpl
+    # For all other sizes, we just use 'sorted()'
+    if is_tuple:
+        # Note that sorted() is just list(iterable).sort()
+        lst_or_tpl = list(lst_or_tpl)
+    lst_or_tpl.sort(key=get_key, reverse=reverse)
+    return lst_or_tpl
+
+
 cdef class _MergeSorter
 
 cdef class KnownGraph:
@@ -216,6 +274,19 @@
                 PyList_Append(tails, node)
         return tails
 
+    def _find_tips(self):
+        cdef PyObject *temp_node
+        cdef _KnownGraphNode node
+        cdef Py_ssize_t pos
+
+        tips = []
+        pos = 0
+        while PyDict_Next(self._nodes, &pos, NULL, &temp_node):
+            node = <_KnownGraphNode>temp_node
+            if PyList_GET_SIZE(node.children) == 0:
+                PyList_Append(tips, node)
+        return tips
+
     def _find_gdfo(self):
         cdef _KnownGraphNode node
         cdef _KnownGraphNode child
@@ -322,7 +393,7 @@
                 continue
             if node.parents is not None and PyTuple_GET_SIZE(node.parents) > 0:
                 for pos from 0 <= pos < PyTuple_GET_SIZE(node.parents):
-                    parent_node = _get_parent(node.parents, pos)
+                    parent_node = _get_tuple_node(node.parents, pos)
                     last_item = last_item + 1
                     if last_item < PyList_GET_SIZE(pending):
                         Py_INCREF(parent_node) # SetItem steals a ref
@@ -397,6 +468,77 @@
         # We started from the parents, so we don't need to do anymore work
         return topo_order
 
+    def gc_sort(self):
+        """Return a reverse topological ordering which is 'stable'.
+
+        There are a few constraints:
+          1) Reverse topological (all children before all parents)
+          2) Grouped by prefix
+          3) 'stable' sorting, so that we get the same result, independent of
+             machine, or extra data.
+        To do this, we use the same basic algorithm as topo_sort, but when we
+        aren't sure what node to access next, we sort them lexicographically.
+        """
+        cdef PyObject *temp
+        cdef Py_ssize_t pos, last_item
+        cdef _KnownGraphNode node, node2, parent_node
+
+        tips = self._find_tips()
+        # Split the tips based on prefix
+        prefix_tips = {}
+        for pos from 0 <= pos < PyList_GET_SIZE(tips):
+            node = _get_list_node(tips, pos)
+            if PyString_CheckExact(node.key) or len(node.key) == 1:
+                prefix = ''
+            else:
+                prefix = node.key[0]
+            temp = PyDict_GetItem(prefix_tips, prefix)
+            if temp == NULL:
+                prefix_tips[prefix] = [node]
+            else:
+                tip_nodes = <object>temp
+                PyList_Append(tip_nodes, node)
+
+        result = []
+        for prefix in sorted(prefix_tips):
+            temp = PyDict_GetItem(prefix_tips, prefix)
+            assert temp != NULL
+            tip_nodes = <object>temp
+            pending = _sort_list_nodes(tip_nodes, 1)
+            last_item = PyList_GET_SIZE(pending) - 1
+            while last_item >= 0:
+                node = _get_list_node(pending, last_item)
+                last_item = last_item - 1
+                if node.parents is None:
+                    # Ghost
+                    continue
+                PyList_Append(result, node.key)
+                # Sorting the parent keys isn't strictly necessary for stable
+                # sorting of a given graph. But it does help minimize the
+                # differences between graphs
+                # For bzr.dev ancestry:
+                #   4.73ms  no sort
+                #   7.73ms  RichCompareBool sort
+                parents = _sort_list_nodes(node.parents, 1)
+                for pos from 0 <= pos < len(parents):
+                    if PyTuple_CheckExact(parents):
+                        parent_node = _get_tuple_node(parents, pos)
+                    else:
+                        parent_node = _get_list_node(parents, pos)
+                    # TODO: GraphCycle detection
+                    parent_node.seen = parent_node.seen + 1
+                    if (parent_node.seen
+                        == PyList_GET_SIZE(parent_node.children)):
+                        # All children have been processed, queue up this
+                        # parent
+                        last_item = last_item + 1
+                        if last_item < PyList_GET_SIZE(pending):
+                            Py_INCREF(parent_node) # SetItem steals a ref
+                            PyList_SetItem(pending, last_item, parent_node)
+                        else:
+                            PyList_Append(pending, parent_node)
+                        parent_node.seen = 0
+        return result
 
     def merge_sort(self, tip_key):
         """Compute the merge sorted graph output."""
@@ -522,7 +664,7 @@
             raise RuntimeError('ghost nodes should not be pushed'
                                ' onto the stack: %s' % (node,))
         if PyTuple_GET_SIZE(node.parents) > 0:
-            parent_node = _get_parent(node.parents, 0)
+            parent_node = _get_tuple_node(node.parents, 0)
             ms_node.left_parent = parent_node
             if parent_node.parents is None: # left-hand ghost
                 ms_node.left_pending_parent = None
@@ -532,7 +674,7 @@
         if PyTuple_GET_SIZE(node.parents) > 1:
             ms_node.pending_parents = []
             for pos from 1 <= pos < PyTuple_GET_SIZE(node.parents):
-                parent_node = _get_parent(node.parents, pos)
+                parent_node = _get_tuple_node(node.parents, pos)
                 if parent_node.parents is None: # ghost
                     continue
                 PyList_Append(ms_node.pending_parents, parent_node)

=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py	2009-08-26 03:20:32 +0000
+++ bzrlib/builtins.py	2009-08-28 05:00:33 +0000
@@ -3382,6 +3382,8 @@
                      Option('lsprof-timed',
                             help='Generate lsprof output for benchmarked'
                                  ' sections of code.'),
+                     Option('lsprof-tests',
+                            help='Generate lsprof output for each test.'),
                      Option('cache-dir', type=str,
                             help='Cache intermediate benchmark output in this '
                                  'directory.'),
@@ -3428,7 +3430,7 @@
             first=False, list_only=False,
             randomize=None, exclude=None, strict=False,
             load_list=None, debugflag=None, starting_with=None, subunit=False,
-            parallel=None):
+            parallel=None, lsprof_tests=False):
         from bzrlib.tests import selftest
         import bzrlib.benchmarks as benchmarks
         from bzrlib.benchmarks import tree_creator
@@ -3468,6 +3470,7 @@
                               "transport": transport,
                               "test_suite_factory": test_suite_factory,
                               "lsprof_timed": lsprof_timed,
+                              "lsprof_tests": lsprof_tests,
                               "bench_history": benchfile,
                               "matching_tests_first": first,
                               "list_only": list_only,

=== modified file 'bzrlib/groupcompress.py'
--- bzrlib/groupcompress.py	2009-08-26 16:47:51 +0000
+++ bzrlib/groupcompress.py	2009-09-03 15:25:36 +0000
@@ -457,7 +457,6 @@
                 # There are code paths that first extract as fulltext, and then
                 # extract as storage_kind (smart fetch). So we don't break the
                 # refcycle here, but instead in manager.get_record_stream()
-                # self._manager = None
             if storage_kind == 'fulltext':
                 return self._bytes
             else:
@@ -469,6 +468,14 @@
 class _LazyGroupContentManager(object):
     """This manages a group of _LazyGroupCompressFactory objects."""
 
+    _max_cut_fraction = 0.75 # We allow a block to be trimmed to 75% of
+                             # current size, and still be considered
+                             # resuable
+    _full_block_size = 4*1024*1024
+    _full_mixed_block_size = 2*1024*1024
+    _full_enough_block_size = 3*1024*1024 # size at which we won't repack
+    _full_enough_mixed_block_size = 2*768*1024 # 1.5MB
+
     def __init__(self, block):
         self._block = block
         # We need to preserve the ordering
@@ -546,22 +553,23 @@
         # time (self._block._content) is a little expensive.
         self._block._ensure_content(self._last_byte)
 
-    def _check_rebuild_block(self):
+    def _check_rebuild_action(self):
         """Check to see if our block should be repacked."""
         total_bytes_used = 0
         last_byte_used = 0
         for factory in self._factories:
             total_bytes_used += factory._end - factory._start
-            last_byte_used = max(last_byte_used, factory._end)
-        # If we are using most of the bytes from the block, we have nothing
-        # else to check (currently more that 1/2)
+            if last_byte_used < factory._end:
+                last_byte_used = factory._end
+        # If we are using more than half of the bytes from the block, we have
+        # nothing else to check
         if total_bytes_used * 2 >= self._block._content_length:
-            return
-        # Can we just strip off the trailing bytes? If we are going to be
-        # transmitting more than 50% of the front of the content, go ahead
+            return None, last_byte_used, total_bytes_used
+        # We are using less than 50% of the content. Is the content we are
+        # using at the beginning of the block? If so, we can just trim the
+        # tail, rather than rebuilding from scratch.
         if total_bytes_used * 2 > last_byte_used:
-            self._trim_block(last_byte_used)
-            return
+            return 'trim', last_byte_used, total_bytes_used
 
         # We are using a small amount of the data, and it isn't just packed
         # nicely at the front, so rebuild the content.
@@ -574,7 +582,77 @@
         #       expanding many deltas into fulltexts, as well.
         #       If we build a cheap enough 'strip', then we could try a strip,
         #       if that expands the content, we then rebuild.
-        self._rebuild_block()
+        return 'rebuild', last_byte_used, total_bytes_used
+
+    def check_is_well_utilized(self):
+        """Is the current block considered 'well utilized'?
+
+        This is a bit of a heuristic, but it basically asks if the current
+        block considers itself to be a fully developed group, rather than just
+        a loose collection of data.
+        """
+        if len(self._factories) == 1:
+            # A block of length 1 is never considered 'well utilized' :)
+            return False
+        action, last_byte_used, total_bytes_used = self._check_rebuild_action()
+        block_size = self._block._content_length
+        if total_bytes_used < block_size * self._max_cut_fraction:
+            # This block wants to trim itself small enough that we want to
+            # consider it under-utilized.
+            return False
+        # TODO: This code is meant to be the twin of _insert_record_stream's
+        #       'start_new_block' logic. It would probably be better to factor
+        #       out that logic into a shared location, so that it stays
+        #       together better
+        # We currently assume a block is properly utilized whenever it is >75%
+        # of the size of a 'full' block. In normal operation, a block is
+        # considered full when it hits 4MB of same-file content. So any block
+        # >3MB is 'full enough'.
+        # The only time this isn't true is when a given block has large-object
+        # content. (a single file >4MB, etc.)
+        # Under these circumstances, we allow a block to grow to
+        # 2 x largest_content.  Which means that if a given block had a large
+        # object, it may actually be under-utilized. However, given that this
+        # is 'pack-on-the-fly' it is probably reasonable to not repack large
+        # contet blobs on-the-fly.
+        if block_size >= self._full_enough_block_size:
+            return True
+        # If a block is <3MB, it still may be considered 'full' if it contains
+        # mixed content. The current rule is 2MB of mixed content is considered
+        # full. So check to see if this block contains mixed content, and
+        # set the threshold appropriately.
+        common_prefix = None
+        for factory in self._factories:
+            prefix = factory.key[:-1]
+            if common_prefix is None:
+                common_prefix = prefix
+            elif prefix != common_prefix:
+                # Mixed content, check the size appropriately
+                if block_size >= self._full_enough_mixed_block_size:
+                    return True
+                break
+        # The content failed both the mixed check and the single-content check
+        # so obviously it is not fully utilized
+        # TODO: there is one other constraint that isn't being checked
+        #       namely, that the entries in the block are in the appropriate
+        #       order. For example, you could insert the entries in exactly
+        #       reverse groupcompress order, and we would think that is ok.
+        #       (all the right objects are in one group, and it is fully
+        #       utilized, etc.) For now, we assume that case is rare,
+        #       especially since we should always fetch in 'groupcompress'
+        #       order.
+        return False
+
+    def _check_rebuild_block(self):
+        action, last_byte_used, total_bytes_used = self._check_rebuild_action()
+        if action is None:
+            return
+        if action == 'trim':
+            self._trim_block(last_byte_used)
+        elif action == 'rebuild':
+            self._rebuild_block()
+        else:
+            raise ValueError('unknown rebuild action: %r' % (action,))
 
     def _wire_bytes(self):
         """Return a byte stream suitable for transmitting over the wire."""
@@ -1570,6 +1648,7 @@
         block_length = None
         # XXX: TODO: remove this, it is just for safety checking for now
         inserted_keys = set()
+        reuse_this_block = reuse_blocks
         for record in stream:
             # Raise an error when a record is missing.
             if record.storage_kind == 'absent':
@@ -1583,10 +1662,20 @@
             if reuse_blocks:
                 # If the reuse_blocks flag is set, check to see if we can just
                 # copy a groupcompress block as-is.
+                # We only check on the first record (groupcompress-block) not
+                # on all of the (groupcompress-block-ref) entries.
+                # The reuse_this_block flag is then kept for as long as
+                if record.storage_kind == 'groupcompress-block':
+                    # Check to see if we really want to re-use this block
+                    insert_manager = record._manager
+                    reuse_this_block = insert_manager.check_is_well_utilized()
+            else:
+                reuse_this_block = False
+            if reuse_this_block:
+                # We still want to reuse this block
                 if record.storage_kind == 'groupcompress-block':
                     # Insert the raw block into the target repo
                     insert_manager = record._manager
-                    insert_manager._check_rebuild_block()
                     bytes = record._manager._block.to_bytes()
                     _, start, length = self._access.add_raw_records(
                         [(None, len(bytes))], bytes)[0]
@@ -1597,6 +1686,11 @@
                                            'groupcompress-block-ref'):
                     if insert_manager is None:
                         raise AssertionError('No insert_manager set')
+                    if insert_manager is not record._manager:
+                        raise AssertionError('insert_manager does not match'
+                            ' the current record, we cannot be positive'
+                            ' that the appropriate content was inserted.'
+                            )
                     value = "%d %d %d %d" % (block_start, block_length,
                                              record._start, record._end)
                     nodes = [(record.key, value, (record.parents,))]

=== modified file 'bzrlib/lsprof.py'
--- bzrlib/lsprof.py	2009-03-08 06:18:06 +0000
+++ bzrlib/lsprof.py	2009-08-24 21:05:09 +0000
@@ -13,45 +13,74 @@
 
 __all__ = ['profile', 'Stats']
 
-_g_threadmap = {}
-
-
-def _thread_profile(f, *args, **kwds):
-    # we lose the first profile point for a new thread in order to trampoline
-    # a new Profile object into place
-    global _g_threadmap
-    thr = thread.get_ident()
-    _g_threadmap[thr] = p = Profiler()
-    # this overrides our sys.setprofile hook:
-    p.enable(subcalls=True, builtins=True)
-
-
 def profile(f, *args, **kwds):
     """Run a function profile.
 
     Exceptions are not caught: If you need stats even when exceptions are to be
-    raised, passing in a closure that will catch the exceptions and transform
-    them appropriately for your driver function.
+    raised, pass in a closure that will catch the exceptions and transform them
+    appropriately for your driver function.
 
     :return: The functions return value and a stats object.
     """
-    global _g_threadmap
-    p = Profiler()
-    p.enable(subcalls=True)
-    threading.setprofile(_thread_profile)
+    profiler = BzrProfiler()
+    profiler.start()
     try:
         ret = f(*args, **kwds)
     finally:
-        p.disable()
-        for pp in _g_threadmap.values():
+        stats = profiler.stop()
+    return ret, stats
+
+
+class BzrProfiler(object):
+    """Bzr utility wrapper around Profiler.
+    
+    For most uses the module level 'profile()' function will be suitable.
+    However profiling when a simple wrapped function isn't available may
+    be easier to accomplish using this class.
+
+    To use it, create a BzrProfiler and call start() on it. Some arbitrary
+    time later call stop() to stop profiling and retrieve the statistics
+    from the code executed in the interim.
+    """
+
+    def start(self):
+        """Start profiling.
+        
+        This hooks into threading and will record all calls made until
+        stop() is called.
+        """
+        self._g_threadmap = {}
+        self.p = Profiler()
+        self.p.enable(subcalls=True)
+        threading.setprofile(self._thread_profile)
+
+    def stop(self):
+        """Stop profiling.
+
+        This unhooks from threading and cleans up the profiler, returning
+        the gathered Stats object.
+
+        :return: A bzrlib.lsprof.Stats object.
+        """
+        self.p.disable()
+        for pp in self._g_threadmap.values():
             pp.disable()
         threading.setprofile(None)
+        p = self.p
+        self.p = None
+        threads = {}
+        for tid, pp in self._g_threadmap.items():
+            threads[tid] = Stats(pp.getstats(), {})
+        self._g_threadmap = None
+        return Stats(p.getstats(), threads)
 
-    threads = {}
-    for tid, pp in _g_threadmap.items():
-        threads[tid] = Stats(pp.getstats(), {})
-    _g_threadmap = {}
-    return ret, Stats(p.getstats(), threads)
+    def _thread_profile(self, f, *args, **kwds):
+        # we lose the first profile point for a new thread in order to
+        # trampoline a new Profile object into place
+        thr = thread.get_ident()
+        self._g_threadmap[thr] = p = Profiler()
+        # this overrides our sys.setprofile hook:
+        p.enable(subcalls=True, builtins=True)
 
 
 class Stats(object):

=== modified file 'bzrlib/repofmt/groupcompress_repo.py'
--- bzrlib/repofmt/groupcompress_repo.py	2009-08-24 19:34:13 +0000
+++ bzrlib/repofmt/groupcompress_repo.py	2009-09-01 06:10:24 +0000
@@ -932,7 +932,7 @@
         super(GroupCHKStreamSource, self).__init__(from_repository, to_format)
         self._revision_keys = None
         self._text_keys = None
-        # self._text_fetch_order = 'unordered'
+        self._text_fetch_order = 'groupcompress'
         self._chk_id_roots = None
         self._chk_p_id_roots = None
 
@@ -949,7 +949,7 @@
             p_id_roots_set = set()
             source_vf = self.from_repository.inventories
             stream = source_vf.get_record_stream(inventory_keys,
-                                                 'unordered', True)
+                                                 'groupcompress', True)
             for record in stream:
                 if record.storage_kind == 'absent':
                     if allow_absent:

=== modified file 'bzrlib/repository.py'
--- bzrlib/repository.py	2009-08-30 22:02:45 +0000
+++ bzrlib/repository.py	2009-09-03 15:26:27 +0000
@@ -3844,6 +3844,9 @@
                 possible_trees.append((basis_id, cache[basis_id]))
             basis_id, delta = self._get_delta_for_revision(tree, parent_ids,
                                                            possible_trees)
+            revision = self.source.get_revision(current_revision_id)
+            pending_deltas.append((basis_id, delta,
+                current_revision_id, revision.parent_ids))
             if self._converting_to_rich_root:
                 self._revision_id_to_root_id[current_revision_id] = \
                     tree.get_root_id()
@@ -3878,9 +3881,6 @@
                     if entry.revision == file_revision:
                         texts_possibly_new_in_tree.remove(file_key)
             text_keys.update(texts_possibly_new_in_tree)
-            revision = self.source.get_revision(current_revision_id)
-            pending_deltas.append((basis_id, delta,
-                current_revision_id, revision.parent_ids))
             pending_revisions.append(revision)
             cache[current_revision_id] = tree
             basis_id = current_revision_id

=== modified file 'bzrlib/smart/repository.py'
--- bzrlib/smart/repository.py	2009-08-14 00:55:42 +0000
+++ bzrlib/smart/repository.py	2009-09-02 22:29:55 +0000
@@ -519,36 +519,92 @@
     yield pack_writer.end()
 
 
+class _ByteStreamDecoder(object):
+    """Helper for _byte_stream_to_stream.
+
+    Broadly this class has to unwrap two layers of iterators:
+    (type, substream)
+    (substream details)
+
+    This is complicated by wishing to return type, iterator_for_type, but
+    getting the data for iterator_for_type when we find out type: we can't
+    simply pass a generator down to the NetworkRecordStream parser, instead
+    we have a little local state to seed each NetworkRecordStream instance,
+    and gather the type that we'll be yielding.
+
+    :ivar byte_stream: The byte stream being decoded.
+    :ivar stream_decoder: A pack parser used to decode the bytestream
+    :ivar current_type: The current type, used to join adjacent records of the
+        same type into a single stream.
+    :ivar first_bytes: The first bytes to give the next NetworkRecordStream.
+    """
+
+    def __init__(self, byte_stream):
+        """Create a _ByteStreamDecoder."""
+        self.stream_decoder = pack.ContainerPushParser()
+        self.current_type = None
+        self.first_bytes = None
+        self.byte_stream = byte_stream
+
+    def iter_stream_decoder(self):
+        """Iterate the contents of the pack from stream_decoder."""
+        # dequeue pending items
+        for record in self.stream_decoder.read_pending_records():
+            yield record
+        # Pull bytes of the wire, decode them to records, yield those records.
+        for bytes in self.byte_stream:
+            self.stream_decoder.accept_bytes(bytes)
+            for record in self.stream_decoder.read_pending_records():
+                yield record
+
+    def iter_substream_bytes(self):
+        if self.first_bytes is not None:
+            yield self.first_bytes
+            # If we run out of pack records, single the outer layer to stop.
+            self.first_bytes = None
+        for record in self.iter_pack_records:
+            record_names, record_bytes = record
+            record_name, = record_names
+            substream_type = record_name[0]
+            if substream_type != self.current_type:
+                # end of a substream, seed the next substream.
+                self.current_type = substream_type
+                self.first_bytes = record_bytes
+                return
+            yield record_bytes
+
+    def record_stream(self):
+        """Yield substream_type, substream from the byte stream."""
+        self.seed_state()
+        # Make and consume sub generators, one per substream type:
+        while self.first_bytes is not None:
+            substream = NetworkRecordStream(self.iter_substream_bytes())
+            # after substream is fully consumed, self.current_type is set to
+            # the next type, and self.first_bytes is set to the matching bytes.
+            yield self.current_type, substream.read()
+
+    def seed_state(self):
+        """Prepare the _ByteStreamDecoder to decode from the pack stream."""
+        # Set a single generator we can use to get data from the pack stream.
+        self.iter_pack_records = self.iter_stream_decoder()
+        # Seed the very first subiterator with content; after this each one
+        # seeds the next.
+        list(self.iter_substream_bytes())
+
+
 def _byte_stream_to_stream(byte_stream):
     """Convert a byte stream into a format and a stream.
 
     :param byte_stream: A bytes iterator, as output by _stream_to_byte_stream.
     :return: (RepositoryFormat, stream_generator)
     """
-    stream_decoder = pack.ContainerPushParser()
-    def record_stream():
-        """Closure to return the substreams."""
-        # May have fully parsed records already.
-        for record in stream_decoder.read_pending_records():
-            record_names, record_bytes = record
-            record_name, = record_names
-            substream_type = record_name[0]
-            substream = NetworkRecordStream([record_bytes])
-            yield substream_type, substream.read()
-        for bytes in byte_stream:
-            stream_decoder.accept_bytes(bytes)
-            for record in stream_decoder.read_pending_records():
-                record_names, record_bytes = record
-                record_name, = record_names
-                substream_type = record_name[0]
-                substream = NetworkRecordStream([record_bytes])
-                yield substream_type, substream.read()
+    decoder = _ByteStreamDecoder(byte_stream)
     for bytes in byte_stream:
-        stream_decoder.accept_bytes(bytes)
-        for record in stream_decoder.read_pending_records(max=1):
+        decoder.stream_decoder.accept_bytes(bytes)
+        for record in decoder.stream_decoder.read_pending_records(max=1):
             record_names, src_format_name = record
             src_format = network_format_registry.get(src_format_name)
-            return src_format, record_stream()
+            return src_format, decoder.record_stream()
 
 
 class SmartServerRepositoryUnlock(SmartServerRepositoryRequest):

=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py	2009-08-24 20:30:18 +0000
+++ bzrlib/tests/__init__.py	2009-08-28 21:05:31 +0000
@@ -28,6 +28,7 @@
 
 import atexit
 import codecs
+from copy import copy
 from cStringIO import StringIO
 import difflib
 import doctest
@@ -174,17 +175,47 @@
         self._overall_start_time = time.time()
         self._strict = strict
 
-    def done(self):
-        # nb: called stopTestRun in the version of this that Python merged
-        # upstream, according to lifeless 20090803
+    def stopTestRun(self):
+        run = self.testsRun
+        actionTaken = "Ran"
+        stopTime = time.time()
+        timeTaken = stopTime - self.startTime
+        self.printErrors()
+        self.stream.writeln(self.separator2)
+        self.stream.writeln("%s %d test%s in %.3fs" % (actionTaken,
+                            run, run != 1 and "s" or "", timeTaken))
+        self.stream.writeln()
+        if not self.wasSuccessful():
+            self.stream.write("FAILED (")
+            failed, errored = map(len, (self.failures, self.errors))
+            if failed:
+                self.stream.write("failures=%d" % failed)
+            if errored:
+                if failed: self.stream.write(", ")
+                self.stream.write("errors=%d" % errored)
+            if self.known_failure_count:
+                if failed or errored: self.stream.write(", ")
+                self.stream.write("known_failure_count=%d" %
+                    self.known_failure_count)
+            self.stream.writeln(")")
+        else:
+            if self.known_failure_count:
+                self.stream.writeln("OK (known_failures=%d)" %
+                    self.known_failure_count)
+            else:
+                self.stream.writeln("OK")
+        if self.skip_count > 0:
+            skipped = self.skip_count
+            self.stream.writeln('%d test%s skipped' %
+                                (skipped, skipped != 1 and "s" or ""))
+        if self.unsupported:
+            for feature, count in sorted(self.unsupported.items()):
+                self.stream.writeln("Missing feature '%s' skipped %d tests." %
+                    (feature, count))
         if self._strict:
             ok = self.wasStrictlySuccessful()
         else:
             ok = self.wasSuccessful()
-        if ok:
-            self.stream.write('tests passed\n')
-        else:
-            self.stream.write('tests failed\n')
         if TestCase._first_thread_leaker_id:
             self.stream.write(
                 '%s is leaking threads among %d leaking tests.\n' % (
@@ -382,12 +413,12 @@
         else:
             raise errors.BzrError("Unknown whence %r" % whence)
 
-    def finished(self):
-        pass
-
     def report_cleaning_up(self):
         pass
 
+    def startTestRun(self):
+        self.startTime = time.time()
+
     def report_success(self, test):
         pass
 
@@ -420,15 +451,14 @@
         self.pb.update_latency = 0
         self.pb.show_transport_activity = False
 
-    def done(self):
+    def stopTestRun(self):
         # called when the tests that are going to run have run
         self.pb.clear()
-        super(TextTestResult, self).done()
-
-    def finished(self):
         self.pb.finished()
+        super(TextTestResult, self).stopTestRun()
 
-    def report_starting(self):
+    def startTestRun(self):
+        super(TextTestResult, self).startTestRun()
         self.pb.update('[test 0/%d] Starting' % (self.num_tests))
 
     def printErrors(self):
@@ -513,7 +543,8 @@
             result = a_string
         return result.ljust(final_width)
 
-    def report_starting(self):
+    def startTestRun(self):
+        super(VerboseTestResult, self).startTestRun()
         self.stream.write('running %d tests...\n' % self.num_tests)
 
     def report_test_start(self, test):
@@ -577,88 +608,57 @@
                  descriptions=0,
                  verbosity=1,
                  bench_history=None,
-                 list_only=False,
                  strict=False,
+                 result_decorators=None,
                  ):
+        """Create a TextTestRunner.
+
+        :param result_decorators: An optional list of decorators to apply
+            to the result object being used by the runner. Decorators are
+            applied left to right - the first element in the list is the 
+            innermost decorator.
+        """
         self.stream = unittest._WritelnDecorator(stream)
         self.descriptions = descriptions
         self.verbosity = verbosity
         self._bench_history = bench_history
-        self.list_only = list_only
         self._strict = strict
+        self._result_decorators = result_decorators or []
 
     def run(self, test):
         "Run the given test case or test suite."
-        startTime = time.time()
         if self.verbosity == 1:
             result_class = TextTestResult
         elif self.verbosity >= 2:
             result_class = VerboseTestResult
-        result = result_class(self.stream,
+        original_result = result_class(self.stream,
                               self.descriptions,
                               self.verbosity,
                               bench_history=self._bench_history,
                               strict=self._strict,
                               )
-        result.stop_early = self.stop_on_failure
-        result.report_starting()
-        if self.list_only:
-            if self.verbosity >= 2:
-                self.stream.writeln("Listing tests only ...\n")
-            run = 0
-            for t in iter_suite_tests(test):
-                self.stream.writeln("%s" % (t.id()))
-                run += 1
-            return None
-        else:
-            try:
-                import testtools
-            except ImportError:
-                test.run(result)
-            else:
-                if isinstance(test, testtools.ConcurrentTestSuite):
-                    # We need to catch bzr specific behaviors
-                    test.run(BZRTransformingResult(result))
-                else:
-                    test.run(result)
-            run = result.testsRun
-            actionTaken = "Ran"
-        stopTime = time.time()
-        timeTaken = stopTime - startTime
-        result.printErrors()
-        self.stream.writeln(result.separator2)
-        self.stream.writeln("%s %d test%s in %.3fs" % (actionTaken,
-                            run, run != 1 and "s" or "", timeTaken))
-        self.stream.writeln()
-        if not result.wasSuccessful():
-            self.stream.write("FAILED (")
-            failed, errored = map(len, (result.failures, result.errors))
-            if failed:
-                self.stream.write("failures=%d" % failed)
-            if errored:
-                if failed: self.stream.write(", ")
-                self.stream.write("errors=%d" % errored)
-            if result.known_failure_count:
-                if failed or errored: self.stream.write(", ")
-                self.stream.write("known_failure_count=%d" %
-                    result.known_failure_count)
-            self.stream.writeln(")")
-        else:
-            if result.known_failure_count:
-                self.stream.writeln("OK (known_failures=%d)" %
-                    result.known_failure_count)
-            else:
-                self.stream.writeln("OK")
-        if result.skip_count > 0:
-            skipped = result.skip_count
-            self.stream.writeln('%d test%s skipped' %
-                                (skipped, skipped != 1 and "s" or ""))
-        if result.unsupported:
-            for feature, count in sorted(result.unsupported.items()):
-                self.stream.writeln("Missing feature '%s' skipped %d tests." %
-                    (feature, count))
-        result.finished()
-        return result
+        # Signal to result objects that look at stop early policy to stop,
+        original_result.stop_early = self.stop_on_failure
+        result = original_result
+        for decorator in self._result_decorators:
+            result = decorator(result)
+            result.stop_early = self.stop_on_failure
+        try:
+            import testtools
+        except ImportError:
+            pass
+        else:
+            if isinstance(test, testtools.ConcurrentTestSuite):
+                # We need to catch bzr specific behaviors
+                result = BZRTransformingResult(result)
+        result.startTestRun()
+        try:
+            test.run(result)
+        finally:
+            result.stopTestRun()
+        # higher level code uses our extended protocol to determine
+        # what exit code to give.
+        return original_result
 
 
 def iter_suite_tests(suite):
@@ -928,6 +928,18 @@
     def _lock_broken(self, result):
         self._lock_actions.append(('broken', result))
 
+    def start_server(self, transport_server, backing_server=None):
+        """Start transport_server for this test.
+
+        This starts the server, registers a cleanup for it and permits the
+        server's urls to be used.
+        """
+        if backing_server is None:
+            transport_server.setUp()
+        else:
+            transport_server.setUp(backing_server)
+        self.addCleanup(transport_server.tearDown)
+
     def _ndiff_strings(self, a, b):
         """Return ndiff between two strings containing lines.
 
@@ -2067,13 +2079,12 @@
         if self.__readonly_server is None:
             if self.transport_readonly_server is None:
                 # readonly decorator requested
-                # bring up the server
                 self.__readonly_server = ReadonlyServer()
-                self.__readonly_server.setUp(self.get_vfs_only_server())
             else:
+                # explicit readonly transport.
                 self.__readonly_server = self.create_transport_readonly_server()
-                self.__readonly_server.setUp(self.get_vfs_only_server())
-            self.addCleanup(self.__readonly_server.tearDown)
+            self.start_server(self.__readonly_server,
+                self.get_vfs_only_server())
         return self.__readonly_server
 
     def get_readonly_url(self, relpath=None):
@@ -2098,8 +2109,7 @@
         """
         if self.__vfs_server is None:
             self.__vfs_server = MemoryServer()
-            self.__vfs_server.setUp()
-            self.addCleanup(self.__vfs_server.tearDown)
+            self.start_server(self.__vfs_server)
         return self.__vfs_server
 
     def get_server(self):
@@ -2112,19 +2122,13 @@
         then the self.get_vfs_server is returned.
         """
         if self.__server is None:
-            if self.transport_server is None or self.transport_server is self.vfs_transport_factory:
-                return self.get_vfs_only_server()
+            if (self.transport_server is None or self.transport_server is
+                self.vfs_transport_factory):
+                self.__server = self.get_vfs_only_server()
             else:
                 # bring up a decorated means of access to the vfs only server.
                 self.__server = self.transport_server()
-                try:
-                    self.__server.setUp(self.get_vfs_only_server())
-                except TypeError, e:
-                    # This should never happen; the try:Except here is to assist
-                    # developers having to update code rather than seeing an
-                    # uninformative TypeError.
-                    raise Exception, "Old server API in use: %s, %s" % (self.__server, e)
-            self.addCleanup(self.__server.tearDown)
+                self.start_server(self.__server, self.get_vfs_only_server())
         return self.__server
 
     def _adjust_url(self, base, relpath):
@@ -2263,9 +2267,8 @@
 
     def make_smart_server(self, path):
         smart_server = server.SmartTCPServer_for_testing()
-        smart_server.setUp(self.get_server())
+        self.start_server(smart_server, self.get_server())
         remote_transport = get_transport(smart_server.get_url()).clone(path)
-        self.addCleanup(smart_server.tearDown)
         return remote_transport
 
     def make_branch_and_memory_tree(self, relpath, format=None):
@@ -2472,8 +2475,7 @@
         """
         if self.__vfs_server is None:
             self.__vfs_server = self.vfs_transport_factory()
-            self.__vfs_server.setUp()
-            self.addCleanup(self.__vfs_server.tearDown)
+            self.start_server(self.__vfs_server)
         return self.__vfs_server
 
     def make_branch_and_tree(self, relpath, format=None):
@@ -2486,6 +2488,15 @@
         repository will also be accessed locally. Otherwise a lightweight
         checkout is created and returned.
 
+        We do this because we can't physically create a tree in the local
+        path, with a branch reference to the transport_factory url, and
+        a branch + repository in the vfs_transport, unless the vfs_transport
+        namespace is distinct from the local disk - the two branch objects
+        would collide. While we could construct a tree with its branch object
+        pointing at the transport_factory transport in memory, reopening it
+        would behaving unexpectedly, and has in the past caused testing bugs
+        when we tried to do it that way.
+
         :param format: The BzrDirFormat.
         :returns: the WorkingTree.
         """
@@ -2762,7 +2773,9 @@
               strict=False,
               runner_class=None,
               suite_decorators=None,
-              stream=None):
+              stream=None,
+              result_decorators=None,
+              ):
     """Run a test suite for bzr selftest.
 
     :param runner_class: The class of runner to use. Must support the
@@ -2783,8 +2796,8 @@
                             descriptions=0,
                             verbosity=verbosity,
                             bench_history=bench_history,
-                            list_only=list_only,
                             strict=strict,
+                            result_decorators=result_decorators,
                             )
     runner.stop_on_failure=stop_on_failure
     # built in decorator factories:
@@ -2805,10 +2818,15 @@
         decorators.append(CountingDecorator)
     for decorator in decorators:
         suite = decorator(suite)
-    result = runner.run(suite)
     if list_only:
+        # Done after test suite decoration to allow randomisation etc
+        # to take effect, though that is of marginal benefit.
+        if verbosity >= 2:
+            stream.write("Listing tests only ...\n")
+        for t in iter_suite_tests(suite):
+            stream.write("%s\n" % (t.id()))
         return True
-    result.done()
+    result = runner.run(suite)
     if strict:
         return result.wasStrictlySuccessful()
     else:
@@ -3131,7 +3149,7 @@
     return result
 
 
-class BZRTransformingResult(unittest.TestResult):
+class ForwardingResult(unittest.TestResult):
 
     def __init__(self, target):
         unittest.TestResult.__init__(self)
@@ -3143,6 +3161,27 @@
     def stopTest(self, test):
         self.result.stopTest(test)
 
+    def startTestRun(self):
+        self.result.startTestRun()
+
+    def stopTestRun(self):
+        self.result.stopTestRun()
+
+    def addSkip(self, test, reason):
+        self.result.addSkip(test, reason)
+
+    def addSuccess(self, test):
+        self.result.addSuccess(test)
+
+    def addError(self, test, err):
+        self.result.addError(test, err)
+
+    def addFailure(self, test, err):
+        self.result.addFailure(test, err)
+
+
+class BZRTransformingResult(ForwardingResult):
+
     def addError(self, test, err):
         feature = self._error_looks_like('UnavailableFeature: ', err)
         if feature is not None:
@@ -3158,12 +3197,6 @@
         else:
             self.result.addFailure(test, err)
 
-    def addSkip(self, test, reason):
-        self.result.addSkip(test, reason)
-
-    def addSuccess(self, test):
-        self.result.addSuccess(test)
-
     def _error_looks_like(self, prefix, err):
         """Deserialize exception and returns the stringify value."""
         import subunit
@@ -3181,6 +3214,38 @@
         return value
 
 
+class ProfileResult(ForwardingResult):
+    """Generate profiling data for all activity between start and success.
+    
+    The profile data is appended to the test's _benchcalls attribute and can
+    be accessed by the forwarded-to TestResult.
+
+    While it might be cleaner do accumulate this in stopTest, addSuccess is
+    where our existing output support for lsprof is, and this class aims to
+    fit in with that: while it could be moved it's not necessary to accomplish
+    test profiling, nor would it be dramatically cleaner.
+    """
+
+    def startTest(self, test):
+        self.profiler = bzrlib.lsprof.BzrProfiler()
+        self.profiler.start()
+        ForwardingResult.startTest(self, test)
+
+    def addSuccess(self, test):
+        stats = self.profiler.stop()
+        try:
+            calls = test._benchcalls
+        except AttributeError:
+            test._benchcalls = []
+            calls = test._benchcalls
+        calls.append(((test.id(), "", ""), stats))
+        ForwardingResult.addSuccess(self, test)
+
+    def stopTest(self, test):
+        ForwardingResult.stopTest(self, test)
+        self.profiler = None
+
+
 # Controlled by "bzr selftest -E=..." option
 # Currently supported:
 #   -Eallow_debug           Will no longer clear debug.debug_flags() so it
@@ -3208,6 +3273,7 @@
              runner_class=None,
              suite_decorators=None,
              stream=None,
+             lsprof_tests=False,
              ):
     """Run the whole test suite under the enhanced runner"""
     # XXX: Very ugly way to do this...
@@ -3242,6 +3308,9 @@
         if starting_with:
             # But always filter as requested.
             suite = filter_suite_by_id_startswith(suite, starting_with)
+        result_decorators = []
+        if lsprof_tests:
+            result_decorators.append(ProfileResult)
         return run_suite(suite, 'testbzr', verbose=verbose, pattern=pattern,
                      stop_on_failure=stop_on_failure,
                      transport=transport,
@@ -3255,6 +3324,7 @@
                      runner_class=runner_class,
                      suite_decorators=suite_decorators,
                      stream=stream,
+                     result_decorators=result_decorators,
                      )
     finally:
         default_transport = old_transport
@@ -3416,6 +3486,206 @@
 test_prefix_alias_registry.register('bp', 'bzrlib.plugins')
 
 
+def _test_suite_testmod_names():
+    """Return the standard list of test module names to test."""
+    return [
+        'bzrlib.doc',
+        'bzrlib.tests.blackbox',
+        'bzrlib.tests.commands',
+        'bzrlib.tests.per_branch',
+        'bzrlib.tests.per_bzrdir',
+        'bzrlib.tests.per_interrepository',
+        'bzrlib.tests.per_intertree',
+        'bzrlib.tests.per_inventory',
+        'bzrlib.tests.per_interbranch',
+        'bzrlib.tests.per_lock',
+        'bzrlib.tests.per_transport',
+        'bzrlib.tests.per_tree',
+        'bzrlib.tests.per_pack_repository',
+        'bzrlib.tests.per_repository',
+        'bzrlib.tests.per_repository_chk',
+        'bzrlib.tests.per_repository_reference',
+        'bzrlib.tests.per_versionedfile',
+        'bzrlib.tests.per_workingtree',
+        'bzrlib.tests.test__annotator',
+        'bzrlib.tests.test__chk_map',
+        'bzrlib.tests.test__dirstate_helpers',
+        'bzrlib.tests.test__groupcompress',
+        'bzrlib.tests.test__known_graph',
+        'bzrlib.tests.test__rio',
+        'bzrlib.tests.test__walkdirs_win32',
+        'bzrlib.tests.test_ancestry',
+        'bzrlib.tests.test_annotate',
+        'bzrlib.tests.test_api',
+        'bzrlib.tests.test_atomicfile',
+        'bzrlib.tests.test_bad_files',
+        'bzrlib.tests.test_bencode',
+        'bzrlib.tests.test_bisect_multi',
+        'bzrlib.tests.test_branch',
+        'bzrlib.tests.test_branchbuilder',
+        'bzrlib.tests.test_btree_index',
+        'bzrlib.tests.test_bugtracker',
+        'bzrlib.tests.test_bundle',
+        'bzrlib.tests.test_bzrdir',
+        'bzrlib.tests.test__chunks_to_lines',
+        'bzrlib.tests.test_cache_utf8',
+        'bzrlib.tests.test_chk_map',
+        'bzrlib.tests.test_chk_serializer',
+        'bzrlib.tests.test_chunk_writer',
+        'bzrlib.tests.test_clean_tree',
+        'bzrlib.tests.test_commands',
+        'bzrlib.tests.test_commit',
+        'bzrlib.tests.test_commit_merge',
+        'bzrlib.tests.test_config',
+        'bzrlib.tests.test_conflicts',
+        'bzrlib.tests.test_counted_lock',
+        'bzrlib.tests.test_crash',
+        'bzrlib.tests.test_decorators',
+        'bzrlib.tests.test_delta',
+        'bzrlib.tests.test_debug',
+        'bzrlib.tests.test_deprecated_graph',
+        'bzrlib.tests.test_diff',
+        'bzrlib.tests.test_directory_service',
+        'bzrlib.tests.test_dirstate',
+        'bzrlib.tests.test_email_message',
+        'bzrlib.tests.test_eol_filters',
+        'bzrlib.tests.test_errors',
+        'bzrlib.tests.test_export',
+        'bzrlib.tests.test_extract',
+        'bzrlib.tests.test_fetch',
+        'bzrlib.tests.test_fifo_cache',
+        'bzrlib.tests.test_filters',
+        'bzrlib.tests.test_ftp_transport',
+        'bzrlib.tests.test_foreign',
+        'bzrlib.tests.test_generate_docs',
+        'bzrlib.tests.test_generate_ids',
+        'bzrlib.tests.test_globbing',
+        'bzrlib.tests.test_gpg',
+        'bzrlib.tests.test_graph',
+        'bzrlib.tests.test_groupcompress',
+        'bzrlib.tests.test_hashcache',
+        'bzrlib.tests.test_help',
+        'bzrlib.tests.test_hooks',
+        'bzrlib.tests.test_http',
+        'bzrlib.tests.test_http_response',
+        'bzrlib.tests.test_https_ca_bundle',
+        'bzrlib.tests.test_identitymap',
+        'bzrlib.tests.test_ignores',
+        'bzrlib.tests.test_index',
+        'bzrlib.tests.test_info',
+        'bzrlib.tests.test_inv',
+        'bzrlib.tests.test_inventory_delta',
+        'bzrlib.tests.test_knit',
+        'bzrlib.tests.test_lazy_import',
+        'bzrlib.tests.test_lazy_regex',
+        'bzrlib.tests.test_lock',
+        'bzrlib.tests.test_lockable_files',
+        'bzrlib.tests.test_lockdir',
+        'bzrlib.tests.test_log',
+        'bzrlib.tests.test_lru_cache',
+        'bzrlib.tests.test_lsprof',
+        'bzrlib.tests.test_mail_client',
+        'bzrlib.tests.test_memorytree',
+        'bzrlib.tests.test_merge',
+        'bzrlib.tests.test_merge3',
+        'bzrlib.tests.test_merge_core',
+        'bzrlib.tests.test_merge_directive',
+        'bzrlib.tests.test_missing',
+        'bzrlib.tests.test_msgeditor',
+        'bzrlib.tests.test_multiparent',
+        'bzrlib.tests.test_mutabletree',
+        'bzrlib.tests.test_nonascii',
+        'bzrlib.tests.test_options',
+        'bzrlib.tests.test_osutils',
+        'bzrlib.tests.test_osutils_encodings',
+        'bzrlib.tests.test_pack',
+        'bzrlib.tests.test_patch',
+        'bzrlib.tests.test_patches',
+        'bzrlib.tests.test_permissions',
+        'bzrlib.tests.test_plugins',
+        'bzrlib.tests.test_progress',
+        'bzrlib.tests.test_read_bundle',
+        'bzrlib.tests.test_reconcile',
+        'bzrlib.tests.test_reconfigure',
+        'bzrlib.tests.test_registry',
+        'bzrlib.tests.test_remote',
+        'bzrlib.tests.test_rename_map',
+        'bzrlib.tests.test_repository',
+        'bzrlib.tests.test_revert',
+        'bzrlib.tests.test_revision',
+        'bzrlib.tests.test_revisionspec',
+        'bzrlib.tests.test_revisiontree',
+        'bzrlib.tests.test_rio',
+        'bzrlib.tests.test_rules',
+        'bzrlib.tests.test_sampler',
+        'bzrlib.tests.test_selftest',
+        'bzrlib.tests.test_serializer',
+        'bzrlib.tests.test_setup',
+        'bzrlib.tests.test_sftp_transport',
+        'bzrlib.tests.test_shelf',
+        'bzrlib.tests.test_shelf_ui',
+        'bzrlib.tests.test_smart',
+        'bzrlib.tests.test_smart_add',
+        'bzrlib.tests.test_smart_request',
+        'bzrlib.tests.test_smart_transport',
+        'bzrlib.tests.test_smtp_connection',
+        'bzrlib.tests.test_source',
+        'bzrlib.tests.test_ssh_transport',
+        'bzrlib.tests.test_status',
+        'bzrlib.tests.test_store',
+        'bzrlib.tests.test_strace',
+        'bzrlib.tests.test_subsume',
+        'bzrlib.tests.test_switch',
+        'bzrlib.tests.test_symbol_versioning',
+        'bzrlib.tests.test_tag',
+        'bzrlib.tests.test_testament',
+        'bzrlib.tests.test_textfile',
+        'bzrlib.tests.test_textmerge',
+        'bzrlib.tests.test_timestamp',
+        'bzrlib.tests.test_trace',
+        'bzrlib.tests.test_transactions',
+        'bzrlib.tests.test_transform',
+        'bzrlib.tests.test_transport',
+        'bzrlib.tests.test_transport_log',
+        'bzrlib.tests.test_tree',
+        'bzrlib.tests.test_treebuilder',
+        'bzrlib.tests.test_tsort',
+        'bzrlib.tests.test_tuned_gzip',
+        'bzrlib.tests.test_ui',
+        'bzrlib.tests.test_uncommit',
+        'bzrlib.tests.test_upgrade',
+        'bzrlib.tests.test_upgrade_stacked',
+        'bzrlib.tests.test_urlutils',
+        'bzrlib.tests.test_version',
+        'bzrlib.tests.test_version_info',
+        'bzrlib.tests.test_weave',
+        'bzrlib.tests.test_whitebox',
+        'bzrlib.tests.test_win32utils',
+        'bzrlib.tests.test_workingtree',
+        'bzrlib.tests.test_workingtree_4',
+        'bzrlib.tests.test_wsgi',
+        'bzrlib.tests.test_xml',
+        ]
+
+
+def _test_suite_modules_to_doctest():
+    """Return the list of modules to doctest."""   
+    return [
+        'bzrlib',
+        'bzrlib.branchbuilder',
+        'bzrlib.export',
+        'bzrlib.inventory',
+        'bzrlib.iterablefile',
+        'bzrlib.lockdir',
+        'bzrlib.merge3',
+        'bzrlib.option',
+        'bzrlib.symbol_versioning',
+        'bzrlib.tests',
+        'bzrlib.timestamp',
+        'bzrlib.version_info_formats.format_custom',
+        ]
+
+
 def test_suite(keep_only=None, starting_with=None):
     """Build and return TestSuite for the whole of bzrlib.
 
@@ -3427,184 +3697,6 @@
     This function can be replaced if you need to change the default test
     suite on a global basis, but it is not encouraged.
     """
-    testmod_names = [
-                   'bzrlib.doc',
-                   'bzrlib.tests.blackbox',
-                   'bzrlib.tests.commands',
-                   'bzrlib.tests.per_branch',
-                   'bzrlib.tests.per_bzrdir',
-                   'bzrlib.tests.per_interrepository',
-                   'bzrlib.tests.per_intertree',
-                   'bzrlib.tests.per_inventory',
-                   'bzrlib.tests.per_interbranch',
-                   'bzrlib.tests.per_lock',
-                   'bzrlib.tests.per_transport',
-                   'bzrlib.tests.per_tree',
-                   'bzrlib.tests.per_pack_repository',
-                   'bzrlib.tests.per_repository',
-                   'bzrlib.tests.per_repository_chk',
-                   'bzrlib.tests.per_repository_reference',
-                   'bzrlib.tests.per_versionedfile',
-                   'bzrlib.tests.per_workingtree',
-                   'bzrlib.tests.test__annotator',
-                   'bzrlib.tests.test__chk_map',
-                   'bzrlib.tests.test__dirstate_helpers',
-                   'bzrlib.tests.test__groupcompress',
-                   'bzrlib.tests.test__known_graph',
-                   'bzrlib.tests.test__rio',
-                   'bzrlib.tests.test__walkdirs_win32',
-                   'bzrlib.tests.test_ancestry',
-                   'bzrlib.tests.test_annotate',
-                   'bzrlib.tests.test_api',
-                   'bzrlib.tests.test_atomicfile',
-                   'bzrlib.tests.test_bad_files',
-                   'bzrlib.tests.test_bencode',
-                   'bzrlib.tests.test_bisect_multi',
-                   'bzrlib.tests.test_branch',
-                   'bzrlib.tests.test_branchbuilder',
-                   'bzrlib.tests.test_btree_index',
-                   'bzrlib.tests.test_bugtracker',
-                   'bzrlib.tests.test_bundle',
-                   'bzrlib.tests.test_bzrdir',
-                   'bzrlib.tests.test__chunks_to_lines',
-                   'bzrlib.tests.test_cache_utf8',
-                   'bzrlib.tests.test_chk_map',
-                   'bzrlib.tests.test_chk_serializer',
-                   'bzrlib.tests.test_chunk_writer',
-                   'bzrlib.tests.test_clean_tree',
-                   'bzrlib.tests.test_commands',
-                   'bzrlib.tests.test_commit',
-                   'bzrlib.tests.test_commit_merge',
-                   'bzrlib.tests.test_config',
-                   'bzrlib.tests.test_conflicts',
-                   'bzrlib.tests.test_counted_lock',
-                   'bzrlib.tests.test_crash',
-                   'bzrlib.tests.test_decorators',
-                   'bzrlib.tests.test_delta',
-                   'bzrlib.tests.test_debug',
-                   'bzrlib.tests.test_deprecated_graph',
-                   'bzrlib.tests.test_diff',
-                   'bzrlib.tests.test_directory_service',
-                   'bzrlib.tests.test_dirstate',
-                   'bzrlib.tests.test_email_message',
-                   'bzrlib.tests.test_eol_filters',
-                   'bzrlib.tests.test_errors',
-                   'bzrlib.tests.test_export',
-                   'bzrlib.tests.test_extract',
-                   'bzrlib.tests.test_fetch',
-                   'bzrlib.tests.test_fifo_cache',
-                   'bzrlib.tests.test_filters',
-                   'bzrlib.tests.test_ftp_transport',
-                   'bzrlib.tests.test_foreign',
-                   'bzrlib.tests.test_generate_docs',
-                   'bzrlib.tests.test_generate_ids',
-                   'bzrlib.tests.test_globbing',
-                   'bzrlib.tests.test_gpg',
-                   'bzrlib.tests.test_graph',
-                   'bzrlib.tests.test_groupcompress',
-                   'bzrlib.tests.test_hashcache',
-                   'bzrlib.tests.test_help',
-                   'bzrlib.tests.test_hooks',
-                   'bzrlib.tests.test_http',
-                   'bzrlib.tests.test_http_response',
-                   'bzrlib.tests.test_https_ca_bundle',
-                   'bzrlib.tests.test_identitymap',
-                   'bzrlib.tests.test_ignores',
-                   'bzrlib.tests.test_index',
-                   'bzrlib.tests.test_info',
-                   'bzrlib.tests.test_inv',
-                   'bzrlib.tests.test_inventory_delta',
-                   'bzrlib.tests.test_knit',
-                   'bzrlib.tests.test_lazy_import',
-                   'bzrlib.tests.test_lazy_regex',
-                   'bzrlib.tests.test_lock',
-                   'bzrlib.tests.test_lockable_files',
-                   'bzrlib.tests.test_lockdir',
-                   'bzrlib.tests.test_log',
-                   'bzrlib.tests.test_lru_cache',
-                   'bzrlib.tests.test_lsprof',
-                   'bzrlib.tests.test_mail_client',
-                   'bzrlib.tests.test_memorytree',
-                   'bzrlib.tests.test_merge',
-                   'bzrlib.tests.test_merge3',
-                   'bzrlib.tests.test_merge_core',
-                   'bzrlib.tests.test_merge_directive',
-                   'bzrlib.tests.test_missing',
-                   'bzrlib.tests.test_msgeditor',
-                   'bzrlib.tests.test_multiparent',
-                   'bzrlib.tests.test_mutabletree',
-                   'bzrlib.tests.test_nonascii',
-                   'bzrlib.tests.test_options',
-                   'bzrlib.tests.test_osutils',
-                   'bzrlib.tests.test_osutils_encodings',
-                   'bzrlib.tests.test_pack',
-                   'bzrlib.tests.test_patch',
-                   'bzrlib.tests.test_patches',
-                   'bzrlib.tests.test_permissions',
-                   'bzrlib.tests.test_plugins',
-                   'bzrlib.tests.test_progress',
-                   'bzrlib.tests.test_read_bundle',
-                   'bzrlib.tests.test_reconcile',
-                   'bzrlib.tests.test_reconfigure',
-                   'bzrlib.tests.test_registry',
-                   'bzrlib.tests.test_remote',
-                   'bzrlib.tests.test_rename_map',
-                   'bzrlib.tests.test_repository',
-                   'bzrlib.tests.test_revert',
-                   'bzrlib.tests.test_revision',
-                   'bzrlib.tests.test_revisionspec',
-                   'bzrlib.tests.test_revisiontree',
-                   'bzrlib.tests.test_rio',
-                   'bzrlib.tests.test_rules',
-                   'bzrlib.tests.test_sampler',
-                   'bzrlib.tests.test_selftest',
-                   'bzrlib.tests.test_serializer',
-                   'bzrlib.tests.test_setup',
-                   'bzrlib.tests.test_sftp_transport',
-                   'bzrlib.tests.test_shelf',
-                   'bzrlib.tests.test_shelf_ui',
-                   'bzrlib.tests.test_smart',
-                   'bzrlib.tests.test_smart_add',
-                   'bzrlib.tests.test_smart_request',
-                   'bzrlib.tests.test_smart_transport',
-                   'bzrlib.tests.test_smtp_connection',
-                   'bzrlib.tests.test_source',
-                   'bzrlib.tests.test_ssh_transport',
-                   'bzrlib.tests.test_status',
-                   'bzrlib.tests.test_store',
-                   'bzrlib.tests.test_strace',
-                   'bzrlib.tests.test_subsume',
-                   'bzrlib.tests.test_switch',
-                   'bzrlib.tests.test_symbol_versioning',
-                   'bzrlib.tests.test_tag',
-                   'bzrlib.tests.test_testament',
-                   'bzrlib.tests.test_textfile',
-                   'bzrlib.tests.test_textmerge',
-                   'bzrlib.tests.test_timestamp',
-                   'bzrlib.tests.test_trace',
-                   'bzrlib.tests.test_transactions',
-                   'bzrlib.tests.test_transform',
-                   'bzrlib.tests.test_transport',
-                   'bzrlib.tests.test_transport_log',
-                   'bzrlib.tests.test_tree',
-                   'bzrlib.tests.test_treebuilder',
-                   'bzrlib.tests.test_tsort',
-                   'bzrlib.tests.test_tuned_gzip',
-                   'bzrlib.tests.test_ui',
-                   'bzrlib.tests.test_uncommit',
-                   'bzrlib.tests.test_upgrade',
-                   'bzrlib.tests.test_upgrade_stacked',
-                   'bzrlib.tests.test_urlutils',
-                   'bzrlib.tests.test_version',
-                   'bzrlib.tests.test_version_info',
-                   'bzrlib.tests.test_weave',
-                   'bzrlib.tests.test_whitebox',
-                   'bzrlib.tests.test_win32utils',
-                   'bzrlib.tests.test_workingtree',
-                   'bzrlib.tests.test_workingtree_4',
-                   'bzrlib.tests.test_wsgi',
-                   'bzrlib.tests.test_xml',
-                   ]
 
     loader = TestUtil.TestLoader()
 
@@ -3639,24 +3731,9 @@
     suite = loader.suiteClass()
 
     # modules building their suite with loadTestsFromModuleNames
-    suite.addTest(loader.loadTestsFromModuleNames(testmod_names))
-
-    modules_to_doctest = [
-        'bzrlib',
-        'bzrlib.branchbuilder',
-        'bzrlib.export',
-        'bzrlib.inventory',
-        'bzrlib.iterablefile',
-        'bzrlib.lockdir',
-        'bzrlib.merge3',
-        'bzrlib.option',
-        'bzrlib.symbol_versioning',
-        'bzrlib.tests',
-        'bzrlib.timestamp',
-        'bzrlib.version_info_formats.format_custom',
-        ]
-
-    for mod in modules_to_doctest:
+    suite.addTest(loader.loadTestsFromModuleNames(_test_suite_testmod_names()))
+
+    for mod in _test_suite_modules_to_doctest():
         if not interesting_module(mod):
             # No tests to keep here, move along
             continue
@@ -3803,8 +3880,7 @@
     :param new_id: The id to assign to it.
     :return: The new test.
     """
-    from copy import deepcopy
-    new_test = deepcopy(test)
+    new_test = copy(test)
     new_test.id = lambda: new_id
     return new_test
 

=== modified file 'bzrlib/tests/blackbox/test_filesystem_cicp.py'
--- bzrlib/tests/blackbox/test_filesystem_cicp.py	2009-04-06 08:17:53 +0000
+++ bzrlib/tests/blackbox/test_filesystem_cicp.py	2009-08-26 09:06:02 +0000
@@ -216,12 +216,19 @@
 
 
 class TestMisc(TestCICPBase):
+
     def test_status(self):
         wt = self._make_mixed_case_tree()
         self.run_bzr('add')
 
-        self.check_output('added:\n  CamelCaseParent/CamelCase\n  lowercaseparent/lowercase\n',
-                          'status camelcaseparent/camelcase LOWERCASEPARENT/LOWERCASE')
+        self.check_output(
+            """added:
+  CamelCaseParent/
+  CamelCaseParent/CamelCase
+  lowercaseparent/
+  lowercaseparent/lowercase
+""",
+            'status camelcaseparent/camelcase LOWERCASEPARENT/LOWERCASE')
 
     def test_ci(self):
         wt = self._make_mixed_case_tree()

=== modified file 'bzrlib/tests/blackbox/test_info.py'
--- bzrlib/tests/blackbox/test_info.py	2009-08-17 03:47:03 +0000
+++ bzrlib/tests/blackbox/test_info.py	2009-08-25 23:38:10 +0000
@@ -1328,6 +1328,10 @@
     def test_info_locking_oslocks(self):
         if sys.platform == "win32":
             raise TestSkipped("don't use oslocks on win32 in unix manner")
+        # This test tests old (all-in-one, OS lock using) behaviour which
+        # simply cannot work on windows (and is indeed why we changed our
+        # design. As such, don't try to remove the thisFailsStrictLockCheck
+        # call here.
         self.thisFailsStrictLockCheck()
 
         tree = self.make_branch_and_tree('branch',

=== modified file 'bzrlib/tests/blackbox/test_push.py'
--- bzrlib/tests/blackbox/test_push.py	2009-08-20 04:09:58 +0000
+++ bzrlib/tests/blackbox/test_push.py	2009-08-27 22:17:35 +0000
@@ -576,9 +576,7 @@
     def setUp(self):
         tests.TestCaseWithTransport.setUp(self)
         self.memory_server = RedirectingMemoryServer()
-        self.memory_server.setUp()
-        self.addCleanup(self.memory_server.tearDown)
-
+        self.start_server(self.memory_server)
         # Make the branch and tree that we'll be pushing.
         t = self.make_branch_and_tree('tree')
         self.build_tree(['tree/file'])

=== modified file 'bzrlib/tests/blackbox/test_selftest.py'
--- bzrlib/tests/blackbox/test_selftest.py	2009-08-24 05:23:11 +0000
+++ bzrlib/tests/blackbox/test_selftest.py	2009-08-24 22:32:53 +0000
@@ -172,3 +172,7 @@
             outputs_nothing(['selftest', '--list-only', '--exclude', 'selftest'])
         finally:
             tests.selftest = original_selftest
+
+    def test_lsprof_tests(self):
+        params = self.get_params_passed_to_core('selftest --lsprof-tests')
+        self.assertEqual(True, params[1]["lsprof_tests"])

=== modified file 'bzrlib/tests/blackbox/test_serve.py'
--- bzrlib/tests/blackbox/test_serve.py	2009-07-20 11:27:05 +0000
+++ bzrlib/tests/blackbox/test_serve.py	2009-08-27 22:17:35 +0000
@@ -209,8 +209,7 @@
         ssh_server = SFTPServer(StubSSHServer)
         # XXX: We *don't* want to override the default SSH vendor, so we set
         # _vendor to what _get_ssh_vendor returns.
-        ssh_server.setUp()
-        self.addCleanup(ssh_server.tearDown)
+        self.start_server(ssh_server)
         port = ssh_server._listener.port
 
         # Access the branch via a bzr+ssh URL.  The BZR_REMOTE_PATH environment

=== modified file 'bzrlib/tests/blackbox/test_split.py'
--- bzrlib/tests/blackbox/test_split.py	2009-06-08 02:02:08 +0000
+++ bzrlib/tests/blackbox/test_split.py	2009-08-27 21:48:33 +0000
@@ -31,7 +31,7 @@
         wt.add(['b', 'b/c'])
         wt.commit('rev1')
         self.run_bzr('split a/b')
-        self.run_bzr_error(('.* is not versioned',), 'split q')
+        self.run_bzr_error(('.* is not versioned',), 'split q', working_dir='a')
 
     def test_split_repo_failure(self):
         repo = self.make_repository('branch', shared=True, format='knit')

=== modified file 'bzrlib/tests/http_utils.py'
--- bzrlib/tests/http_utils.py	2009-05-04 14:48:21 +0000
+++ bzrlib/tests/http_utils.py	2009-08-27 22:17:35 +0000
@@ -133,8 +133,7 @@
         """Get the server instance for the secondary transport."""
         if self.__secondary_server is None:
             self.__secondary_server = self.create_transport_secondary_server()
-            self.__secondary_server.setUp()
-            self.addCleanup(self.__secondary_server.tearDown)
+            self.start_server(self.__secondary_server)
         return self.__secondary_server
 
 

=== modified file 'bzrlib/tests/per_branch/test_push.py'
--- bzrlib/tests/per_branch/test_push.py	2009-08-14 00:55:42 +0000
+++ bzrlib/tests/per_branch/test_push.py	2009-08-27 22:17:35 +0000
@@ -394,8 +394,7 @@
         # Create a smart server that publishes whatever the backing VFS server
         # does.
         self.smart_server = server.SmartTCPServer_for_testing()
-        self.smart_server.setUp(self.get_server())
-        self.addCleanup(self.smart_server.tearDown)
+        self.start_server(self.smart_server, self.get_server())
         # Make two empty branches, 'empty' and 'target'.
         self.empty_branch = self.make_branch('empty')
         self.make_branch('target')

=== modified file 'bzrlib/tests/per_pack_repository.py'
--- bzrlib/tests/per_pack_repository.py	2009-08-14 00:55:42 +0000
+++ bzrlib/tests/per_pack_repository.py	2009-08-27 22:17:35 +0000
@@ -271,8 +271,7 @@
         # failing to delete obsolete packs is not fatal
         format = self.get_format()
         server = fakenfs.FakeNFSServer()
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         transport = get_transport(server.get_url())
         bzrdir = self.get_format().initialize_on_transport(transport)
         repo = bzrdir.create_repository()
@@ -1020,8 +1019,7 @@
         # Create a smart server that publishes whatever the backing VFS server
         # does.
         self.smart_server = server.SmartTCPServer_for_testing()
-        self.smart_server.setUp(self.get_server())
-        self.addCleanup(self.smart_server.tearDown)
+        self.start_server(self.smart_server, self.get_server())
         # Log all HPSS calls into self.hpss_calls.
         client._SmartClient.hooks.install_named_hook(
             'call', self.capture_hpss_call, None)

=== modified file 'bzrlib/tests/per_repository/test_repository.py'
--- bzrlib/tests/per_repository/test_repository.py	2009-08-18 22:03:18 +0000
+++ bzrlib/tests/per_repository/test_repository.py	2009-08-27 22:17:35 +0000
@@ -823,9 +823,8 @@
         be created at the given path."""
         repo = self.make_repository(path, shared=shared)
         smart_server = server.SmartTCPServer_for_testing()
-        smart_server.setUp(self.get_server())
+        self.start_server(smart_server, self.get_server())
         remote_transport = get_transport(smart_server.get_url()).clone(path)
-        self.addCleanup(smart_server.tearDown)
         remote_bzrdir = bzrdir.BzrDir.open_from_transport(remote_transport)
         remote_repo = remote_bzrdir.open_repository()
         return remote_repo
@@ -897,14 +896,6 @@
         local_repo = local_bzrdir.open_repository()
         self.assertEqual(remote_backing_repo._format, local_repo._format)
 
-    # XXX: this helper probably belongs on TestCaseWithTransport
-    def make_smart_server(self, path):
-        smart_server = server.SmartTCPServer_for_testing()
-        smart_server.setUp(self.get_server())
-        remote_transport = get_transport(smart_server.get_url()).clone(path)
-        self.addCleanup(smart_server.tearDown)
-        return remote_transport
-
     def test_clone_to_hpss(self):
         pre_metadir_formats = [RepositoryFormat5(), RepositoryFormat6()]
         if self.repository_format in pre_metadir_formats:

=== modified file 'bzrlib/tests/per_workingtree/test_flush.py'
--- bzrlib/tests/per_workingtree/test_flush.py	2009-07-31 17:42:29 +0000
+++ bzrlib/tests/per_workingtree/test_flush.py	2009-08-25 23:38:10 +0000
@@ -16,7 +16,9 @@
 
 """Tests for WorkingTree.flush."""
 
+import sys
 from bzrlib import errors, inventory
+from bzrlib.tests import TestSkipped
 from bzrlib.tests.per_workingtree import TestCaseWithWorkingTree
 
 
@@ -31,8 +33,14 @@
             tree.unlock()
 
     def test_flush_when_inventory_is_modified(self):
+        if sys.platform == "win32":
+            raise TestSkipped("don't use oslocks on win32 in unix manner")
         # This takes a write lock on the source tree, then opens a second copy
-        # and tries to grab a read lock, which is a bit bogus
+        # and tries to grab a read lock. This works on Unix and is a reasonable
+        # way to detect when the file is actually written to, but it won't work
+        # (as a test) on Windows. It might be nice to instead stub out the
+        # functions used to write and that way do both less work and also be
+        # able to execute on Windows.
         self.thisFailsStrictLockCheck()
         # when doing a flush the inventory should be written if needed.
         # we test that by changing the inventory (using

=== modified file 'bzrlib/tests/per_workingtree/test_locking.py'
--- bzrlib/tests/per_workingtree/test_locking.py	2009-07-31 17:42:29 +0000
+++ bzrlib/tests/per_workingtree/test_locking.py	2009-08-25 23:38:10 +0000
@@ -16,11 +16,14 @@
 
 """Tests for the (un)lock interfaces on all working tree implemenations."""
 
+import sys
+
 from bzrlib import (
     branch,
     errors,
     lockdir,
     )
+from bzrlib.tests import TestSkipped
 from bzrlib.tests.per_workingtree import TestCaseWithWorkingTree
 
 
@@ -105,8 +108,14 @@
 
         :param methodname: The lock method to use to establish locks.
         """
-        # This write locks the local tree, and then grabs a read lock on a
-        # copy, which is bogus and the test just needs to be rewritten.
+        if sys.platform == "win32":
+            raise TestSkipped("don't use oslocks on win32 in unix manner")
+        # This helper takes a write lock on the source tree, then opens a
+        # second copy and tries to grab a read lock. This works on Unix and is
+        # a reasonable way to detect when the file is actually written to, but
+        # it won't work (as a test) on Windows. It might be nice to instead
+        # stub out the functions used to write and that way do both less work
+        # and also be able to execute on Windows.
         self.thisFailsStrictLockCheck()
         # when unlocking the last lock count from tree_write_lock,
         # the tree should do a flush().

=== modified file 'bzrlib/tests/per_workingtree/test_set_root_id.py'
--- bzrlib/tests/per_workingtree/test_set_root_id.py	2009-08-21 01:48:13 +0000
+++ bzrlib/tests/per_workingtree/test_set_root_id.py	2009-08-28 05:00:33 +0000
@@ -16,13 +16,18 @@
 
 """Tests for WorkingTree.set_root_id"""
 
+import sys
+
 from bzrlib import inventory
+from bzrlib.tests import TestSkipped
 from bzrlib.tests.per_workingtree import TestCaseWithWorkingTree
 
 
 class TestSetRootId(TestCaseWithWorkingTree):
 
     def test_set_and_read_unicode(self):
+        if sys.platform == "win32":
+            raise TestSkipped("don't use oslocks on win32 in unix manner")
         # This test tests that setting the root doesn't flush, so it
         # deliberately tests concurrent access that isn't possible on windows.
         self.thisFailsStrictLockCheck()

=== modified file 'bzrlib/tests/test__known_graph.py'
--- bzrlib/tests/test__known_graph.py	2009-08-26 16:03:59 +0000
+++ bzrlib/tests/test__known_graph.py	2009-09-02 13:32:52 +0000
@@ -768,3 +768,70 @@
                 },
                 'E',
                 [])
+
+
+class TestKnownGraphStableReverseTopoSort(TestCaseWithKnownGraph):
+    """Test the sort order returned by gc_sort."""
+
+    def assertSorted(self, expected, parent_map):
+        graph = self.make_known_graph(parent_map)
+        value = graph.gc_sort()
+        if expected != value:
+            self.assertEqualDiff(pprint.pformat(expected),
+                                 pprint.pformat(value))
+
+    def test_empty(self):
+        self.assertSorted([], {})
+
+    def test_single(self):
+        self.assertSorted(['a'], {'a':()})
+        self.assertSorted([('a',)], {('a',):()})
+        self.assertSorted([('F', 'a')], {('F', 'a'):()})
+
+    def test_linear(self):
+        self.assertSorted(['c', 'b', 'a'], {'a':(), 'b':('a',), 'c':('b',)})
+        self.assertSorted([('c',), ('b',), ('a',)],
+                          {('a',):(), ('b',): (('a',),), ('c',): (('b',),)})
+        self.assertSorted([('F', 'c'), ('F', 'b'), ('F', 'a')],
+                          {('F', 'a'):(), ('F', 'b'): (('F', 'a'),),
+                           ('F', 'c'): (('F', 'b'),)})
+
+    def test_mixed_ancestries(self):
+        # Each prefix should be sorted separately
+        self.assertSorted([('F', 'c'), ('F', 'b'), ('F', 'a'),
+                           ('G', 'c'), ('G', 'b'), ('G', 'a'),
+                           ('Q', 'c'), ('Q', 'b'), ('Q', 'a'),
+                          ],
+                          {('F', 'a'):(), ('F', 'b'): (('F', 'a'),),
+                           ('F', 'c'): (('F', 'b'),),
+                           ('G', 'a'):(), ('G', 'b'): (('G', 'a'),),
+                           ('G', 'c'): (('G', 'b'),),
+                           ('Q', 'a'):(), ('Q', 'b'): (('Q', 'a'),),
+                           ('Q', 'c'): (('Q', 'b'),),
+                          })
+
+    def test_stable_sorting(self):
+        # the sort order should be stable even when extra nodes are added
+        self.assertSorted(['b', 'c', 'a'],
+                          {'a':(), 'b':('a',), 'c':('a',)})
+        self.assertSorted(['b', 'c', 'd', 'a'],
+                          {'a':(), 'b':('a',), 'c':('a',), 'd':('a',)})
+        self.assertSorted(['b', 'c', 'd', 'a'],
+                          {'a':(), 'b':('a',), 'c':('a',), 'd':('a',)})
+        self.assertSorted(['Z', 'b', 'c', 'd', 'a'],
+                          {'a':(), 'b':('a',), 'c':('a',), 'd':('a',),
+                           'Z':('a',)})
+        self.assertSorted(['e', 'b', 'c', 'f', 'Z', 'd', 'a'],
+                          {'a':(), 'b':('a',), 'c':('a',), 'd':('a',),
+                           'Z':('a',),
+                           'e':('b', 'c', 'd'),
+                           'f':('d', 'Z'),
+                           })
+
+    def test_skip_ghost(self):
+        self.assertSorted(['b', 'c', 'a'],
+                          {'a':(), 'b':('a', 'ghost'), 'c':('a',)})
+
+    def test_skip_mainline_ghost(self):
+        self.assertSorted(['b', 'c', 'a'],
+                          {'a':(), 'b':('ghost', 'a'), 'c':('a',)})

=== modified file 'bzrlib/tests/test_bundle.py'
--- bzrlib/tests/test_bundle.py	2009-08-04 14:10:09 +0000
+++ bzrlib/tests/test_bundle.py	2009-08-27 22:17:35 +0000
@@ -1830,9 +1830,8 @@
         """
         from bzrlib.tests.blackbox.test_push import RedirectingMemoryServer
         server = RedirectingMemoryServer()
-        server.setUp()
+        self.start_server(server)
         url = server.get_url() + 'infinite-loop'
-        self.addCleanup(server.tearDown)
         self.assertRaises(errors.NotABundle, read_mergeable_from_url, url)
 
     def test_smart_server_connection_reset(self):
@@ -1841,8 +1840,7 @@
         """
         # Instantiate a server that will provoke a ConnectionReset
         sock_server = _DisconnectingTCPServer()
-        sock_server.setUp()
-        self.addCleanup(sock_server.tearDown)
+        self.start_server(sock_server)
         # We don't really care what the url is since the server will close the
         # connection without interpreting it
         url = sock_server.get_url()

=== modified file 'bzrlib/tests/test_crash.py'
--- bzrlib/tests/test_crash.py	2009-08-20 04:45:48 +0000
+++ bzrlib/tests/test_crash.py	2009-08-28 12:38:01 +0000
@@ -18,20 +18,17 @@
 from StringIO import StringIO
 import sys
 
-
-from bzrlib.crash import (
-    report_bug,
-    _write_apport_report_to_file,
+from bzrlib import (
+    crash,
+    tests,
     )
-from bzrlib.tests import TestCase
-from bzrlib.tests.features import ApportFeature
-
-
-class TestApportReporting(TestCase):
-
-    def setUp(self):
-        TestCase.setUp(self)
-        self.requireFeature(ApportFeature)
+
+from bzrlib.tests import features
+
+
+class TestApportReporting(tests.TestCase):
+
+    _test_needs_features = [features.ApportFeature]
 
     def test_apport_report_contents(self):
         try:
@@ -39,19 +36,13 @@
         except AssertionError, e:
             pass
         outf = StringIO()
-        _write_apport_report_to_file(sys.exc_info(),
-            outf)
+        crash._write_apport_report_to_file(sys.exc_info(), outf)
         report = outf.getvalue()
 
-        self.assertContainsRe(report,
-            '(?m)^BzrVersion:')
+        self.assertContainsRe(report, '(?m)^BzrVersion:')
         # should be in the traceback
-        self.assertContainsRe(report,
-            'my error')
-        self.assertContainsRe(report,
-            'AssertionError')
-        self.assertContainsRe(report,
-            'test_apport_report_contents')
+        self.assertContainsRe(report, 'my error')
+        self.assertContainsRe(report, 'AssertionError')
+        self.assertContainsRe(report, 'test_apport_report_contents')
         # should also be in there
-        self.assertContainsRe(report,
-            '(?m)^CommandLine:.*selftest')
+        self.assertContainsRe(report, '(?m)^CommandLine:')

=== modified file 'bzrlib/tests/test_groupcompress.py'
--- bzrlib/tests/test_groupcompress.py	2009-06-29 14:51:13 +0000
+++ bzrlib/tests/test_groupcompress.py	2009-09-03 15:26:27 +0000
@@ -538,7 +538,7 @@
                     'as-requested', False)]
         self.assertEqual([('b',), ('a',), ('d',), ('c',)], keys)
 
-    def test_insert_record_stream_re_uses_blocks(self):
+    def test_insert_record_stream_reuses_blocks(self):
         vf = self.make_test_vf(True, dir='source')
         def grouped_stream(revision_ids, first_parents=()):
             parents = first_parents
@@ -582,8 +582,14 @@
         vf2 = self.make_test_vf(True, dir='target')
         # ordering in 'groupcompress' order, should actually swap the groups in
         # the target vf, but the groups themselves should not be disturbed.
-        vf2.insert_record_stream(vf.get_record_stream(
-            [(r,) for r in 'abcdefgh'], 'groupcompress', False))
+        def small_size_stream():
+            for record in vf.get_record_stream([(r,) for r in 'abcdefgh'],
+                                               'groupcompress', False):
+                record._manager._full_enough_block_size = \
+                    record._manager._block._content_length
+                yield record
+                        
+        vf2.insert_record_stream(small_size_stream())
         stream = vf2.get_record_stream([(r,) for r in 'abcdefgh'],
                                        'groupcompress', False)
         vf2.writer.end()
@@ -594,6 +600,44 @@
                              record._manager._block._z_content)
         self.assertEqual(8, num_records)
 
+    def test_insert_record_stream_packs_on_the_fly(self):
+        vf = self.make_test_vf(True, dir='source')
+        def grouped_stream(revision_ids, first_parents=()):
+            parents = first_parents
+            for revision_id in revision_ids:
+                key = (revision_id,)
+                record = versionedfile.FulltextContentFactory(
+                    key, parents, None,
+                    'some content that is\n'
+                    'identical except for\n'
+                    'revision_id:%s\n' % (revision_id,))
+                yield record
+                parents = (key,)
+        # One group, a-d
+        vf.insert_record_stream(grouped_stream(['a', 'b', 'c', 'd']))
+        # Second group, e-h
+        vf.insert_record_stream(grouped_stream(['e', 'f', 'g', 'h'],
+                                               first_parents=(('d',),)))
+        # Now copy the blocks into another vf, and see that the
+        # insert_record_stream rebuilt a new block on-the-fly because of
+        # under-utilization
+        vf2 = self.make_test_vf(True, dir='target')
+        vf2.insert_record_stream(vf.get_record_stream(
+            [(r,) for r in 'abcdefgh'], 'groupcompress', False))
+        stream = vf2.get_record_stream([(r,) for r in 'abcdefgh'],
+                                       'groupcompress', False)
+        vf2.writer.end()
+        num_records = 0
+        # All of the records should be recombined into a single block
+        block = None
+        for record in stream:
+            num_records += 1
+            if block is None:
+                block = record._manager._block
+            else:
+                self.assertIs(block, record._manager._block)
+        self.assertEqual(8, num_records)
+
     def test__insert_record_stream_no_reuse_block(self):
         vf = self.make_test_vf(True, dir='source')
         def grouped_stream(revision_ids, first_parents=()):
@@ -702,19 +746,128 @@
                               " 0 8', \(\(\('a',\),\),\)\)")
 
 
+class StubGCVF(object):
+    def __init__(self, canned_get_blocks=None):
+        self._group_cache = {}
+        self._canned_get_blocks = canned_get_blocks or []
+    def _get_blocks(self, read_memos):
+        return iter(self._canned_get_blocks)
+    
+
+class Test_BatchingBlockFetcher(TestCaseWithGroupCompressVersionedFiles):
+    """Simple whitebox unit tests for _BatchingBlockFetcher."""
+    
+    def test_add_key_new_read_memo(self):
+        """Adding a key with an uncached read_memo new to this batch adds that
+        read_memo to the list of memos to fetch.
+        """
+        # locations are: index_memo, ignored, parents, ignored
+        # where index_memo is: (idx, offset, len, factory_start, factory_end)
+        # and (idx, offset, size) is known as the 'read_memo', identifying the
+        # raw bytes needed.
+        read_memo = ('fake index', 100, 50)
+        locations = {
+            ('key',): (read_memo + (None, None), None, None, None)}
+        batcher = groupcompress._BatchingBlockFetcher(StubGCVF(), locations)
+        total_size = batcher.add_key(('key',))
+        self.assertEqual(50, total_size)
+        self.assertEqual([('key',)], batcher.keys)
+        self.assertEqual([read_memo], batcher.memos_to_get)
+
+    def test_add_key_duplicate_read_memo(self):
+        """read_memos that occur multiple times in a batch will only be fetched
+        once.
+        """
+        read_memo = ('fake index', 100, 50)
+        # Two keys, both sharing the same read memo (but different overall
+        # index_memos).
+        locations = {
+            ('key1',): (read_memo + (0, 1), None, None, None),
+            ('key2',): (read_memo + (1, 2), None, None, None)}
+        batcher = groupcompress._BatchingBlockFetcher(StubGCVF(), locations)
+        total_size = batcher.add_key(('key1',))
+        total_size = batcher.add_key(('key2',))
+        self.assertEqual(50, total_size)
+        self.assertEqual([('key1',), ('key2',)], batcher.keys)
+        self.assertEqual([read_memo], batcher.memos_to_get)
+
+    def test_add_key_cached_read_memo(self):
+        """Adding a key with a cached read_memo will not cause that read_memo
+        to be added to the list to fetch.
+        """
+        read_memo = ('fake index', 100, 50)
+        gcvf = StubGCVF()
+        gcvf._group_cache[read_memo] = 'fake block'
+        locations = {
+            ('key',): (read_memo + (None, None), None, None, None)}
+        batcher = groupcompress._BatchingBlockFetcher(gcvf, locations)
+        total_size = batcher.add_key(('key',))
+        self.assertEqual(0, total_size)
+        self.assertEqual([('key',)], batcher.keys)
+        self.assertEqual([], batcher.memos_to_get)
+
+    def test_yield_factories_empty(self):
+        """An empty batch yields no factories."""
+        batcher = groupcompress._BatchingBlockFetcher(StubGCVF(), {})
+        self.assertEqual([], list(batcher.yield_factories()))
+
+    def test_yield_factories_calls_get_blocks(self):
+        """Uncached memos are retrieved via get_blocks."""
+        read_memo1 = ('fake index', 100, 50)
+        read_memo2 = ('fake index', 150, 40)
+        gcvf = StubGCVF(
+            canned_get_blocks=[
+                (read_memo1, groupcompress.GroupCompressBlock()),
+                (read_memo2, groupcompress.GroupCompressBlock())])
+        locations = {
+            ('key1',): (read_memo1 + (None, None), None, None, None),
+            ('key2',): (read_memo2 + (None, None), None, None, None)}
+        batcher = groupcompress._BatchingBlockFetcher(gcvf, locations)
+        batcher.add_key(('key1',))
+        batcher.add_key(('key2',))
+        factories = list(batcher.yield_factories(full_flush=True))
+        self.assertLength(2, factories)
+        keys = [f.key for f in factories]
+        kinds = [f.storage_kind for f in factories]
+        self.assertEqual([('key1',), ('key2',)], keys)
+        self.assertEqual(['groupcompress-block', 'groupcompress-block'], kinds)
+
+    def test_yield_factories_flushing(self):
+        """yield_factories holds back on yielding results from the final block
+        unless passed full_flush=True.
+        """
+        fake_block = groupcompress.GroupCompressBlock()
+        read_memo = ('fake index', 100, 50)
+        gcvf = StubGCVF()
+        gcvf._group_cache[read_memo] = fake_block
+        locations = {
+            ('key',): (read_memo + (None, None), None, None, None)}
+        batcher = groupcompress._BatchingBlockFetcher(gcvf, locations)
+        batcher.add_key(('key',))
+        self.assertEqual([], list(batcher.yield_factories()))
+        factories = list(batcher.yield_factories(full_flush=True))
+        self.assertLength(1, factories)
+        self.assertEqual(('key',), factories[0].key)
+        self.assertEqual('groupcompress-block', factories[0].storage_kind)
+
+
 class TestLazyGroupCompress(tests.TestCaseWithTransport):
 
     _texts = {
         ('key1',): "this is a text\n"
-                   "with a reasonable amount of compressible bytes\n",
+                   "with a reasonable amount of compressible bytes\n"
+                   "which can be shared between various other texts\n",
         ('key2',): "another text\n"
-                   "with a reasonable amount of compressible bytes\n",
+                   "with a reasonable amount of compressible bytes\n"
+                   "which can be shared between various other texts\n",
         ('key3',): "yet another text which won't be extracted\n"
-                   "with a reasonable amount of compressible bytes\n",
+                   "with a reasonable amount of compressible bytes\n"
+                   "which can be shared between various other texts\n",
         ('key4',): "this will be extracted\n"
                    "but references most of its bytes from\n"
                    "yet another text which won't be extracted\n"
-                   "with a reasonable amount of compressible bytes\n",
+                   "with a reasonable amount of compressible bytes\n"
+                   "which can be shared between various other texts\n",
     }
     def make_block(self, key_to_text):
         """Create a GroupCompressBlock, filling it with the given texts."""
@@ -732,6 +885,13 @@
         start, end = locations[key]
         manager.add_factory(key, (), start, end)
 
+    def make_block_and_full_manager(self, texts):
+        locations, block = self.make_block(texts)
+        manager = groupcompress._LazyGroupContentManager(block)
+        for key in sorted(texts):
+            self.add_key_to_manager(key, locations, block, manager)
+        return block, manager
+
     def test_get_fulltexts(self):
         locations, block = self.make_block(self._texts)
         manager = groupcompress._LazyGroupContentManager(block)
@@ -788,8 +948,8 @@
         header_len = int(header_len)
         block_len = int(block_len)
         self.assertEqual('groupcompress-block', storage_kind)
-        self.assertEqual(33, z_header_len)
-        self.assertEqual(25, header_len)
+        self.assertEqual(34, z_header_len)
+        self.assertEqual(26, header_len)
         self.assertEqual(len(block_bytes), block_len)
         z_header = rest[:z_header_len]
         header = zlib.decompress(z_header)
@@ -829,13 +989,7 @@
         self.assertEqual([('key1',), ('key4',)], result_order)
 
     def test__check_rebuild_no_changes(self):
-        locations, block = self.make_block(self._texts)
-        manager = groupcompress._LazyGroupContentManager(block)
-        # Request all the keys, which ensures that we won't rebuild
-        self.add_key_to_manager(('key1',), locations, block, manager)
-        self.add_key_to_manager(('key2',), locations, block, manager)
-        self.add_key_to_manager(('key3',), locations, block, manager)
-        self.add_key_to_manager(('key4',), locations, block, manager)
+        block, manager = self.make_block_and_full_manager(self._texts)
         manager._check_rebuild_block()
         self.assertIs(block, manager._block)
 
@@ -866,3 +1020,50 @@
             self.assertEqual(('key4',), record.key)
             self.assertEqual(self._texts[record.key],
                              record.get_bytes_as('fulltext'))
+
+    def test_check_is_well_utilized_all_keys(self):
+        block, manager = self.make_block_and_full_manager(self._texts)
+        self.assertFalse(manager.check_is_well_utilized())
+        # Though we can fake it by changing the recommended minimum size
+        manager._full_enough_block_size = block._content_length
+        self.assertTrue(manager.check_is_well_utilized())
+        # Setting it just above causes it to fail
+        manager._full_enough_block_size = block._content_length + 1
+        self.assertFalse(manager.check_is_well_utilized())
+        # Setting the mixed-block size doesn't do anything, because the content
+        # is considered to not be 'mixed'
+        manager._full_enough_mixed_block_size = block._content_length
+        self.assertFalse(manager.check_is_well_utilized())
+
+    def test_check_is_well_utilized_mixed_keys(self):
+        texts = {}
+        f1k1 = ('f1', 'k1')
+        f1k2 = ('f1', 'k2')
+        f2k1 = ('f2', 'k1')
+        f2k2 = ('f2', 'k2')
+        texts[f1k1] = self._texts[('key1',)]
+        texts[f1k2] = self._texts[('key2',)]
+        texts[f2k1] = self._texts[('key3',)]
+        texts[f2k2] = self._texts[('key4',)]
+        block, manager = self.make_block_and_full_manager(texts)
+        self.assertFalse(manager.check_is_well_utilized())
+        manager._full_enough_block_size = block._content_length
+        self.assertTrue(manager.check_is_well_utilized())
+        manager._full_enough_block_size = block._content_length + 1
+        self.assertFalse(manager.check_is_well_utilized())
+        manager._full_enough_mixed_block_size = block._content_length
+        self.assertTrue(manager.check_is_well_utilized())
+
+    def test_check_is_well_utilized_partial_use(self):
+        locations, block = self.make_block(self._texts)
+        manager = groupcompress._LazyGroupContentManager(block)
+        manager._full_enough_block_size = block._content_length
+        self.add_key_to_manager(('key1',), locations, block, manager)
+        self.add_key_to_manager(('key2',), locations, block, manager)
+        # Just using the content from key1 and 2 is not enough to be considered
+        # 'complete'
+        self.assertFalse(manager.check_is_well_utilized())
+        # However if we add key3, then we have enough, as we only require 75%
+        # consumption
+        self.add_key_to_manager(('key4',), locations, block, manager)
+        self.assertTrue(manager.check_is_well_utilized())

=== modified file 'bzrlib/tests/test_http.py'
--- bzrlib/tests/test_http.py	2009-08-19 16:33:39 +0000
+++ bzrlib/tests/test_http.py	2009-08-27 22:17:35 +0000
@@ -304,7 +304,7 @@
 
         server = http_server.HttpServer(BogusRequestHandler)
         try:
-            self.assertRaises(httplib.UnknownProtocol,server.setUp)
+            self.assertRaises(httplib.UnknownProtocol, server.setUp)
         except:
             server.tearDown()
             self.fail('HTTP Server creation did not raise UnknownProtocol')
@@ -312,7 +312,7 @@
     def test_force_invalid_protocol(self):
         server = http_server.HttpServer(protocol_version='HTTP/0.1')
         try:
-            self.assertRaises(httplib.UnknownProtocol,server.setUp)
+            self.assertRaises(httplib.UnknownProtocol, server.setUp)
         except:
             server.tearDown()
             self.fail('HTTP Server creation did not raise UnknownProtocol')
@@ -320,8 +320,10 @@
     def test_server_start_and_stop(self):
         server = http_server.HttpServer()
         server.setUp()
-        self.assertTrue(server._http_running)
-        server.tearDown()
+        try:
+            self.assertTrue(server._http_running)
+        finally:
+            server.tearDown()
         self.assertFalse(server._http_running)
 
     def test_create_http_server_one_zero(self):
@@ -330,8 +332,7 @@
             protocol_version = 'HTTP/1.0'
 
         server = http_server.HttpServer(RequestHandlerOneZero)
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         self.assertIsInstance(server._httpd, http_server.TestingHTTPServer)
 
     def test_create_http_server_one_one(self):
@@ -340,8 +341,7 @@
             protocol_version = 'HTTP/1.1'
 
         server = http_server.HttpServer(RequestHandlerOneOne)
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         self.assertIsInstance(server._httpd,
                               http_server.TestingThreadingHTTPServer)
 
@@ -352,8 +352,7 @@
 
         server = http_server.HttpServer(RequestHandlerOneZero,
                                         protocol_version='HTTP/1.1')
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         self.assertIsInstance(server._httpd,
                               http_server.TestingThreadingHTTPServer)
 
@@ -364,8 +363,7 @@
 
         server = http_server.HttpServer(RequestHandlerOneOne,
                                         protocol_version='HTTP/1.0')
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         self.assertIsInstance(server._httpd,
                               http_server.TestingHTTPServer)
 
@@ -431,8 +429,8 @@
     def test_http_impl_urls(self):
         """There are servers which ask for particular clients to connect"""
         server = self._server()
+        server.setUp()
         try:
-            server.setUp()
             url = server.get_url()
             self.assertTrue(url.startswith('%s://' % self._qualified_prefix))
         finally:
@@ -544,8 +542,7 @@
 
     def test_post_body_is_received(self):
         server = RecordingServer(expect_body_tail='end-of-body')
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         scheme = self._qualified_prefix
         url = '%s://%s:%s/' % (scheme, server.host, server.port)
         http_transport = self._transport(url)
@@ -780,8 +777,7 @@
 
     def test_send_receive_bytes(self):
         server = RecordingServer(expect_body_tail='c')
-        server.setUp()
-        self.addCleanup(server.tearDown)
+        self.start_server(server)
         sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         sock.connect((server.host, server.port))
         sock.sendall('abc')

=== modified file 'bzrlib/tests/test_lsprof.py'
--- bzrlib/tests/test_lsprof.py	2009-03-23 14:59:43 +0000
+++ bzrlib/tests/test_lsprof.py	2009-08-24 21:05:09 +0000
@@ -92,3 +92,22 @@
         self.stats.save(f)
         data1 = cPickle.load(open(f))
         self.assertEqual(type(data1), bzrlib.lsprof.Stats)
+
+
+class TestBzrProfiler(tests.TestCase):
+
+    _test_needs_features = [LSProfFeature]
+
+    def test_start_call_stuff_stop(self):
+        profiler = bzrlib.lsprof.BzrProfiler()
+        profiler.start()
+        try:
+            def a_function():
+                pass
+            a_function()
+        finally:
+            stats = profiler.stop()
+        stats.freeze()
+        lines = [str(data) for data in stats.data]
+        lines = [line for line in lines if 'a_function' in line]
+        self.assertLength(1, lines)

=== modified file 'bzrlib/tests/test_remote.py'
--- bzrlib/tests/test_remote.py	2009-08-27 05:22:14 +0000
+++ bzrlib/tests/test_remote.py	2009-08-30 21:34:42 +0000
@@ -1945,8 +1945,7 @@
     def test_allows_new_revisions(self):
         """get_parent_map's results can be updated by commit."""
         smart_server = server.SmartTCPServer_for_testing()
-        smart_server.setUp()
-        self.addCleanup(smart_server.tearDown)
+        self.start_server(smart_server)
         self.make_branch('branch')
         branch = Branch.open(smart_server.get_url() + '/branch')
         tree = branch.create_checkout('tree', lightweight=True)
@@ -2781,8 +2780,7 @@
         stacked_branch.set_stacked_on_url('../base')
         # start a server looking at this
         smart_server = server.SmartTCPServer_for_testing()
-        smart_server.setUp()
-        self.addCleanup(smart_server.tearDown)
+        self.start_server(smart_server)
         remote_bzrdir = BzrDir.open(smart_server.get_url() + '/stacked')
         # can get its branch and repository
         remote_branch = remote_bzrdir.open_branch()
@@ -2943,8 +2941,7 @@
         # Create a smart server that publishes whatever the backing VFS server
         # does.
         self.smart_server = server.SmartTCPServer_for_testing()
-        self.smart_server.setUp(self.get_server())
-        self.addCleanup(self.smart_server.tearDown)
+        self.start_server(self.smart_server, self.get_server())
         # Log all HPSS calls into self.hpss_calls.
         _SmartClient.hooks.install_named_hook(
             'call', self.capture_hpss_call, None)

=== modified file 'bzrlib/tests/test_repository.py'
--- bzrlib/tests/test_repository.py	2009-08-17 23:15:55 +0000
+++ bzrlib/tests/test_repository.py	2009-09-01 21:21:53 +0000
@@ -683,6 +683,28 @@
 
 class Test2a(TestCaseWithTransport):
 
+    def test_fetch_combines_groups(self):
+        builder = self.make_branch_builder('source', format='2a')
+        builder.start_series()
+        builder.build_snapshot('1', None, [
+            ('add', ('', 'root-id', 'directory', '')),
+            ('add', ('file', 'file-id', 'file', 'content\n'))])
+        builder.build_snapshot('2', ['1'], [
+            ('modify', ('file-id', 'content-2\n'))])
+        builder.finish_series()
+        source = builder.get_branch()
+        target = self.make_repository('target', format='2a')
+        target.fetch(source.repository)
+        target.lock_read()
+        self.addCleanup(target.unlock)
+        details = target.texts._index.get_build_details(
+            [('file-id', '1',), ('file-id', '2',)])
+        file_1_details = details[('file-id', '1')]
+        file_2_details = details[('file-id', '2')]
+        # The index, and what to read off disk, should be the same for both
+        # versions of the file.
+        self.assertEqual(file_1_details[0][:3], file_2_details[0][:3])
+
     def test_format_pack_compresses_True(self):
         repo = self.make_repository('repo', format='2a')
         self.assertTrue(repo._format.pack_compresses)

=== modified file 'bzrlib/tests/test_selftest.py'
--- bzrlib/tests/test_selftest.py	2009-08-24 05:35:28 +0000
+++ bzrlib/tests/test_selftest.py	2009-08-26 23:25:28 +0000
@@ -687,6 +687,26 @@
         self.assertEqual(url, t.clone('..').base)
 
 
+class TestProfileResult(tests.TestCase):
+
+    def test_profiles_tests(self):
+        self.requireFeature(test_lsprof.LSProfFeature)
+        terminal = unittest.TestResult()
+        result = tests.ProfileResult(terminal)
+        class Sample(tests.TestCase):
+            def a(self):
+                self.sample_function()
+            def sample_function(self):
+                pass
+        test = Sample("a")
+        test.attrs_to_keep = test.attrs_to_keep + ('_benchcalls',)
+        test.run(result)
+        self.assertLength(1, test._benchcalls)
+        # We must be able to unpack it as the test reporting code wants
+        (_, _, _), stats = test._benchcalls[0]
+        self.assertTrue(callable(stats.pprint))
+
+
 class TestTestResult(tests.TestCase):
 
     def check_timing(self, test_case, expected_re):
@@ -800,7 +820,7 @@
     def test_known_failure(self):
         """A KnownFailure being raised should trigger several result actions."""
         class InstrumentedTestResult(tests.ExtendedTestResult):
-            def done(self): pass
+            def stopTestRun(self): pass
             def startTests(self): pass
             def report_test_start(self, test): pass
             def report_known_failure(self, test, err):
@@ -854,7 +874,7 @@
     def test_add_not_supported(self):
         """Test the behaviour of invoking addNotSupported."""
         class InstrumentedTestResult(tests.ExtendedTestResult):
-            def done(self): pass
+            def stopTestRun(self): pass
             def startTests(self): pass
             def report_test_start(self, test): pass
             def report_unsupported(self, test, feature):
@@ -898,7 +918,7 @@
     def test_unavailable_exception(self):
         """An UnavailableFeature being raised should invoke addNotSupported."""
         class InstrumentedTestResult(tests.ExtendedTestResult):
-            def done(self): pass
+            def stopTestRun(self): pass
             def startTests(self): pass
             def report_test_start(self, test): pass
             def addNotSupported(self, test, feature):
@@ -981,11 +1001,14 @@
         because of our use of global state.
         """
         old_root = tests.TestCaseInTempDir.TEST_ROOT
+        old_leak = tests.TestCase._first_thread_leaker_id
         try:
             tests.TestCaseInTempDir.TEST_ROOT = None
+            tests.TestCase._first_thread_leaker_id = None
             return testrunner.run(test)
         finally:
             tests.TestCaseInTempDir.TEST_ROOT = old_root
+            tests.TestCase._first_thread_leaker_id = old_leak
 
     def test_known_failure_failed_run(self):
         # run a test that generates a known failure which should be printed in
@@ -1031,6 +1054,20 @@
             '\n'
             'OK \\(known_failures=1\\)\n')
 
+    def test_result_decorator(self):
+        # decorate results
+        calls = []
+        class LoggingDecorator(tests.ForwardingResult):
+            def startTest(self, test):
+                tests.ForwardingResult.startTest(self, test)
+                calls.append('start')
+        test = unittest.FunctionTestCase(lambda:None)
+        stream = StringIO()
+        runner = tests.TextTestRunner(stream=stream,
+            result_decorators=[LoggingDecorator])
+        result = self.run_test_runner(runner, test)
+        self.assertLength(1, calls)
+
     def test_skipped_test(self):
         # run a test that is skipped, and check the suite as a whole still
         # succeeds.
@@ -1103,10 +1140,6 @@
         self.assertContainsRe(out.getvalue(),
                 r'(?m)^    this test never runs')
 
-    def test_not_applicable_demo(self):
-        # just so you can see it in the test output
-        raise tests.TestNotApplicable('this test is just a demonstation')
-
     def test_unsupported_features_listed(self):
         """When unsupported features are encountered they are detailed."""
         class Feature1(tests.Feature):
@@ -1261,6 +1294,34 @@
         self.assertContainsRe(log, 'this will be kept')
         self.assertEqual(log, test._log_contents)
 
+    def test_startTestRun(self):
+        """run should call result.startTestRun()"""
+        calls = []
+        class LoggingDecorator(tests.ForwardingResult):
+            def startTestRun(self):
+                tests.ForwardingResult.startTestRun(self)
+                calls.append('startTestRun')
+        test = unittest.FunctionTestCase(lambda:None)
+        stream = StringIO()
+        runner = tests.TextTestRunner(stream=stream,
+            result_decorators=[LoggingDecorator])
+        result = self.run_test_runner(runner, test)
+        self.assertLength(1, calls)
+
+    def test_stopTestRun(self):
+        """run should call result.stopTestRun()"""
+        calls = []
+        class LoggingDecorator(tests.ForwardingResult):
+            def stopTestRun(self):
+                tests.ForwardingResult.stopTestRun(self)
+                calls.append('stopTestRun')
+        test = unittest.FunctionTestCase(lambda:None)
+        stream = StringIO()
+        runner = tests.TextTestRunner(stream=stream,
+            result_decorators=[LoggingDecorator])
+        result = self.run_test_runner(runner, test)
+        self.assertLength(1, calls)
+
 
 class SampleTestCase(tests.TestCase):
 
@@ -1480,6 +1541,7 @@
         self.assertEqual((time.sleep, (0.003,), {}), self._benchcalls[1][0])
         self.assertIsInstance(self._benchcalls[0][1], bzrlib.lsprof.Stats)
         self.assertIsInstance(self._benchcalls[1][1], bzrlib.lsprof.Stats)
+        del self._benchcalls[:]
 
     def test_knownFailure(self):
         """Self.knownFailure() should raise a KnownFailure exception."""
@@ -1742,16 +1804,16 @@
         tree = self.make_branch_and_memory_tree('a')
         self.assertIsInstance(tree, bzrlib.memorytree.MemoryTree)
 
-
-class TestSFTPMakeBranchAndTree(test_sftp_transport.TestCaseWithSFTPServer):
-
-    def test_make_tree_for_sftp_branch(self):
-        """Transports backed by local directories create local trees."""
-        # NB: This is arguably a bug in the definition of make_branch_and_tree.
+    def test_make_tree_for_local_vfs_backed_transport(self):
+        # make_branch_and_tree has to use local branch and repositories
+        # when the vfs transport and local disk are colocated, even if
+        # a different transport is in use for url generation.
+        from bzrlib.transport.fakevfat import FakeVFATServer
+        self.transport_server = FakeVFATServer
+        self.assertFalse(self.get_url('t1').startswith('file://'))
         tree = self.make_branch_and_tree('t1')
         base = tree.bzrdir.root_transport.base
-        self.failIf(base.startswith('sftp'),
-                'base %r is on sftp but should be local' % base)
+        self.assertStartsWith(base, 'file://')
         self.assertEquals(tree.bzrdir.root_transport,
                 tree.branch.bzrdir.root_transport)
         self.assertEquals(tree.bzrdir.root_transport,
@@ -1817,6 +1879,20 @@
         self.assertNotContainsRe("Test.b", output.getvalue())
         self.assertLength(2, output.readlines())
 
+    def test_lsprof_tests(self):
+        self.requireFeature(test_lsprof.LSProfFeature)
+        calls = []
+        class Test(object):
+            def __call__(test, result):
+                test.run(result)
+            def run(test, result):
+                self.assertIsInstance(result, tests.ForwardingResult)
+                calls.append("called")
+            def countTestCases(self):
+                return 1
+        self.run_selftest(test_suite_factory=Test, lsprof_tests=True)
+        self.assertLength(1, calls)
+
     def test_random(self):
         # test randomising by listing a number of tests.
         output_123 = self.run_selftest(test_suite_factory=self.factory,
@@ -1877,8 +1953,8 @@
     def test_transport_sftp(self):
         try:
             import bzrlib.transport.sftp
-        except ParamikoNotPresent:
-            raise TestSkipped("Paramiko not present")
+        except errors.ParamikoNotPresent:
+            raise tests.TestSkipped("Paramiko not present")
         self.check_transport_set(bzrlib.transport.sftp.SFTPAbsoluteServer)
 
     def test_transport_memory(self):
@@ -2072,7 +2148,8 @@
         return self.out, self.err
 
 
-class TestRunBzrSubprocess(tests.TestCaseWithTransport):
+class TestWithFakedStartBzrSubprocess(tests.TestCaseWithTransport):
+    """Base class for tests testing how we might run bzr."""
 
     def setUp(self):
         tests.TestCaseWithTransport.setUp(self)
@@ -2089,6 +2166,9 @@
             'working_dir':working_dir, 'allow_plugins':allow_plugins})
         return self.next_subprocess
 
+
+class TestRunBzrSubprocess(TestWithFakedStartBzrSubprocess):
+
     def assertRunBzrSubprocess(self, expected_args, process, *args, **kwargs):
         """Run run_bzr_subprocess with args and kwargs using a stubbed process.
 
@@ -2157,6 +2237,32 @@
             StubProcess(), '', allow_plugins=True)
 
 
+class TestFinishBzrSubprocess(TestWithFakedStartBzrSubprocess):
+
+    def test_finish_bzr_subprocess_with_error(self):
+        """finish_bzr_subprocess allows specification of the desired exit code.
+        """
+        process = StubProcess(err="unknown command", retcode=3)
+        result = self.finish_bzr_subprocess(process, retcode=3)
+        self.assertEqual('', result[0])
+        self.assertContainsRe(result[1], 'unknown command')
+
+    def test_finish_bzr_subprocess_ignoring_retcode(self):
+        """finish_bzr_subprocess allows the exit code to be ignored."""
+        process = StubProcess(err="unknown command", retcode=3)
+        result = self.finish_bzr_subprocess(process, retcode=None)
+        self.assertEqual('', result[0])
+        self.assertContainsRe(result[1], 'unknown command')
+
+    def test_finish_subprocess_with_unexpected_retcode(self):
+        """finish_bzr_subprocess raises self.failureException if the retcode is
+        not the expected one.
+        """
+        process = StubProcess(err="unknown command", retcode=3)
+        self.assertRaises(self.failureException, self.finish_bzr_subprocess,
+                          process)
+
+
 class _DontSpawnProcess(Exception):
     """A simple exception which just allows us to skip unnecessary steps"""
 
@@ -2240,39 +2346,8 @@
         self.assertEqual(['foo', 'current'], chdirs)
 
 
-class TestBzrSubprocess(tests.TestCaseWithTransport):
-
-    def test_start_and_stop_bzr_subprocess(self):
-        """We can start and perform other test actions while that process is
-        still alive.
-        """
-        process = self.start_bzr_subprocess(['--version'])
-        result = self.finish_bzr_subprocess(process)
-        self.assertContainsRe(result[0], 'is free software')
-        self.assertEqual('', result[1])
-
-    def test_start_and_stop_bzr_subprocess_with_error(self):
-        """finish_bzr_subprocess allows specification of the desired exit code.
-        """
-        process = self.start_bzr_subprocess(['--versionn'])
-        result = self.finish_bzr_subprocess(process, retcode=3)
-        self.assertEqual('', result[0])
-        self.assertContainsRe(result[1], 'unknown command')
-
-    def test_start_and_stop_bzr_subprocess_ignoring_retcode(self):
-        """finish_bzr_subprocess allows the exit code to be ignored."""
-        process = self.start_bzr_subprocess(['--versionn'])
-        result = self.finish_bzr_subprocess(process, retcode=None)
-        self.assertEqual('', result[0])
-        self.assertContainsRe(result[1], 'unknown command')
-
-    def test_start_and_stop_bzr_subprocess_with_unexpected_retcode(self):
-        """finish_bzr_subprocess raises self.failureException if the retcode is
-        not the expected one.
-        """
-        process = self.start_bzr_subprocess(['--versionn'])
-        self.assertRaises(self.failureException, self.finish_bzr_subprocess,
-                          process)
+class TestActuallyStartBzrSubprocess(tests.TestCaseWithTransport):
+    """Tests that really need to do things with an external bzr."""
 
     def test_start_and_stop_bzr_subprocess_send_signal(self):
         """finish_bzr_subprocess raises self.failureException if the retcode is
@@ -2286,14 +2361,6 @@
         self.assertEqual('', result[0])
         self.assertEqual('bzr: interrupted\n', result[1])
 
-    def test_start_and_stop_working_dir(self):
-        cwd = osutils.getcwd()
-        self.make_branch_and_tree('one')
-        process = self.start_bzr_subprocess(['root'], working_dir='one')
-        result = self.finish_bzr_subprocess(process, universal_newlines=True)
-        self.assertEndsWith(result[0], 'one\n')
-        self.assertEqual('', result[1])
-
 
 class TestKnownFailure(tests.TestCase):
 
@@ -2681,10 +2748,52 @@
 
 class TestTestSuite(tests.TestCase):
 
+    def test__test_suite_testmod_names(self):
+        # Test that a plausible list of test module names are returned
+        # by _test_suite_testmod_names.
+        test_list = tests._test_suite_testmod_names()
+        self.assertSubset([
+            'bzrlib.tests.blackbox',
+            'bzrlib.tests.per_transport',
+            'bzrlib.tests.test_selftest',
+            ],
+            test_list)
+
+    def test__test_suite_modules_to_doctest(self):
+        # Test that a plausible list of modules to doctest is returned
+        # by _test_suite_modules_to_doctest.
+        test_list = tests._test_suite_modules_to_doctest()
+        self.assertSubset([
+            'bzrlib.timestamp',
+            ],
+            test_list)
+
     def test_test_suite(self):
-        # This test is slow - it loads the entire test suite to operate, so we
-        # do a single test with one test in each category
-        test_list = [
+        # test_suite() loads the entire test suite to operate. To avoid this
+        # overhead, and yet still be confident that things are happening,
+        # we temporarily replace two functions used by test_suite with 
+        # test doubles that supply a few sample tests to load, and check they
+        # are loaded.
+        calls = []
+        def _test_suite_testmod_names():
+            calls.append("testmod_names")
+            return [
+                'bzrlib.tests.blackbox.test_branch',
+                'bzrlib.tests.per_transport',
+                'bzrlib.tests.test_selftest',
+                ]
+        original_testmod_names = tests._test_suite_testmod_names
+        def _test_suite_modules_to_doctest():
+            calls.append("modules_to_doctest")
+            return ['bzrlib.timestamp']
+        orig_modules_to_doctest = tests._test_suite_modules_to_doctest
+        def restore_names():
+            tests._test_suite_testmod_names = original_testmod_names
+            tests._test_suite_modules_to_doctest = orig_modules_to_doctest
+        self.addCleanup(restore_names)
+        tests._test_suite_testmod_names = _test_suite_testmod_names
+        tests._test_suite_modules_to_doctest = _test_suite_modules_to_doctest
+        expected_test_list = [
             # testmod_names
             'bzrlib.tests.blackbox.test_branch.TestBranch.test_branch',
             ('bzrlib.tests.per_transport.TransportTests'
@@ -2695,13 +2804,16 @@
             # plugins can't be tested that way since selftest may be run with
             # --no-plugins
             ]
-        suite = tests.test_suite(test_list)
-        self.assertEquals(test_list, _test_ids(suite))
+        suite = tests.test_suite()
+        self.assertEqual(set(["testmod_names", "modules_to_doctest"]),
+            set(calls))
+        self.assertSubset(expected_test_list, _test_ids(suite))
 
     def test_test_suite_list_and_start(self):
         # We cannot test this at the same time as the main load, because we want
-        # to know that starting_with == None works. So a second full load is
-        # incurred.
+        # to know that starting_with == None works. So a second load is
+        # incurred - note that the starting_with parameter causes a partial load
+        # rather than a full load so this test should be pretty quick.
         test_list = ['bzrlib.tests.test_selftest.TestTestSuite.test_test_suite']
         suite = tests.test_suite(test_list,
                                  ['bzrlib.tests.test_selftest.TestTestSuite'])
@@ -2853,19 +2965,3 @@
                                                 self.verbosity)
         tests.run_suite(suite, runner_class=MyRunner, stream=StringIO())
         self.assertLength(1, calls)
-
-    def test_done(self):
-        """run_suite should call result.done()"""
-        self.calls = 0
-        def one_more_call(): self.calls += 1
-        def test_function():
-            pass
-        test = unittest.FunctionTestCase(test_function)
-        class InstrumentedTestResult(tests.ExtendedTestResult):
-            def done(self): one_more_call()
-        class MyRunner(tests.TextTestRunner):
-            def run(self, test):
-                return InstrumentedTestResult(self.stream, self.descriptions,
-                                              self.verbosity)
-        tests.run_suite(test, runner_class=MyRunner, stream=StringIO())
-        self.assertEquals(1, self.calls)

=== modified file 'bzrlib/tests/test_shelf.py'
--- bzrlib/tests/test_shelf.py	2009-08-26 07:40:38 +0000
+++ bzrlib/tests/test_shelf.py	2009-08-28 05:00:33 +0000
@@ -476,6 +476,8 @@
     def test_shelve_skips_added_root(self):
         """Skip adds of the root when iterating through shelvable changes."""
         tree = self.make_branch_and_tree('tree')
+        tree.lock_tree_write()
+        self.addCleanup(tree.unlock)
         creator = shelf.ShelfCreator(tree, tree.basis_tree())
         self.addCleanup(creator.finalize)
         self.assertEqual([], list(creator.iter_shelvable()))

=== modified file 'bzrlib/tests/test_smart.py'
--- bzrlib/tests/test_smart.py	2009-08-17 23:15:55 +0000
+++ bzrlib/tests/test_smart.py	2009-09-03 15:26:27 +0000
@@ -36,6 +36,7 @@
     smart,
     tests,
     urlutils,
+    versionedfile,
     )
 from bzrlib.branch import Branch, BranchReferenceFormat
 import bzrlib.smart.branch
@@ -87,8 +88,7 @@
         if self._chroot_server is None:
             backing_transport = tests.TestCaseWithTransport.get_transport(self)
             self._chroot_server = chroot.ChrootServer(backing_transport)
-            self._chroot_server.setUp()
-            self.addCleanup(self._chroot_server.tearDown)
+            self.start_server(self._chroot_server)
         t = get_transport(self._chroot_server.get_url())
         if relpath is not None:
             t = t.clone(relpath)
@@ -113,6 +113,25 @@
         return self.get_transport().get_smart_medium()
 
 
+class TestByteStreamToStream(tests.TestCase):
+
+    def test_repeated_substreams_same_kind_are_one_stream(self):
+        # Make a stream - an iterable of bytestrings.
+        stream = [('text', [versionedfile.FulltextContentFactory(('k1',), None,
+            None, 'foo')]),('text', [
+            versionedfile.FulltextContentFactory(('k2',), None, None, 'bar')])]
+        fmt = bzrdir.format_registry.get('pack-0.92')().repository_format
+        bytes = smart.repository._stream_to_byte_stream(stream, fmt)
+        streams = []
+        # Iterate the resulting iterable; checking that we get only one stream
+        # out.
+        fmt, stream = smart.repository._byte_stream_to_stream(bytes)
+        for kind, substream in stream:
+            streams.append((kind, list(substream)))
+        self.assertLength(1, streams)
+        self.assertLength(2, streams[0][1])
+
+
 class TestSmartServerResponse(tests.TestCase):
 
     def test__eq__(self):

=== modified file 'bzrlib/tests/test_transport.py'
--- bzrlib/tests/test_transport.py	2009-03-23 14:59:43 +0000
+++ bzrlib/tests/test_transport.py	2009-08-27 22:17:35 +0000
@@ -363,24 +363,22 @@
     def test_abspath(self):
         # The abspath is always relative to the chroot_url.
         server = ChrootServer(get_transport('memory:///foo/bar/'))
-        server.setUp()
+        self.start_server(server)
         transport = get_transport(server.get_url())
         self.assertEqual(server.get_url(), transport.abspath('/'))
 
         subdir_transport = transport.clone('subdir')
         self.assertEqual(server.get_url(), subdir_transport.abspath('/'))
-        server.tearDown()
 
     def test_clone(self):
         server = ChrootServer(get_transport('memory:///foo/bar/'))
-        server.setUp()
+        self.start_server(server)
         transport = get_transport(server.get_url())
         # relpath from root and root path are the same
         relpath_cloned = transport.clone('foo')
         abspath_cloned = transport.clone('/foo')
         self.assertEqual(server, relpath_cloned.server)
         self.assertEqual(server, abspath_cloned.server)
-        server.tearDown()
 
     def test_chroot_url_preserves_chroot(self):
         """Calling get_transport on a chroot transport's base should produce a
@@ -393,12 +391,11 @@
             new_transport = get_transport(parent_url)
         """
         server = ChrootServer(get_transport('memory:///path/subpath'))
-        server.setUp()
+        self.start_server(server)
         transport = get_transport(server.get_url())
         new_transport = get_transport(transport.base)
         self.assertEqual(transport.server, new_transport.server)
         self.assertEqual(transport.base, new_transport.base)
-        server.tearDown()
 
     def test_urljoin_preserves_chroot(self):
         """Using urlutils.join(url, '..') on a chroot URL should not produce a
@@ -410,11 +407,10 @@
             new_transport = get_transport(parent_url)
         """
         server = ChrootServer(get_transport('memory:///path/'))
-        server.setUp()
+        self.start_server(server)
         transport = get_transport(server.get_url())
         self.assertRaises(
             InvalidURLJoin, urlutils.join, transport.base, '..')
-        server.tearDown()
 
 
 class ChrootServerTest(TestCase):
@@ -428,7 +424,10 @@
         backing_transport = MemoryTransport()
         server = ChrootServer(backing_transport)
         server.setUp()
-        self.assertTrue(server.scheme in _get_protocol_handlers().keys())
+        try:
+            self.assertTrue(server.scheme in _get_protocol_handlers().keys())
+        finally:
+            server.tearDown()
 
     def test_tearDown(self):
         backing_transport = MemoryTransport()
@@ -441,8 +440,10 @@
         backing_transport = MemoryTransport()
         server = ChrootServer(backing_transport)
         server.setUp()
-        self.assertEqual('chroot-%d:///' % id(server), server.get_url())
-        server.tearDown()
+        try:
+            self.assertEqual('chroot-%d:///' % id(server), server.get_url())
+        finally:
+            server.tearDown()
 
 
 class ReadonlyDecoratorTransportTest(TestCase):
@@ -460,15 +461,12 @@
         import bzrlib.transport.readonly as readonly
         # connect to '.' via http which is not listable
         server = HttpServer()
-        server.setUp()
-        try:
-            transport = get_transport('readonly+' + server.get_url())
-            self.failUnless(isinstance(transport,
-                                       readonly.ReadonlyTransportDecorator))
-            self.assertEqual(False, transport.listable())
-            self.assertEqual(True, transport.is_readonly())
-        finally:
-            server.tearDown()
+        self.start_server(server)
+        transport = get_transport('readonly+' + server.get_url())
+        self.failUnless(isinstance(transport,
+                                   readonly.ReadonlyTransportDecorator))
+        self.assertEqual(False, transport.listable())
+        self.assertEqual(True, transport.is_readonly())
 
 
 class FakeNFSDecoratorTests(TestCaseInTempDir):
@@ -492,31 +490,24 @@
         from bzrlib.tests.http_server import HttpServer
         # connect to '.' via http which is not listable
         server = HttpServer()
-        server.setUp()
-        try:
-            transport = self.get_nfs_transport(server.get_url())
-            self.assertIsInstance(
-                transport, bzrlib.transport.fakenfs.FakeNFSTransportDecorator)
-            self.assertEqual(False, transport.listable())
-            self.assertEqual(True, transport.is_readonly())
-        finally:
-            server.tearDown()
+        self.start_server(server)
+        transport = self.get_nfs_transport(server.get_url())
+        self.assertIsInstance(
+            transport, bzrlib.transport.fakenfs.FakeNFSTransportDecorator)
+        self.assertEqual(False, transport.listable())
+        self.assertEqual(True, transport.is_readonly())
 
     def test_fakenfs_server_default(self):
         # a FakeNFSServer() should bring up a local relpath server for itself
         import bzrlib.transport.fakenfs as fakenfs
         server = fakenfs.FakeNFSServer()
-        server.setUp()
-        try:
-            # the url should be decorated appropriately
-            self.assertStartsWith(server.get_url(), 'fakenfs+')
-            # and we should be able to get a transport for it
-            transport = get_transport(server.get_url())
-            # which must be a FakeNFSTransportDecorator instance.
-            self.assertIsInstance(
-                transport, fakenfs.FakeNFSTransportDecorator)
-        finally:
-            server.tearDown()
+        self.start_server(server)
+        # the url should be decorated appropriately
+        self.assertStartsWith(server.get_url(), 'fakenfs+')
+        # and we should be able to get a transport for it
+        transport = get_transport(server.get_url())
+        # which must be a FakeNFSTransportDecorator instance.
+        self.assertIsInstance(transport, fakenfs.FakeNFSTransportDecorator)
 
     def test_fakenfs_rename_semantics(self):
         # a FakeNFS transport must mangle the way rename errors occur to
@@ -587,8 +578,7 @@
     def setUp(self):
         super(TestTransportImplementation, self).setUp()
         self._server = self.transport_server()
-        self._server.setUp()
-        self.addCleanup(self._server.tearDown)
+        self.start_server(self._server)
 
     def get_transport(self, relpath=None):
         """Return a connected transport to the local directory.

=== modified file 'doc/_templates/index.html'
--- doc/_templates/index.html	2009-07-22 14:36:38 +0000
+++ doc/_templates/index.html	2009-08-18 00:10:19 +0000
@@ -26,19 +26,17 @@
       <p class="biglink"><a class="biglink" href="{{ pathto("en/upgrade-guide/index") }}">Upgrade Guide</a><br/>
       <span class="linkdescr">moving to Bazaar 2.x</span>
       </p>
-      <p class="biglink"><a class="biglink" href="{{ pathto("en/migration/index") }}">Migration Docs</a><br/>
+      <p class="biglink"><a class="biglink" href="http://doc.bazaar-vcs.org/migration/en/">Migration Docs</a><br/>
       <span class="linkdescr">for refugees of other tools</span>
       </p>
-      <p class="biglink"><a class="biglink" href="{{ pathto("developers/index") }}">Developer Docs</a><br/>
-      <span class="linkdescr">polices and tools for giving back</span>
+      <p class="biglink"><a class="biglink" href="http://doc.bazaar-vcs.org/plugins/en/">Plugins Guide</a><br/>
+      <span class="linkdescr">help on popular plugins</span>
       </p>
     </td></tr>
   </table>
 
-  <p>Other languages:
-      <a href="{{ pathto("index.es") }}">Spanish</a>,
-      <a href="{{ pathto("index.ru") }}">Russian</a>
-  </p>
+  <p>Keen to help? See the <a href="{{ pathto("developers/index") }}">Developer Docs</a>
+      for policies and tools on contributing code, tests and documentation.</p>
 
 
   <h2>Related Links</h2>
@@ -59,4 +57,9 @@
     </td></tr>
   </table>
 
+  <hr>
+  <p>Other languages:
+      <a href="{{ pathto("index.es") }}">Spanish</a>,
+      <a href="{{ pathto("index.ru") }}">Russian</a>
+  </p>
 {% endblock %}

=== modified file 'doc/contents.txt'
--- doc/contents.txt	2009-07-22 13:41:01 +0000
+++ doc/contents.txt	2009-08-18 00:10:19 +0000
@@ -20,7 +20,6 @@
 
    en/release-notes/index
    en/upgrade-guide/index
-   en/migration/index
    developers/index
 
 

=== modified file 'doc/developers/bug-handling.txt'
--- doc/developers/bug-handling.txt	2009-08-24 00:29:31 +0000
+++ doc/developers/bug-handling.txt	2009-08-24 20:16:15 +0000
@@ -142,12 +142,8 @@
     it's not a good idea for a developer to spend time reproducing the bug
     until they're going to work on it.)
 Triaged
-    This is an odd state - one we consider a bug in launchpad, as it really
-    means "Importance has been set". We use this to mean the same thing
-    as confirmed, and set no preference on whether Confirmed or Triaged are
-    used. Please do not change a "Confirmed" bug to "Triaged" or vice verca -
-    any reports we create or use will always search for both "Confirmed" and
-    "Triaged" or neither "Confirmed" nor "Triaged".
+    We don't use this status.  If it is set, it means the same as
+    Confirmed.
 In Progress
     Someone has started working on this.
 Won't Fix

=== removed directory 'doc/en/migration'
=== removed file 'doc/en/migration/index.txt'
--- doc/en/migration/index.txt	2009-07-22 13:41:01 +0000
+++ doc/en/migration/index.txt	1970-01-01 00:00:00 +0000
@@ -1,6 +0,0 @@
-Bazaar Migration Guide
-======================
-
-This guide is under development. For notes collected so far, see
-http://bazaar-vcs.org/BzrMigration/.
-