Merge lp:~jameinel/bzr/2.1-static-tuple-btree-string-intern into lp:~jameinel/bzr/2.1-static-tuple-btree

[John A Meinel]

[John A Meinel (jameinel)]

This is a fairly minor update to my "btree uses StaticTuples", though it is mostly orthogonal.

This changes the code that decides to intern strings. Specifically it:

1) Doesn't intern key bits that start with 'sha1:'.
   a) There are *lots* of these (500k)
   b) They are rarely duplicated (no parent list, and only duplicated in the chk maps)
   c) They are never a sub part of a key. revision_id may show up as (revision_id,) or may show up
      as (file_id, revision_id), sha1: is never used like that [yet]

  This saves us 24 bytes per string in the string interned dict. At 500k sha1's that is 11MB.
  Note that at the moment, we still intern the StaticTuple('sha1:...'). I may want to revisit
  that, since there are no parent lists. However, for something like fetch, we will load the key
  from the index, and then load it again in the chk map. So I think interning is a net win.

2) Does intern some of the 'values' if they indicate the value is for a 'null content' record. We
   get a null content record for all of the entries for directories. And since our upgrade to
   rich-root decided to add an entry for TREE_ROOT for every revision, we have *lots* of these.
   (I think it was something like 1/4th of all entries in the per-file graph was these root keys.)

   The code comment mentions that when loading bzr.dev it saves 1.2MB, and for launchpad it saves
   around 4MB. Which was right about 1.7% of peak memory. Not huge, but a fairly cheap win.

This is a little bit of a layering inversion, because it is adding domain specific logic into btree index, when they should be considered generic containers. However, the memory savings is visible. It also isn't a correctness thing, as it only decides whether we intern() or not.

[Andrew Bennetts (spiv)]

review: Approve

[John A Meinel (jameinel)]

Bumping this back to 'work-in-progress' for now.

Running more of the test suite has shown some failures. Some really *random* failures on PQM, that I cannot reproduce locally, but at least I did get some local failures that I can work on fixing.

=== modified file 'Makefile'
--- Makefile	2009-10-01 02:46:17 +0000
+++ Makefile	2009-10-15 18:31:14 +0000
@@ -409,7 +409,7 @@
 	$(MAKE) clean && \
 	$(MAKE) && \
 	bzr export $$expdir && \
-	cp bzrlib/*.c $$expdir/bzrlib/. && \
+	cp bzrlib/*.c bzrlib/*.h $$expdir/bzrlib/. && \
 	tar cfz $$tarball -C $$expbasedir bzr-$$version && \
 	gpg --detach-sign $$tarball && \
 	rm -rf $$expbasedir
 
=== modified file 'NEWS'
--- NEWS	2009-10-06 15:58:12 +0000
+++ NEWS	2009-10-15 18:31:14 +0000
@@ -6,11 +6,113 @@
    :depth: 1
 
 
-2.1.0 series (not released yet)
-###############################
-
-Compatibility Breaks
-********************
+bzr 2.1.0b2 (not released yet)
+##############################
+
+:Codename:
+:2.1.0b2: ???
+
+
+Compatibility Breaks
+********************
+
+New Features
+************
+
+Bug Fixes
+*********
+
+Improvements
+************
+
+* When reading index files, we now use a ``StaticTuple`` rather than a
+  plain ``tuple`` object. This generally gives a 20% decrease in peak
+  memory, and can give a performance boost up to 40% on large projects.
+  (John Arbash Meinel)
+
+Documentation
+*************
+
+API Changes
+***********
+
+* ``UIFactory`` now has new ``show_error``, ``show_message`` and
+  ``show_warning`` methods, which can be hooked by non-text UIs.  
+  (Martin Pool)
+
+Internals
+*********
+
+* Added ``bzrlib._simple_set_pyx``. This is a hybrid between a Set and a
+  Dict (it only holds keys, but you can lookup the object located at a
+  given key). It has significantly reduced memory consumption versus the
+  builtin objects (1/2 the size of Set, 1/3rd the size of Dict). This is
+  used as the interning structure for StaticTuple objects.
+  (John Arbash Meinel)
+
+* ``bzrlib._static_tuple_pyx.StaticTuple`` is now available and used by
+  the btree index parser. This class functions similarly to ``tuple``
+  objects. However, it can only point at other ``StaticTuple`` instances
+  or strings. This allows us to remove it from the garbage collector (it
+  cannot be in a cycle), it also allows us to intern the objects. In
+  testing, this can reduce peak memory by 20-40%, and significantly
+  improve performance by removing objects from being inspected by the
+  garbage collector. (John Arbash Meinel)
+
+Testing
+*******
+
+
+bzr 2.0.2 (not released yet)
+############################
+
+:Codename:
+:2.0.2: ???
+
+Compatibility Breaks
+********************
+
+New Features
+************
+
+Bug Fixes
+*********
+
+Improvements
+************
+
+Documentation
+*************
+
+API Changes
+***********
+
+Internals
+*********
+
+Testing
+*******
+
+
+bzr 2.1.0b1
+###########
+
+:Codename: While the cat is away
+:2.1.0b1: 2009-10-14
+
+This is the first development release in the new split "stable" and
+"development" series. As such, the release is a snapshot of bzr.dev
+without creating a release candidate first. This release includes a
+fair amount of internal changes, with deprecated code being removed,
+and several new feature developments. People looking for a stable code
+base with only bugfixes should focus on the 2.0.1 release. All bugfixes
+present in 2.0.1 are present in 2.1.0b1.
+
+Highlights include support for ``bzr+ssh://host/~/homedir`` style urls,
+finer control over the plugin search path via extended BZR_PLUGIN_PATH
+syntax, visible warnings when extension modules fail to load, and improved
+error handling during unlocking.
+
 
 New Features
 ************
@@ -37,6 +139,10 @@
   automatically benefit from this feature when ``bzr`` on the server is
   upgraded.  (Andrew Bennetts, #109143)
 
+* Extensions can now be compiled if either Cython or Pyrex is available.
+  Currently Pyrex is preferred, but that may change in the future.
+  (Arkanes)
+
 * Give more control on BZR_PLUGIN_PATH by providing a way to refer to or
   disable the user, site and core plugin directories.
   (Vincent Ladeuil, #412930, #316192, #145612)
@@ -56,54 +162,24 @@
   filename will issue a warning and skip over those files.
   (Robert Collins, #3918)
 
-* ``bzr check`` in pack-0.92, 1.6 and 1.9 format repositories will no
-  longer report incorrect errors about ``Missing inventory ('TREE_ROOT', ...)``
-  (Robert Collins, #416732)
-
 * ``bzr dpush`` now aborts if uncommitted changes (including pending merges)
   are present in the working tree. The configuration option ``dpush_strict``
   can be used to set the default for this behavior.
   (Vincent Ladeuil, #438158)
 
-* ``bzr info -v`` on a 2a format still claimed that it was a "Development
-  format" (John Arbash Meinel, #424392)
-
 * ``bzr merge`` and ``bzr remove-tree`` now requires --force if pending
   merges are present in the working tree.
   (Vincent Ladeuil, #426344)
 
-* bzr will attempt to authenticate with SSH servers that support
-  ``keyboard-interactive`` auth but not ``password`` auth when using
-  Paramiko.   (Andrew Bennetts, #433846)
-
 * Clearer message when Bazaar runs out of memory, instead of a ``MemoryError``
   traceback.  (Martin Pool, #109115)
 
-* Conversion to 2a will create a single pack for all the new revisions (as
-  long as it ran without interruption). This improves both ``bzr upgrade``
-  and ``bzr pull`` or ``bzr merge`` from local branches in older formats.
-  The autopack logic that occurs every 100 revisions during local
-  conversions was not returning that pack's identifier, which resulted in
-  the partial packs created during the conversion not being consolidated
-  at the end of the conversion process. (Robert Collins, #423818)
-
 * Don't give a warning on Windows when failing to import ``_readdir_pyx``
   as it is never built. (John Arbash Meinel, #430645)
 
 * Don't restrict the command name used to run the test suite.
   (Vincent Ladeuil, #419950)
 
-* Fetches from 2a to 2a are now again requested in 'groupcompress' order.
-  Groups that are seen as 'underutilized' will be repacked on-the-fly.
-  This means that when the source is fully packed, there is minimal
-  overhead during the fetch, but if the source is poorly packed the result
-  is a fairly well packed repository (not as good as 'bzr pack' but
-  good-enough.) (Robert Collins, John Arbash Meinel, #402652)
-
-* Fixed fetches from a stacked branch on a smart server that were failing
-  with some combinations of remote and local formats.  This was causing
-  "unknown object type identifier 60" errors.  (Andrew Bennetts, #427736)
-
 * ftp transports were built differently when the kerberos python module was
   present leading to obscure failures related to ASCII/BINARY modes.
   (Vincent Ladeuil, #443041)
@@ -111,23 +187,6 @@
 * Network streams now decode adjacent records of the same type into a
   single stream, reducing layering churn. (Robert Collins)
 
-* Make sure that we unlock the tree if we fail to create a TreeTransform
-  object when doing a merge, and there is limbo, or pending-deletions
-  directory.  (Gary van der Merwe, #427773)
-
-* Occasional IndexError on renamed files have been fixed. Operations that
-  set a full inventory in the working tree will now go via the
-  apply_inventory_delta code path which is simpler and easier to
-  understand than dirstates set_state_from_inventory method. This may
-  have a small performance impact on operations built on _write_inventory,
-  but such operations are already doing full tree scans, so no radical
-  performance change should be observed. (Robert Collins, #403322)
-
-* Prevent some kinds of incomplete data from being committed to a 2a
-  repository, such as revisions without inventories or inventories without
-  chk_bytes root records.
-  (Andrew Bennetts, #423506)
-
 * PreviewTree behaves correctly when get_file_mtime is invoked on an unmodified
   file. (Aaron Bentley, #251532)
 
@@ -138,9 +197,6 @@
   domains or user ids embedding '.sig'. Now they can.
   (Matthew Fuller, Vincent Ladeuil, #430868)
 
-* When a file kind becomes unversionable after being added, a sensible
-  error will be shown instead of a traceback. (Robert Collins, #438569)
-
 Improvements
 ************
 
@@ -153,6 +209,12 @@
   See also <https://answers.launchpad.net/bzr/+faq/703>.
   (Martin Pool, #406113, #430529)
 
+* Secondary errors that occur during Branch.unlock and Repository.unlock
+  no longer obscure the original error.  These methods now use a new
+  decorator, ``only_raises``.  This fixes many causes of
+  ``TooManyConcurrentRequests`` and similar errors.
+  (Andrew Bennetts, #429747)
+
 Documentation
 *************
 
@@ -164,21 +226,37 @@
 API Changes
 ***********
 
-* ``ProgressTask.note`` is deprecated.
-  (Martin Pool)
-
 * ``bzrlib.user_encoding`` has been removed; use
   ``bzrlib.osutils.get_user_encoding`` instead.  (Martin Pool)
 
 * ``bzrlib.tests`` now uses ``stopTestRun`` for its ``TestResult``
   subclasses - the same as python's unittest module. (Robert Collins)
+  
+* ``diff._get_trees_to_diff`` has been renamed to 
+  ``diff.get_trees_and_branches_to_diff``. It is now a public API, and it 
+  returns the old and new branches. (Gary van der Merwe)
 
 * ``bzrlib.trace.log_error``, ``error`` and ``info`` have been deprecated.
   (Martin Pool)
 
+* ``MutableTree.has_changes()`` does not require a tree parameter anymore. It
+  now defaults to comparing to the basis tree. It now checks for pending
+  merges too.  ``Merger.check_basis`` has been deprecated and replaced by the
+  corresponding has_changes() calls. ``Merge.compare_basis``,
+  ``Merger.file_revisions`` and ``Merger.ensure_revision_trees`` have also
+  been deprecated.
+  (Vincent Ladeuil, #440631)
+
+* ``ProgressTask.note`` is deprecated.
+  (Martin Pool)
+
 Internals
 *********
 
+* Added ``-Drelock`` debug flag.  It will ``note`` a message every time a
+  repository or branch object is unlocked then relocked the same way.
+  (Andrew Bennetts)
+  
 * ``BTreeLeafParser.extract_key`` has been tweaked slightly to reduce
   mallocs while parsing the index (approx 3=>1 mallocs per key read).
   This results in a 10% speedup while reading an index.
@@ -225,8 +303,16 @@
   present. (Vincent Ladeuil, #430749)
 
 
-bzr 2.0.1 (Not Released Yet)
-############################
+bzr 2.0.1
+#########
+
+:Codename: Stability First
+:2.0.1: 2009-10-14
+
+The first of our new ongoing bugfix-only stable releases has arrived. It
+includes a collection of 12 bugfixes applied to bzr 2.0.0, but does not
+include any of the feature development in the 2.1.0 series.
+
 
 Bug Fixes
 *********
@@ -243,6 +329,18 @@
   with some combinations of remote and local formats.  This was causing
   "unknown object type identifier 60" errors.  (Andrew Bennetts, #427736)
 
+* Fixed ``ObjectNotLocked`` errors when doing some log and diff operations
+  on branches via a smart server.  (Andrew Bennetts, #389413)
+
+* Handle things like ``bzr add foo`` and ``bzr rm foo`` when the tree is
+  at the root of a drive. ``osutils._cicp_canonical_relpath`` always
+  assumed that ``abspath()`` returned a path that did not have a trailing
+  ``/``, but that is not true when working at the root of the filesystem.
+  (John Arbash Meinel, Jason Spashett, #322807)
+
+* Hide deprecation warnings for 'final' releases for python2.6.
+  (John Arbash Meinel, #440062)
+
 * Improve the time for ``bzr log DIR`` for 2a format repositories.
   We had been using the same code path as for <2a formats, which required
   iterating over all objects in all revisions.
@@ -260,12 +358,25 @@
   but such operations are already doing full tree scans, so no radical
   performance change should be observed. (Robert Collins, #403322)
 
-* When a file kind becomes unversionable after being added, a sensible
-  error will be shown instead of a traceback. (Robert Collins, #438569)
-
 * Retrieving file text or mtime from a _PreviewTree has good performance when
   there are many changes.  (Aaron Bentley)
 
+* The CHK index pages now use an unlimited cache size. With a limited
+  cache and a large project, the random access of chk pages could cause us
+  to download the entire cix file many times.
+  (John Arbash Meinel, #402623)
+
+* When a file kind becomes unversionable after being added, a sensible
+  error will be shown instead of a traceback. (Robert Collins, #438569)
+
+Documentation
+*************
+
+* Improved README. (Ian Clatworthy)
+
+* Improved upgrade documentation for Launchpad branches.
+  (Barry Warsaw)
+
 
 bzr 2.0.0
 #########
@@ -10886,5 +10997,38 @@
 * Storage of local versions: init, add, remove, rm, info, log,
   diff, status, etc.
 
+
+bzr ?.?.? (not released yet)
+############################
+
+:Codename: template
+:2.0.2: ???
+
+Compatibility Breaks
+********************
+
+New Features
+************
+
+Bug Fixes
+*********
+
+Improvements
+************
+
+Documentation
+*************
+
+API Changes
+***********
+
+Internals
+*********
+
+Testing
+*******
+
+
+
 ..
    vim: tw=74 ft=rst ff=unix
 
=== modified file 'README'
--- README	2008-03-16 14:01:20 +0000
+++ README	2009-10-15 18:31:14 +0000
@@ -3,72 +3,44 @@
 =================
 
 Bazaar (``bzr``) is a decentralized revision control system, designed to be
-easy for developers and end users alike.  Bazaar is part of the GNU project to
-develop a complete free operating system.
-
-To install Bazaar from source, follow the instructions in the INSTALL
-file. Otherwise, you may want to check your distribution package manager
-for ready-to-install packages, or http://bazaar-vcs.org/DistroDownloads.
-
-To learn how to use Bazaar, check the documentation in the doc/ directory.
-Once installed, you can also run 'bzr help'. An always up-to-date and more
-complete set of documents can be found in the Bazaar website, at:
-
-    http://bazaar-vcs.org/Documentation
+easy for developers and end users alike.  Bazaar is part of the GNU project
+to develop a complete free operating system.
+
+To install Bazaar, follow the instructions given at
+http://bazaar-vcs.org/Download. Ready-to-install packages are available
+for most popular operating systems or you can install from source.
+
+To learn how to use Bazaar, see the official documentation at:
+
+    http://doc.bazaar-vcs.org/en/
+
+For additional training materials including screencasts and slides,
+visit our community wiki documentation page at:
+
+    http://bazaar-vcs.org/Documentation/
 
 Bazaar is written in Python, and is sponsored by Canonical Limited, the
 founders of Ubuntu and Launchpad. Bazaar is Free Software, and is released
 under the GNU General Public License.
 
-Bazaar was formerly known as Bazaar-NG.  It's the successor to ``baz``, a fork
-of GNU arch, but shares no code.  (To upgrade from Baz, use the ``baz-import``
-command in the bzrtools plugin.)
-
 Bazaar highlights
 =================
 
-* Easy to use and intuitive.
-
-  Only five commands are needed to do all basic operations, and all
-  commands have documentation accessible via 'bzr help command'.
-  Bazaar's interface is also easy to learn for CVS and Subversion users.
-
-* Robust and reliable.
-
-  Bazaar is developed under an extensive test suite. Branches can be
-  checked and verified for integrity at any time, and revisions can be
-  signed with PGP/GnuPG.
-
-* Publish branches with HTTP.
-
-  Branches can be hosted on an HTTP server with no need for special
-  software on the server side. Branches can be uploaded by bzr itself
-  over SSH (SFTP), or with rsync.
-
-* Adapts to multiple environments.
-
-  Bazaar runs on Linux and Windows, fully supports Unicode filenames,
-  and suits different development models, including centralized.
-
-* Easily extended and customized.
-
-  A rich Python interface is provided for extending and embedding,
-  including a plugin interface. There are already many available plugins,
-  most of them registered at http://bazaar-vcs.org/PluginRegistry.
-
-* Smart merging.
-
-  Changes will never be merged more than once, conflicts will be
-  minimized, and identical changes are dealt with well.
-
-* Vibrant and active community.
-
-  Help with Bazaar is obtained easily, via the mailing list, or the IRC
-  channel.
-
-
-Registration and Feedback
-=========================
+Bazaar directly supports both central version control (like cvs/svn) and
+distributed version control (like git/hg). Developers can organize their
+workspace in whichever way they want on a per project basis including:
+
+* checkouts (like svn)
+* feature branches (like hg)
+* shared working tree (like git).
+
+It also directly supports and encourages a large number of development best
+practices like refactoring and pre-commit regression testing. Users can
+choose between our command line tool and our cross-platform GUI application.
+For further details, see our website at http://bazaar-vcs.org/en.
+
+Feedback
+========
 
 If you encounter any problems with Bazaar, need help understanding it, or would
 like to offer suggestions or feedback, please get in touch with us:
@@ -76,7 +48,7 @@
 * Ask a question through our web support interface, at 
   https://answers.launchpad.net/bzr/
 
-* Report bugs at https://bugs.edge.launchpad.net/bzr/+filebug
+* Report bugs at https://bugs.launchpad.net/bzr/+filebug
 
 * Write to us at [e-mail] 
   You can join the list at <https://lists.ubuntu.com/mailman/listinfo/bazaar>.
@@ -85,12 +57,8 @@
 
 * Talk to us in irc://irc.ubuntu.com/bzr
 
-* And see http://bazaar-vcs.org/BzrSupport for more.
-
-If you would like to help us improve Bazaar by telling us about yourself and
-what we could do better, please register and complete the online survey here:
-http://www.surveymonkey.com/s.aspx?sm=L94RvLswhKdktrxiHWiX3g_3d_3d.  
-Registration is completely optional. 
-
-Enjoy,
+Our mission is to make a version control tool that developers LOVE to use
+and that casual contributors feel confident with. Please let us know how
+we're going.
+
 The Bazaar Team
 
=== modified file 'bzrlib/__init__.py'
--- bzrlib/__init__.py	2009-09-25 21:24:21 +0000
+++ bzrlib/__init__.py	2009-10-15 18:31:14 +0000
@@ -44,7 +44,7 @@
 # Python version 2.0 is (2, 0, 0, 'final', 0)."  Additionally we use a
 # releaselevel of 'dev' for unreleased under-development code.
 
-version_info = (2, 1, 0, 'dev', 0)
+version_info = (2, 1, 0, 'dev', 2)
 
 # API compatibility version: bzrlib is currently API compatible with 1.15.
 api_minimum_version = (2, 1, 0)
 
=== modified file 'bzrlib/_bencode_pyx.pyx'
--- bzrlib/_bencode_pyx.pyx	2009-06-05 01:48:32 +0000
+++ bzrlib/_bencode_pyx.pyx	2009-10-15 18:31:14 +0000
@@ -58,6 +58,13 @@
     void D_UPDATE_TAIL(Decoder, int n)
     void E_UPDATE_TAIL(Encoder, int n)
 
+# To maintain compatibility with older versions of pyrex, we have to use the
+# relative import here, rather than 'bzrlib._static_tuple_c'
+from _static_tuple_c cimport StaticTuple, StaticTuple_CheckExact, \
+    import_static_tuple_c
+
+import_static_tuple_c()
+
 
 cdef class Decoder:
     """Bencode decoder"""
@@ -371,7 +378,8 @@
                 self._encode_int(x)
             elif PyLong_CheckExact(x):
                 self._encode_long(x)
-            elif PyList_CheckExact(x) or PyTuple_CheckExact(x):
+            elif (PyList_CheckExact(x) or PyTuple_CheckExact(x)
+                  or StaticTuple_CheckExact(x)):
                 self._encode_list(x)
             elif PyDict_CheckExact(x):
                 self._encode_dict(x)
 
=== modified file 'bzrlib/_btree_serializer_pyx.pyx'
--- bzrlib/_btree_serializer_pyx.pyx	2009-10-08 15:44:41 +0000
+++ bzrlib/_btree_serializer_pyx.pyx	2009-10-15 18:31:14 +0000
@@ -173,7 +173,13 @@
                                                    last - self._start)))
                     raise AssertionError(failure_string)
             # capture the key string
-            key_element = safe_interned_string_from_size(self._start,
+            if (self.key_length == 1
+                and (temp_ptr - self._start) == 45
+                and strncmp(self._start, 'sha1:', 5) == 0):
+                key_element = safe_string_from_size(self._start,
+                                                    temp_ptr - self._start)
+            else:
+                key_element = safe_interned_string_from_size(self._start,
                                                          temp_ptr - self._start)
             # advance our pointer
             self._start = temp_ptr + 1
@@ -189,6 +195,7 @@
         cdef char *ref_ptr
         cdef char *next_start
         cdef int loop_counter
+        cdef Py_ssize_t str_len
 
         self._start = self._cur_str
         # Find the next newline
@@ -224,8 +231,20 @@
             # Invalid line
             raise AssertionError("Failed to find the value area")
         else:
-            # capture the value string
-            value = safe_string_from_size(temp_ptr + 1, last - temp_ptr - 1)
+            # Because of how conversions were done, we ended up with *lots* of
+            # values that are identical. These are all of the 0-length nodes
+            # that are referred to by the TREE_ROOT (and likely some other
+            # directory nodes.) For example, bzr has 25k references to
+            # something like '12607215 328306 0 0', which ends up consuming 1MB
+            # of memory, just for those strings.
+            str_len = last - temp_ptr - 1
+            if (str_len > 4
+                and strncmp(" 0 0", last - 4, 4) == 0):
+                # This drops peak mem for bzr.dev from 87.4MB => 86.2MB
+                # For Launchpad 236MB => 232MB
+                value = safe_interned_string_from_size(temp_ptr + 1, str_len)
+            else:
+                value = safe_string_from_size(temp_ptr + 1, str_len)
             # shrink the references end point
             last = temp_ptr
 
@@ -332,8 +351,17 @@
     elif node_len < 3:
         raise ValueError('Without ref_lists, we need at least 3 entries not: %s'
             % len(node))
-    # I don't expect that we can do faster than string.join()
-    string_key = '\0'.join(node[1])# <object>PyTuple_GET_ITEM_ptr_object(node, 1))
+    # TODO: We can probably do better than string.join(), namely
+    #       when key has only 1 item, we can just grab that string
+    #       And when there are 2 items, we could do a single malloc + len() + 1
+    #       also, doing .join() requires a PyObject_GetAttrString call, which
+    #       we could also avoid.
+    # TODO: Note that pyrex 0.9.6 generates fairly crummy code here, using the
+    #       python object interface, versus 0.9.8+ which uses a helper that
+    #       checks if this supports the sequence interface.
+    #       We *could* do more work on our own, and grab the actual items
+    #       lists. For now, just ask people to use a better compiler. :)
+    string_key = '\0'.join(node[1])
 
     # TODO: instead of using string joins, precompute the final string length,
     #       and then malloc a single string and copy everything in.
@@ -350,7 +378,7 @@
     refs_len = 0
     if have_reference_lists:
         # Figure out how many bytes it will take to store the references
-        ref_lists = node[3]# <object>PyTuple_GET_ITEM_ptr_object(node, 3)
+        ref_lists = node[3]
         next_len = len(ref_lists) # TODO: use a Py function
         if next_len > 0:
             # If there are no nodes, we don't need to do any work
@@ -374,8 +402,7 @@
                             # We will need (len - 1) '\x00' characters to
                             # separate the reference key
                             refs_len = refs_len + (next_len - 1)
-                            for i from 0 <= i < next_len:
-                                ref_bit = reference[i]
+                            for ref_bit in reference:
                                 if not PyString_CheckExact(ref_bit):
                                     raise TypeError('We expect reference bits'
                                         ' to be strings not: %s'
@@ -384,7 +411,7 @@
 
     # So we have the (key NULL refs NULL value LF)
     key_len = PyString_Size(string_key)
-    val = node[2] # PyTuple_GET_ITEM_ptr_object(node, 2)
+    val = node[2]
     if not PyString_CheckExact(val):
         raise TypeError('Expected a plain str for value not: %s'
                         % type(val))
@@ -416,7 +443,7 @@
                     if i != 0:
                         out[0] = c'\x00'
                         out = out + 1
-                    ref_bit = reference[i] #PyTuple_GET_ITEM_ptr_object(reference, i)
+                    ref_bit = reference[i]
                     ref_bit_len = PyString_GET_SIZE(ref_bit)
                     memcpy(out, PyString_AS_STRING(ref_bit), ref_bit_len)
                     out = out + ref_bit_len
 
=== modified file 'bzrlib/_export_c_api.h'
--- bzrlib/_export_c_api.h	2009-10-02 02:21:12 +0000
+++ bzrlib/_export_c_api.h	2009-10-15 18:31:14 +0000
@@ -45,14 +45,17 @@
     PyObject *d = NULL;
     PyObject *c_obj = NULL;
 
-    d = PyObject_GetAttrString(module, _C_API_NAME);
+    /* (char *) is because python2.4 declares this api as 'char *' rather than
+     * const char* which it really is.
+     */
+    d = PyObject_GetAttrString(module, (char *)_C_API_NAME);
     if (!d) {
         PyErr_Clear();
         d = PyDict_New();
         if (!d)
             goto bad;
         Py_INCREF(d);
-        if (PyModule_AddObject(module, _C_API_NAME, d) < 0)
+        if (PyModule_AddObject(module, (char *)_C_API_NAME, d) < 0)
             goto bad;
     }
     c_obj = PyCObject_FromVoidPtrAndDesc(func, signature, 0);
 
=== modified file 'bzrlib/_import_c_api.h'
--- bzrlib/_import_c_api.h	2009-10-01 21:34:36 +0000
+++ bzrlib/_import_c_api.h	2009-10-15 18:31:14 +0000
@@ -47,7 +47,10 @@
     PyObject *c_obj = NULL;
     const char *desc = NULL;
 
-    d = PyObject_GetAttrString(module, _C_API_NAME);
+    /* (char *) because Python2.4 defines this as (char *) rather than
+     * (const char *)
+     */
+    d = PyObject_GetAttrString(module, (char *)_C_API_NAME);
     if (!d) {
         // PyObject_GetAttrString sets an appropriate exception
         goto bad;
@@ -94,7 +97,7 @@
 {
     PyObject *type = NULL;
 
-    type = PyObject_GetAttrString(module, class_name);
+    type = PyObject_GetAttrString(module, (char *)class_name);
     if (!type) {
         goto bad;
     }
@@ -149,7 +152,7 @@
     struct type_description *cur_type;
     int ret_code;
     
-    module = PyImport_ImportModule(module_name);
+    module = PyImport_ImportModule((char *)module_name);
     if (!module)
         goto bad;
     if (functions != NULL) {
 
=== modified file 'bzrlib/_simple_set_pyx.pxd'
--- bzrlib/_simple_set_pyx.pxd	2009-10-08 04:40:16 +0000
+++ bzrlib/_simple_set_pyx.pxd	2009-10-15 18:31:13 +0000
@@ -40,11 +40,36 @@
     (like a dict), but we also don't implement the complete list of 'set'
     operations (difference, intersection, etc).
     """
+    # Data structure definition:
+    #   This is a basic hash table using open addressing.
+    #       http://en.wikipedia.org/wiki/Open_addressing
+    #   Basically that means we keep an array of pointers to Python objects
+    #   (called a table). Each location in the array is called a 'slot'.
+    #
+    #   An empty slot holds a NULL pointer, a slot where there was an item
+    #   which was then deleted will hold a pointer to _dummy, and a filled slot
+    #   points at the actual object which fills that slot.
+    #
+    #   The table is always a power of two, and the default location where an
+    #   object is inserted is at hash(object) & (table_size - 1)
+    #
+    #   If there is a collision, then we search for another location. The
+    #   specific algorithm is in _lookup. We search until we:
+    #       find the object
+    #       find an equivalent object (by tp_richcompare(obj1, obj2, Py_EQ))
+    #       find a NULL slot
+    #
+    #   When an object is deleted, we set its slot to _dummy. this way we don't
+    #   have to track whether there was a collision, and find the corresponding
+    #   keys. (The collision resolution algorithm makes that nearly impossible
+    #   anyway, because it depends on the upper bits of the hash.)
+    #   The main effect of this, is that if we find _dummy, then we can insert
+    #   an object there, but we have to keep searching until we find NULL to
+    #   know that the object is not present elsewhere.
 
     cdef Py_ssize_t _used   # active
     cdef Py_ssize_t _fill   # active + dummy
-    cdef Py_ssize_t _mask   # Table contains (mask+1) slots, a power
-                                     # of 2
+    cdef Py_ssize_t _mask   # Table contains (mask+1) slots, a power of 2
     cdef PyObject **_table  # Pyrex/Cython doesn't support arrays to 'object'
                             # so we manage it manually
 
@@ -57,7 +82,6 @@
 
 # TODO: might want to export the C api here, though it is all available from
 #       the class object...
-cdef api object SimpleSet_Add(object self, object key)
 cdef api SimpleSet SimpleSet_New()
 cdef api object SimpleSet_Add(object self, object key)
 cdef api int SimpleSet_Contains(object self, object key) except -1
 
=== modified file 'bzrlib/_simple_set_pyx.pyx'
--- bzrlib/_simple_set_pyx.pyx	2009-10-08 04:40:16 +0000
+++ bzrlib/_simple_set_pyx.pyx	2009-10-15 18:31:13 +0000
@@ -16,15 +16,16 @@
 
 """Definition of a class that is similar to Set with some small changes."""
 
+cdef extern from "python-compat.h":
+    pass
+
 cdef extern from "Python.h":
     ctypedef unsigned long size_t
-    ctypedef long (*hashfunc)(PyObject*)
-    ctypedef PyObject *(*richcmpfunc)(PyObject *, PyObject *, int)
+    ctypedef long (*hashfunc)(PyObject*) except -1
+    ctypedef object (*richcmpfunc)(PyObject *, PyObject *, int)
     ctypedef int (*visitproc)(PyObject *, void *)
     ctypedef int (*traverseproc)(PyObject *, visitproc, void *)
     int Py_EQ
-    PyObject *Py_True
-    PyObject *Py_NotImplemented
     void Py_INCREF(PyObject *)
     void Py_DECREF(PyObject *)
     ctypedef struct PyTypeObject:
@@ -33,38 +34,54 @@
         traverseproc tp_traverse
 
     PyTypeObject *Py_TYPE(PyObject *)
+    # Note: *Don't* use hash(), Pyrex 0.9.8.5 thinks it returns an 'int', and
+    #       thus silently truncates to 32-bits on 64-bit machines.
+    long PyObject_Hash(PyObject *) except -1
         
     void *PyMem_Malloc(size_t nbytes)
     void PyMem_Free(void *)
     void memset(void *, int, size_t)
 
 
+# Dummy is an object used to mark nodes that have been deleted. Since
+# collisions require us to move a node to an alternative location, if we just
+# set an entry to NULL on delete, we won't find any relocated nodes.
+# We have to use _dummy_obj because we need to keep a refcount to it, but we
+# also use _dummy as a pointer, because it avoids having to put <PyObject*> all
+# over the code base.
 cdef object _dummy_obj
 cdef PyObject *_dummy
 _dummy_obj = object()
 _dummy = <PyObject *>_dummy_obj
 
 
-cdef int _is_equal(PyObject *this, long this_hash, PyObject *other):
+cdef object _NotImplemented
+_NotImplemented = NotImplemented
+
+
+cdef int _is_equal(PyObject *this, long this_hash, PyObject *other) except -1:
     cdef long other_hash
-    cdef PyObject *res
 
     if this == other:
         return 1
-    other_hash = Py_TYPE(other).tp_hash(other)
+    other_hash = PyObject_Hash(other)
     if other_hash != this_hash:
         return 0
+
+    # This implements a subset of the PyObject_RichCompareBool functionality.
+    # Namely it:
+    #   1) Doesn't try to do anything with old-style classes
+    #   2) Assumes that both objects have a tp_richcompare implementation, and
+    #      that if that is not enough to compare equal, then they are not
+    #      equal. (It doesn't try to cast them both to some intermediate form
+    #      that would compare equal.)
     res = Py_TYPE(this).tp_richcompare(this, other, Py_EQ)
-    if res == Py_True:
-        Py_DECREF(res)
-        return 1
-    if res == Py_NotImplemented:
-        Py_DECREF(res)
+    if res is _NotImplemented:
         res = Py_TYPE(other).tp_richcompare(other, this, Py_EQ)
-    if res == Py_True:
-        Py_DECREF(res)
+        if res is _NotImplemented:
+            return 0
+    if res:
         return 1
-    Py_DECREF(res)
     return 0
 
 
@@ -84,7 +101,6 @@
     """
     # Attributes are defined in the .pxd file
     DEF DEFAULT_SIZE=1024
-    DEF PERTURB_SHIFT=5
 
     def __init__(self):
         cdef Py_ssize_t size, n_bytes
@@ -170,27 +186,25 @@
         as it makes a lot of assuptions about keys not already being present,
         and there being no dummy entries.
         """
-        cdef size_t i, perturb, mask
+        cdef size_t i, n_lookup
         cdef long the_hash
-        cdef PyObject **table, **entry
+        cdef PyObject **table, **slot
+        cdef Py_ssize_t mask
 
         mask = self._mask
         table = self._table
 
-        the_hash = Py_TYPE(key).tp_hash(key)
-        i = the_hash & mask
-        entry = &table[i]
-        perturb = the_hash
-        # Because we know that we made all items unique before, we can just
-        # iterate as long as the target location is not empty, we don't have to
-        # do any comparison, etc.
-        while entry[0] != NULL:
-            i = (i << 2) + i + perturb + 1
-            entry = &table[i & mask]
-            perturb >>= PERTURB_SHIFT
-        entry[0] = key
-        self._fill += 1
-        self._used += 1
+        the_hash = PyObject_Hash(key)
+        i = the_hash
+        for n_lookup from 0 <= n_lookup <= <size_t>mask: # Don't loop forever
+            slot = &table[i & mask]
+            if slot[0] == NULL:
+                slot[0] = key
+                self._fill = self._fill + 1
+                self._used = self._used + 1
+                return 1
+            i = i + 1 + n_lookup
+        raise RuntimeError('ran out of slots.')
 
     def _py_resize(self, min_used):
         """Do not use this directly, it is only exposed for testing."""
@@ -206,7 +220,7 @@
         :return: The new size of the internal table
         """
         cdef Py_ssize_t new_size, n_bytes, remaining
-        cdef PyObject **new_table, **old_table, **entry
+        cdef PyObject **new_table, **old_table, **slot
 
         new_size = DEFAULT_SIZE
         while new_size <= min_used and new_size > 0:
@@ -236,16 +250,16 @@
 
         # Moving everything to the other table is refcount neutral, so we don't
         # worry about it.
-        entry = old_table
+        slot = old_table
         while remaining > 0:
-            if entry[0] == NULL: # unused slot
+            if slot[0] == NULL: # unused slot
                 pass 
-            elif entry[0] == _dummy: # dummy slot
-                remaining -= 1
+            elif slot[0] == _dummy: # dummy slot
+                remaining = remaining - 1
             else: # active slot
-                remaining -= 1
-                self._insert_clean(entry[0])
-            entry += 1
+                remaining = remaining - 1
+                self._insert_clean(slot[0])
+            slot = slot + 1
         PyMem_Free(old_table)
         return new_size
 
@@ -262,20 +276,24 @@
         cdef PyObject **slot, *py_key
         cdef int added
 
+        py_key = <PyObject *>key
+        if (Py_TYPE(py_key).tp_richcompare == NULL
+            or Py_TYPE(py_key).tp_hash == NULL):
+            raise TypeError('Types added to SimpleSet must implement'
+                            ' both tp_richcompare and tp_hash')
         added = 0
         # We need at least one empty slot
         assert self._used < self._mask
         slot = _lookup(self, key)
-        py_key = <PyObject *>key
         if (slot[0] == NULL):
             Py_INCREF(py_key)
-            self._fill += 1
-            self._used += 1
+            self._fill = self._fill + 1
+            self._used = self._used + 1
             slot[0] = py_key
             added = 1
         elif (slot[0] == _dummy):
             Py_INCREF(py_key)
-            self._used += 1
+            self._used = self._used + 1
             slot[0] = py_key
             added = 1
         # No else: clause. If _lookup returns a pointer to
@@ -291,11 +309,13 @@
         return retval
 
     def discard(self, key):
-        """Remove key from the dict, whether it exists or not.
+        """Remove key from the set, whether it exists or not.
 
-        :return: 0 if the item did not exist, 1 if it did
+        :return: False if the item did not exist, True if it did
         """
-        return self._discard(key)
+        if self._discard(key):
+            return True
+        return False
 
     cdef int _discard(self, key) except -1:
         cdef PyObject **slot, *py_key
@@ -303,7 +323,7 @@
         slot = _lookup(self, key)
         if slot[0] == NULL or slot[0] == _dummy:
             return 0
-        self._used -= 1
+        self._used = self._used - 1
         Py_DECREF(slot[0])
         slot[0] = _dummy
         # PySet uses the heuristic: If more than 1/5 are dummies, then resize
@@ -320,16 +340,6 @@
             self._resize(self._used * 2)
         return 1
 
-    def __delitem__(self, key):
-        """Remove the given item from the dict.
-
-        Raise a KeyError if the key was not present.
-        """
-        cdef int exists
-        exists = self._discard(key)
-        if not exists:
-            raise KeyError('Key %s not present' % (key,))
-
     def __iter__(self):
         return _SimpleSet_iterator(self)
 
@@ -353,7 +363,7 @@
 
     def __next__(self):
         cdef Py_ssize_t mask, i
-        cdef PyObject **table
+        cdef PyObject *key
 
         if self.set is None:
             raise StopIteration
@@ -361,21 +371,13 @@
             # Force this exception to continue to be raised
             self._used = -1
             raise RuntimeError("Set size changed during iteration")
-        i = self.pos
-        mask = self.set._mask
-        table = self.set._table
-        assert i >= 0
-        while i <= mask and (table[i] == NULL or table[i] == _dummy):
-            i += 1
-        self.pos = i + 1
-        if i > mask:
-            # we walked to the end
+        if not SimpleSet_Next(self.set, &self.pos, &key):
             self.set = None
             raise StopIteration
-        # We must have found one
-        key = <object>(table[i])
-        self.len -= 1
-        return key
+        # we found something
+        the_key = <object>key # INCREF
+        self.len = self.len - 1
+        return the_key
 
     def __length_hint__(self):
         if self.set is not None and self._used == self.set._used:
@@ -411,51 +413,68 @@
 
     :param key: An object we are looking up
     :param hash: The hash for key
-    :return: The location in self.table where key should be put
-        should never be NULL, but may reference a NULL (PyObject*)
+    :return: The location in self.table where key should be put.
+        location == NULL is an exception, but (*location) == NULL just
+        indicates the slot is empty and can be used.
     """
-    # This is the heart of most functions, which is why it is pulled out as an
-    # cdef inline function.
-    cdef size_t i, perturb
+    # This uses Quadratic Probing:
+    #  http://en.wikipedia.org/wiki/Quadratic_probing
+    # with c1 = c2 = 1/2
+    # This leads to probe locations at:
+    #  h0 = hash(k1)
+    #  h1 = h0 + 1
+    #  h2 = h0 + 3 = h1 + 1 + 1
+    #  h3 = h0 + 6 = h2 + 1 + 2
+    #  h4 = h0 + 10 = h2 + 1 + 3
+    # Note that all of these are '& mask', but that is computed *after* the
+    # offset.
+    # This differs from the algorithm used by Set and Dict. Which, effectively,
+    # use double-hashing, and a step size that starts large, but dwindles to
+    # stepping one-by-one.
+    # This gives more 'locality' in that if you have a collision at offset X,
+    # the first fallback is X+1, which is fast to check. However, that means
+    # that an object w/ hash X+1 will also check there, and then X+2 next.
+    # However, for objects with differing hashes, their chains are different.
+    # The former checks X, X+1, X+3, ... the latter checks X+1, X+2, X+4, ...
+    # So different hashes diverge quickly.
+    # A bigger problem is that we *only* ever use the lowest bits of the hash
+    # So all integers (x + SIZE*N) will resolve into the same bucket, and all
+    # use the same collision resolution. We may want to try to find a way to
+    # incorporate the upper bits of the hash with quadratic probing. (For
+    # example, X, X+1, X+3+some_upper_bits, X+6+more_upper_bits, etc.)
+    cdef size_t i, n_lookup
     cdef Py_ssize_t mask
     cdef long key_hash
-    cdef long this_hash
-    cdef PyObject **table, **cur, **free_slot, *py_key
+    cdef PyObject **table, **slot, *cur, **free_slot, *py_key
 
-    key_hash = hash(key)
+    py_key = <PyObject *>key
+    # Note: avoid using hash(obj) because of a bug w/ pyrex 0.9.8.5 and 64-bit
+    #       (it treats hash() as returning an 'int' rather than a 'long')
+    key_hash = PyObject_Hash(py_key)
+    i = <size_t>key_hash
     mask = self._mask
     table = self._table
-    i = key_hash & mask
-    cur = &table[i]
-    py_key = <PyObject *>key
-    if cur[0] == NULL:
-        # Found a blank spot, or found the exact key
-        return cur
-    if cur[0] == py_key:
-        return cur
-    if cur[0] == _dummy:
-        free_slot = cur
-    else:
-        if _is_equal(py_key, key_hash, cur[0]):
-            # Both py_key and cur[0] belong in this slot, return it
-            return cur
-        free_slot = NULL
-    # size_t is unsigned, hash is signed...
-    perturb = key_hash
-    while True:
-        i = (i << 2) + i + perturb + 1
-        cur = &table[i & mask]
-        if cur[0] == NULL: # Found an empty spot
-            if free_slot: # Did we find a _dummy earlier?
+    free_slot = NULL
+    for n_lookup from 0 <= n_lookup <= <size_t>mask: # Don't loop forever
+        slot = &table[i & mask]
+        cur = slot[0]
+        if cur == NULL:
+            # Found a blank spot
+            if free_slot != NULL:
+                # Did we find an earlier _dummy entry?
                 return free_slot
             else:
-                return cur
-        if (cur[0] == py_key # exact match
-            or _is_equal(py_key, key_hash, cur[0])): # Equivalent match
-            return cur
-        if (cur[0] == _dummy and free_slot == NULL):
-            free_slot = cur
-        perturb >>= PERTURB_SHIFT
+                return slot
+        if cur == py_key:
+            # Found an exact pointer to the key
+            return slot
+        if cur == _dummy:
+            if free_slot == NULL:
+                free_slot = slot
+        elif _is_equal(py_key, key_hash, cur):
+            # Both py_key and cur belong in this slot, return it
+            return slot
+        i = i + 1 + n_lookup
     raise AssertionError('should never get here')
 
 
@@ -521,7 +540,6 @@
     return _check_self(self)._used
 
 
-# TODO: this should probably have direct tests, since it isn't used by __iter__
 cdef api int SimpleSet_Next(object self, Py_ssize_t *pos, PyObject **key):
     """Walk over items in a SimpleSet.
 
@@ -540,7 +558,7 @@
     mask = true_self._mask
     table= true_self._table
     while (i <= mask and (table[i] == NULL or table[i] == _dummy)):
-        i += 1
+        i = i + 1
     pos[0] = i + 1
     if (i > mask):
         return 0 # All done
@@ -565,8 +583,7 @@
         ret = visit(next_key, arg)
         if ret:
             return ret
-
-    return 0;
+    return 0
 
 # It is a little bit ugly to do this, but it works, and means that Meliae can
 # dump the total memory consumed by all child objects.
 
=== modified file 'bzrlib/_static_tuple_c.c'
--- bzrlib/_static_tuple_c.c	2009-10-07 19:31:39 +0000
+++ bzrlib/_static_tuple_c.c	2009-10-15 18:31:14 +0000
@@ -20,12 +20,20 @@
  */
 #define STATIC_TUPLE_MODULE
 
+#include <Python.h>
+#include "python-compat.h"
+
 #include "_static_tuple_c.h"
 #include "_export_c_api.h"
+
+/* Pyrex 0.9.6.4 exports _simple_set_pyx_api as
+ * import__simple_set_pyx(), while Pyrex 0.9.8.5 and Cython 0.11.3 export them
+ * as import_bzrlib___simple_set_pyx(). As such, we just #define one to be
+ * equivalent to the other in our internal code.
+ */
+#define import__simple_set_pyx import_bzrlib___simple_set_pyx
 #include "_simple_set_pyx_api.h"
 
-#include "python-compat.h"
-
 #if defined(__GNUC__)
 #   define inline __inline__
 #elif defined(_MSC_VER)
@@ -74,7 +82,7 @@
 static StaticTuple *
 StaticTuple_Intern(StaticTuple *self)
 {
-    PyObject *unique_key = NULL;
+    PyObject *canonical_tuple = NULL;
 
     if (_interned_tuples == NULL || _StaticTuple_is_interned(self)) {
         Py_INCREF(self);
@@ -83,20 +91,18 @@
     /* SimpleSet_Add returns whatever object is present at self
      * or the new object if it needs to add it.
      */
-    unique_key = SimpleSet_Add(_interned_tuples, (PyObject *)self);
-    if (!unique_key) {
-        // Suppress any error and just return the object
-        PyErr_Clear();
-        Py_INCREF(self);
-        return self;
+    canonical_tuple = SimpleSet_Add(_interned_tuples, (PyObject *)self);
+    if (!canonical_tuple) {
+        // Some sort of exception, propogate it.
+        return NULL;
     }
-    if (unique_key != (PyObject *)self) {
-        // There was already a key at that location
-        return (StaticTuple *)unique_key;
+    if (canonical_tuple != (PyObject *)self) {
+        // There was already a tuple with that value
+        return (StaticTuple *)canonical_tuple;
     }
     self->flags |= STATIC_TUPLE_INTERNED_FLAG;
-    // The two references in the dict do not count, so that the StaticTuple object
-    // does not become immortal just because it was interned.
+    // The two references in the dict do not count, so that the StaticTuple
+    // object does not become immortal just because it was interned.
     Py_REFCNT(self) -= 1;
     return self;
 }
@@ -169,7 +175,7 @@
 
 
 static PyObject *
-StaticTuple_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
+StaticTuple_new_constructor(PyTypeObject *type, PyObject *args, PyObject *kwds)
 {
     StaticTuple *self;
     PyObject *obj = NULL;
@@ -187,7 +193,7 @@
     if (len < 0 || len > 255) {
         /* Too big or too small */
         PyErr_SetString(PyExc_ValueError, "StaticTuple.__init__(...)"
-            " takes from 0 to 255 key bits");
+            " takes from 0 to 255 items");
         return NULL;
     }
     self = (StaticTuple *)StaticTuple_New(len);
@@ -199,8 +205,7 @@
         if (!PyString_CheckExact(obj)) {
             if (!StaticTuple_CheckExact(obj)) {
                 PyErr_SetString(PyExc_TypeError, "StaticTuple.__init__(...)"
-                    " requires that all key bits are strings or StaticTuple.");
-                /* TODO: What is the proper way to dealloc ? */
+                    " requires that all items are strings or StaticTuple.");
                 type->tp_dealloc((PyObject *)self);
                 return NULL;
             }
@@ -236,21 +241,21 @@
     /* adapted from tuplehash(), is the specific hash value considered
      * 'stable'?
      */
-	register long x, y;
-	Py_ssize_t len = self->size;
-	PyObject **p;
-	long mult = 1000003L;
+    register long x, y;
+    Py_ssize_t len = self->size;
+    PyObject **p;
+    long mult = 1000003L;
 
 #if STATIC_TUPLE_HAS_HASH
     if (self->hash != -1) {
         return self->hash;
     }
 #endif
-	x = 0x345678L;
-	p = self->items;
+    x = 0x345678L;
+    p = self->items;
     // TODO: We could set specific flags if we know that, for example, all the
-    //       keys are strings. I haven't seen a real-world benefit to that yet,
-    //       though.
+    //       items are strings. I haven't seen a real-world benefit to that
+    //       yet, though.
     while (--len >= 0) {
         y = PyObject_Hash(*p++);
         if (y == -1) /* failure */
@@ -259,18 +264,13 @@
         /* the cast might truncate len; that doesn't change hash stability */
         mult += (long)(82520L + len + len);
     }
-	x += 97531L;
-	if (x == -1)
-		x = -2;
+    x += 97531L;
+    if (x == -1)
+        x = -2;
 #if STATIC_TUPLE_HAS_HASH
-    if (self->hash != -1) {
-        if (self->hash != x) {
-            fprintf(stderr, "hash changed: %d => %d\n", self->hash, x);
-        }
-    }
     self->hash = x;
 #endif
-	return x;
+    return x;
 }
 
 static PyObject *
@@ -281,25 +281,39 @@
     
     vt = StaticTuple_as_tuple((StaticTuple *)v);
     if (vt == NULL) {
-        goto Done;
+        goto done;
     }
     if (!PyTuple_Check(wt)) {
         PyErr_BadInternalCall();
-        result = NULL;
-        goto Done;
+        goto done;
     }
     /* Now we have 2 tuples to compare, do it */
     result = PyTuple_Type.tp_richcompare(vt, wt, op);
-Done:
+done:
     Py_XDECREF(vt);
     return result;
 }
 
+/** Compare two objects to determine if they are equivalent.
+ * The basic flow is as follows
+ *  1) First make sure that both objects are StaticTuple instances. If they
+ *     aren't then cast self to a tuple, and have the tuple do the comparison.
+ *  2) Special case comparison to Py_None, because it happens to occur fairly
+ *     often in the test suite.
+ *  3) Special case when v and w are the same pointer. As we know the answer to
+ *     all queries without walking individual items.
+ *  4) For all operations, we then walk the items to find the first paired
+ *     items that are not equal.
+ *  5) If all items found are equal, we then check the length of self and
+ *     other to determine equality.
+ *  6) If an item differs, then we apply "op" to those last two items. (eg.
+ *     StaticTuple(A, B) > StaticTuple(A, C) iff B > C)
+ */
 
 static PyObject *
 StaticTuple_richcompare(PyObject *v, PyObject *w, int op)
 {
-    StaticTuple *vk, *wk;
+    StaticTuple *v_st, *w_st;
     Py_ssize_t vlen, wlen, min_len, i;
     PyObject *v_obj, *w_obj;
     richcmpfunc string_richcompare;
@@ -313,10 +327,10 @@
         Py_INCREF(Py_NotImplemented);
         return Py_NotImplemented;
     }
-    vk = (StaticTuple *)v;
+    v_st = (StaticTuple *)v;
     if (StaticTuple_CheckExact(w)) {
         /* The most common case */
-        wk = (StaticTuple*)w;
+        w_st = (StaticTuple*)w;
     } else if (PyTuple_Check(w)) {
         /* One of v or w is a tuple, so we go the 'slow' route and cast up to
          * tuples to compare.
@@ -325,17 +339,19 @@
          *       We probably want to optimize comparing self to other when
          *       other is a tuple.
          */
-        return StaticTuple_richcompare_to_tuple(vk, w, op);
+        return StaticTuple_richcompare_to_tuple(v_st, w, op);
     } else if (w == Py_None) {
         // None is always less than the object
-		switch (op) {
-		case Py_NE:case Py_GT:case Py_GE:
+        switch (op) {
+        case Py_NE:case Py_GT:case Py_GE:
             Py_INCREF(Py_True);
             return Py_True;
         case Py_EQ:case Py_LT:case Py_LE:
             Py_INCREF(Py_False);
             return Py_False;
-		}
+    default: // Should never happen
+        return Py_NotImplemented;
+        }
     } else {
         /* We don't special case this comparison, we just let python handle
          * it.
@@ -344,38 +360,49 @@
          return Py_NotImplemented;
     }
     /* Now we know that we have 2 StaticTuple objects, so let's compare them.
-     * This code is somewhat borrowed from tuplerichcompare, except we know our
+     * This code is inspired from tuplerichcompare, except we know our
      * objects are limited in scope, so we can inline some comparisons.
      */
     if (v == w) {
         /* Identical pointers, we can shortcut this easily. */
-		switch (op) {
-		case Py_EQ:case Py_LE:case Py_GE:
+        switch (op) {
+        case Py_EQ:case Py_LE:case Py_GE:
             Py_INCREF(Py_True);
             return Py_True;
-		case Py_NE:case Py_LT:case Py_GT:
+        case Py_NE:case Py_LT:case Py_GT:
             Py_INCREF(Py_False);
             return Py_False;
-		}
-    }
-    /* TODO: if STATIC_TUPLE_INTERNED_FLAG is set on both objects and they are
-     *       not the same pointer, then we know they aren't the same object
-     *       without having to do sub-by-sub comparison.
-     */
+        }
+    }
+    if (op == Py_EQ
+        && _StaticTuple_is_interned(v_st)
+        && _StaticTuple_is_interned(w_st))
+    {
+        /* If both objects are interned, we know they are different if the
+         * pointer is not the same, which would have been handled by the
+         * previous if. No need to compare the entries.
+         */
+        Py_INCREF(Py_False);
+        return Py_False;
+    }
 
-    /* It will be rare that we compare tuples of different lengths, so we don't
-     * start by optimizing the length comparision, same as the tuple code
-     * TODO: Interning may change this, because we'll be comparing lots of
-     *       different StaticTuple objects in the intern dict
+    /* The only time we are likely to compare items of different lengths is in
+     * something like the interned_keys set. However, the hash is good enough
+     * that it is rare. Note that 'tuple_richcompare' also does not compare
+     * lengths here.
      */
-    vlen = vk->size;
-    wlen = wk->size;
-	min_len = (vlen < wlen) ? vlen : wlen;
+    vlen = v_st->size;
+    wlen = w_st->size;
+    min_len = (vlen < wlen) ? vlen : wlen;
     string_richcompare = PyString_Type.tp_richcompare;
     for (i = 0; i < min_len; i++) {
         PyObject *result = NULL;
-        v_obj = StaticTuple_GET_ITEM(vk, i);
-        w_obj = StaticTuple_GET_ITEM(wk, i);
+        v_obj = StaticTuple_GET_ITEM(v_st, i);
+        w_obj = StaticTuple_GET_ITEM(w_st, i);
+        if (v_obj == w_obj) {
+            /* Shortcut case, these must be identical */
+            continue;
+        }
         if (PyString_CheckExact(v_obj) && PyString_CheckExact(w_obj)) {
             result = string_richcompare(v_obj, w_obj, Py_EQ);
         } else if (StaticTuple_CheckExact(v_obj) &&
@@ -391,9 +418,15 @@
             return NULL; /* There seems to be an error */
         }
         if (result == Py_NotImplemented) {
-            PyErr_BadInternalCall();
             Py_DECREF(result);
-            return NULL;
+            /* One side must have had a string and the other a StaticTuple.
+             * This clearly means that they are not equal.
+             */
+            if (op == Py_EQ) {
+                Py_INCREF(Py_False);
+                return Py_False;
+            }
+            result = PyObject_RichCompare(v_obj, w_obj, Py_EQ);
         }
         if (result == Py_False) {
             /* This entry is not identical
@@ -415,28 +448,28 @@
         }
         Py_DECREF(result);
     }
-	if (i >= vlen || i >= wlen) {
+    if (i >= min_len) {
         /* We walked off one of the lists, but everything compared equal so
          * far. Just compare the size.
          */
-		int cmp;
-		PyObject *res;
-		switch (op) {
-		case Py_LT: cmp = vlen <  wlen; break;
-		case Py_LE: cmp = vlen <= wlen; break;
-		case Py_EQ: cmp = vlen == wlen; break;
-		case Py_NE: cmp = vlen != wlen; break;
-		case Py_GT: cmp = vlen >  wlen; break;
-		case Py_GE: cmp = vlen >= wlen; break;
-		default: return NULL; /* cannot happen */
-		}
-		if (cmp)
-			res = Py_True;
-		else
-			res = Py_False;
-		Py_INCREF(res);
-		return res;
-	}
+        int cmp;
+        PyObject *res;
+        switch (op) {
+        case Py_LT: cmp = vlen <  wlen; break;
+        case Py_LE: cmp = vlen <= wlen; break;
+        case Py_EQ: cmp = vlen == wlen; break;
+        case Py_NE: cmp = vlen != wlen; break;
+        case Py_GT: cmp = vlen >  wlen; break;
+        case Py_GE: cmp = vlen >= wlen; break;
+        default: return NULL; /* cannot happen */
+        }
+        if (cmp)
+            res = Py_True;
+        else
+            res = Py_False;
+        Py_INCREF(res);
+        return res;
+    }
     /* The last item differs, shortcut the Py_NE case */
     if (op == Py_NE) {
         Py_INCREF(Py_True);
@@ -477,15 +510,22 @@
 }
 
 static char StaticTuple__is_interned_doc[] = "_is_interned() => True/False\n"
-    "Check to see if this key has been interned.\n";
+    "Check to see if this tuple has been interned.\n";
 
 
 static PyObject *
 StaticTuple_item(StaticTuple *self, Py_ssize_t offset)
 {
     PyObject *obj;
-    if (offset < 0 || offset >= self->size) {
-        PyErr_SetString(PyExc_IndexError, "StaticTuple index out of range");
+    /* We cast to (int) to avoid worrying about whether Py_ssize_t is a
+     * long long, etc. offsets should never be >2**31 anyway.
+     */
+    if (offset < 0) {
+        PyErr_Format(PyExc_IndexError, "StaticTuple_item does not support"
+            " negative indices: %d\n", (int)offset);
+    } else if (offset >= self->size) {
+        PyErr_Format(PyExc_IndexError, "StaticTuple index out of range"
+            " %d >= %d", (int)offset, (int)self->size);
         return NULL;
     }
     obj = (PyObject *)self->items[offset];
@@ -519,9 +559,12 @@
 
 static char StaticTuple_doc[] =
     "C implementation of a StaticTuple structure."
-    "\n This is used as StaticTuple(key_bit_1, key_bit_2, key_bit_3, ...)"
-    "\n This is similar to tuple, just less flexible in what it"
-    "\n supports, but also lighter memory consumption.";
+    "\n This is used as StaticTuple(item1, item2, item3)"
+    "\n This is similar to tuple, less flexible in what it"
+    "\n supports, but also lighter memory consumption."
+    "\n Note that the constructor mimics the () form of tuples"
+    "\n Rather than the 'tuple()' constructor."
+    "\n  eg. StaticTuple(a, b) == (a, b) == tuple((a, b))";
 
 static PyMethodDef StaticTuple_methods[] = {
     {"as_tuple", (PyCFunction)StaticTuple_as_tuple, METH_NOARGS, StaticTuple_as_tuple_doc},
@@ -542,6 +585,12 @@
     0,                              /* sq_contains */
 };
 
+/* TODO: Implement StaticTuple_as_mapping.
+ *       The only thing we really want to support from there is mp_subscript,
+ *       so that we could support extended slicing (foo[::2]). Not worth it
+ *       yet, though.
+ */
+
 
 PyTypeObject StaticTuple_Type = {
     PyObject_HEAD_INIT(NULL)
@@ -561,7 +610,7 @@
     (hashfunc)StaticTuple_hash,                  /* tp_hash */
     0,                                           /* tp_call */
     0,                                           /* tp_str */
-    PyObject_GenericGetAttr,                     /* tp_getattro */
+    0,                                           /* tp_getattro */
     0,                                           /* tp_setattro */
     0,                                           /* tp_as_buffer */
     Py_TPFLAGS_DEFAULT,                          /* tp_flags*/
@@ -590,7 +639,7 @@
     0,                                           /* tp_dictoffset */
     0,                                           /* tp_init */
     0,                                           /* tp_alloc */
-    StaticTuple_new,                             /* tp_new */
+    StaticTuple_new_constructor,                 /* tp_new */
 };
 
 
@@ -646,11 +695,57 @@
 }
 
 
+static int
+_workaround_pyrex_096(void)
+{
+    /* Work around an incompatibility in how pyrex 0.9.6 exports a module,
+     * versus how pyrex 0.9.8 and cython 0.11 export it.
+     * Namely 0.9.6 exports import__simple_set_pyx and tries to
+     * "import _simple_set_pyx" but it is available only as
+     * "import bzrlib._simple_set_pyx"
+     * It is a shame to hack up sys.modules, but that is what we've got to do.
+     */
+    PyObject *sys_module = NULL, *modules = NULL, *set_module = NULL;
+    int retval = -1;
+
+    /* Clear out the current ImportError exception, and try again. */
+    PyErr_Clear();
+    /* Note that this only seems to work if somewhere else imports
+     * bzrlib._simple_set_pyx before importing bzrlib._static_tuple_c
+     */
+    set_module = PyImport_ImportModule("bzrlib._simple_set_pyx");
+    if (set_module == NULL) {
+	// fprintf(stderr, "Failed to import bzrlib._simple_set_pyx\n");
+        goto end;
+    }
+    /* Add the _simple_set_pyx into sys.modules at the appropriate location. */
+    sys_module = PyImport_ImportModule("sys");
+    if (sys_module == NULL) {
+    	// fprintf(stderr, "Failed to import sys\n");
+        goto end;
+    }
+    modules = PyObject_GetAttrString(sys_module, "modules");
+    if (modules == NULL || !PyDict_Check(modules)) {
+    	// fprintf(stderr, "Failed to find sys.modules\n");
+        goto end;
+    }
+    PyDict_SetItemString(modules, "_simple_set_pyx", set_module);
+    /* Now that we have hacked it in, try the import again. */
+    retval = import_bzrlib___simple_set_pyx();
+end:
+    Py_XDECREF(set_module);
+    Py_XDECREF(sys_module);
+    Py_XDECREF(modules);
+    return retval;
+}
+
+
 PyMODINIT_FUNC
 init_static_tuple_c(void)
 {
     PyObject* m;
 
+    StaticTuple_Type.tp_getattro = PyObject_GenericGetAttr;
     if (PyType_Ready(&StaticTuple_Type) < 0)
         return;
 
@@ -661,8 +756,9 @@
 
     Py_INCREF(&StaticTuple_Type);
     PyModule_AddObject(m, "StaticTuple", (PyObject *)&StaticTuple_Type);
-    if (import_bzrlib___simple_set_pyx() == -1) {
-        // We failed to set up, stop early
+    if (import_bzrlib___simple_set_pyx() == -1
+        && _workaround_pyrex_096() == -1)
+    {
         return;
     }
     setup_interned_tuples(m);
 
=== modified file 'bzrlib/_static_tuple_py.py'
--- bzrlib/_static_tuple_py.py	2009-10-07 16:12:50 +0000
+++ bzrlib/_static_tuple_py.py	2009-10-15 18:31:14 +0000
@@ -52,6 +52,8 @@
         return _interned_tuples.setdefault(self, self)
 
 
+# Have to set it to None first, so that __new__ can determine whether
+# the _empty_tuple singleton has been created yet or not.
 _empty_tuple = None
 _empty_tuple = StaticTuple()
 _interned_tuples = {}
 
=== modified file 'bzrlib/branch.py'
--- bzrlib/branch.py	2009-08-19 18:04:49 +0000
+++ bzrlib/branch.py	2009-10-15 18:31:14 +0000
@@ -46,9 +46,10 @@
     )
 """)
 
-from bzrlib.decorators import needs_read_lock, needs_write_lock
+from bzrlib.decorators import needs_read_lock, needs_write_lock, only_raises
 from bzrlib.hooks import HookPoint, Hooks
 from bzrlib.inter import InterObject
+from bzrlib.lock import _RelockDebugMixin
 from bzrlib import registry
 from bzrlib.symbol_versioning import (
     deprecated_in,
@@ -2079,7 +2080,7 @@
     _legacy_formats[0].network_name(), _legacy_formats[0].__class__)
 
 
-class BzrBranch(Branch):
+class BzrBranch(Branch, _RelockDebugMixin):
     """A branch stored in the actual filesystem.
 
     Note that it's "local" in the context of the filesystem; it doesn't
@@ -2131,6 +2132,8 @@
         return self.control_files.is_locked()
 
     def lock_write(self, token=None):
+        if not self.is_locked():
+            self._note_lock('w')
         # All-in-one needs to always unlock/lock.
         repo_control = getattr(self.repository, 'control_files', None)
         if self.control_files == repo_control or not self.is_locked():
@@ -2146,6 +2149,8 @@
             raise
 
     def lock_read(self):
+        if not self.is_locked():
+            self._note_lock('r')
         # All-in-one needs to always unlock/lock.
         repo_control = getattr(self.repository, 'control_files', None)
         if self.control_files == repo_control or not self.is_locked():
@@ -2160,6 +2165,7 @@
                 self.repository.unlock()
             raise
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         try:
             self.control_files.unlock()
 
=== modified file 'bzrlib/btree_index.py'
--- bzrlib/btree_index.py	2009-09-22 02:18:24 +0000
+++ bzrlib/btree_index.py	2009-10-15 18:31:13 +0000
@@ -163,6 +163,7 @@
         node_refs, _ = self._check_key_ref_value(key, references, value)
         if key in self._nodes:
             raise errors.BadIndexDuplicateKey(key, self)
+        # TODO: StaticTuple
         self._nodes[key] = (node_refs, value)
         self._keys.add(key)
         if self._nodes_by_key is not None and self._key_length > 1:
@@ -625,6 +626,7 @@
         for line in lines[2:]:
             if line == '':
                 break
+            # TODO: Switch to StaticTuple here.
             nodes.append(tuple(map(intern, line.split('\0'))))
         return nodes
 
@@ -636,7 +638,7 @@
     memory except when very large walks are done.
     """
 
-    def __init__(self, transport, name, size):
+    def __init__(self, transport, name, size, unlimited_cache=False):
         """Create a B+Tree index object on the index name.
 
         :param transport: The transport to read data for the index from.
@@ -646,6 +648,9 @@
             the initial read (to read the root node header) can be done
             without over-reading even on empty indices, and on small indices
             allows single-IO to read the entire index.
+        :param unlimited_cache: If set to True, then instead of using an
+            LRUCache with size _NODE_CACHE_SIZE, we will use a dict and always
+            cache all leaf nodes.
         """
         self._transport = transport
         self._name = name
@@ -655,12 +660,15 @@
         self._root_node = None
         # Default max size is 100,000 leave values
         self._leaf_value_cache = None # lru_cache.LRUCache(100*1000)
-        self._leaf_node_cache = lru_cache.LRUCache(_NODE_CACHE_SIZE)
-        # We could limit this, but even a 300k record btree has only 3k leaf
-        # nodes, and only 20 internal nodes. So the default of 100 nodes in an
-        # LRU would mean we always cache everything anyway, no need to pay the
-        # overhead of LRU
-        self._internal_node_cache = fifo_cache.FIFOCache(100)
+        if unlimited_cache:
+            self._leaf_node_cache = {}
+            self._internal_node_cache = {}
+        else:
+            self._leaf_node_cache = lru_cache.LRUCache(_NODE_CACHE_SIZE)
+            # We use a FIFO here just to prevent possible blowout. However, a
+            # 300k record btree has only 3k leaf nodes, and only 20 internal
+            # nodes. A value of 100 scales to ~100*100*100 = 1M records.
+            self._internal_node_cache = fifo_cache.FIFOCache(100)
         self._key_count = None
         self._row_lengths = None
         self._row_offsets = None # Start of each row, [-1] is the end
@@ -698,9 +706,9 @@
                 if start_of_leaves is None:
                     start_of_leaves = self._row_offsets[-2]
                 if node_pos < start_of_leaves:
-                    self._internal_node_cache.add(node_pos, node)
+                    self._internal_node_cache[node_pos] = node
                 else:
-                    self._leaf_node_cache.add(node_pos, node)
+                    self._leaf_node_cache[node_pos] = node
             found[node_pos] = node
         return found
 
 
=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py	2009-09-24 19:51:37 +0000
+++ bzrlib/builtins.py	2009-10-15 18:31:14 +0000
@@ -431,7 +431,10 @@
         for node in bt.iter_all_entries():
             # Node is made up of:
             # (index, key, value, [references])
-            self.outf.write('%s\n' % (node[1:],))
+            refs_as_tuples = tuple([tuple([tuple(ref) for ref in ref_list])
+                                   for ref_list in node[3]])
+            as_tuple = (tuple(node[1]), node[2], refs_as_tuples)
+            self.outf.write('%s\n' % (as_tuple,))
 
 
 class cmd_remove_tree(Command):
@@ -461,8 +464,7 @@
             raise errors.BzrCommandError("You cannot remove the working tree"
                                          " of a remote path")
         if not force:
-            if (working.has_changes(working.basis_tree())
-                or len(working.get_parent_ids()) > 1):
+            if (working.has_changes()):
                 raise errors.UncommittedChanges(working)
 
         working_path = working.bzrdir.root_transport.base
@@ -1109,8 +1111,7 @@
         else:
             revision_id = None
         if strict and tree is not None and revision_id is None:
-            if (tree.has_changes(tree.basis_tree())
-                or len(tree.get_parent_ids()) > 1):
+            if (tree.has_changes()):
                 raise errors.UncommittedChanges(
                     tree, more='Use --no-strict to force the push.')
             if tree.last_revision() != tree.branch.last_revision():
@@ -1887,7 +1888,7 @@
     @display_command
     def run(self, revision=None, file_list=None, diff_options=None,
             prefix=None, old=None, new=None, using=None):
-        from bzrlib.diff import _get_trees_to_diff, show_diff_trees
+        from bzrlib.diff import get_trees_and_branches_to_diff, show_diff_trees
 
         if (prefix is None) or (prefix == '0'):
             # diff -p0 format
@@ -1907,9 +1908,10 @@
             raise errors.BzrCommandError('bzr diff --revision takes exactly'
                                          ' one or two revision specifiers')
 
-        old_tree, new_tree, specific_files, extra_trees = \
-                _get_trees_to_diff(file_list, revision, old, new,
-                apply_view=True)
+        (old_tree, new_tree,
+         old_branch, new_branch,
+         specific_files, extra_trees) = get_trees_and_branches_to_diff(
+            file_list, revision, old, new, apply_view=True)
         return show_diff_trees(old_tree, new_tree, sys.stdout,
                                specific_files=specific_files,
                                external_diff_options=diff_options,
@@ -3663,7 +3665,7 @@
 
         # die as quickly as possible if there are uncommitted changes
         if not force:
-            if tree.has_changes(basis_tree) or len(tree.get_parent_ids()) > 1:
+            if tree.has_changes():
                 raise errors.UncommittedChanges(tree)
 
         view_info = _get_view_info_for_change_reporter(tree)
@@ -3720,7 +3722,10 @@
                                        merger.other_rev_id)
                     result.report(self.outf)
                     return 0
-            merger.check_basis(False)
+            if merger.this_basis is None:
+                raise errors.BzrCommandError(
+                    "This branch has no commits."
+                    " (perhaps you would prefer 'bzr pull')")
             if preview:
                 return self._do_preview(merger, cleanups)
             elif interactive:
 
=== modified file 'bzrlib/bundle/apply_bundle.py'
--- bzrlib/bundle/apply_bundle.py	2009-03-23 14:59:43 +0000
+++ bzrlib/bundle/apply_bundle.py	2009-10-15 18:31:14 +0000
@@ -56,7 +56,8 @@
                         change_reporter=change_reporter)
         merger.pp = pp
         merger.pp.next_phase()
-        merger.check_basis(check_clean, require_commits=False)
+        if check_clean and tree.has_changes():
+            raise errors.UncommittedChanges(self)
         merger.other_rev_id = reader.target
         merger.other_tree = merger.revision_tree(reader.target)
         merger.other_basis = reader.target
 
=== modified file 'bzrlib/commands.py'
--- bzrlib/commands.py	2009-09-17 07:16:12 +0000
+++ bzrlib/commands.py	2009-10-15 18:31:14 +0000
@@ -1097,7 +1097,7 @@
 
     # Is this a final release version? If so, we should suppress warnings
     if bzrlib.version_info[3] == 'final':
-        suppress_deprecation_warnings(override=False)
+        suppress_deprecation_warnings(override=True)
     if argv is None:
         argv = osutils.get_unicode_argv()
     else:
 
=== modified file 'bzrlib/decorators.py'
--- bzrlib/decorators.py	2009-03-23 14:59:43 +0000
+++ bzrlib/decorators.py	2009-10-15 18:31:14 +0000
@@ -24,6 +24,8 @@
 
 import sys
 
+from bzrlib import trace
+
 
 def _get_parameters(func):
     """Recreate the parameters for a function using introspection.
@@ -204,6 +206,31 @@
     return write_locked
 
 
+def only_raises(*errors):
+    """Make a decorator that will only allow the given error classes to be
+    raised.  All other errors will be logged and then discarded.
+
+    Typical use is something like::
+
+        @only_raises(LockNotHeld, LockBroken)
+        def unlock(self):
+            # etc
+    """
+    def decorator(unbound):
+        def wrapped(*args, **kwargs):
+            try:
+                return unbound(*args, **kwargs)
+            except errors:
+                raise
+            except:
+                trace.mutter('Error suppressed by only_raises:')
+                trace.log_exception_quietly()
+        wrapped.__doc__ = unbound.__doc__
+        wrapped.__name__ = unbound.__name__
+        return wrapped
+    return decorator
+
+
 # Default is more functionality, 'bzr' the commandline will request fast
 # versions.
 needs_read_lock = _pretty_needs_read_lock
 
=== modified file 'bzrlib/diff.py'
--- bzrlib/diff.py	2009-07-29 21:35:05 +0000
+++ bzrlib/diff.py	2009-10-15 18:31:13 +0000
@@ -277,8 +277,8 @@
                         new_abspath, e)
 
 
-def _get_trees_to_diff(path_list, revision_specs, old_url, new_url,
-    apply_view=True):
+def get_trees_and_branches_to_diff(path_list, revision_specs, old_url, new_url,
+                                   apply_view=True):
     """Get the trees and specific files to diff given a list of paths.
 
     This method works out the trees to be diff'ed and the files of
@@ -299,9 +299,9 @@
         if True and a view is set, apply the view or check that the paths
         are within it
     :returns:
-        a tuple of (old_tree, new_tree, specific_files, extra_trees) where
-        extra_trees is a sequence of additional trees to search in for
-        file-ids.
+        a tuple of (old_tree, new_tree, old_branch, new_branch,
+        specific_files, extra_trees) where extra_trees is a sequence of
+        additional trees to search in for file-ids.
     """
     # Get the old and new revision specs
     old_revision_spec = None
@@ -341,6 +341,7 @@
             views.check_path_in_view(working_tree, relpath)
         specific_files.append(relpath)
     old_tree = _get_tree_to_diff(old_revision_spec, working_tree, branch)
+    old_branch = branch
 
     # Get the new location
     if new_url is None:
@@ -354,6 +355,7 @@
             specific_files.append(relpath)
     new_tree = _get_tree_to_diff(new_revision_spec, working_tree, branch,
         basis_is_default=working_tree is None)
+    new_branch = branch
 
     # Get the specific files (all files is None, no files is [])
     if make_paths_wt_relative and working_tree is not None:
@@ -378,7 +380,8 @@
     extra_trees = None
     if working_tree is not None and working_tree not in (old_tree, new_tree):
         extra_trees = (working_tree,)
-    return old_tree, new_tree, specific_files, extra_trees
+    return old_tree, new_tree, old_branch, new_branch, specific_files, extra_trees
+
 
 def _get_tree_to_diff(spec, tree=None, branch=None, basis_is_default=True):
     if branch is None and tree is not None:
 
=== modified file 'bzrlib/foreign.py'
--- bzrlib/foreign.py	2009-10-02 09:11:43 +0000
+++ bzrlib/foreign.py	2009-10-15 18:31:14 +0000
@@ -305,8 +305,7 @@
                 ).get_user_option_as_bool('dpush_strict')
         if strict is None: strict = True # default value
         if strict and source_wt is not None:
-            if (source_wt.has_changes(source_wt.basis_tree())
-                or len(source_wt.get_parent_ids()) > 1):
+            if (source_wt.has_changes()):
                 raise errors.UncommittedChanges(
                     source_wt, more='Use --no-strict to force the push.')
             if source_wt.last_revision() != source_wt.branch.last_revision():
 
=== modified file 'bzrlib/help_topics/__init__.py'
--- bzrlib/help_topics/__init__.py	2009-06-19 09:06:56 +0000
+++ bzrlib/help_topics/__init__.py	2009-10-15 18:31:14 +0000
@@ -644,38 +644,20 @@
 formats may also be introduced to improve performance and
 scalability.
 
-Use the following guidelines to select a format (stopping
-as soon as a condition is true):
-
-* If you are working on an existing project, use whatever
-  format that project is using. (Bazaar will do this for you
-  by default).
-
-* If you are using bzr-svn to interoperate with a Subversion
-  repository, use 1.14-rich-root.
-
-* If you are working on a project with big trees (5000+ paths)
-  or deep history (5000+ revisions), use 1.14.
-
-* Otherwise, use the default format - it is good enough for
-  most projects.
-
-If some of your developers are unable to use the most recent
-version of Bazaar (due to distro package availability say), be
-sure to adjust the guidelines above accordingly. For example,
-you may need to select 1.9 instead of 1.14 if your project has
-standardized on Bazaar 1.13.1 say.
-
-Note: Many of the currently supported formats have two variants:
+The newest format, 2a, is highly recommended. If your
+project is not using 2a, then you should suggest to the
+project owner to upgrade.
+
+
+Note: Some of the older formats have two variants:
 a plain one and a rich-root one. The latter include an additional
 field about the root of the tree. There is no performance cost
 for using a rich-root format but you cannot easily merge changes
 from a rich-root format into a plain format. As a consequence,
 moving a project to a rich-root format takes some co-ordination
 in that all contributors need to upgrade their repositories
-around the same time. (It is for this reason that we have delayed
-making a rich-root format the default so far, though we will do
-so at some appropriate time in the future.)
+around the same time. 2a and all future formats will be
+implicitly rich-root.
 
 See ``bzr help current-formats`` for the complete list of
 currently supported formats. See ``bzr help other-formats`` for
 
=== modified file 'bzrlib/help_topics/en/debug-flags.txt'
--- bzrlib/help_topics/en/debug-flags.txt	2009-08-20 05:02:45 +0000
+++ bzrlib/help_topics/en/debug-flags.txt	2009-10-15 18:31:14 +0000
@@ -22,16 +22,18 @@
 -Dhttp            Trace http connections, requests and responses.
 -Dindex           Trace major index operations.
 -Dknit            Trace knit operations.
--Dstrict_locks    Trace when OS locks are potentially used in a non-portable
-                  manner.
 -Dlock            Trace when lockdir locks are taken or released.
 -Dprogress        Trace progress bar operations.
 -Dmerge           Emit information for debugging merges.
 -Dno_apport       Don't use apport to report crashes.
--Dunlock          Some errors during unlock are treated as warnings.
 -Dpack            Emit information about pack operations.
+-Drelock          Emit a message every time a branch or repository object is
+                  unlocked then relocked the same way.
 -Dsftp            Trace SFTP internals.
 -Dstream          Trace fetch streams.
+-Dstrict_locks    Trace when OS locks are potentially used in a non-portable
+                  manner.
+-Dunlock          Some errors during unlock are treated as warnings.
 -DIDS_never       Never use InterDifferingSerializer when fetching.
 -DIDS_always      Always use InterDifferingSerializer to fetch if appropriate
                   for the format, even for non-local fetches.
 
=== modified file 'bzrlib/index.py'
--- bzrlib/index.py	2009-10-08 15:44:41 +0000
+++ bzrlib/index.py	2009-10-15 18:31:14 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2007, 2008 Canonical Ltd
+# Copyright (C) 2007, 2008, 2009 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -40,7 +40,7 @@
     debug,
     errors,
     )
-from bzrlib._static_tuple_c import StaticTuple
+from bzrlib.static_tuple import StaticTuple
 
 _HEADER_READV = (0, 200)
 _OPTION_KEY_ELEMENTS = "key_elements="
@@ -203,7 +203,9 @@
                 if reference not in self._nodes:
                     self._check_key(reference)
                     absent_references.append(reference)
+            # TODO: StaticTuple
             node_refs.append(tuple(reference_list))
+        # TODO: StaticTuple
         return tuple(node_refs), absent_references
 
     def add_node(self, key, value, references=()):
@@ -369,7 +371,7 @@
     suitable for production use. :XXX
     """
 
-    def __init__(self, transport, name, size):
+    def __init__(self, transport, name, size, unlimited_cache=False):
         """Open an index called name on transport.
 
         :param transport: A bzrlib.transport.Transport.
 
=== modified file 'bzrlib/lock.py'
--- bzrlib/lock.py	2009-07-31 16:51:48 +0000
+++ bzrlib/lock.py	2009-10-15 18:31:13 +0000
@@ -518,3 +518,24 @@
 # We default to using the first available lock class.
 _lock_type, WriteLock, ReadLock = _lock_classes[0]
 
+
+class _RelockDebugMixin(object):
+    """Mixin support for -Drelock flag.
+
+    Add this as a base class then call self._note_lock with 'r' or 'w' when
+    acquiring a read- or write-lock.  If this object was previously locked (and
+    locked the same way), and -Drelock is set, then this will trace.note a
+    message about it.
+    """
+    
+    _prev_lock = None
+
+    def _note_lock(self, lock_type):
+        if 'relock' in debug.debug_flags and self._prev_lock == lock_type:
+            if lock_type == 'r':
+                type_name = 'read'
+            else:
+                type_name = 'write'
+            trace.note('%r was %s locked again', self, type_name)
+        self._prev_lock = lock_type
+
 
=== modified file 'bzrlib/lockable_files.py'
--- bzrlib/lockable_files.py	2009-07-27 05:39:01 +0000
+++ bzrlib/lockable_files.py	2009-10-15 18:31:14 +0000
@@ -32,8 +32,7 @@
 """)
 
 from bzrlib.decorators import (
-    needs_read_lock,
-    needs_write_lock,
+    only_raises,
     )
 from bzrlib.symbol_versioning import (
     deprecated_in,
@@ -221,6 +220,7 @@
         """Setup a write transaction."""
         self._set_transaction(transactions.WriteTransaction())
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         if not self._lock_mode:
             return lock.cant_unlock_not_held(self)
 
=== modified file 'bzrlib/lockdir.py'
--- bzrlib/lockdir.py	2009-07-27 05:24:02 +0000
+++ bzrlib/lockdir.py	2009-10-15 18:31:14 +0000
@@ -112,6 +112,7 @@
     lock,
     )
 import bzrlib.config
+from bzrlib.decorators import only_raises
 from bzrlib.errors import (
         DirectoryNotEmpty,
         FileExists,
@@ -286,6 +287,7 @@
                                             info_bytes)
         return tmpname
 
+    @only_raises(LockNotHeld, LockBroken)
     def unlock(self):
         """Release a held lock
         """
 
=== modified file 'bzrlib/merge.py'
--- bzrlib/merge.py	2009-10-06 12:25:59 +0000
+++ bzrlib/merge.py	2009-10-15 18:31:14 +0000
@@ -35,6 +35,10 @@
     ui,
     versionedfile
     )
+from bzrlib.symbol_versioning import (
+    deprecated_in,
+    deprecated_method,
+    )
 # TODO: Report back as changes are merged in
 
 
@@ -226,6 +230,7 @@
         revision_id = _mod_revision.ensure_null(revision_id)
         return branch, self.revision_tree(revision_id, branch)
 
+    @deprecated_method(deprecated_in((2, 1, 0)))
     def ensure_revision_trees(self):
         if self.this_revision_tree is None:
             self.this_basis_tree = self.revision_tree(self.this_basis)
@@ -239,6 +244,7 @@
             other_rev_id = self.other_basis
             self.other_tree = other_basis_tree
 
+    @deprecated_method(deprecated_in((2, 1, 0)))
     def file_revisions(self, file_id):
         self.ensure_revision_trees()
         def get_id(tree, file_id):
@@ -252,6 +258,7 @@
         trees = (self.this_basis_tree, self.other_tree)
         return [get_id(tree, file_id) for tree in trees]
 
+    @deprecated_method(deprecated_in((2, 1, 0)))
     def check_basis(self, check_clean, require_commits=True):
         if self.this_basis is None and require_commits is True:
             raise errors.BzrCommandError(
@@ -262,6 +269,7 @@
             if self.this_basis != self.this_rev_id:
                 raise errors.UncommittedChanges(self.this_tree)
 
+    @deprecated_method(deprecated_in((2, 1, 0)))
     def compare_basis(self):
         try:
             basis_tree = self.revision_tree(self.this_tree.last_revision())
@@ -274,7 +282,8 @@
         self.interesting_files = file_list
 
     def set_pending(self):
-        if not self.base_is_ancestor or not self.base_is_other_ancestor or self.other_rev_id is None:
+        if (not self.base_is_ancestor or not self.base_is_other_ancestor
+            or self.other_rev_id is None):
             return
         self._add_parent()
 
 
=== modified file 'bzrlib/mutabletree.py'
--- bzrlib/mutabletree.py	2009-10-06 12:25:59 +0000
+++ bzrlib/mutabletree.py	2009-10-15 18:31:14 +0000
@@ -233,12 +233,20 @@
         raise NotImplementedError(self._gather_kinds)
 
     @needs_read_lock
-    def has_changes(self, from_tree):
-        """Quickly check that the tree contains at least one change.
+    def has_changes(self, _from_tree=None):
+        """Quickly check that the tree contains at least one commitable change.
+
+        :param _from_tree: tree to compare against to find changes (default to
+            the basis tree and is intended to be used by tests).
 
         :return: True if a change is found. False otherwise
         """
-        changes = self.iter_changes(from_tree)
+        # Check pending merges
+        if len(self.get_parent_ids()) > 1:
+            return True
+        if _from_tree is None:
+            _from_tree = self.basis_tree()
+        changes = self.iter_changes(_from_tree)
         try:
             change = changes.next()
             # Exclude root (talk about black magic... --vila 20090629)
 
=== modified file 'bzrlib/osutils.py'
--- bzrlib/osutils.py	2009-09-19 01:33:10 +0000
+++ bzrlib/osutils.py	2009-10-15 18:31:14 +0000
@@ -1132,7 +1132,14 @@
     bit_iter = iter(rel.split('/'))
     for bit in bit_iter:
         lbit = bit.lower()
-        for look in _listdir(current):
+        try:
+            next_entries = _listdir(current)
+        except OSError: # enoent, eperm, etc
+            # We can't find this in the filesystem, so just append the
+            # remaining bits.
+            current = pathjoin(current, bit, *list(bit_iter))
+            break
+        for look in next_entries:
             if lbit == look.lower():
                 current = pathjoin(current, look)
                 break
@@ -1142,7 +1149,7 @@
             # the target of a move, for example).
             current = pathjoin(current, bit, *list(bit_iter))
             break
-    return current[len(abs_base)+1:]
+    return current[len(abs_base):].lstrip('/')
 
 # XXX - TODO - we need better detection/integration of case-insensitive
 # file-systems; Linux often sees FAT32 devices (or NFS-mounted OSX
 
=== modified file 'bzrlib/python-compat.h'
--- bzrlib/python-compat.h	2009-09-30 18:00:42 +0000
+++ bzrlib/python-compat.h	2009-10-15 18:31:14 +0000
@@ -28,6 +28,9 @@
 /* http://www.python.org/dev/peps/pep-0353/ */
 #if PY_VERSION_HEX < 0x02050000 && !defined(PY_SSIZE_T_MIN)
     typedef int Py_ssize_t;
+    typedef Py_ssize_t (*lenfunc)(PyObject *);
+    typedef PyObject * (*ssizeargfunc)(PyObject *, Py_ssize_t);
+    typedef PyObject * (*ssizessizeargfunc)(PyObject *, Py_ssize_t, Py_ssize_t);
     #define PY_SSIZE_T_MAX INT_MAX
     #define PY_SSIZE_T_MIN INT_MIN
     #define PyInt_FromSsize_t(z) PyInt_FromLong(z)
@@ -77,5 +80,8 @@
 #ifndef Py_TYPE
 #  define Py_TYPE(o) ((o)->ob_type)
 #endif
+#ifndef Py_REFCNT
+#  define Py_REFCNT(o) ((o)->ob_refcnt)
+#endif
 
 #endif /* _BZR_PYTHON_COMPAT_H */
 
=== modified file 'bzrlib/reconfigure.py'
--- bzrlib/reconfigure.py	2009-07-24 03:15:56 +0000
+++ bzrlib/reconfigure.py	2009-10-15 18:31:14 +0000
@@ -265,9 +265,7 @@
 
     def _check(self):
         """Raise if reconfiguration would destroy local changes"""
-        if self._destroy_tree:
-            # XXX: What about pending merges ? -- vila 20090629
-            if self.tree.has_changes(self.tree.basis_tree()):
+        if self._destroy_tree and self.tree.has_changes():
                 raise errors.UncommittedChanges(self.tree)
         if self._create_reference and self.local_branch is not None:
             reference_branch = branch.Branch.open(self._select_bind_location())
 
=== modified file 'bzrlib/remote.py'
--- bzrlib/remote.py	2009-10-02 05:43:41 +0000
+++ bzrlib/remote.py	2009-10-15 18:31:14 +0000
@@ -33,7 +33,7 @@
 )
 from bzrlib.branch import BranchReferenceFormat
 from bzrlib.bzrdir import BzrDir, RemoteBzrDirFormat
-from bzrlib.decorators import needs_read_lock, needs_write_lock
+from bzrlib.decorators import needs_read_lock, needs_write_lock, only_raises
 from bzrlib.errors import (
     NoSuchRevision,
     SmartProtocolError,
@@ -619,7 +619,7 @@
         return self._custom_format._serializer
 
 
-class RemoteRepository(_RpcHelper):
+class RemoteRepository(_RpcHelper, lock._RelockDebugMixin):
     """Repository accessed over rpc.
 
     For the moment most operations are performed using local transport-backed
@@ -949,6 +949,7 @@
     def lock_read(self):
         # wrong eventually - want a local lock cache context
         if not self._lock_mode:
+            self._note_lock('r')
             self._lock_mode = 'r'
             self._lock_count = 1
             self._unstacked_provider.enable_cache(cache_misses=True)
@@ -974,6 +975,7 @@
 
     def lock_write(self, token=None, _skip_rpc=False):
         if not self._lock_mode:
+            self._note_lock('w')
             if _skip_rpc:
                 if self._lock_token is not None:
                     if token != self._lock_token:
@@ -1082,6 +1084,7 @@
         else:
             raise errors.UnexpectedSmartServerResponse(response)
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         if not self._lock_count:
             return lock.cant_unlock_not_held(self)
@@ -2081,7 +2084,7 @@
         return self._custom_format.supports_set_append_revisions_only()
 
 
-class RemoteBranch(branch.Branch, _RpcHelper):
+class RemoteBranch(branch.Branch, _RpcHelper, lock._RelockDebugMixin):
     """Branch stored on a server accessed by HPSS RPC.
 
     At the moment most operations are mapped down to simple file operations.
@@ -2318,6 +2321,7 @@
     def lock_read(self):
         self.repository.lock_read()
         if not self._lock_mode:
+            self._note_lock('r')
             self._lock_mode = 'r'
             self._lock_count = 1
             if self._real_branch is not None:
@@ -2343,6 +2347,7 @@
 
     def lock_write(self, token=None):
         if not self._lock_mode:
+            self._note_lock('w')
             # Lock the branch and repo in one remote call.
             remote_tokens = self._remote_lock_write(token)
             self._lock_token, self._repo_lock_token = remote_tokens
@@ -2383,6 +2388,7 @@
             return
         raise errors.UnexpectedSmartServerResponse(response)
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         try:
             self._lock_count -= 1
@@ -2428,6 +2434,7 @@
             raise NotImplementedError(self.dont_leave_lock_in_place)
         self._leave_lock = False
 
+    @needs_read_lock
     def get_rev_id(self, revno, history=None):
         if revno == 0:
             return _mod_revision.NULL_REVISION
 
=== modified file 'bzrlib/repofmt/pack_repo.py'
--- bzrlib/repofmt/pack_repo.py	2009-09-08 05:51:36 +0000
+++ bzrlib/repofmt/pack_repo.py	2009-10-15 18:31:13 +0000
@@ -54,7 +54,7 @@
     revision as _mod_revision,
     )
 
-from bzrlib.decorators import needs_write_lock
+from bzrlib.decorators import needs_write_lock, only_raises
 from bzrlib.btree_index import (
     BTreeGraphIndex,
     BTreeBuilder,
@@ -73,6 +73,7 @@
     )
 from bzrlib.trace import (
     mutter,
+    note,
     warning,
     )
 
@@ -224,10 +225,14 @@
         return self.index_name('text', name)
 
     def _replace_index_with_readonly(self, index_type):
+        unlimited_cache = False
+        if index_type == 'chk':
+            unlimited_cache = True
         setattr(self, index_type + '_index',
             self.index_class(self.index_transport,
                 self.index_name(index_type, self.name),
-                self.index_sizes[self.index_offset(index_type)]))
+                self.index_sizes[self.index_offset(index_type)],
+                unlimited_cache=unlimited_cache))
 
 
 class ExistingPack(Pack):
@@ -1674,7 +1679,7 @@
             txt_index = self._make_index(name, '.tix')
             sig_index = self._make_index(name, '.six')
             if self.chk_index is not None:
-                chk_index = self._make_index(name, '.cix')
+                chk_index = self._make_index(name, '.cix', unlimited_cache=True)
             else:
                 chk_index = None
             result = ExistingPack(self._pack_transport, name, rev_index,
@@ -1699,7 +1704,8 @@
             txt_index = self._make_index(name, '.tix', resume=True)
             sig_index = self._make_index(name, '.six', resume=True)
             if self.chk_index is not None:
-                chk_index = self._make_index(name, '.cix', resume=True)
+                chk_index = self._make_index(name, '.cix', resume=True,
+                                             unlimited_cache=True)
             else:
                 chk_index = None
             result = self.resumed_pack_factory(name, rev_index, inv_index,
@@ -1735,7 +1741,7 @@
         return self._index_class(self.transport, 'pack-names', None
                 ).iter_all_entries()
 
-    def _make_index(self, name, suffix, resume=False):
+    def _make_index(self, name, suffix, resume=False, unlimited_cache=False):
         size_offset = self._suffix_offsets[suffix]
         index_name = name + suffix
         if resume:
@@ -1744,7 +1750,8 @@
         else:
             transport = self._index_transport
             index_size = self._names[name][size_offset]
-        return self._index_class(transport, index_name, index_size)
+        return self._index_class(transport, index_name, index_size,
+                                 unlimited_cache=unlimited_cache)
 
     def _max_pack_count(self, total_revisions):
         """Return the maximum number of packs to use for total revisions.
@@ -2300,6 +2307,9 @@
         if self._write_lock_count == 1:
             self._transaction = transactions.WriteTransaction()
         if not locked:
+            if 'relock' in debug.debug_flags and self._prev_lock == 'w':
+                note('%r was write locked again', self)
+            self._prev_lock = 'w'
             for repo in self._fallback_repositories:
                 # Writes don't affect fallback repos
                 repo.lock_read()
@@ -2312,6 +2322,9 @@
         else:
             self.control_files.lock_read()
         if not locked:
+            if 'relock' in debug.debug_flags and self._prev_lock == 'r':
+                note('%r was read locked again', self)
+            self._prev_lock = 'r'
             for repo in self._fallback_repositories:
                 repo.lock_read()
             self._refresh_data()
@@ -2345,6 +2358,7 @@
         packer = ReconcilePacker(collection, packs, extension, revs)
         return packer.pack(pb)
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         if self._write_lock_count == 1 and self._write_group is not None:
             self.abort_write_group()
 
=== modified file 'bzrlib/repository.py'
--- bzrlib/repository.py	2009-09-24 04:54:19 +0000
+++ bzrlib/repository.py	2009-10-15 18:31:14 +0000
@@ -49,7 +49,8 @@
 from bzrlib.testament import Testament
 """)
 
-from bzrlib.decorators import needs_read_lock, needs_write_lock
+from bzrlib.decorators import needs_read_lock, needs_write_lock, only_raises
+from bzrlib.lock import _RelockDebugMixin
 from bzrlib.inter import InterObject
 from bzrlib.inventory import (
     Inventory,
@@ -856,7 +857,7 @@
 # Repositories
 
 
-class Repository(object):
+class Repository(_RelockDebugMixin):
     """Repository holding history for one or more branches.
 
     The repository holds and retrieves historical information including
@@ -1381,6 +1382,7 @@
         locked = self.is_locked()
         result = self.control_files.lock_write(token=token)
         if not locked:
+            self._note_lock('w')
             for repo in self._fallback_repositories:
                 # Writes don't affect fallback repos
                 repo.lock_read()
@@ -1391,6 +1393,7 @@
         locked = self.is_locked()
         self.control_files.lock_read()
         if not locked:
+            self._note_lock('r')
             for repo in self._fallback_repositories:
                 repo.lock_read()
             self._refresh_data()
@@ -1720,6 +1723,7 @@
         self.start_write_group()
         return result
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         if (self.control_files._lock_count == 1 and
             self.control_files._lock_mode == 'w'):
@@ -4315,6 +4319,13 @@
                 ):
                 if versioned_file is None:
                     continue
+                # TODO: key is often going to be a StaticTuple object
+                #       I don't believe we can define a method by which
+                #       (prefix,) + StaticTuple will work, though we could
+                #       define a StaticTuple.sq_concat that would allow you to
+                #       pass in either a tuple or a StaticTuple as the second
+                #       object, so instead we could have:
+                #       StaticTuple(prefix) + key here...
                 missing_keys.update((prefix,) + key for key in
                     versioned_file.get_missing_compression_parent_keys())
         except NotImplementedError:
 
=== modified file 'bzrlib/send.py'
--- bzrlib/send.py	2009-07-17 14:41:02 +0000
+++ bzrlib/send.py	2009-10-15 18:31:13 +0000
@@ -115,14 +115,13 @@
                     ).get_user_option_as_bool('send_strict')
             if strict is None: strict = True # default value
             if strict and tree is not None:
-                if (tree.has_changes(tree.basis_tree())
-                    or len(tree.get_parent_ids()) > 1):
+                if (tree.has_changes()):
                     raise errors.UncommittedChanges(
                         tree, more='Use --no-strict to force the send.')
                 if tree.last_revision() != tree.branch.last_revision():
                     # The tree has lost sync with its branch, there is little
                     # chance that the user is aware of it but he can still force
-                    # the push with --no-strict
+                    # the send with --no-strict
                     raise errors.OutOfDateTree(
                         tree, more='Use --no-strict to force the send.')
             revision_id = branch.last_revision()
 
=== added file 'bzrlib/static_tuple.py'
--- bzrlib/static_tuple.py	1970-01-01 00:00:00 +0000
+++ bzrlib/static_tuple.py	2009-10-15 18:31:14 +0000
@@ -0,0 +1,25 @@
+# Copyright (C) 2009 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Interface thunk for a StaticTuple implementation."""
+
+try:
+    from bzrlib._static_tuple_c import StaticTuple
+except ImportError, e:
+    from bzrlib import osutils
+    osutils.failed_to_load_extension(e)
+    from bzrlib._static_tuple_py import StaticTuple
+
 
=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py	2009-10-08 14:59:54 +0000
+++ bzrlib/tests/__init__.py	2009-10-15 18:31:14 +0000
@@ -222,6 +222,10 @@
                 '%s is leaking threads among %d leaking tests.\n' % (
                 TestCase._first_thread_leaker_id,
                 TestCase._leaking_threads_tests))
+            # We don't report the main thread as an active one.
+            self.stream.write(
+                '%d non-main threads were left active in the end.\n'
+                % (TestCase._active_threads - 1))
 
     def _extractBenchmarkTime(self, testCase):
         """Add a benchmark time for the current test case."""
@@ -846,7 +850,13 @@
         active = threading.activeCount()
         leaked_threads = active - TestCase._active_threads
         TestCase._active_threads = active
-        if leaked_threads:
+        # If some tests make the number of threads *decrease*, we'll consider
+        # that they are just observing old threads dieing, not agressively kill
+        # random threads. So we don't report these tests as leaking. The risk
+        # is that we have false positives that way (the test see 2 threads
+        # going away but leak one) but it seems less likely than the actual
+        # false positives (the test see threads going away and does not leak).
+        if leaked_threads > 0:
             TestCase._leaking_threads_tests += 1
             if TestCase._first_thread_leaker_id is None:
                 TestCase._first_thread_leaker_id = self.id()
@@ -1146,6 +1156,25 @@
             self.fail("Incorrect length: wanted %d, got %d for %r" % (
                 length, len(obj_with_len), obj_with_len))
 
+    def assertLogsError(self, exception_class, func, *args, **kwargs):
+        """Assert that func(*args, **kwargs) quietly logs a specific exception.
+        """
+        from bzrlib import trace
+        captured = []
+        orig_log_exception_quietly = trace.log_exception_quietly
+        try:
+            def capture():
+                orig_log_exception_quietly()
+                captured.append(sys.exc_info())
+            trace.log_exception_quietly = capture
+            func(*args, **kwargs)
+        finally:
+            trace.log_exception_quietly = orig_log_exception_quietly
+        self.assertLength(1, captured)
+        err = captured[0][1]
+        self.assertIsInstance(err, exception_class)
+        return err
+
     def assertPositive(self, val):
         """Assert that val is greater than 0."""
         self.assertTrue(val > 0, 'expected a positive value, but got %s' % val)
@@ -3661,6 +3690,7 @@
         'bzrlib.tests.per_repository',
         'bzrlib.tests.per_repository_chk',
         'bzrlib.tests.per_repository_reference',
+        'bzrlib.tests.per_uifactory',
         'bzrlib.tests.per_versionedfile',
         'bzrlib.tests.per_workingtree',
         'bzrlib.tests.test__annotator',
 
=== modified file 'bzrlib/tests/blackbox/test_merge.py'
--- bzrlib/tests/blackbox/test_merge.py	2009-09-09 15:43:52 +0000
+++ bzrlib/tests/blackbox/test_merge.py	2009-10-15 18:31:13 +0000
@@ -605,7 +605,7 @@
 
     def test_merge_force(self):
         self.tree_a.commit('empty change to allow merge to run')
-        # Second merge on top if the uncommitted one
+        # Second merge on top of the uncommitted one
         self.run_bzr(['merge', '../a', '--force'], working_dir='b')
 
 
 
=== modified file 'bzrlib/tests/blackbox/test_uncommit.py'
--- bzrlib/tests/blackbox/test_uncommit.py	2009-04-04 02:50:01 +0000
+++ bzrlib/tests/blackbox/test_uncommit.py	2009-10-15 18:31:14 +0000
@@ -233,14 +233,14 @@
         tree3.commit('unchanged', rev_id='c3')
 
         wt.merge_from_branch(tree2.branch)
-        wt.merge_from_branch(tree3.branch)
+        wt.merge_from_branch(tree3.branch, force=True)
         wt.commit('merge b3, c3', rev_id='a3')
 
         tree2.commit('unchanged', rev_id='b4')
         tree3.commit('unchanged', rev_id='c4')
 
         wt.merge_from_branch(tree3.branch)
-        wt.merge_from_branch(tree2.branch)
+        wt.merge_from_branch(tree2.branch, force=True)
         wt.commit('merge b4, c4', rev_id='a4')
 
         self.assertEqual(['a4'], wt.get_parent_ids())
 
=== modified file 'bzrlib/tests/lock_helpers.py'
--- bzrlib/tests/lock_helpers.py	2009-04-15 07:30:34 +0000
+++ bzrlib/tests/lock_helpers.py	2009-10-15 18:31:14 +0000
@@ -17,6 +17,7 @@
 """Helper functions/classes for testing locking"""
 
 from bzrlib import errors
+from bzrlib.decorators import only_raises
 
 
 class TestPreventLocking(errors.LockError):
@@ -68,6 +69,7 @@
             return self._other.lock_write()
         raise TestPreventLocking('lock_write disabled')
 
+    @only_raises(errors.LockNotHeld, errors.LockBroken)
     def unlock(self):
         self._sequence.append((self._other_id, 'ul', self._allow_unlock))
         if self._allow_unlock:
 
=== modified file 'bzrlib/tests/per_branch/test_locking.py'
--- bzrlib/tests/per_branch/test_locking.py	2009-07-10 10:46:00 +0000
+++ bzrlib/tests/per_branch/test_locking.py	2009-10-15 18:31:14 +0000
@@ -139,7 +139,7 @@
         try:
             self.assertTrue(b.is_locked())
             self.assertTrue(b.repository.is_locked())
-            self.assertRaises(TestPreventLocking, b.unlock)
+            self.assertLogsError(TestPreventLocking, b.unlock)
             if self.combined_control:
                 self.assertTrue(b.is_locked())
             else:
@@ -183,7 +183,7 @@
         try:
             self.assertTrue(b.is_locked())
             self.assertTrue(b.repository.is_locked())
-            self.assertRaises(TestPreventLocking, b.unlock)
+            self.assertLogsError(TestPreventLocking, b.unlock)
             self.assertTrue(b.is_locked())
             self.assertTrue(b.repository.is_locked())
 
 
=== modified file 'bzrlib/tests/per_repository/test_write_group.py'
--- bzrlib/tests/per_repository/test_write_group.py	2009-09-08 06:25:26 +0000
+++ bzrlib/tests/per_repository/test_write_group.py	2009-10-15 18:31:14 +0000
@@ -84,7 +84,7 @@
         # don't need a specific exception for now - this is
         # really to be sure it's used right, not for signalling
         # semantic information.
-        self.assertRaises(errors.BzrError, repo.unlock)
+        self.assertLogsError(errors.BzrError, repo.unlock)
         # after this error occurs, the repository is unlocked, and the write
         # group is gone.  you've had your chance, and you blew it. ;-)
         self.assertFalse(repo.is_locked())
 
=== modified file 'bzrlib/tests/per_repository_chk/test_supported.py'
--- bzrlib/tests/per_repository_chk/test_supported.py	2009-09-08 06:25:26 +0000
+++ bzrlib/tests/per_repository_chk/test_supported.py	2009-10-15 18:31:14 +0000
@@ -17,8 +17,10 @@
 """Tests for repositories that support CHK indices."""
 
 from bzrlib import (
+    btree_index,
     errors,
     osutils,
+    repository,
     )
 from bzrlib.versionedfile import VersionedFiles
 from bzrlib.tests.per_repository_chk import TestCaseWithRepositoryCHK
@@ -108,6 +110,39 @@
         finally:
             repo.unlock()
 
+    def test_chk_bytes_are_fully_buffered(self):
+        repo = self.make_repository('.')
+        repo.lock_write()
+        self.addCleanup(repo.unlock)
+        repo.start_write_group()
+        try:
+            sha1, len, _ = repo.chk_bytes.add_lines((None,),
+                None, ["foo\n", "bar\n"], random_id=True)
+            self.assertEqual('4e48e2c9a3d2ca8a708cb0cc545700544efb5021',
+                sha1)
+            self.assertEqual(
+                set([('sha1:4e48e2c9a3d2ca8a708cb0cc545700544efb5021',)]),
+                repo.chk_bytes.keys())
+        except:
+            repo.abort_write_group()
+            raise
+        else:
+            repo.commit_write_group()
+        # This may not always be correct if we change away from BTreeGraphIndex
+        # in the future. But for now, lets check that chk_bytes are fully
+        # buffered
+        index = repo.chk_bytes._index._graph_index._indices[0]
+        self.assertIsInstance(index, btree_index.BTreeGraphIndex)
+        self.assertIs(type(index._leaf_node_cache), dict)
+        # Re-opening the repository should also have a repo with everything
+        # fully buffered
+        repo2 = repository.Repository.open(self.get_url())
+        repo2.lock_read()
+        self.addCleanup(repo2.unlock)
+        index = repo2.chk_bytes._index._graph_index._indices[0]
+        self.assertIsInstance(index, btree_index.BTreeGraphIndex)
+        self.assertIs(type(index._leaf_node_cache), dict)
+
 
 class TestCommitWriteGroupIntegrityCheck(TestCaseWithRepositoryCHK):
     """Tests that commit_write_group prevents various kinds of invalid data
 
=== added directory 'bzrlib/tests/per_uifactory'
=== added file 'bzrlib/tests/per_uifactory/__init__.py'
--- bzrlib/tests/per_uifactory/__init__.py	1970-01-01 00:00:00 +0000
+++ bzrlib/tests/per_uifactory/__init__.py	2009-10-15 18:31:14 +0000
@@ -0,0 +1,148 @@
+# Copyright (C) 2009 Canonical Ltd
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+"""Tests run per UIFactory."""
+
+# Testing UIFactories is a bit interesting because we require they all support a
+# common interface, but the way they implement it can vary very widely.  Between
+# text, batch-mode, graphical and other potential UIFactories, the requirements
+# to set up a factory, to make it respond to requests, and to simulate user
+# input can vary a lot.
+#
+# We want tests that therefore allow for the evaluation of the result to vary
+# per implementation, but we want to check that the supported facilities are
+# the same across all UIFactorys, unless they're specifically skipped.
+#
+# Our normal approach is to use test scenarios but that seems to just end up
+# creating test-like objects inside the scenario.  Therefore we fall back to
+# the older method of putting the common tests in a mixin.
+#
+# Plugins that add new UIFactorys can create their own subclasses.
+
+
+from cStringIO import StringIO
+import unittest
+
+
+from bzrlib import (
+    tests,
+    ui,
+    )
+
+
+class UIFactoryTestMixin(object):
+    """Common tests for UIFactories.
+
+    These are supposed to be expressed with no assumptions about how the
+    UIFactory implements the method, only that it does implement them (or
+    fails cleanly), and that the concrete subclass will make arrangements to
+    build a factory and to examine its behaviour.
+
+    Note that this is *not* a TestCase, because it can't be directly run, but
+    the concrete subclasses should be.
+    """
+
+    def test_note(self):
+        self.factory.note("a note to the user")
+        self._check_note("a note to the user")
+
+    def test_show_error(self):
+        msg = 'an error occurred'
+        self.factory.show_error(msg)
+        self._check_show_error(msg)
+
+    def test_show_message(self):
+        msg = 'a message'
+        self.factory.show_message(msg)
+        self._check_show_message(msg)
+
+    def test_show_warning(self):
+        msg = 'a warning'
+        self.factory.show_warning(msg)
+        self._check_show_warning(msg)
+
+
+class TestTextUIFactory(tests.TestCase, UIFactoryTestMixin):
+
+    def setUp(self):
+        super(TestTextUIFactory, self).setUp()
+        self.stdin = StringIO()
+        self.stdout = StringIO()
+        self.stderr = StringIO()
+        self.factory = ui.text.TextUIFactory(self.stdin, self.stdout,
+            self.stderr)
+
+    def _check_note(self, note_text):
+        self.assertEquals("%s\n" % note_text,
+            self.stdout.getvalue())
+
+    def _check_show_error(self, msg):
+        self.assertEquals("bzr: error: %s\n" % msg,
+            self.stderr.getvalue())
+        self.assertEquals("", self.stdout.getvalue())
+
+    def _check_show_message(self, msg):
+        self.assertEquals("%s\n" % msg,
+            self.stdout.getvalue())
+        self.assertEquals("", self.stderr.getvalue())
+
+    def _check_show_warning(self, msg):
+        self.assertEquals("bzr: warning: %s\n" % msg,
+            self.stderr.getvalue())
+        self.assertEquals("", self.stdout.getvalue())
+
+
+class TestSilentUIFactory(tests.TestCase, UIFactoryTestMixin):
+    # discards output, therefore tests for output expect nothing
+
+    def setUp(self):
+        super(TestSilentUIFactory, self).setUp()
+        self.factory = ui.SilentUIFactory()
+
+    def _check_note(self, note_text):
+        # it's just discarded
+        pass
+
+    def _check_show_error(self, msg):
+        pass
+
+    def _check_show_message(self, msg):
+        pass
+
+    def _check_show_warning(self, msg):
+        pass
+
+
+class TestCannedInputUIFactory(tests.TestCase, UIFactoryTestMixin):
+    # discards output, reads input from variables
+
+    def setUp(self):
+        super(TestCannedInputUIFactory, self).setUp()
+        self.factory = ui.CannedInputUIFactory([])
+
+    def _check_note(self, note_text):
+        pass
+
+    def _check_show_error(self, msg):
+        pass
+
+    def _check_show_message(self, msg):
+        pass
+
+    def _check_show_warning(self, msg):
+        pass
+
+
 
=== modified file 'bzrlib/tests/test__simple_set.py'
--- bzrlib/tests/test__simple_set.py	2009-10-08 04:35:01 +0000
+++ bzrlib/tests/test__simple_set.py	2009-10-15 18:31:14 +0000
@@ -30,6 +30,51 @@
     _simple_set_pyx = None
 
 
+class _Hashable(object):
+    """A simple object which has a fixed hash value.
+
+    We could have used an 'int', but it turns out that Int objects don't
+    implement tp_richcompare...
+    """
+
+    def __init__(self, the_hash):
+        self.hash = the_hash
+
+    def __hash__(self):
+        return self.hash
+
+    def __eq__(self, other):
+        if not isinstance(other, _Hashable):
+            return NotImplemented
+        return other.hash == self.hash
+
+
+class _BadSecondHash(_Hashable):
+
+    def __init__(self, the_hash):
+        _Hashable.__init__(self, the_hash)
+        self._first = True
+
+    def __hash__(self):
+        if self._first:
+            self._first = False
+            return self.hash
+        else:
+            raise ValueError('I can only be hashed once.')
+
+
+class _BadCompare(_Hashable):
+
+    def __eq__(self, other):
+        raise RuntimeError('I refuse to play nice')
+
+
+class _NoImplementCompare(_Hashable):
+
+    def __eq__(self, other):
+        return NotImplemented
+
+
 # Even though this is an extension, we don't permute the tests for a python
 # version. As the plain python version is just a dict or set
 
@@ -62,6 +107,9 @@
     def assertFillState(self, used, fill, mask, obj):
         self.assertEqual((used, fill, mask), (obj.used, obj.fill, obj.mask))
 
+    def assertLookup(self, offset, value, obj, key):
+        self.assertEqual((offset, value), obj._test_lookup(key))
+
     def assertRefcount(self, count, obj):
         """Assert that the refcount for obj is what we expect.
 
@@ -81,58 +129,88 @@
         self.assertFillState(0, 0, 0x3ff, obj)
 
     def test__lookup(self):
-        # The tuple hash function is rather good at entropy. For all integers
-        # 0=>1023, hash((i,)) & 1023 maps to a unique output, and hash((i,j))
-        # maps to all 1024 fields evenly.
-        # However, hash((c,d))& 1023 for characters has an uneven distribution
-        # of collisions, for example:
-        #  ('a', 'a'), ('f', '4'), ('p', 'r'), ('q', '1'), ('F', 'T'),
-        #  ('Q', 'Q'), ('V', 'd'), ('7', 'C')
-        # all collide @ 643
-        obj = self.module.SimpleSet()
-        offset, val = obj._test_lookup(('a', 'a'))
-        self.assertEqual(643, offset)
-        self.assertEqual('<null>', val)
-        offset, val = obj._test_lookup(('f', '4'))
-        self.assertEqual(643, offset)
-        self.assertEqual('<null>', val)
-        offset, val = obj._test_lookup(('p', 'r'))
-        self.assertEqual(643, offset)
-        self.assertEqual('<null>', val)
+        # These are carefully chosen integers to force hash collisions in the
+        # algorithm, based on the initial set size of 1024
+        obj = self.module.SimpleSet()
+        self.assertLookup(643, '<null>', obj, _Hashable(643))
+        self.assertLookup(643, '<null>', obj, _Hashable(643 + 1024))
+        self.assertLookup(643, '<null>', obj, _Hashable(643 + 50*1024))
+
+    def test__lookup_collision(self):
+        obj = self.module.SimpleSet()
+        k1 = _Hashable(643)
+        k2 = _Hashable(643 + 1024)
+        self.assertLookup(643, '<null>', obj, k1)
+        self.assertLookup(643, '<null>', obj, k2)
+        obj.add(k1)
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(644, '<null>', obj, k2)
+
+    def test__lookup_after_resize(self):
+        obj = self.module.SimpleSet()
+        k1 = _Hashable(643)
+        k2 = _Hashable(643 + 1024)
+        obj.add(k1)
+        obj.add(k2)
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(644, k2, obj, k2)
+        obj._py_resize(2047) # resized to 2048
+        self.assertEqual(2048, obj.mask + 1)
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(643+1024, k2, obj, k2)
+        obj._py_resize(1023) # resized back to 1024
+        self.assertEqual(1024, obj.mask + 1)
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(644, k2, obj, k2)
 
     def test_get_set_del_with_collisions(self):
         obj = self.module.SimpleSet()
-        k1 = ('a', 'a')
-        k2 = ('f', '4') # collides
-        k3 = ('p', 'r')
-        k4 = ('q', '1')
-        self.assertEqual((643, '<null>'), obj._test_lookup(k1))
-        self.assertEqual((643, '<null>'), obj._test_lookup(k2))
-        self.assertEqual((643, '<null>'), obj._test_lookup(k3))
-        self.assertEqual((643, '<null>'), obj._test_lookup(k4))
+
+        h1 = 643
+        h2 = 643 + 1024
+        h3 = 643 + 1024*50
+        h4 = 643 + 1024*25
+        h5 = 644
+        h6 = 644 + 1024
+
+        k1 = _Hashable(h1)
+        k2 = _Hashable(h2)
+        k3 = _Hashable(h3)
+        k4 = _Hashable(h4)
+        k5 = _Hashable(h5)
+        k6 = _Hashable(h6)
+        self.assertLookup(643, '<null>', obj, k1)
+        self.assertLookup(643, '<null>', obj, k2)
+        self.assertLookup(643, '<null>', obj, k3)
+        self.assertLookup(643, '<null>', obj, k4)
+        self.assertLookup(644, '<null>', obj, k5)
+        self.assertLookup(644, '<null>', obj, k6)
         obj.add(k1)
         self.assertIn(k1, obj)
         self.assertNotIn(k2, obj)
         self.assertNotIn(k3, obj)
         self.assertNotIn(k4, obj)
-        self.assertEqual((643, k1), obj._test_lookup(k1))
-        self.assertEqual((787, '<null>'), obj._test_lookup(k2))
-        self.assertEqual((787, '<null>'), obj._test_lookup(k3))
-        self.assertEqual((787, '<null>'), obj._test_lookup(k4))
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(644, '<null>', obj, k2)
+        self.assertLookup(644, '<null>', obj, k3)
+        self.assertLookup(644, '<null>', obj, k4)
+        self.assertLookup(644, '<null>', obj, k5)
+        self.assertLookup(644, '<null>', obj, k6)
         self.assertIs(k1, obj[k1])
-        obj.add(k2)
+        self.assertIs(k2, obj.add(k2))
         self.assertIs(k2, obj[k2])
-        self.assertEqual((643, k1), obj._test_lookup(k1))
-        self.assertEqual((787, k2), obj._test_lookup(k2))
-        self.assertEqual((660, '<null>'), obj._test_lookup(k3))
-        # Even though k4 collides for the first couple of iterations, the hash
-        # perturbation uses the full width hash (not just the masked value), so
-        # it now diverges
-        self.assertEqual((180, '<null>'), obj._test_lookup(k4))
-        self.assertEqual((643, k1), obj._test_lookup(('a', 'a')))
-        self.assertEqual((787, k2), obj._test_lookup(('f', '4')))
-        self.assertEqual((660, '<null>'), obj._test_lookup(('p', 'r')))
-        self.assertEqual((180, '<null>'), obj._test_lookup(('q', '1')))
+        self.assertLookup(643, k1, obj, k1)
+        self.assertLookup(644, k2, obj, k2)
+        self.assertLookup(646, '<null>', obj, k3)
+        self.assertLookup(646, '<null>', obj, k4)
+        self.assertLookup(645, '<null>', obj, k5)
+        self.assertLookup(645, '<null>', obj, k6)
+        self.assertLookup(643, k1, obj, _Hashable(h1))
+        self.assertLookup(644, k2, obj, _Hashable(h2))
+        self.assertLookup(646, '<null>', obj, _Hashable(h3))
+        self.assertLookup(646, '<null>', obj, _Hashable(h4))
+        self.assertLookup(645, '<null>', obj, _Hashable(h5))
+        self.assertLookup(645, '<null>', obj, _Hashable(h6))
         obj.add(k3)
         self.assertIs(k3, obj[k3])
         self.assertIn(k1, obj)
@@ -140,11 +218,11 @@
         self.assertIn(k3, obj)
         self.assertNotIn(k4, obj)
 
-        del obj[k1]
-        self.assertEqual((643, '<dummy>'), obj._test_lookup(k1))
-        self.assertEqual((787, k2), obj._test_lookup(k2))
-        self.assertEqual((660, k3), obj._test_lookup(k3))
-        self.assertEqual((643, '<dummy>'), obj._test_lookup(k4))
+        obj.discard(k1)
+        self.assertLookup(643, '<dummy>', obj, k1)
+        self.assertLookup(644, k2, obj, k2)
+        self.assertLookup(646, k3, obj, k3)
+        self.assertLookup(643, '<dummy>', obj, k4)
         self.assertNotIn(k1, obj)
         self.assertIn(k2, obj)
         self.assertIn(k3, obj)
@@ -179,7 +257,7 @@
         self.assertRefcount(2, k1)
         self.assertRefcount(1, k2)
         # Deleting an entry should remove the fill, but not the used
-        del obj[k1]
+        obj.discard(k1)
         self.assertFillState(0, 1, 0x3ff, obj)
         self.assertRefcount(1, k1)
         k3 = tuple(['bar'])
@@ -210,23 +288,6 @@
         self.assertEqual(1, obj.discard(k3))
         self.assertRefcount(1, k3)
 
-    def test__delitem__(self):
-        obj = self.module.SimpleSet()
-        k1 = tuple(['foo'])
-        k2 = tuple(['foo'])
-        k3 = tuple(['bar'])
-        self.assertRefcount(1, k1)
-        self.assertRefcount(1, k2)
-        self.assertRefcount(1, k3)
-        obj.add(k1)
-        self.assertRefcount(2, k1)
-        self.assertRaises(KeyError, obj.__delitem__, k3)
-        self.assertRefcount(1, k3)
-        obj.add(k3)
-        self.assertRefcount(2, k3)
-        del obj[k3]
-        self.assertRefcount(1, k3)
-
     def test__resize(self):
         obj = self.module.SimpleSet()
         k1 = ('foo',)
@@ -235,13 +296,13 @@
         obj.add(k1)
         obj.add(k2)
         obj.add(k3)
-        del obj[k2]
+        obj.discard(k2)
         self.assertFillState(2, 3, 0x3ff, obj)
         self.assertEqual(1024, obj._py_resize(500))
         # Doesn't change the size, but does change the content
         self.assertFillState(2, 2, 0x3ff, obj)
         obj.add(k2)
-        del obj[k3]
+        obj.discard(k3)
         self.assertFillState(2, 3, 0x3ff, obj)
         self.assertEqual(4096, obj._py_resize(4095))
         self.assertFillState(2, 2, 0xfff, obj)
@@ -250,13 +311,44 @@
         self.assertNotIn(k3, obj)
         obj.add(k2)
         self.assertIn(k2, obj)
-        del obj[k2]
+        obj.discard(k2)
         self.assertEqual((591, '<dummy>'), obj._test_lookup(k2))
         self.assertFillState(1, 2, 0xfff, obj)
         self.assertEqual(2048, obj._py_resize(1024))
         self.assertFillState(1, 1, 0x7ff, obj)
         self.assertEqual((591, '<null>'), obj._test_lookup(k2))
 
+    def test_second_hash_failure(self):
+        obj = self.module.SimpleSet()
+        k1 = _BadSecondHash(200)
+        k2 = _Hashable(200)
+        # Should only call hash() one time
+        obj.add(k1)
+        self.assertFalse(k1._first)
+        self.assertRaises(ValueError, obj.add, k2)
+
+    def test_richcompare_failure(self):
+        obj = self.module.SimpleSet()
+        k1 = _Hashable(200)
+        k2 = _BadCompare(200)
+        obj.add(k1)
+        # Tries to compare with k1, fails
+        self.assertRaises(RuntimeError, obj.add, k2)
+
+    def test_richcompare_not_implemented(self):
+        obj = self.module.SimpleSet()
+        # Even though their hashes are the same, tp_richcompare returns
+        # NotImplemented, which means we treat them as not equal
+        k1 = _NoImplementCompare(200)
+        k2 = _NoImplementCompare(200)
+        self.assertLookup(200, '<null>', obj, k1)
+        self.assertLookup(200, '<null>', obj, k2)
+        self.assertIs(k1, obj.add(k1))
+        self.assertLookup(200, k1, obj, k1)
+        self.assertLookup(201, '<null>', obj, k2)
+        self.assertIs(k2, obj.add(k2))
+        self.assertIs(k1, obj[k1])
+
     def test_add_and_remove_lots_of_items(self):
         obj = self.module.SimpleSet()
         chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890'
@@ -295,5 +387,5 @@
         # Set changed size
         self.assertRaises(RuntimeError, iterator.next)
         # And even removing an item still causes it to fail
-        del obj[k2]
+        obj.discard(k2)
         self.assertRaises(RuntimeError, iterator.next)
 
=== modified file 'bzrlib/tests/test__static_tuple.py'
--- bzrlib/tests/test__static_tuple.py	2009-10-07 19:34:22 +0000
+++ bzrlib/tests/test__static_tuple.py	2009-10-15 18:31:14 +0000
@@ -23,6 +23,7 @@
     _static_tuple_py,
     errors,
     osutils,
+    static_tuple,
     tests,
     )
 
@@ -140,6 +141,13 @@
         self.assertEqual('foo', k[0])
         self.assertEqual('z', k[6])
         self.assertEqual('z', k[-1])
+        self.assertRaises(IndexError, k.__getitem__, 7)
+        self.assertRaises(IndexError, k.__getitem__, 256+7)
+        self.assertRaises(IndexError, k.__getitem__, 12024)
+        # Python's [] resolver handles the negative arguments, so we can't
+        # really test StaticTuple_item() with negative values.
+        self.assertRaises(TypeError, k.__getitem__, 'not-an-int')
+        self.assertRaises(TypeError, k.__getitem__, '5')
 
     def test_refcount(self):
         f = 'fo' + 'oo'
@@ -201,13 +209,29 @@
         self.assertTrue(k_small <= k_big)
         self.assertTrue(k_small < k_big)
 
+    def assertCompareNoRelation(self, k1, k2):
+        """Run the comparison operators, make sure they do something.
+
+        However, we don't actually care what comes first or second. This is
+        stuff like cross-class comparisons. We don't want to segfault/raise an
+        exception, but we don't care about the sort order.
+        """
+        self.assertFalse(k1 == k2)
+        self.assertTrue(k1 != k2)
+        # Do the comparison, but we don't care about the result
+        k1 >= k2
+        k1 > k2
+        k1 <= k2
+        k1 < k2
+
     def test_compare_vs_none(self):
         k1 = self.module.StaticTuple('baz', 'bing')
         self.assertCompareDifferent(None, k1)
-        self.assertCompareDifferent(10, k1)
-        # Comparison with a string is poorly-defined, I seem to get failures
-        # regardless of which one comes first...
-        # self.assertCompareDifferent('baz', k1)
+    
+    def test_compare_cross_class(self):
+        k1 = self.module.StaticTuple('baz', 'bing')
+        self.assertCompareNoRelation(10, k1)
+        self.assertCompareNoRelation('baz', k1)
 
     def test_compare_all_different_same_width(self):
         k1 = self.module.StaticTuple('baz', 'bing')
@@ -255,6 +279,16 @@
         self.assertCompareEqual(k3, (k1, ('foo', 'bar')))
         self.assertCompareEqual((k1, ('foo', 'bar')), k3)
 
+    def test_compare_mixed_depths(self):
+        stuple = self.module.StaticTuple
+        k1 = stuple(stuple('a',), stuple('b',))
+        k2 = stuple(stuple(stuple('c',), stuple('d',)),
+                    stuple('b',))
+        # This requires comparing a StaticTuple to a 'string', and then
+        # interpreting that value in the next higher StaticTuple. This used to
+        # generate a PyErr_BadIternalCall. We now fall back to *something*.
+        self.assertCompareNoRelation(k1, k2)
+
     def test_hash(self):
         k = self.module.StaticTuple('foo')
         self.assertEqual(hash(k), hash(('foo',)))
@@ -276,12 +310,22 @@
         k = self.module.StaticTuple('foo', 'bar', 'baz', 'bing')
         self.assertEqual(('foo', 'bar'), k[:2])
         self.assertEqual(('baz',), k[2:-1])
+        try:
+            val = k[::2]
+        except TypeError:
+            # C implementation raises a TypeError, we don't need the
+            # implementation yet, so allow this to pass
+            pass
+        else:
+            # Python implementation uses a regular Tuple, so make sure it gives
+            # the right result
+            self.assertEqual(('foo', 'baz'), val)
 
     def test_referents(self):
         # We implement tp_traverse so that things like 'meliae' can measure the
         # amount of referenced memory. Unfortunately gc.get_referents() first
-        # checks the IS_GC flag before it traverses anything. So there isn't a
-        # way to expose it that I can see.
+        # checks the IS_GC flag before it traverses anything. We could write a
+        # helper func, but that won't work for the generic implementation...
         self.requireFeature(Meliae)
         from meliae import scanner
         strs = ['foo', 'bar', 'baz', 'bing']
@@ -383,3 +427,13 @@
         if self.module is _static_tuple_py:
             return
         self.assertIsNot(None, self.module._C_API)
+
+    def test_static_tuple_thunk(self):
+        # Make sure the right implementation is available from
+        # bzrlib.static_tuple.StaticTuple.
+        if self.module is _static_tuple_py:
+            if CompiledStaticTuple.available():
+                # We will be using the C version
+                return
+        self.assertIs(static_tuple.StaticTuple, 
+                      self.module.StaticTuple)
 
=== modified file 'bzrlib/tests/test_btree_index.py'
--- bzrlib/tests/test_btree_index.py	2009-08-13 19:56:26 +0000
+++ bzrlib/tests/test_btree_index.py	2009-10-15 18:31:13 +0000
@@ -23,6 +23,8 @@
 from bzrlib import (
     btree_index,
     errors,
+    fifo_cache,
+    lru_cache,
     osutils,
     tests,
     )
@@ -1115,6 +1117,43 @@
         self.assertEqual({}, parent_map)
         self.assertEqual(set([('one',), ('two',)]), missing_keys)
 
+    def test_supports_unlimited_cache(self):
+        builder = btree_index.BTreeBuilder(reference_lists=0, key_elements=1)
+        # We need enough nodes to cause a page split (so we have both an
+        # internal node and a couple leaf nodes. 500 seems to be enough.)
+        nodes = self.make_nodes(500, 1, 0)
+        for node in nodes:
+            builder.add_node(*node)
+        stream = builder.finish()
+        trans = get_transport(self.get_url())
+        size = trans.put_file('index', stream)
+        index = btree_index.BTreeGraphIndex(trans, 'index', size)
+        self.assertEqual(500, index.key_count())
+        # We have an internal node
+        self.assertEqual(2, len(index._row_lengths))
+        # We have at least 2 leaf nodes
+        self.assertTrue(index._row_lengths[-1] >= 2)
+        self.assertIsInstance(index._leaf_node_cache, lru_cache.LRUCache)
+        self.assertEqual(btree_index._NODE_CACHE_SIZE,
+                         index._leaf_node_cache._max_cache)
+        self.assertIsInstance(index._internal_node_cache, fifo_cache.FIFOCache)
+        self.assertEqual(100, index._internal_node_cache._max_cache)
+        # No change if unlimited_cache=False is passed
+        index = btree_index.BTreeGraphIndex(trans, 'index', size,
+                                            unlimited_cache=False)
+        self.assertIsInstance(index._leaf_node_cache, lru_cache.LRUCache)
+        self.assertEqual(btree_index._NODE_CACHE_SIZE,
+                         index._leaf_node_cache._max_cache)
+        self.assertIsInstance(index._internal_node_cache, fifo_cache.FIFOCache)
+        self.assertEqual(100, index._internal_node_cache._max_cache)
+        index = btree_index.BTreeGraphIndex(trans, 'index', size,
+                                            unlimited_cache=True)
+        self.assertIsInstance(index._leaf_node_cache, dict)
+        self.assertIs(type(index._internal_node_cache), dict)
+        # Exercise the lookup code
+        entries = set(index.iter_entries([n[0] for n in nodes]))
+        self.assertEqual(500, len(entries))
+
 
 class TestBTreeNodes(BTreeTestCase):
 
 
=== modified file 'bzrlib/tests/test_decorators.py'
--- bzrlib/tests/test_decorators.py	2009-03-23 14:59:43 +0000
+++ bzrlib/tests/test_decorators.py	2009-10-15 18:31:14 +0000
@@ -23,11 +23,15 @@
 from bzrlib.tests import TestCase
 
 
-def create_decorator_sample(style, except_in_unlock=False):
+class SampleUnlockError(Exception):
+    pass
+
+
+def create_decorator_sample(style, unlock_error=None):
     """Create a DecoratorSample object, using specific lock operators.
 
     :param style: The type of lock decorators to use (fast/pretty/None)
-    :param except_in_unlock: If True, raise an exception during unlock
+    :param unlock_error: If specified, an error to raise from unlock.
     :return: An instantiated DecoratorSample object.
     """
 
@@ -58,10 +62,11 @@
         def lock_write(self):
             self.actions.append('lock_write')
 
+        @decorators.only_raises(SampleUnlockError)
         def unlock(self):
-            if except_in_unlock:
+            if unlock_error:
                 self.actions.append('unlock_fail')
-                raise KeyError('during unlock')
+                raise unlock_error
             else:
                 self.actions.append('unlock')
 
@@ -119,28 +124,28 @@
 
     def test_read_lock_raises_original_error(self):
         sam = create_decorator_sample(self._decorator_style,
-                                      except_in_unlock=True)
+                                      unlock_error=SampleUnlockError())
         self.assertRaises(TypeError, sam.fail_during_read)
         self.assertEqual(['lock_read', 'fail_during_read', 'unlock_fail'],
                          sam.actions)
 
     def test_write_lock_raises_original_error(self):
         sam = create_decorator_sample(self._decorator_style,
-                                      except_in_unlock=True)
+                                      unlock_error=SampleUnlockError())
         self.assertRaises(TypeError, sam.fail_during_write)
         self.assertEqual(['lock_write', 'fail_during_write', 'unlock_fail'],
                          sam.actions)
 
     def test_read_lock_raises_unlock_error(self):
         sam = create_decorator_sample(self._decorator_style,
-                                      except_in_unlock=True)
-        self.assertRaises(KeyError, sam.frob)
+                                      unlock_error=SampleUnlockError())
+        self.assertRaises(SampleUnlockError, sam.frob)
         self.assertEqual(['lock_read', 'frob', 'unlock_fail'], sam.actions)
 
     def test_write_lock_raises_unlock_error(self):
         sam = create_decorator_sample(self._decorator_style,
-                                      except_in_unlock=True)
-        self.assertRaises(KeyError, sam.bank, 'bar', biz='bing')
+                                      unlock_error=SampleUnlockError())
+        self.assertRaises(SampleUnlockError, sam.bank, 'bar', biz='bing')
         self.assertEqual(['lock_write', ('bank', 'bar', 'bing'),
                           'unlock_fail'], sam.actions)
 
@@ -276,3 +281,21 @@
         finally:
             decorators.needs_read_lock = cur_read
             decorators.needs_write_lock = cur_write
+
+
+class TestOnlyRaisesDecorator(TestCase):
+
+    def raise_ZeroDivisionError(self):
+        1/0
+        
+    def test_raises_approved_error(self):
+        decorator = decorators.only_raises(ZeroDivisionError)
+        decorated_meth = decorator(self.raise_ZeroDivisionError)
+        self.assertRaises(ZeroDivisionError, decorated_meth)
+
+    def test_quietly_logs_unapproved_errors(self):
+        decorator = decorators.only_raises(IOError)
+        decorated_meth = decorator(self.raise_ZeroDivisionError)
+        self.assertLogsError(ZeroDivisionError, decorated_meth)
+        
+
 
=== modified file 'bzrlib/tests/test_diff.py'
--- bzrlib/tests/test_diff.py	2009-03-31 00:12:10 +0000
+++ bzrlib/tests/test_diff.py	2009-10-15 18:31:14 +0000
@@ -32,6 +32,7 @@
     external_diff,
     internal_diff,
     show_diff_trees,
+    get_trees_and_branches_to_diff,
     )
 from bzrlib.errors import BinaryFile, NoDiff, ExecutableMissing
 import bzrlib.osutils as osutils
@@ -41,6 +42,8 @@
 import bzrlib._patiencediff_py
 from bzrlib.tests import (Feature, TestCase, TestCaseWithTransport,
                           TestCaseInTempDir, TestSkipped)
+from bzrlib.revisiontree import RevisionTree
+from bzrlib.revisionspec import RevisionSpec
 
 
 class _AttribFeature(Feature):
@@ -1382,3 +1385,46 @@
             self.assertTrue(os.path.samefile('tree/newname', new_path))
         # make sure we can create files with the same parent directories
         diff_obj._prepare_files('file2-id', 'oldname2', 'newname2')
+
+
+class TestGetTreesAndBranchesToDiff(TestCaseWithTransport):
+
+    def test_basic(self):
+        tree = self.make_branch_and_tree('tree')
+        (old_tree, new_tree,
+         old_branch, new_branch,
+         specific_files, extra_trees) = \
+            get_trees_and_branches_to_diff(['tree'], None, None, None)
+
+        self.assertIsInstance(old_tree, RevisionTree)
+        #print dir (old_tree)
+        self.assertEqual(_mod_revision.NULL_REVISION, old_tree.get_revision_id())
+        self.assertEqual(tree.basedir, new_tree.basedir)
+        self.assertEqual(tree.branch.base, old_branch.base)
+        self.assertEqual(tree.branch.base, new_branch.base)
+        self.assertIs(None, specific_files)
+        self.assertIs(None, extra_trees)
+
+    def test_with_rev_specs(self):
+        tree = self.make_branch_and_tree('tree')
+        self.build_tree_contents([('tree/file', 'oldcontent')])
+        tree.add('file', 'file-id')
+        tree.commit('old tree', timestamp=0, rev_id="old-id")
+        self.build_tree_contents([('tree/file', 'newcontent')])
+        tree.commit('new tree', timestamp=0, rev_id="new-id")
+
+        revisions = [RevisionSpec.from_string('1'),
+                     RevisionSpec.from_string('2')]
+        (old_tree, new_tree,
+         old_branch, new_branch,
+         specific_files, extra_trees) = \
+            get_trees_and_branches_to_diff(['tree'], revisions, None, None)
+
+        self.assertIsInstance(old_tree, RevisionTree)
+        self.assertEqual("old-id", old_tree.get_revision_id())
+        self.assertIsInstance(new_tree, RevisionTree)
+        self.assertEqual("new-id", new_tree.get_revision_id())
+        self.assertEqual(tree.branch.base, old_branch.base)
+        self.assertEqual(tree.branch.base, new_branch.base)
+        self.assertIs(None, specific_files)
+        self.assertEqual(tree.basedir, extra_trees[0].basedir)
 
=== modified file 'bzrlib/tests/test_index.py'
--- bzrlib/tests/test_index.py	2009-08-13 19:56:26 +0000
+++ bzrlib/tests/test_index.py	2009-10-15 18:31:14 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2007 Canonical Ltd
+# Copyright (C) 2007, 2009 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -1006,6 +1006,15 @@
         self.assertEqual(set(), missing_keys)
         self.assertEqual(set(), search_keys)
 
+    def test_supports_unlimited_cache(self):
+        builder = GraphIndexBuilder(0, key_elements=1)
+        stream = builder.finish()
+        trans = get_transport(self.get_url())
+        size = trans.put_file('index', stream)
+        # It doesn't matter what unlimited_cache does here, just that it can be
+        # passed
+        index = GraphIndex(trans, 'index', size, unlimited_cache=True)
+
 
 class TestCombinedGraphIndex(TestCaseWithMemoryTransport):
 
 
=== modified file 'bzrlib/tests/test_msgeditor.py'
--- bzrlib/tests/test_msgeditor.py	2009-08-20 04:09:58 +0000
+++ bzrlib/tests/test_msgeditor.py	2009-10-15 18:31:14 +0000
@@ -93,7 +93,7 @@
         tree3.commit('Feature Y, based on initial X work.',
                      timestamp=1233285960, timezone=0)
         tree.merge_from_branch(tree2.branch)
-        tree.merge_from_branch(tree3.branch)
+        tree.merge_from_branch(tree3.branch, force=True)
         return tree
 
     def test_commit_template_pending_merges(self):
 
=== modified file 'bzrlib/tests/test_mutabletree.py'
--- bzrlib/tests/test_mutabletree.py	2009-09-07 23:14:05 +0000
+++ bzrlib/tests/test_mutabletree.py	2009-10-15 18:31:13 +0000
@@ -19,15 +19,18 @@
 Most functionality of MutableTree is tested as part of WorkingTree.
 """
 
-from bzrlib.tests import TestCase
-from bzrlib.mutabletree import MutableTree, MutableTreeHooks
-
-class TestHooks(TestCase):
+from bzrlib import (
+    mutabletree,
+    tests,
+    )
+
+
+class TestHooks(tests.TestCase):
 
     def test_constructor(self):
         """Check that creating a MutableTreeHooks instance has the right
         defaults."""
-        hooks = MutableTreeHooks()
+        hooks = mutabletree.MutableTreeHooks()
         self.assertTrue("start_commit" in hooks,
                         "start_commit not in %s" % hooks)
         self.assertTrue("post_commit" in hooks,
@@ -36,7 +39,25 @@
     def test_installed_hooks_are_MutableTreeHooks(self):
         """The installed hooks object should be a MutableTreeHooks."""
         # the installed hooks are saved in self._preserved_hooks.
-        self.assertIsInstance(self._preserved_hooks[MutableTree][1],
-                              MutableTreeHooks)
-
-
+        self.assertIsInstance(self._preserved_hooks[mutabletree.MutableTree][1],
+                              mutabletree.MutableTreeHooks)
+
+
+class TestHasChanges(tests.TestCaseWithTransport):
+
+    def setUp(self):
+        super(TestHasChanges, self).setUp()
+        self.tree = self.make_branch_and_tree('tree')
+
+    def test_with_uncommitted_changes(self):
+        self.build_tree(['tree/file'])
+        self.tree.add('file')
+        self.assertTrue(self.tree.has_changes())
+
+    def test_with_pending_merges(self):
+        other_tree = self.tree.bzrdir.sprout('other').open_workingtree()
+        self.build_tree(['other/file'])
+        other_tree.add('file')
+        other_tree.commit('added file')
+        self.tree.merge_from_branch(other_tree.branch)
+        self.assertTrue(self.tree.has_changes())
 
=== modified file 'bzrlib/tests/test_osutils.py'
--- bzrlib/tests/test_osutils.py	2009-09-19 16:14:10 +0000
+++ bzrlib/tests/test_osutils.py	2009-10-15 18:31:14 +0000
@@ -457,6 +457,49 @@
         self.failUnlessEqual('work/MixedCaseParent/nochild', actual)
 
 
+class Test_CICPCanonicalRelpath(tests.TestCaseWithTransport):
+
+    def assertRelpath(self, expected, base, path):
+        actual = osutils._cicp_canonical_relpath(base, path)
+        self.assertEqual(expected, actual)
+
+    def test_simple(self):
+        self.build_tree(['MixedCaseName'])
+        base = osutils.realpath(self.get_transport('.').local_abspath('.'))
+        self.assertRelpath('MixedCaseName', base, 'mixedcAsename')
+
+    def test_subdir_missing_tail(self):
+        self.build_tree(['MixedCaseParent/', 'MixedCaseParent/a_child'])
+        base = osutils.realpath(self.get_transport('.').local_abspath('.'))
+        self.assertRelpath('MixedCaseParent/a_child', base,
+                           'MixedCaseParent/a_child')
+        self.assertRelpath('MixedCaseParent/a_child', base,
+                           'MixedCaseParent/A_Child')
+        self.assertRelpath('MixedCaseParent/not_child', base,
+                           'MixedCaseParent/not_child')
+
+    def test_at_root_slash(self):
+        # We can't test this on Windows, because it has a 'MIN_ABS_PATHLENGTH'
+        # check...
+        if osutils.MIN_ABS_PATHLENGTH > 1:
+            raise tests.TestSkipped('relpath requires %d chars'
+                                    % osutils.MIN_ABS_PATHLENGTH)
+        self.assertRelpath('foo', '/', '/foo')
+
+    def test_at_root_drive(self):
+        if sys.platform != 'win32':
+            raise tests.TestNotApplicable('we can only test drive-letter relative'
+                                          ' paths on Windows where we have drive'
+                                          ' letters.')
+        # see bug #322807
+        # The specific issue is that when at the root of a drive, 'abspath'
+        # returns "C:/" or just "/". However, the code assumes that abspath
+        # always returns something like "C:/foo" or "/foo" (no trailing slash).
+        self.assertRelpath('foo', 'C:/', 'C:/foo')
+        self.assertRelpath('foo', 'X:/', 'X:/foo')
+        self.assertRelpath('foo', 'X:/', 'X://foo')
+
+
 class TestPumpFile(tests.TestCase):
     """Test pumpfile method."""
 
 
=== modified file 'bzrlib/tests/test_reconfigure.py'
--- bzrlib/tests/test_reconfigure.py	2009-04-28 20:12:44 +0000
+++ bzrlib/tests/test_reconfigure.py	2009-10-15 18:31:14 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2007 Canonical Ltd
+# Copyright (C) 2007, 2008, 2009 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -44,6 +44,19 @@
         self.assertRaises(errors.NoWorkingTree, workingtree.WorkingTree.open,
                           'tree')
 
+    def test_tree_with_pending_merge_to_branch(self):
+        tree = self.make_branch_and_tree('tree')
+        other_tree = tree.bzrdir.sprout('other').open_workingtree()
+        self.build_tree(['other/file'])
+        other_tree.add('file')
+        other_tree.commit('file added')
+        tree.merge_from_branch(other_tree.branch)
+        reconfiguration = reconfigure.Reconfigure.to_branch(tree.bzrdir)
+        self.assertRaises(errors.UncommittedChanges, reconfiguration.apply)
+        reconfiguration.apply(force=True)
+        self.assertRaises(errors.NoWorkingTree, workingtree.WorkingTree.open,
+                          'tree')
+
     def test_branch_to_branch(self):
         branch = self.make_branch('branch')
         self.assertRaises(errors.AlreadyBranch,
 
=== modified file 'bzrlib/tests/test_remote.py'
--- bzrlib/tests/test_remote.py	2009-09-24 05:31:23 +0000
+++ bzrlib/tests/test_remote.py	2009-10-15 18:31:14 +0000
@@ -2201,6 +2201,26 @@
             repo.get_rev_id_for_revno, 5, (42, 'rev-foo'))
         self.assertFinished(client)
 
+    def test_branch_fallback_locking(self):
+        """RemoteBranch.get_rev_id takes a read lock, and tries to call the
+        get_rev_id_for_revno verb.  If the verb is unknown the VFS fallback
+        will be invoked, which will fail if the repo is unlocked.
+        """
+        self.setup_smart_server_with_call_log()
+        tree = self.make_branch_and_memory_tree('.')
+        tree.lock_write()
+        rev1 = tree.commit('First')
+        rev2 = tree.commit('Second')
+        tree.unlock()
+        branch = tree.branch
+        self.assertFalse(branch.is_locked())
+        self.reset_smart_call_log()
+        verb = 'Repository.get_rev_id_for_revno'
+        self.disable_verb(verb)
+        self.assertEqual(rev1, branch.get_rev_id(1))
+        self.assertLength(1, [call for call in self.hpss_calls if
+                              call.call.method == verb])
+
 
 class TestRepositoryIsShared(TestRemoteRepository):
 
 
=== modified file 'bzrlib/tests/test_status.py'
--- bzrlib/tests/test_status.py	2009-08-20 04:09:58 +0000
+++ bzrlib/tests/test_status.py	2009-10-15 18:31:13 +0000
@@ -53,7 +53,7 @@
         tree2.commit('commit 3b', timestamp=1196796819, timezone=0)
         tree3.commit('commit 3c', timestamp=1196796819, timezone=0)
         tree.merge_from_branch(tree2.branch)
-        tree.merge_from_branch(tree3.branch)
+        tree.merge_from_branch(tree3.branch, force=True)
         return tree
 
     def test_multiple_pending(self):
 
=== modified file 'bzrlib/tests/test_ui.py'
--- bzrlib/tests/test_ui.py	2009-09-24 08:56:52 +0000
+++ bzrlib/tests/test_ui.py	2009-10-15 18:31:14 +0000
@@ -54,7 +54,7 @@
     )
 
 
-class UITests(tests.TestCase):
+class TestTextUIFactory(tests.TestCase):
 
     def test_text_factory_ascii_password(self):
         ui = tests.TestUIFactory(stdin='secret\n',
@@ -100,56 +100,6 @@
         finally:
             pb.finished()
 
-    def test_progress_construction(self):
-        """TextUIFactory constructs the right progress view.
-        """
-        for (file_class, term, pb, expected_pb_class) in (
-            # on an xterm, either use them or not as the user requests,
-            # otherwise default on
-            (_TTYStringIO, 'xterm', 'none', NullProgressView),
-            (_TTYStringIO, 'xterm', 'text', TextProgressView),
-            (_TTYStringIO, 'xterm', None, TextProgressView),
-            # on a dumb terminal, again if there's explicit configuration do
-            # it, otherwise default off
-            (_TTYStringIO, 'dumb', 'none', NullProgressView),
-            (_TTYStringIO, 'dumb', 'text', TextProgressView),
-            (_TTYStringIO, 'dumb', None, NullProgressView),
-            # on a non-tty terminal, it's null regardless of $TERM
-            (StringIO, 'xterm', None, NullProgressView),
-            (StringIO, 'dumb', None, NullProgressView),
-            # however, it can still be forced on
-            (StringIO, 'dumb', 'text', TextProgressView),
-            ):
-            os.environ['TERM'] = term
-            if pb is None:
-                if 'BZR_PROGRESS_BAR' in os.environ:
-                    del os.environ['BZR_PROGRESS_BAR']
-            else:
-                os.environ['BZR_PROGRESS_BAR'] = pb
-            stdin = file_class('')
-            stderr = file_class()
-            stdout = file_class()
-            uif = make_ui_for_terminal(stdin, stdout, stderr)
-            self.assertIsInstance(uif, TextUIFactory,
-                "TERM=%s BZR_PROGRESS_BAR=%s uif=%r" % (term, pb, uif,))
-            self.assertIsInstance(uif.make_progress_view(),
-                expected_pb_class,
-                "TERM=%s BZR_PROGRESS_BAR=%s uif=%r" % (term, pb, uif,))
-
-    def test_text_ui_non_terminal(self):
-        """Even on non-ttys, make_ui_for_terminal gives a text ui."""
-        stdin = _NonTTYStringIO('')
-        stderr = _NonTTYStringIO()
-        stdout = _NonTTYStringIO()
-        for term_type in ['dumb', None, 'xterm']:
-            if term_type is None:
-                del os.environ['TERM']
-            else:
-                os.environ['TERM'] = term_type
-            uif = make_ui_for_terminal(stdin, stdout, stderr)
-            self.assertIsInstance(uif, TextUIFactory,
-                'TERM=%r' % (term_type,))
-
     def test_progress_note(self):
         stderr = StringIO()
         stdout = StringIO()
@@ -304,6 +254,59 @@
             pb.finished()
 
 
+class UITests(tests.TestCase):
+
+    def test_progress_construction(self):
+        """TextUIFactory constructs the right progress view.
+        """
+        for (file_class, term, pb, expected_pb_class) in (
+            # on an xterm, either use them or not as the user requests,
+            # otherwise default on
+            (_TTYStringIO, 'xterm', 'none', NullProgressView),
+            (_TTYStringIO, 'xterm', 'text', TextProgressView),
+            (_TTYStringIO, 'xterm', None, TextProgressView),
+            # on a dumb terminal, again if there's explicit configuration do
+            # it, otherwise default off
+            (_TTYStringIO, 'dumb', 'none', NullProgressView),
+            (_TTYStringIO, 'dumb', 'text', TextProgressView),
+            (_TTYStringIO, 'dumb', None, NullProgressView),
+            # on a non-tty terminal, it's null regardless of $TERM
+            (StringIO, 'xterm', None, NullProgressView),
+            (StringIO, 'dumb', None, NullProgressView),
+            # however, it can still be forced on
+            (StringIO, 'dumb', 'text', TextProgressView),
+            ):
+            os.environ['TERM'] = term
+            if pb is None:
+                if 'BZR_PROGRESS_BAR' in os.environ:
+                    del os.environ['BZR_PROGRESS_BAR']
+            else:
+                os.environ['BZR_PROGRESS_BAR'] = pb
+            stdin = file_class('')
+            stderr = file_class()
+            stdout = file_class()
+            uif = make_ui_for_terminal(stdin, stdout, stderr)
+            self.assertIsInstance(uif, TextUIFactory,
+                "TERM=%s BZR_PROGRESS_BAR=%s uif=%r" % (term, pb, uif,))
+            self.assertIsInstance(uif.make_progress_view(),
+                expected_pb_class,
+                "TERM=%s BZR_PROGRESS_BAR=%s uif=%r" % (term, pb, uif,))
+
+    def test_text_ui_non_terminal(self):
+        """Even on non-ttys, make_ui_for_terminal gives a text ui."""
+        stdin = _NonTTYStringIO('')
+        stderr = _NonTTYStringIO()
+        stdout = _NonTTYStringIO()
+        for term_type in ['dumb', None, 'xterm']:
+            if term_type is None:
+                del os.environ['TERM']
+            else:
+                os.environ['TERM'] = term_type
+            uif = make_ui_for_terminal(stdin, stdout, stderr)
+            self.assertIsInstance(uif, TextUIFactory,
+                'TERM=%r' % (term_type,))
+
+
 class CLIUITests(TestCase):
 
     def test_cli_factory_deprecated(self):
 
=== modified file 'bzrlib/ui/__init__.py'
--- bzrlib/ui/__init__.py	2009-07-22 07:34:08 +0000
+++ bzrlib/ui/__init__.py	2009-10-15 18:31:14 +0000
@@ -22,18 +22,18 @@
 Several levels are supported, and you can also register new factories such as
 for a GUI.
 
-UIFactory
+bzrlib.ui.UIFactory
     Semi-abstract base class
 
-SilentUIFactory
+bzrlib.ui.SilentUIFactory
     Produces no output and cannot take any input; useful for programs using
     bzrlib in batch mode or for programs such as loggerhead.
 
-CannedInputUIFactory
+bzrlib.ui.CannedInputUIFactory
     For use in testing; the input values to be returned are provided 
     at construction.
 
-TextUIFactory
+bzrlib.ui.text.TextUIFactory
     Standard text command-line interface, with stdin, stdout, stderr.
     May make more or less advanced use of them, eg in drawing progress bars,
     depending on the detected capabilities of the terminal.
@@ -208,6 +208,22 @@
         """
         pass
 
+    def show_error(self, msg):
+        """Show an error message (not an exception) to the user.
+        
+        The message should not have an error prefix or trailing newline.  That
+        will be added by the factory if appropriate. 
+        """
+        raise NotImplementedError(self.show_error)
+
+    def show_message(self, msg):
+        """Show a message to the user."""
+        raise NotImplementedError(self.show_message)
+
+    def show_warning(self, msg):
+        """Show a warning to the user."""
+        raise NotImplementedError(self.show_warning)
+
 
 
 class CLIUIFactory(UIFactory):
@@ -318,6 +334,15 @@
     def get_username(self, prompt, **kwargs):
         return None
 
+    def show_error(self, msg):
+        pass
+
+    def show_message(self, msg):
+        pass
+
+    def show_warning(self, msg):
+        pass
+
 
 class CannedInputUIFactory(SilentUIFactory):
     """A silent UI that return canned input."""
 
=== modified file 'bzrlib/ui/text.py'
--- bzrlib/ui/text.py	2009-08-06 02:23:37 +0000
+++ bzrlib/ui/text.py	2009-10-15 18:31:14 +0000
@@ -49,9 +49,6 @@
                  stdout=None,
                  stderr=None):
         """Create a TextUIFactory.
-
-        :param bar_type: The type of progress bar to create.  Deprecated
-            and ignored; a TextProgressView is always used.
         """
         super(TextUIFactory, self).__init__()
         # TODO: there's no good reason not to pass all three streams, maybe we
@@ -176,6 +173,17 @@
         self._progress_view.show_transport_activity(transport,
             direction, byte_count)
 
+    def show_error(self, msg):
+        self.clear_term()
+        self.stderr.write("bzr: error: %s\n" % msg)
+
+    def show_message(self, msg):
+        self.note(msg)
+
+    def show_warning(self, msg):
+        self.clear_term()
+        self.stderr.write("bzr: warning: %s\n" % msg)
+
     def _progress_updated(self, task):
         """A task has been updated and wants to be displayed.
         """
 
=== modified file 'bzrlib/util/_bencode_py.py'
--- bzrlib/util/_bencode_py.py	2009-06-10 03:56:49 +0000
+++ bzrlib/util/_bencode_py.py	2009-10-15 18:31:14 +0000
@@ -154,6 +154,13 @@
         encode_int(int(x), r)
     encode_func[BooleanType] = encode_bool
 
+try:
+    from bzrlib._static_tuple_c import StaticTuple
+except ImportError:
+    pass
+else:
+    encode_func[StaticTuple] = encode_list
+
 
 def bencode(x):
     r = []
 
=== modified file 'bzrlib/workingtree.py'
--- bzrlib/workingtree.py	2009-08-26 05:38:16 +0000
+++ bzrlib/workingtree.py	2009-10-15 18:31:14 +0000
@@ -896,7 +896,7 @@
 
     @needs_write_lock # because merge pulls data into the branch.
     def merge_from_branch(self, branch, to_revision=None, from_revision=None,
-        merge_type=None):
+                          merge_type=None, force=False):
         """Merge from a branch into this working tree.
 
         :param branch: The branch to merge from.
@@ -911,9 +911,9 @@
             merger = Merger(self.branch, this_tree=self, pb=pb)
             merger.pp = ProgressPhase("Merge phase", 5, pb)
             merger.pp.next_phase()
-            # check that there are no
-            # local alterations
-            merger.check_basis(check_clean=True, require_commits=False)
+            # check that there are no local alterations
+            if not force and self.has_changes():
+                raise errors.UncommittedChanges(self)
             if to_revision is None:
                 to_revision = _mod_revision.ensure_null(branch.last_revision())
             merger.other_rev_id = to_revision
 
=== modified file 'doc/developers/HACKING.txt'
--- doc/developers/HACKING.txt	2009-09-30 14:56:21 +0000
+++ doc/developers/HACKING.txt	2009-10-15 18:31:14 +0000
@@ -671,6 +671,19 @@
     may not catch every case but it's still useful sometimes.
 
 
+Cleanup methods
+===============
+
+Often when something has failed later code, including cleanups invoked
+from ``finally`` blocks, will fail too.  These secondary failures are
+generally uninteresting compared to the original exception.  So use the
+``only_raises`` decorator (from ``bzrlib.decorators``) for methods that
+are typically called in ``finally`` blocks, such as ``unlock`` methods.
+For example, ``@only_raises(LockNotHeld, LockBroken)``.  All errors that
+are unlikely to be a knock-on failure from an previous failure should be
+allowed.
+
+
 Factories
 =========
 
 
=== modified file 'doc/developers/releasing.txt'
--- doc/developers/releasing.txt	2009-09-16 08:59:26 +0000
+++ doc/developers/releasing.txt	2009-10-15 18:31:14 +0000
@@ -212,9 +212,20 @@
 we have a releasable product.  The next step is to make it generally
 available to the world.
 
-#. Link from http://bazaar-vcs.org/Download to the tarball and signature.
-
-#. Announce on the `Bazaar home page <http://bazaar-vcs.org/>`_.
+go to the release
+
+#. Within that release, upload the source tarball and zipfile and the GPG
+   signature.  Or, if you prefer, use the
+   ``tools/packaging/lp-upload-release`` script to do this.
+
+#. Link from http://bazaar-vcs.org/SourceDownloads to the tarball and
+   signature.
+
+#. Announce on the `Bazaar website <http://bazaar-vcs.org/>`_.
+   This page is edited via the lp:bzr-website branch. (Changes
+   pushed to this branch are refreshed by a cron job on escudero.)
+
+#. Announce on the `Bazaar wiki <http://bazaar-vcs.org/Welcome>`_.
 
 #. Check that the documentation for this release is available in
    <http://doc.bazaar-vcs.org>.  It should be automatically build when the
 
=== modified file 'doc/en/upgrade-guide/data_migration.txt'
--- doc/en/upgrade-guide/data_migration.txt	2009-09-09 13:34:08 +0000
+++ doc/en/upgrade-guide/data_migration.txt	2009-10-15 18:31:13 +0000
@@ -30,7 +30,7 @@
 * **upgrade** - migrate data to a different format.
 
 **reconcile** is rarely needed but it's good practice to run **check**
-before and after runing **upgrade**.
+before and after running **upgrade**.
 
 For detailed help on these commands, see the `Bazaar User Reference`_.
 
@@ -40,7 +40,7 @@
 Communicating with your community
 ---------------------------------
 
-To enable a smooth transistion to the new format, you should:
+To enable a smooth transition to the new format, you should:
 
 1. Make one person responsible for migrating the trunk.
 
@@ -97,22 +97,54 @@
 Migrating branches on Launchpad
 -------------------------------
 
+You have two options for upgrading your Launchpad branches.  You can either
+upgrade them remotely or you can upgrade them locally and push the migrated
+branch to Launchpad.  We recommend the latter.  Upgrading remotely currently
+requires a fast, rock solid network connection to the Launchpad servers, and
+any interruption in that connection can leave you with a partially upgraded
+branch.  The instructions below are the safest and often fastest way to
+upgrade your Launchpad branches.
+
 To allow isolation between public and private branches, Launchpad
 uses stacked branches rather than shared repositories as the core
 technology for efficient branch storage. The process for migrating
 to a new format for projects using Launchpad code hosting is therefore
 different to migrating a personal or in-house project.
 
+In Launchpad, a project can define a *development series* and associate a
+branch with that series.  The branch then becomes the *focus of development*
+and gets special treatment and a shortcut url.  By default, if anybody
+branches your project's focus of development and pushes changes back to
+Launchpad, their branch will be stacked on your development focus branch.
+Also, branches can be associated with other Launchpad artifacts such as bugs
+and merge proposals.  All of these things mean that upgrading your focus of
+development branch is trickier.
+
 Here are the steps to follow:
 
-1. The nominated person grabs a copy of trunk and does the migration.
+1. The nominated person grabs a copy of trunk and does the migration locally.
 
 2. On Launchpad, unset the current trunk from being the development focus.
    (This *must* be done or the following step won't work as expected.)
 
-3. Push the migrated trunk to Launchpad.
-
-4. Set it as the development focus.
+   1. Go to your project's home page on Launchpad
+
+   2. Look for "XXX is the current focus of development"
+
+   3. Click on the edit (pencil) icon
+
+   4. Click on "Change details" in the portlet on the right
+
+   5. Scroll down to where it says "Branch: (Optional)"
+
+   6. Blank out this input field and click "Change"
+
+3. Push the migrated trunk to Launchpad.  See below if you want your
+   new migrated development focus branch to have the same name as your old
+   pre-migration development focus branch.
+
+4. Set it as the development focus.  Follow the instructions above but at step
+   5, enter the name of the newly migrated branch you just pushed.
 
 5. Ask users subscribed to the old trunk to subscribe to the new one.
 
@@ -124,6 +156,20 @@
 You are now ready to tell your community that the new trunk is available
 and to give them instructions on migrating any local branches they have.
 
+If you want your new migrated development focus branch to have the same name
+as your old pre-migration branch, you need to do a few extra things before you
+establish the new development focus.
+
+1. Rename your old pre-migration branch; use something like
+   **foo-obsolete-do-not-use**.  You will really not want to delete this
+   because there will be artifacts (bugs, merge proposals, etc.) associated
+   with it.
+
+2. Rename the new migrated branch to the pre-migration branch's old name.
+
+3. Re-establish the development focus branch using the new migrated branch's
+   new name (i.e. the old pre-migration branch's original name).
+
 
 Migrating local branches after a central trunk has migrated
 -----------------------------------------------------------
 
=== modified file 'doc/en/user-guide/branching_a_project.txt'
--- doc/en/user-guide/branching_a_project.txt	2009-09-09 15:30:59 +0000
+++ doc/en/user-guide/branching_a_project.txt	2009-10-15 18:31:14 +0000
@@ -16,11 +16,11 @@
   ===========         ======================================================
   Prefix              Description
   ===========         ======================================================
-  file://             Access using the standard filesystem (default)
-  sftp://             Access using SFTP (most SSH servers provide SFTP).
-  bzr://              Fast access using the Bazaar smart server.
-  ftp://              Access using passive FTP.
-  http://             Read-only access to branches exported by a web server.
+  \file://            Access using the standard filesystem (default)
+  \sftp://            Access using SFTP (most SSH servers provide SFTP).
+  \bzr://             Fast access using the Bazaar smart server.
+  \ftp://             Access using passive FTP.
+  \http://            Read-only access to branches exported by a web server.
   ===========         ======================================================
 
 As indicated above, branches are identified using URLs with the
 
=== modified file 'setup.py'
--- setup.py	2009-10-08 15:44:41 +0000
+++ setup.py	2009-10-15 18:31:14 +0000
@@ -167,7 +167,13 @@
 from distutils.extension import Extension
 ext_modules = []
 try:
-    from Pyrex.Distutils import build_ext
+    try:
+        from Pyrex.Distutils import build_ext
+        from Pyrex.Compiler.Version import version as pyrex_version
+    except ImportError:
+        print "No Pyrex, trying Cython..."
+        from Cython.Distutils import build_ext
+        from Cython.Compiler.Version import version as pyrex_version
 except ImportError:
     have_pyrex = False
     # try to build the extension from the prior generated source.
@@ -180,7 +186,6 @@
     from distutils.command.build_ext import build_ext
 else:
     have_pyrex = True
-    from Pyrex.Compiler.Version import version as pyrex_version
 
 
 class build_ext_if_possible(build_ext):