Merge lp:~mbp/bzr/715000-stacking into lp:bzr

Proposed by Martin Pool on 2011-02-08
Status: Merged
Merged at revision: 5657
Proposed branch: lp:~mbp/bzr/715000-stacking
Merge into: lp:bzr
Diff against target: 243 lines (+146/-3) (has conflicts)
8 files modified
bzr (+4/-0)
bzrlib/__init__.py (+4/-0)
bzrlib/groupcompress.py (+2/-2)
bzrlib/knit.py (+1/-1)
bzrlib/tests/per_repository_reference/__init__.py (+1/-0)
bzrlib/tests/per_repository_reference/test_graph.py (+45/-0)
bzrlib/versionedfile.py (+13/-0)
doc/en/release-notes/bzr-2.3.txt (+76/-0)
Text conflict in bzr
Text conflict in bzrlib/__init__.py
Text conflict in doc/en/release-notes/bzr-2.3.txt
To merge this branch: bzr merge lp:~mbp/bzr/715000-stacking
Reviewer Review Type Date Requested Status
Robert Collins (community) 2011-02-08 Approve on 2011-02-08
Review via email: mp+48886@code.launchpad.net

Description of the change

This tries to fix bug 715000 by making sure we get the graph from all fallback repositories.

This is done off lp:bzr/2.2 to keep our options open, but perhaps we should just merge it to 2.3 or 2.4 and deploy that to Launchpad.

To post a comment you must log in.
Andrew Bennetts (spiv) wrote :

Some quick comments (rather than a comprehensive review):

1. "for br in [branch_c]:" is a weird expression (why have a loop when there's exactly 1 item?)

2. this fix appears reasonable, but you only fix one method when there are many other uses of self._fallback_vfs in e.g. GroupCompressVersionedFiles. Either there are more bugs waiting to be fixed here (hopefully easy ones to reveal with tests similar to the one you add in this patch), or the reason why only get_known_graph_ancestry was broken is subtle/unclear. Do you know which? I'm a bit worried that this may correct the immediate traceback without actually making the Launchpad scanner functional because of further issues.

Vincent Ladeuil (vila) wrote :

Fwiw, merging up in 2.3 and trunk pass the whole test suite.

As mentioned by poolie in https://bugs.launchpad.net/bzr/+bug/715000/comments/6 we should investigate for sibling bugs.

Note to reviewers: you can safely ignore the conflicts which are caused by the mp being based on 2.2 while targeting trunk.

John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/8/2011 3:26 AM, Andrew Bennetts wrote:
> Some quick comments (rather than a comprehensive review):
>
> 1. "for br in [branch_c]:" is a weird expression (why have a loop when there's exactly 1 item?)
>
> 2. this fix appears reasonable, but you only fix one method when there are many other uses of self._fallback_vfs in e.g. GroupCompressVersionedFiles. Either there are more bugs waiting to be fixed here (hopefully easy ones to reveal with tests similar to the one you add in this patch), or the reason why only get_known_graph_ancestry was broken is subtle/unclear. Do you know which? I'm a bit worried that this may correct the immediate traceback without actually making the Launchpad scanner functional because of further issues.

I agree, I think he meant:

for br in [branch_a, branch_b, branch_c]:

Though I'm fine with a test of just branch_c, if we are just wanting to
test the transitivity of operations.

I wouldn't be surprised if there are some bugs in other places, but I
think things like "get_parent_map" already recurse via the public api,
rather than recursing to find the transitive fallbacks, and then
iterating. (so branch_c.repo.get_parent_map() calls
branch_b.repo.get_parent_map() which calls
branch_a.repo.get_parent_map() rather than
branch_c.repo.get_parent_map() going through all fallbacks directly.)

I sort of like the public api recursion approach more than the
_transitive_fallbacks approach. If only because it means that
RepoA.do_something() isn't poking at RepoB._fallback_repositories. It
only looks at its own private attributes, and a fallbacks public attributes.

transitive_fallbacks does ok with this, if you consider that the
function is actually a public Repository api. (Which I think it should
be, and should be tested as such, with a per_repository implementation
test.)

Anyway, I'm fine with this as a fix for what we are currently
encountering. I think we can do an audit of the codebase for everywhere
that _fallback_vfs or _fallback_repositories is being encountered, but
that can be separate to Martin's patch.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1RrJkACgkQJdeBCYSNAAMBMACcDC92UyAzUi1q7Zekhf9qmBUD
edEAniQplUIN6tovAJ+R+VgAr3R9N+ry
=HM+5
-----END PGP SIGNATURE-----

Martin Pool (mbp) wrote :

On 8 February 2011 20:26, Andrew Bennetts <email address hidden> wrote:
> Some quick comments (rather than a comprehensive review):
>
> 1. "for br in [branch_c]:" is a weird expression (why have a loop when there's exactly 1 item?)

Ah, for the sake of easier testing I had cut out the others. It
should test all of them.

> 2. this fix appears reasonable, but you only fix one method when there are many other uses of self._fallback_vfs in e.g. GroupCompressVersionedFiles.  Either there are more bugs waiting to be fixed here (hopefully easy ones to reveal with tests similar to the one you add in this patch), or the reason why only get_known_graph_ancestry was broken is subtle/unclear.  Do you know which?  I'm a bit worried that this may correct the immediate traceback without actually making the Launchpad scanner functional because of further issues.

I was hoping someone would tell me :-)

More seriously, I do think all the others need to be checked too, and
also by searching the bug tracker we might find reports where similar
things are failing.

Martin Pool (mbp) wrote :

On 9 February 2011 07:50, John Arbash Meinel <email address hidden> wrote:

> transitive_fallbacks does ok with this, if you consider that the
> function is actually a public Repository api. (Which I think it should
> be, and should be tested as such, with a per_repository implementation
> test.)

I was going to recurse through the public API, and originally took
that approach, but the problem was that in this case the public API
returns a KnownGraph, and I couldn't see how to combine several of
those objects. Perhaps there is a way?

John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/8/2011 4:49 PM, Martin Pool wrote:
> On 9 February 2011 07:50, John Arbash Meinel <email address hidden> wrote:
>
>> transitive_fallbacks does ok with this, if you consider that the
>> function is actually a public Repository api. (Which I think it should
>> be, and should be tested as such, with a per_repository implementation
>> test.)
>
> I was going to recurse through the public API, and originally took
> that approach, but the problem was that in this case the public API
> returns a KnownGraph, and I couldn't see how to combine several of
> those objects. Perhaps there is a way?
>

Not trivially. We have code to add new entries one-by-one, but that
isn't great.

If we wanted to tease this apart more, something that returns a dict,
and then wrapping that in a KnownGraph would be enough. Or expose more
of _find_ancestor (I don't remember the exact 'take one step' function
name).

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1RzqsACgkQJdeBCYSNAAMXPACeIOmDqkT8KfkedqNEGn6Dv89u
9moAoLpnFYmQ0Nnv2rmhGrUJN+x60pkM
=YQNy
-----END PGP SIGNATURE-----

John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/8/2011 4:46 PM, Martin Pool wrote:
> On 8 February 2011 20:26, Andrew Bennetts <email address hidden> wrote:
>> Some quick comments (rather than a comprehensive review):
>>
>> 1. "for br in [branch_c]:" is a weird expression (why have a loop when there's exactly 1 item?)
>
> Ah, for the sake of easier testing I had cut out the others. It
> should test all of them.
>
>> 2. this fix appears reasonable, but you only fix one method when there are many other uses of self._fallback_vfs in e.g. GroupCompressVersionedFiles. Either there are more bugs waiting to be fixed here (hopefully easy ones to reveal with tests similar to the one you add in this patch), or the reason why only get_known_graph_ancestry was broken is subtle/unclear. Do you know which? I'm a bit worried that this may correct the immediate traceback without actually making the Launchpad scanner functional because of further issues.
>
> I was hoping someone would tell me :-)
>
> More seriously, I do think all the others need to be checked too, and
> also by searching the bug tracker we might find reports where similar
> things are failing.
>

I honestly haven't seen much failure of this. Mostly because stacking is
rare outside of launchpad, and stacking on stacked is a whole next-level
of unlikely. Since not even launchpad creates branches like this easily.

I wouldn't be surprised to have internal bugs that are latent. Doing a
quick grep doesn't show anything obvious.

_fallback_vfs is only mentioned in bzrlib/groupcompress.py and knit.py
(since those are the only VersionedFiles that actually support it, it
should make sense.)

Specific references in groupcompress:

get_known_graph_ancestry(), the code in question
_get_parent_map_with_sources(), uses the public api
_find_from_fallback(), public_api
keys(), public_api

in knit.py:

_logical_check(), public_api
get_known_graph_ancestry(), current code
_get_parent_map_with_sources(), public_api (copy & paste because of
                                            inheritance issues)
_get_remaining_record_stream, public_api
get_sha1s(), public_api
insert_record_stream(), just used to tell if we have a stacking
  boundary, and need to do extra sanity checks on the stream
iter_lines_added_or_present_in_keys(), public_api, probably should be
  deprecated anyway
keys(), public_api

_ContentMapGenerator._work, public_api
_KnitAnnotator._get_needed_texts, only used to signal fallback to
  non-optimized code

In the end, what we could do is promote "self._index.find_ancestry()" to
be a public attribute of VersionedFiles rather than hidden on the _index
object, and then it could be done via public api, rather than chaining
the transitive closure of fallback_vfs. That would allow us to
consistently go via public api. Albeit by using a slightly different api
that can chain.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1R0SQACgkQJdeBCYSNAAMZIwCeNQtpjB/0NPaQjNb98wW0Vxg2
T9IAnAl/pJhAoqWGbDgXq8ov5R3oyxkW
=i3cD
-----END PGP SIGNATURE-----

Robert Collins (lifeless) wrote :

This looks fine to me; I agree that renaming _fallback_vfs to _immediate_fallbacks or something would be an improvement.

review: Approve
Martin Pool (mbp) wrote :

Since lp is running 2.2 and wants to deploy this on top of 2.2, and since people seem to agree this is safe (and the change is smaller than I suspected it might be), I'll merge it there.

lp:~mbp/bzr/715000-stacking updated on 2011-02-09
5127. By Martin Pool on 2011-02-09

News for bug 715000

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bzr'
2--- bzr 2011-02-08 13:56:49 +0000
3+++ bzr 2011-02-09 04:20:36 +0000
4@@ -23,7 +23,11 @@
5 import warnings
6
7 # update this on each release
8+<<<<<<< TREE
9 _script_version = (2, 4, 0)
10+=======
11+_script_version = (2, 2, 5)
12+>>>>>>> MERGE-SOURCE
13
14 try:
15 version_info = sys.version_info
16
17=== modified file 'bzrlib/__init__.py'
18--- bzrlib/__init__.py 2011-02-08 13:56:49 +0000
19+++ bzrlib/__init__.py 2011-02-09 04:20:36 +0000
20@@ -52,7 +52,11 @@
21 # Python version 2.0 is (2, 0, 0, 'final', 0)." Additionally we use a
22 # releaselevel of 'dev' for unreleased under-development code.
23
24+<<<<<<< TREE
25 version_info = (2, 4, 0, 'dev', 1)
26+=======
27+version_info = (2, 2, 5, 'dev', 0)
28+>>>>>>> MERGE-SOURCE
29
30 # API compatibility version
31 api_minimum_version = (2, 4, 0)
32
33=== modified file 'bzrlib/groupcompress.py'
34--- bzrlib/groupcompress.py 2010-09-26 20:45:28 +0000
35+++ bzrlib/groupcompress.py 2011-02-09 04:20:36 +0000
36@@ -1,4 +1,4 @@
37-# Copyright (C) 2008, 2009, 2010 Canonical Ltd
38+# Copyright (C) 2008-2011 Canonical Ltd
39 #
40 # This program is free software; you can redistribute it and/or modify
41 # it under the terms of the GNU General Public License as published by
42@@ -1312,7 +1312,7 @@
43 # KnitVersionedFiles.get_known_graph_ancestry, but they don't share
44 # ancestry.
45 parent_map, missing_keys = self._index.find_ancestry(keys)
46- for fallback in self._fallback_vfs:
47+ for fallback in self._transitive_fallbacks():
48 if not missing_keys:
49 break
50 (f_parent_map, f_missing_keys) = fallback._index.find_ancestry(
51
52=== modified file 'bzrlib/knit.py'
53--- bzrlib/knit.py 2011-01-11 20:42:36 +0000
54+++ bzrlib/knit.py 2011-02-09 04:20:36 +0000
55@@ -1196,7 +1196,7 @@
56 def get_known_graph_ancestry(self, keys):
57 """Get a KnownGraph instance with the ancestry of keys."""
58 parent_map, missing_keys = self._index.find_ancestry(keys)
59- for fallback in self._fallback_vfs:
60+ for fallback in self._transitive_fallbacks():
61 if not missing_keys:
62 break
63 (f_parent_map, f_missing_keys) = fallback._index.find_ancestry(
64
65=== modified file 'bzrlib/tests/per_repository_reference/__init__.py'
66--- bzrlib/tests/per_repository_reference/__init__.py 2011-01-10 22:47:59 +0000
67+++ bzrlib/tests/per_repository_reference/__init__.py 2011-02-09 04:20:36 +0000
68@@ -105,6 +105,7 @@
69 'bzrlib.tests.per_repository_reference.test_fetch',
70 'bzrlib.tests.per_repository_reference.test_get_record_stream',
71 'bzrlib.tests.per_repository_reference.test_get_rev_id_for_revno',
72+ 'bzrlib.tests.per_repository_reference.test_graph',
73 'bzrlib.tests.per_repository_reference.test_initialize',
74 'bzrlib.tests.per_repository_reference.test_unlock',
75 ]
76
77=== added file 'bzrlib/tests/per_repository_reference/test_graph.py'
78--- bzrlib/tests/per_repository_reference/test_graph.py 1970-01-01 00:00:00 +0000
79+++ bzrlib/tests/per_repository_reference/test_graph.py 2011-02-09 04:20:36 +0000
80@@ -0,0 +1,45 @@
81+# Copyright (C) 2011 Canonical Ltd
82+#
83+# This program is free software; you can redistribute it and/or modify
84+# it under the terms of the GNU General Public License as published by
85+# the Free Software Foundation; either version 2 of the License, or
86+# (at your option) any later version.
87+#
88+# This program is distributed in the hope that it will be useful,
89+# but WITHOUT ANY WARRANTY; without even the implied warranty of
90+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
91+# GNU General Public License for more details.
92+#
93+# You should have received a copy of the GNU General Public License
94+# along with this program; if not, write to the Free Software
95+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
96+
97+
98+"""Tests for graph operations on stacked repositories."""
99+
100+
101+from bzrlib.tests.per_repository import TestCaseWithRepository
102+
103+
104+class TestGraph(TestCaseWithRepository):
105+
106+ def test_get_known_graph_ancestry_stacked(self):
107+ """get_known_graph_ancestry works correctly on stacking.
108+
109+ See <https://bugs.launchpad.net/bugs/715000>.
110+ """
111+ branch_a, branch_b, branch_c, revid_1 = self.make_double_stacked_branches()
112+ for br in [branch_c]:
113+ self.assertEquals(
114+ [revid_1],
115+ br.repository.get_known_graph_ancestry([revid_1]).topo_sort())
116+
117+ def make_double_stacked_branches(self):
118+ wt_a = self.make_branch_and_tree('a')
119+ branch_a = wt_a.branch
120+ branch_b = self.make_branch('b')
121+ branch_b.set_stacked_on_url('../a')
122+ branch_c = self.make_branch('c')
123+ branch_c.set_stacked_on_url('../b')
124+ revid_1 = wt_a.commit('first commit')
125+ return branch_a, branch_b, branch_c, revid_1
126
127=== modified file 'bzrlib/versionedfile.py'
128--- bzrlib/versionedfile.py 2010-08-28 15:34:24 +0000
129+++ bzrlib/versionedfile.py 2011-02-09 04:20:36 +0000
130@@ -1193,6 +1193,19 @@
131 def _extract_blocks(self, version_id, source, target):
132 return None
133
134+ def _transitive_fallbacks(self):
135+ """Return the whole stack of fallback versionedfiles.
136+
137+ This VersionedFiles may have a list of fallbacks, but it doesn't
138+ necessarily know about the whole stack going down, and it can't know
139+ at open time because they may change after the objects are opened.
140+ """
141+ all_fallbacks = []
142+ for a_vfs in self._fallback_vfs:
143+ all_fallbacks.append(a_vfs)
144+ all_fallbacks.extend(a_vfs._transitive_fallbacks())
145+ return all_fallbacks
146+
147
148 class ThunkedVersionedFiles(VersionedFiles):
149 """Storage for many versioned files thunked onto a 'VersionedFile' class.
150
151=== modified file 'doc/en/release-notes/bzr-2.3.txt'
152--- doc/en/release-notes/bzr-2.3.txt 2011-02-08 13:56:49 +0000
153+++ doc/en/release-notes/bzr-2.3.txt 2011-02-09 04:20:36 +0000
154@@ -2,6 +2,7 @@
155 Bazaar Release Notes
156 ####################
157
158+<<<<<<< TREE
159 .. toctree::
160 :maxdepth: 1
161
162@@ -14,6 +15,81 @@
163 *****************************
164
165 .. These may require users to change the way they use Bazaar.
166+=======
167+.. contents:: List of Releases
168+ :depth: 1
169+
170+bzr 2.2.5
171+#########
172+
173+:Codename: Suggestions welcome
174+:2.2.5: NOT RELEASED YET
175+
176+Compatibility Breaks
177+********************
178+
179+New Features
180+************
181+
182+Bug Fixes
183+*********
184+
185+* Correctly handle ``bzr log`` and `get_known_graph_ancestry` on a
186+ doubly-stacked branch.
187+ (James Westby, Martin Pool, #715000)
188+
189+Improvements
190+************
191+
192+Documentation
193+*************
194+
195+API Changes
196+***********
197+
198+Internals
199+*********
200+
201+Testing
202+*******
203+
204+
205+bzr 2.2.4
206+#########
207+
208+:2.2.4: 2011-02-04
209+
210+This is a bugfix release. Only one bug has been fixed, a regression from 2.2.3
211+involving only certain operations with launchpad. Upgrading is recommended for
212+all users on earlier 2.2 releases.
213+
214+Bug Fixes
215+*********
216+
217+* Fix communications with the Launchpad web service when using
218+ launchpadlib >= 1.5.5. This was a latent bug in bzr's communication
219+ with Launchpad's production instance, which only became a problem when
220+ the default instance was switched from edge to production in bzr 2.2.3.
221+ (Max Bowsher, #707075)
222+
223+
224+bzr 2.2.3
225+#########
226+
227+:2.2.3: 2011-01-20
228+
229+This is a bugfix release. Upgrading is recommended for all users
230+on earlier 2.2 releases.
231+
232+Compatibility Breaks
233+********************
234+
235+* Launchpad has announced that the ``edge.launchpad.net`` instance is
236+ deprecated and may be shut down in the future
237+ <http://blog.launchpad.net/general/edge-is-deprecated>. Bazaar has therefore
238+ been updated in this release to talk to the main (``launchpad.net``) servers,
239+ rather than the ``edge`` ones. (Vincent Ladeuil, #583667)
240+>>>>>>> MERGE-SOURCE
241
242 New Features
243 ************