Merge lp:~jelmer/bzr/hpss-get-inventories into lp:bzr
- hpss-get-inventories
- Merge into bzr.dev
Status: | Merged |
---|---|
Approved by: | Jelmer Vernooij |
Approved revision: | no longer in the source branch. |
Merged at revision: | 6372 |
Proposed branch: | lp:~jelmer/bzr/hpss-get-inventories |
Merge into: | lp:bzr |
Prerequisite: | lp:~jelmer/bzr/hpss-_get-checkout-format |
Diff against target: |
728 lines (+372/-52) 17 files modified
bzrlib/remote.py (+171/-5) bzrlib/smart/repository.py (+58/-1) bzrlib/smart/request.py (+6/-3) bzrlib/tests/blackbox/test_annotate.py (+1/-1) bzrlib/tests/blackbox/test_branch.py (+3/-3) bzrlib/tests/blackbox/test_cat.py (+2/-3) bzrlib/tests/blackbox/test_checkout.py (+3/-8) bzrlib/tests/blackbox/test_export.py (+2/-3) bzrlib/tests/blackbox/test_log.py (+4/-6) bzrlib/tests/blackbox/test_ls.py (+2/-3) bzrlib/tests/blackbox/test_sign_my_commits.py (+4/-12) bzrlib/tests/per_interbranch/test_push.py (+2/-2) bzrlib/tests/per_repository_chk/test_supported.py (+11/-1) bzrlib/tests/per_repository_vf/test_add_inventory_by_delta.py (+1/-1) bzrlib/tests/test_remote.py (+41/-0) bzrlib/tests/test_smart.py (+48/-0) doc/en/release-notes/bzr-2.5.txt (+13/-0) |
To merge this branch: | bzr merge lp:~jelmer/bzr/hpss-get-inventories |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jelmer Vernooij (community) | Approve | ||
Vincent Ladeuil | Approve | ||
Andrew Bennetts | Pending | ||
Review via email: mp+85252@code.launchpad.net |
This proposal supersedes a proposal from 2011-11-29.
Commit message
Add HPSS call for retrieving inventories.
Description of the change
Add a HPSS call for ``Repository.
This massively reduces the number of roundtrips for various commands, and causes a fair number to no longer use VFS calls at all when used against a modern remote server.
Repository.
Unlike my previous attempt at this, this now uses record streams with inventory deltas. It is still a separate verb though.
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal | # |
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal | # |
For comparison (though I should it's a different server):
apt-get source bzr 2.79s user 0.54s system 27% cpu 12.338 total
so we're getting closer.
Andrew Bennetts (spiv) wrote : Posted in a previous version of this proposal | # |
Jelmer Vernooij wrote:
> Add a HPSS call for ``Repository.
I am a bit concerned that by adding verbs like this, with their own ad hoc
record stream-like wire format, that we're growing the maintenance burden
unnecessarily, and not reusing improvements everywhere we could.
I'm glad this is using inventory-deltas. But if it's worth zlib compressing
them here, shouldn't we do that for inventory-deltas from get_record_stream too?
Also, could you implement this via the get_record_stream RPC with a new
parameter that means “just inventories (rather than rev+inv+
given keys)”?
On one hand it might feel a bit ugly to make one RPC do so many things,
get_record_stream, do so many things, but on the other I think it provides a
useful pressure to keep the interface minimal and consistent (e.g. whether
records are zlib-compressed).
Adding this iter_inventories RPC might be the right thing (although putting
“iter” in the name of an RPC feels weird to me, that's surely a property of the
Python API rather than a property of the RPC), but I'd like hear what you think
about the tradeoffs of doing that vs. extending get_record_stream.
-Andrew.
Martin Pool (mbp) wrote : Posted in a previous version of this proposal | # |
On 30 November 2011 09:37, Andrew Bennetts
<email address hidden> wrote:
> Jelmer Vernooij wrote:
>> Add a HPSS call for ``Repository.
>
> I am a bit concerned that by adding verbs like this, with their own ad hoc
> record stream-like wire format, that we're growing the maintenance burden
> unnecessarily, and not reusing improvements everywhere we could.
+1
--
Martin
Vincent Ladeuil (vila) wrote : Posted in a previous version of this proposal | # |
>> I am a bit concerned that by adding verbs like this, with their own ad hoc
>> record stream-like wire format, that we're growing the maintenance burden
>> unnecessarily, and not reusing improvements everywhere we could.
> +1
I kind of had the same feeling when jelmer started adding a bunch of verbs that were basically replacing a vfs roundtrip by a smart request roundtrip.
BUT
If we stop maintaining these verbs on the server side, the clients will fallback to vfs.
So we introduce different/better verbs, we can remove the old ones in both the server and the client and all but the newest clients can still fallback to vfs.
The net effect is at least to get to a point where the most recent client/server do not use vfs at all.
This sounds like a good incremental step to me.
'reusing improvements everywhere we could' is still valuable but doesn't to block progress.
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal | # |
Hi Andrew,
> Jelmer Vernooij wrote:
> > Add a HPSS call for ``Repository.
> I am a bit concerned that by adding verbs like this, with their own ad hoc
> record stream-like wire format, that we're growing the maintenance burden
> unnecessarily, and not reusing improvements everywhere we could.
>
> I'm glad this is using inventory-deltas. But if it's worth zlib compressing
> them here, shouldn't we do that for inventory-deltas from get_record_stream
> too?
It's a pretty big inventory delta in this case - for something like gcc it matters for the delta between null: and the initial revision that was requested. I haven't investigated whether it would help for record streams too, but I imagine it would have less of an effect.
> Also, could you implement this via the get_record_stream RPC with a new
> parameter that means “just inventories (rather than rev+inv+
> for
> given keys)”?
>
> On one hand it might feel a bit ugly to make one RPC do so many things,
> get_record_stream, do so many things, but on the other I think it provides a
> useful pressure to keep the interface minimal and consistent (e.g. whether
> records are zlib-compressed).
>
> Adding this iter_inventories RPC might be the right thing (although putting
> “iter” in the name of an RPC feels weird to me, that's surely a property of
> the Python API rather than a property of the RPC), but I'd like hear what you
> think about the tradeoffs of doing that vs. extending get_record_stream.
With regard to the name: Most of the other calls seem to be named after the equivalent method in the client / server, that's why I went with Repository.
I'm not sure whether this should be part of get_record_stream. I have a hard time understanding the get_record_stream code as it is, so I'd rather not make it even more complex by adding more parameters - for example, get_record_streams seem to be dependent on the repository on-disk format to an extent - the inventory stream is not, as it's using inventory deltas. In other words, adding another verb was simpler, while keeping it all understandable for a mere mortal like me. :-)
Either way, I agree we should be reusing more code between the work I've done recently and the existing record stream
calls. But it seems to me that would best be done by refactoring so they e.g. use the same code for sending a stream of blobs of indeterminate length, rather than by all being a part of the same verb.
What do you think?
Cheers,
Jelmer
Andrew Bennetts (spiv) wrote : Posted in a previous version of this proposal | # |
Jelmer Vernooij wrote:
…
> > I'm glad this is using inventory-deltas. But if it's worth zlib compressing
> > them here, shouldn't we do that for inventory-deltas from get_record_stream
> > too?
> It's a pretty big inventory delta in this case - for something like gcc it
> matters for the delta between null: and the initial revision that was
> requested. I haven't investigated whether it would help for record streams
> too, but I imagine it would have less of an effect.
Well, I know that there are already cases that send deltas from null: —
something to do with stacking perhaps? So reusing this improvement globally
would be nice.
> > Also, could you implement this via the get_record_stream RPC with a new
> > parameter that means “just inventories (rather than rev+inv+
> > for
> > given keys)”?
> >
> > On one hand it might feel a bit ugly to make one RPC do so many things,
> > get_record_stream, do so many things, but on the other I think it provides a
> > useful pressure to keep the interface minimal and consistent (e.g. whether
> > records are zlib-compressed).
> >
> > Adding this iter_inventories RPC might be the right thing (although putting
> > “iter” in the name of an RPC feels weird to me, that's surely a property of
> > the Python API rather than a property of the RPC), but I'd like hear what you
> > think about the tradeoffs of doing that vs. extending get_record_stream.
>
> With regard to the name: Most of the other calls seem to be named after the
> equivalent method in the client / server, that's why I went with
> Repository.
> like "Repository.
> reasonable to me too.
I think of using the name of the Python API as a good guide, not a strict rule.
Certainly there's no insert_
“Repository.
> I'm not sure whether this should be part of get_record_stream. I have a hard
> time understanding the get_record_stream code as it is, so I'd rather not make
> it even more complex by adding more parameters - for example,
> get_record_streams seem to be dependent on the repository on-disk format to an
> extent - the inventory stream is not, as it's using inventory deltas. In other
> words, adding another verb was simpler, while keeping it all understandable
> for a mere mortal like me. :-)
>
> Either way, I agree we should be reusing more code between the work I've done
> recently and the existing record stream calls. But it seems to me that would
> best be done by refactoring so they e.g. use the same code for sending a
> stream of blobs of indeterminate length, rather than by all being a part of
> the same verb.
Well, the path to reuse we've developed *is* record streams. I'm quite willing
to believe that get_record_stream's parameters aren't a convenient way to
express all needs. But I'd really like it the thing that was returned by a new
verb was a true record stream — something you could pass directly to
insert_
Way we have to say “here is a stream of da...
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal | # |
Am 05/12/11 11:31, schrieb Andrew Bennetts:
> Jelmer Vernooij wrote:
> …
>>> I'm glad this is using inventory-deltas. But if it's worth zlib compressing
>>> them here, shouldn't we do that for inventory-deltas from get_record_stream
>>> too?
>> It's a pretty big inventory delta in this case - for something like gcc it
>> matters for the delta between null: and the initial revision that was
>> requested. I haven't investigated whether it would help for record streams
>> too, but I imagine it would have less of an effect.
> Well, I know that there are already cases that send deltas from null: —
> something to do with stacking perhaps? So reusing this improvement globally
> would be nice.
I guess the best way to do this would be to add a zlib pack record kind
for network streams? I.e. a new entry in
NetworkRecordSt
server side?
>>> Also, could you implement this via the get_record_stream RPC with a new
>>> parameter that means “just inventories (rather than rev+inv+
>>> for
>>> given keys)”?
>>>
>>> On one hand it might feel a bit ugly to make one RPC do so many things,
>>> get_record_stream, do so many things, but on the other I think it provides a
>>> useful pressure to keep the interface minimal and consistent (e.g. whether
>>> records are zlib-compressed).
>>>
>>> Adding this iter_inventories RPC might be the right thing (although putting
>>> “iter” in the name of an RPC feels weird to me, that's surely a property of
>>> the Python API rather than a property of the RPC), but I'd like hear what you
>>> think about the tradeoffs of doing that vs. extending get_record_stream.
>> With regard to the name: Most of the other calls seem to be named after the
>> equivalent method in the client / server, that's why I went with
>> Repository.
>> like "Repository.
>> reasonable to me too.
> I think of using the name of the Python API as a good guide, not a strict rule.
> Certainly there's no insert_
>
> “Repository.
I've renamed it.
>
>> I'm not sure whether this should be part of get_record_stream. I have a hard
>> time understanding the get_record_stream code as it is, so I'd rather not make
>> it even more complex by adding more parameters - for example,
>> get_record_streams seem to be dependent on the repository on-disk format to an
>> extent - the inventory stream is not, as it's using inventory deltas. In other
>> words, adding another verb was simpler, while keeping it all understandable
>> for a mere mortal like me. :-)
>>
>> Either way, I agree we should be reusing more code between the work I've done
>> recently and the existing record stream calls. But it seems to me that would
>> best be done by refactoring so they e.g. use the same code for sending a
>> stream of blobs of indeterminate length, rather than by all being a part of
>> the same verb.
> Well, the path to reuse we've developed *is* record streams. I'm quite willing
> to believe that get_record_stream's parameters aren'...
Vincent Ladeuil (vila) wrote : | # |
Based on the tests and the decrease of roundtrips: approved !
16 + def _vfs_checkout_
17 + self._ensure_real()
18 + return self._real_
19 +
20 + def checkout_
21 + medium = self._client.
22 + if medium.
23 + return self._vfs_
_vfs_checkout_
Is that a loop or do I miss something obvious ?
Up to you to wait for spiv's review.
Jelmer Vernooij (jelmer) wrote : | # |
Am 13/12/11 11:14, schrieb Vincent Ladeuil:
> Review: Approve
>
> Based on the tests and the decrease of roundtrips: approved !
>
> 16 + def _vfs_checkout_
> 17 + self._ensure_real()
> 18 + return self._real_
> 19 +
> 20 + def checkout_
> 21 + medium = self._client.
> 22 + if medium.
> 23 + return self._vfs_
>
> _vfs_checkout_
It's not checkout_metadir, but _real_bzrdir.
>
> Is that a loop or do I miss something obvious ?
>
> Up to you to wait for spiv's review.
Cheers,
Jelmer
Jelmer Vernooij (jelmer) wrote : | # |
> Up to you to wait for spiv's review.
Thanks.
I haven't heard from spiv in the last week, and I'd really like to get this landed so it gets some decent testing for 2.5.0. My recent changes should have addressed his concerns regarding the use of record streams, and I'm happy to tweak whatever is necessary later based on post-commit reviews.
Jelmer Vernooij (jelmer) wrote : | # |
sent to pqm by email
Preview Diff
1 | === modified file 'bzrlib/remote.py' | |||
2 | --- bzrlib/remote.py 2011-12-14 12:33:04 +0000 | |||
3 | +++ bzrlib/remote.py 2011-12-14 21:28:31 +0000 | |||
4 | @@ -27,6 +27,7 @@ | |||
5 | 27 | errors, | 27 | errors, |
6 | 28 | gpg, | 28 | gpg, |
7 | 29 | graph, | 29 | graph, |
8 | 30 | inventory_delta, | ||
9 | 30 | lock, | 31 | lock, |
10 | 31 | lockdir, | 32 | lockdir, |
11 | 32 | osutils, | 33 | osutils, |
12 | @@ -1830,9 +1831,125 @@ | |||
13 | 1830 | def get_inventory(self, revision_id): | 1831 | def get_inventory(self, revision_id): |
14 | 1831 | return list(self.iter_inventories([revision_id]))[0] | 1832 | return list(self.iter_inventories([revision_id]))[0] |
15 | 1832 | 1833 | ||
16 | 1834 | def _iter_inventories_rpc(self, revision_ids, ordering): | ||
17 | 1835 | if ordering is None: | ||
18 | 1836 | ordering = 'unordered' | ||
19 | 1837 | path = self.bzrdir._path_for_remote_call(self._client) | ||
20 | 1838 | body = "\n".join(revision_ids) | ||
21 | 1839 | response_tuple, response_handler = ( | ||
22 | 1840 | self._call_with_body_bytes_expecting_body( | ||
23 | 1841 | "VersionedFileRepository.get_inventories", | ||
24 | 1842 | (path, ordering), body)) | ||
25 | 1843 | if response_tuple[0] != "ok": | ||
26 | 1844 | raise errors.UnexpectedSmartServerResponse(response_tuple) | ||
27 | 1845 | deserializer = inventory_delta.InventoryDeltaDeserializer() | ||
28 | 1846 | byte_stream = response_handler.read_streamed_body() | ||
29 | 1847 | decoded = smart_repo._byte_stream_to_stream(byte_stream) | ||
30 | 1848 | if decoded is None: | ||
31 | 1849 | # no results whatsoever | ||
32 | 1850 | return | ||
33 | 1851 | src_format, stream = decoded | ||
34 | 1852 | if src_format.network_name() != self._format.network_name(): | ||
35 | 1853 | raise AssertionError( | ||
36 | 1854 | "Mismatched RemoteRepository and stream src %r, %r" % ( | ||
37 | 1855 | src_format.network_name(), self._format.network_name())) | ||
38 | 1856 | # ignore the src format, it's not really relevant | ||
39 | 1857 | prev_inv = Inventory(root_id=None, | ||
40 | 1858 | revision_id=_mod_revision.NULL_REVISION) | ||
41 | 1859 | # there should be just one substream, with inventory deltas | ||
42 | 1860 | substream_kind, substream = stream.next() | ||
43 | 1861 | if substream_kind != "inventory-deltas": | ||
44 | 1862 | raise AssertionError( | ||
45 | 1863 | "Unexpected stream %r received" % substream_kind) | ||
46 | 1864 | for record in substream: | ||
47 | 1865 | (parent_id, new_id, versioned_root, tree_references, invdelta) = ( | ||
48 | 1866 | deserializer.parse_text_bytes(record.get_bytes_as("fulltext"))) | ||
49 | 1867 | if parent_id != prev_inv.revision_id: | ||
50 | 1868 | raise AssertionError("invalid base %r != %r" % (parent_id, | ||
51 | 1869 | prev_inv.revision_id)) | ||
52 | 1870 | inv = prev_inv.create_by_apply_delta(invdelta, new_id) | ||
53 | 1871 | yield inv, inv.revision_id | ||
54 | 1872 | prev_inv = inv | ||
55 | 1873 | |||
56 | 1874 | def _iter_inventories_vfs(self, revision_ids, ordering=None): | ||
57 | 1875 | self._ensure_real() | ||
58 | 1876 | return self._real_repository._iter_inventories(revision_ids, ordering) | ||
59 | 1877 | |||
60 | 1833 | def iter_inventories(self, revision_ids, ordering=None): | 1878 | def iter_inventories(self, revision_ids, ordering=None): |
63 | 1834 | self._ensure_real() | 1879 | """Get many inventories by revision_ids. |
64 | 1835 | return self._real_repository.iter_inventories(revision_ids, ordering) | 1880 | |
65 | 1881 | This will buffer some or all of the texts used in constructing the | ||
66 | 1882 | inventories in memory, but will only parse a single inventory at a | ||
67 | 1883 | time. | ||
68 | 1884 | |||
69 | 1885 | :param revision_ids: The expected revision ids of the inventories. | ||
70 | 1886 | :param ordering: optional ordering, e.g. 'topological'. If not | ||
71 | 1887 | specified, the order of revision_ids will be preserved (by | ||
72 | 1888 | buffering if necessary). | ||
73 | 1889 | :return: An iterator of inventories. | ||
74 | 1890 | """ | ||
75 | 1891 | if ((None in revision_ids) | ||
76 | 1892 | or (_mod_revision.NULL_REVISION in revision_ids)): | ||
77 | 1893 | raise ValueError('cannot get null revision inventory') | ||
78 | 1894 | for inv, revid in self._iter_inventories(revision_ids, ordering): | ||
79 | 1895 | if inv is None: | ||
80 | 1896 | raise errors.NoSuchRevision(self, revid) | ||
81 | 1897 | yield inv | ||
82 | 1898 | |||
83 | 1899 | def _iter_inventories(self, revision_ids, ordering=None): | ||
84 | 1900 | if len(revision_ids) == 0: | ||
85 | 1901 | return | ||
86 | 1902 | missing = set(revision_ids) | ||
87 | 1903 | if ordering is None: | ||
88 | 1904 | order_as_requested = True | ||
89 | 1905 | invs = {} | ||
90 | 1906 | order = list(revision_ids) | ||
91 | 1907 | order.reverse() | ||
92 | 1908 | next_revid = order.pop() | ||
93 | 1909 | else: | ||
94 | 1910 | order_as_requested = False | ||
95 | 1911 | if ordering != 'unordered' and self._fallback_repositories: | ||
96 | 1912 | raise ValueError('unsupported ordering %r' % ordering) | ||
97 | 1913 | iter_inv_fns = [self._iter_inventories_rpc] + [ | ||
98 | 1914 | fallback._iter_inventories for fallback in | ||
99 | 1915 | self._fallback_repositories] | ||
100 | 1916 | try: | ||
101 | 1917 | for iter_inv in iter_inv_fns: | ||
102 | 1918 | request = [revid for revid in revision_ids if revid in missing] | ||
103 | 1919 | for inv, revid in iter_inv(request, ordering): | ||
104 | 1920 | if inv is None: | ||
105 | 1921 | continue | ||
106 | 1922 | missing.remove(inv.revision_id) | ||
107 | 1923 | if ordering != 'unordered': | ||
108 | 1924 | invs[revid] = inv | ||
109 | 1925 | else: | ||
110 | 1926 | yield inv, revid | ||
111 | 1927 | if order_as_requested: | ||
112 | 1928 | # Yield as many results as we can while preserving order. | ||
113 | 1929 | while next_revid in invs: | ||
114 | 1930 | inv = invs.pop(next_revid) | ||
115 | 1931 | yield inv, inv.revision_id | ||
116 | 1932 | try: | ||
117 | 1933 | next_revid = order.pop() | ||
118 | 1934 | except IndexError: | ||
119 | 1935 | # We still want to fully consume the stream, just | ||
120 | 1936 | # in case it is not actually finished at this point | ||
121 | 1937 | next_revid = None | ||
122 | 1938 | break | ||
123 | 1939 | except errors.UnknownSmartMethod: | ||
124 | 1940 | for inv, revid in self._iter_inventories_vfs(revision_ids, ordering): | ||
125 | 1941 | yield inv, revid | ||
126 | 1942 | return | ||
127 | 1943 | # Report missing | ||
128 | 1944 | if order_as_requested: | ||
129 | 1945 | if next_revid is not None: | ||
130 | 1946 | yield None, next_revid | ||
131 | 1947 | while order: | ||
132 | 1948 | revid = order.pop() | ||
133 | 1949 | yield invs.get(revid), revid | ||
134 | 1950 | else: | ||
135 | 1951 | while missing: | ||
136 | 1952 | yield None, missing.pop() | ||
137 | 1836 | 1953 | ||
138 | 1837 | @needs_read_lock | 1954 | @needs_read_lock |
139 | 1838 | def get_revision(self, revision_id): | 1955 | def get_revision(self, revision_id): |
140 | @@ -2191,6 +2308,8 @@ | |||
141 | 2191 | 2308 | ||
142 | 2192 | @needs_read_lock | 2309 | @needs_read_lock |
143 | 2193 | def _get_inventory_xml(self, revision_id): | 2310 | def _get_inventory_xml(self, revision_id): |
144 | 2311 | # This call is used by older working tree formats, | ||
145 | 2312 | # which stored a serialized basis inventory. | ||
146 | 2194 | self._ensure_real() | 2313 | self._ensure_real() |
147 | 2195 | return self._real_repository._get_inventory_xml(revision_id) | 2314 | return self._real_repository._get_inventory_xml(revision_id) |
148 | 2196 | 2315 | ||
149 | @@ -2235,11 +2354,58 @@ | |||
150 | 2235 | revids.update(set(fallback.all_revision_ids())) | 2354 | revids.update(set(fallback.all_revision_ids())) |
151 | 2236 | return list(revids) | 2355 | return list(revids) |
152 | 2237 | 2356 | ||
153 | 2357 | def _filtered_revision_trees(self, revision_ids, file_ids): | ||
154 | 2358 | """Return Tree for a revision on this branch with only some files. | ||
155 | 2359 | |||
156 | 2360 | :param revision_ids: a sequence of revision-ids; | ||
157 | 2361 | a revision-id may not be None or 'null:' | ||
158 | 2362 | :param file_ids: if not None, the result is filtered | ||
159 | 2363 | so that only those file-ids, their parents and their | ||
160 | 2364 | children are included. | ||
161 | 2365 | """ | ||
162 | 2366 | inventories = self.iter_inventories(revision_ids) | ||
163 | 2367 | for inv in inventories: | ||
164 | 2368 | # Should we introduce a FilteredRevisionTree class rather | ||
165 | 2369 | # than pre-filter the inventory here? | ||
166 | 2370 | filtered_inv = inv.filter(file_ids) | ||
167 | 2371 | yield InventoryRevisionTree(self, filtered_inv, filtered_inv.revision_id) | ||
168 | 2372 | |||
169 | 2238 | @needs_read_lock | 2373 | @needs_read_lock |
170 | 2239 | def get_deltas_for_revisions(self, revisions, specific_fileids=None): | 2374 | def get_deltas_for_revisions(self, revisions, specific_fileids=None): |
174 | 2240 | self._ensure_real() | 2375 | medium = self._client._medium |
175 | 2241 | return self._real_repository.get_deltas_for_revisions(revisions, | 2376 | if medium._is_remote_before((1, 2)): |
176 | 2242 | specific_fileids=specific_fileids) | 2377 | self._ensure_real() |
177 | 2378 | for delta in self._real_repository.get_deltas_for_revisions( | ||
178 | 2379 | revisions, specific_fileids): | ||
179 | 2380 | yield delta | ||
180 | 2381 | return | ||
181 | 2382 | # Get the revision-ids of interest | ||
182 | 2383 | required_trees = set() | ||
183 | 2384 | for revision in revisions: | ||
184 | 2385 | required_trees.add(revision.revision_id) | ||
185 | 2386 | required_trees.update(revision.parent_ids[:1]) | ||
186 | 2387 | |||
187 | 2388 | # Get the matching filtered trees. Note that it's more | ||
188 | 2389 | # efficient to pass filtered trees to changes_from() rather | ||
189 | 2390 | # than doing the filtering afterwards. changes_from() could | ||
190 | 2391 | # arguably do the filtering itself but it's path-based, not | ||
191 | 2392 | # file-id based, so filtering before or afterwards is | ||
192 | 2393 | # currently easier. | ||
193 | 2394 | if specific_fileids is None: | ||
194 | 2395 | trees = dict((t.get_revision_id(), t) for | ||
195 | 2396 | t in self.revision_trees(required_trees)) | ||
196 | 2397 | else: | ||
197 | 2398 | trees = dict((t.get_revision_id(), t) for | ||
198 | 2399 | t in self._filtered_revision_trees(required_trees, | ||
199 | 2400 | specific_fileids)) | ||
200 | 2401 | |||
201 | 2402 | # Calculate the deltas | ||
202 | 2403 | for revision in revisions: | ||
203 | 2404 | if not revision.parent_ids: | ||
204 | 2405 | old_tree = self.revision_tree(_mod_revision.NULL_REVISION) | ||
205 | 2406 | else: | ||
206 | 2407 | old_tree = trees[revision.parent_ids[0]] | ||
207 | 2408 | yield trees[revision.revision_id].changes_from(old_tree) | ||
208 | 2243 | 2409 | ||
209 | 2244 | @needs_read_lock | 2410 | @needs_read_lock |
210 | 2245 | def get_revision_delta(self, revision_id, specific_fileids=None): | 2411 | def get_revision_delta(self, revision_id, specific_fileids=None): |
211 | 2246 | 2412 | ||
212 | === modified file 'bzrlib/smart/repository.py' | |||
213 | --- bzrlib/smart/repository.py 2011-12-11 13:30:10 +0000 | |||
214 | +++ bzrlib/smart/repository.py 2011-12-14 21:28:31 +0000 | |||
215 | @@ -14,7 +14,7 @@ | |||
216 | 14 | # along with this program; if not, write to the Free Software | 14 | # along with this program; if not, write to the Free Software |
217 | 15 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA | 15 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA |
218 | 16 | 16 | ||
220 | 17 | """Server-side repository related request implmentations.""" | 17 | """Server-side repository related request implementations.""" |
221 | 18 | 18 | ||
222 | 19 | import bz2 | 19 | import bz2 |
223 | 20 | import os | 20 | import os |
224 | @@ -28,6 +28,8 @@ | |||
225 | 28 | bencode, | 28 | bencode, |
226 | 29 | errors, | 29 | errors, |
227 | 30 | estimate_compressed_size, | 30 | estimate_compressed_size, |
228 | 31 | inventory as _mod_inventory, | ||
229 | 32 | inventory_delta, | ||
230 | 31 | osutils, | 33 | osutils, |
231 | 32 | pack, | 34 | pack, |
232 | 33 | trace, | 35 | trace, |
233 | @@ -43,6 +45,7 @@ | |||
234 | 43 | from bzrlib.repository import _strip_NULL_ghosts, network_format_registry | 45 | from bzrlib.repository import _strip_NULL_ghosts, network_format_registry |
235 | 44 | from bzrlib import revision as _mod_revision | 46 | from bzrlib import revision as _mod_revision |
236 | 45 | from bzrlib.versionedfile import ( | 47 | from bzrlib.versionedfile import ( |
237 | 48 | ChunkedContentFactory, | ||
238 | 46 | NetworkRecordStream, | 49 | NetworkRecordStream, |
239 | 47 | record_to_fulltext_bytes, | 50 | record_to_fulltext_bytes, |
240 | 48 | ) | 51 | ) |
241 | @@ -1243,3 +1246,57 @@ | |||
242 | 1243 | yield zlib.compress(record.get_bytes_as('fulltext')) | 1246 | yield zlib.compress(record.get_bytes_as('fulltext')) |
243 | 1244 | finally: | 1247 | finally: |
244 | 1245 | self._repository.unlock() | 1248 | self._repository.unlock() |
245 | 1249 | |||
246 | 1250 | |||
247 | 1251 | class SmartServerRepositoryGetInventories(SmartServerRepositoryRequest): | ||
248 | 1252 | """Get the inventory deltas for a set of revision ids. | ||
249 | 1253 | |||
250 | 1254 | This accepts a list of revision ids, and then sends a chain | ||
251 | 1255 | of deltas for the inventories of those revisions. The first | ||
252 | 1256 | revision will be empty. | ||
253 | 1257 | |||
254 | 1258 | The server writes back zlibbed serialized inventory deltas, | ||
255 | 1259 | in the ordering specified. The base for each delta is the | ||
256 | 1260 | inventory generated by the previous delta. | ||
257 | 1261 | |||
258 | 1262 | New in 2.5. | ||
259 | 1263 | """ | ||
260 | 1264 | |||
261 | 1265 | def _inventory_delta_stream(self, repository, ordering, revids): | ||
262 | 1266 | prev_inv = _mod_inventory.Inventory(root_id=None, | ||
263 | 1267 | revision_id=_mod_revision.NULL_REVISION) | ||
264 | 1268 | serializer = inventory_delta.InventoryDeltaSerializer( | ||
265 | 1269 | repository.supports_rich_root(), | ||
266 | 1270 | repository._format.supports_tree_reference) | ||
267 | 1271 | repository.lock_read() | ||
268 | 1272 | try: | ||
269 | 1273 | for inv, revid in repository._iter_inventories(revids, ordering): | ||
270 | 1274 | if inv is None: | ||
271 | 1275 | continue | ||
272 | 1276 | inv_delta = inv._make_delta(prev_inv) | ||
273 | 1277 | lines = serializer.delta_to_lines( | ||
274 | 1278 | prev_inv.revision_id, inv.revision_id, inv_delta) | ||
275 | 1279 | yield ChunkedContentFactory(inv.revision_id, None, None, lines) | ||
276 | 1280 | prev_inv = inv | ||
277 | 1281 | finally: | ||
278 | 1282 | repository.unlock() | ||
279 | 1283 | |||
280 | 1284 | def body_stream(self, repository, ordering, revids): | ||
281 | 1285 | substream = self._inventory_delta_stream(repository, | ||
282 | 1286 | ordering, revids) | ||
283 | 1287 | return _stream_to_byte_stream([('inventory-deltas', substream)], | ||
284 | 1288 | repository._format) | ||
285 | 1289 | |||
286 | 1290 | def do_body(self, body_bytes): | ||
287 | 1291 | return SuccessfulSmartServerResponse(('ok', ), | ||
288 | 1292 | body_stream=self.body_stream(self._repository, self._ordering, | ||
289 | 1293 | body_bytes.splitlines())) | ||
290 | 1294 | |||
291 | 1295 | def do_repository_request(self, repository, ordering): | ||
292 | 1296 | if ordering == 'unordered': | ||
293 | 1297 | # inventory deltas for a topologically sorted stream | ||
294 | 1298 | # are likely to be smaller | ||
295 | 1299 | ordering = 'topological' | ||
296 | 1300 | self._ordering = ordering | ||
297 | 1301 | # Signal that we want a body | ||
298 | 1302 | return None | ||
299 | 1246 | 1303 | ||
300 | === modified file 'bzrlib/smart/request.py' | |||
301 | --- bzrlib/smart/request.py 2011-12-14 12:20:36 +0000 | |||
302 | +++ bzrlib/smart/request.py 2011-12-14 21:28:31 +0000 | |||
303 | @@ -751,15 +751,18 @@ | |||
304 | 751 | 'Repository.check_write_group', 'bzrlib.smart.repository', | 751 | 'Repository.check_write_group', 'bzrlib.smart.repository', |
305 | 752 | 'SmartServerRepositoryCheckWriteGroup', info='read') | 752 | 'SmartServerRepositoryCheckWriteGroup', info='read') |
306 | 753 | request_handlers.register_lazy( | 753 | request_handlers.register_lazy( |
307 | 754 | 'VersionedFileRepository.get_serializer_format', 'bzrlib.smart.repository', | ||
308 | 755 | 'SmartServerRepositoryGetSerializerFormat', info='read') | ||
309 | 756 | request_handlers.register_lazy( | ||
310 | 757 | 'Repository.reconcile', 'bzrlib.smart.repository', | 754 | 'Repository.reconcile', 'bzrlib.smart.repository', |
311 | 758 | 'SmartServerRepositoryReconcile', info='idem') | 755 | 'SmartServerRepositoryReconcile', info='idem') |
312 | 759 | request_handlers.register_lazy( | 756 | request_handlers.register_lazy( |
313 | 760 | 'Repository.tarball', 'bzrlib.smart.repository', | 757 | 'Repository.tarball', 'bzrlib.smart.repository', |
314 | 761 | 'SmartServerRepositoryTarball', info='read') | 758 | 'SmartServerRepositoryTarball', info='read') |
315 | 762 | request_handlers.register_lazy( | 759 | request_handlers.register_lazy( |
316 | 760 | 'VersionedFileRepository.get_serializer_format', 'bzrlib.smart.repository', | ||
317 | 761 | 'SmartServerRepositoryGetSerializerFormat', info='read') | ||
318 | 762 | request_handlers.register_lazy( | ||
319 | 763 | 'VersionedFileRepository.get_inventories', 'bzrlib.smart.repository', | ||
320 | 764 | 'SmartServerRepositoryGetInventories', info='read') | ||
321 | 765 | request_handlers.register_lazy( | ||
322 | 763 | 'rmdir', 'bzrlib.smart.vfs', 'RmdirRequest', info='semivfs') | 766 | 'rmdir', 'bzrlib.smart.vfs', 'RmdirRequest', info='semivfs') |
323 | 764 | request_handlers.register_lazy( | 767 | request_handlers.register_lazy( |
324 | 765 | 'stat', 'bzrlib.smart.vfs', 'StatRequest', info='read') | 768 | 'stat', 'bzrlib.smart.vfs', 'StatRequest', info='read') |
325 | 766 | 769 | ||
326 | === modified file 'bzrlib/tests/blackbox/test_annotate.py' | |||
327 | --- bzrlib/tests/blackbox/test_annotate.py 2011-12-14 12:15:44 +0000 | |||
328 | +++ bzrlib/tests/blackbox/test_annotate.py 2011-12-14 21:28:31 +0000 | |||
329 | @@ -326,6 +326,6 @@ | |||
330 | 326 | # being too low. If rpc_count increases, more network roundtrips have | 326 | # being too low. If rpc_count increases, more network roundtrips have |
331 | 327 | # become necessary for this use case. Please do not adjust this number | 327 | # become necessary for this use case. Please do not adjust this number |
332 | 328 | # upwards without agreement from bzr's network support maintainers. | 328 | # upwards without agreement from bzr's network support maintainers. |
334 | 329 | self.assertLength(19, self.hpss_calls) | 329 | self.assertLength(15, self.hpss_calls) |
335 | 330 | self.expectFailure("annotate accesses inventories, which require VFS access", | 330 | self.expectFailure("annotate accesses inventories, which require VFS access", |
336 | 331 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | 331 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) |
337 | 332 | 332 | ||
338 | === modified file 'bzrlib/tests/blackbox/test_branch.py' | |||
339 | --- bzrlib/tests/blackbox/test_branch.py 2011-12-14 18:17:43 +0000 | |||
340 | +++ bzrlib/tests/blackbox/test_branch.py 2011-12-14 21:28:31 +0000 | |||
341 | @@ -483,7 +483,7 @@ | |||
342 | 483 | # being too low. If rpc_count increases, more network roundtrips have | 483 | # being too low. If rpc_count increases, more network roundtrips have |
343 | 484 | # become necessary for this use case. Please do not adjust this number | 484 | # become necessary for this use case. Please do not adjust this number |
344 | 485 | # upwards without agreement from bzr's network support maintainers. | 485 | # upwards without agreement from bzr's network support maintainers. |
346 | 486 | self.assertLength(40, self.hpss_calls) | 486 | self.assertLength(33, self.hpss_calls) |
347 | 487 | self.expectFailure("branching to the same branch requires VFS access", | 487 | self.expectFailure("branching to the same branch requires VFS access", |
348 | 488 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | 488 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) |
349 | 489 | 489 | ||
350 | @@ -521,8 +521,8 @@ | |||
351 | 521 | # being too low. If rpc_count increases, more network roundtrips have | 521 | # being too low. If rpc_count increases, more network roundtrips have |
352 | 522 | # become necessary for this use case. Please do not adjust this number | 522 | # become necessary for this use case. Please do not adjust this number |
353 | 523 | # upwards without agreement from bzr's network support maintainers. | 523 | # upwards without agreement from bzr's network support maintainers. |
354 | 524 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) | ||
355 | 525 | self.assertLength(15, self.hpss_calls) | 524 | self.assertLength(15, self.hpss_calls) |
356 | 525 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) | ||
357 | 526 | 526 | ||
358 | 527 | def test_branch_from_branch_with_tags(self): | 527 | def test_branch_from_branch_with_tags(self): |
359 | 528 | self.setup_smart_server_with_call_log() | 528 | self.setup_smart_server_with_call_log() |
360 | @@ -539,8 +539,8 @@ | |||
361 | 539 | # being too low. If rpc_count increases, more network roundtrips have | 539 | # being too low. If rpc_count increases, more network roundtrips have |
362 | 540 | # become necessary for this use case. Please do not adjust this number | 540 | # become necessary for this use case. Please do not adjust this number |
363 | 541 | # upwards without agreement from bzr's network support maintainers. | 541 | # upwards without agreement from bzr's network support maintainers. |
364 | 542 | self.assertLength(10, self.hpss_calls) | ||
365 | 542 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) | 543 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
366 | 543 | self.assertLength(10, self.hpss_calls) | ||
367 | 544 | 544 | ||
368 | 545 | def test_branch_to_stacked_from_trivial_branch_streaming_acceptance(self): | 545 | def test_branch_to_stacked_from_trivial_branch_streaming_acceptance(self): |
369 | 546 | self.setup_smart_server_with_call_log() | 546 | self.setup_smart_server_with_call_log() |
370 | 547 | 547 | ||
371 | === modified file 'bzrlib/tests/blackbox/test_cat.py' | |||
372 | --- bzrlib/tests/blackbox/test_cat.py 2011-12-14 12:15:44 +0000 | |||
373 | +++ bzrlib/tests/blackbox/test_cat.py 2011-12-14 21:28:31 +0000 | |||
374 | @@ -239,6 +239,5 @@ | |||
375 | 239 | # being too low. If rpc_count increases, more network roundtrips have | 239 | # being too low. If rpc_count increases, more network roundtrips have |
376 | 240 | # become necessary for this use case. Please do not adjust this number | 240 | # become necessary for this use case. Please do not adjust this number |
377 | 241 | # upwards without agreement from bzr's network support maintainers. | 241 | # upwards without agreement from bzr's network support maintainers. |
381 | 242 | self.assertLength(16, self.hpss_calls) | 242 | self.assertLength(9, self.hpss_calls) |
382 | 243 | self.expectFailure("cat still uses VFS calls", | 243 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
380 | 244 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
383 | 245 | 244 | ||
384 | === modified file 'bzrlib/tests/blackbox/test_checkout.py' | |||
385 | --- bzrlib/tests/blackbox/test_checkout.py 2011-12-14 18:17:43 +0000 | |||
386 | +++ bzrlib/tests/blackbox/test_checkout.py 2011-12-14 21:28:31 +0000 | |||
387 | @@ -179,8 +179,7 @@ | |||
388 | 179 | for count in range(9): | 179 | for count in range(9): |
389 | 180 | t.commit(message='commit %d' % count) | 180 | t.commit(message='commit %d' % count) |
390 | 181 | self.reset_smart_call_log() | 181 | self.reset_smart_call_log() |
393 | 182 | out, err = self.run_bzr(['checkout', self.get_url('from'), | 182 | out, err = self.run_bzr(['checkout', self.get_url('from'), 'target']) |
392 | 183 | 'target']) | ||
394 | 184 | # This figure represent the amount of work to perform this use case. It | 183 | # This figure represent the amount of work to perform this use case. It |
395 | 185 | # is entirely ok to reduce this number if a test fails due to rpc_count | 184 | # is entirely ok to reduce this number if a test fails due to rpc_count |
396 | 186 | # being too low. If rpc_count increases, more network roundtrips have | 185 | # being too low. If rpc_count increases, more network roundtrips have |
397 | @@ -202,9 +201,5 @@ | |||
398 | 202 | # being too low. If rpc_count increases, more network roundtrips have | 201 | # being too low. If rpc_count increases, more network roundtrips have |
399 | 203 | # become necessary for this use case. Please do not adjust this number | 202 | # become necessary for this use case. Please do not adjust this number |
400 | 204 | # upwards without agreement from bzr's network support maintainers. | 203 | # upwards without agreement from bzr's network support maintainers. |
407 | 205 | if len(self.hpss_calls) < 28 or len(self.hpss_calls) > 40: | 204 | self.assertLength(15, self.hpss_calls) |
408 | 206 | self.fail( | 205 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
403 | 207 | "Incorrect length: wanted between 28 and 40, got %d for %r" % ( | ||
404 | 208 | len(self.hpss_calls), self.hpss_calls)) | ||
405 | 209 | self.expectFailure("lightweight checkouts require VFS calls", | ||
406 | 210 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
409 | 211 | 206 | ||
410 | === modified file 'bzrlib/tests/blackbox/test_export.py' | |||
411 | --- bzrlib/tests/blackbox/test_export.py 2011-12-14 12:15:44 +0000 | |||
412 | +++ bzrlib/tests/blackbox/test_export.py 2011-12-14 21:28:31 +0000 | |||
413 | @@ -448,6 +448,5 @@ | |||
414 | 448 | # being too low. If rpc_count increases, more network roundtrips have | 448 | # being too low. If rpc_count increases, more network roundtrips have |
415 | 449 | # become necessary for this use case. Please do not adjust this number | 449 | # become necessary for this use case. Please do not adjust this number |
416 | 450 | # upwards without agreement from bzr's network support maintainers. | 450 | # upwards without agreement from bzr's network support maintainers. |
420 | 451 | self.assertLength(16, self.hpss_calls) | 451 | self.assertLength(7, self.hpss_calls) |
421 | 452 | self.expectFailure("export requires inventory access which requires VFS", | 452 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
419 | 453 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
422 | 454 | 453 | ||
423 | === modified file 'bzrlib/tests/blackbox/test_log.py' | |||
424 | --- bzrlib/tests/blackbox/test_log.py 2011-12-14 12:15:44 +0000 | |||
425 | +++ bzrlib/tests/blackbox/test_log.py 2011-12-14 21:28:31 +0000 | |||
426 | @@ -1086,9 +1086,8 @@ | |||
427 | 1086 | # being too low. If rpc_count increases, more network roundtrips have | 1086 | # being too low. If rpc_count increases, more network roundtrips have |
428 | 1087 | # become necessary for this use case. Please do not adjust this number | 1087 | # become necessary for this use case. Please do not adjust this number |
429 | 1088 | # upwards without agreement from bzr's network support maintainers. | 1088 | # upwards without agreement from bzr's network support maintainers. |
433 | 1089 | self.assertLength(19, self.hpss_calls) | 1089 | self.assertLength(11, self.hpss_calls) |
434 | 1090 | self.expectFailure("verbose log accesses inventories, which require VFS", | 1090 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
432 | 1091 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
435 | 1092 | 1091 | ||
436 | 1093 | def test_per_file(self): | 1092 | def test_per_file(self): |
437 | 1094 | self.setup_smart_server_with_call_log() | 1093 | self.setup_smart_server_with_call_log() |
438 | @@ -1103,6 +1102,5 @@ | |||
439 | 1103 | # being too low. If rpc_count increases, more network roundtrips have | 1102 | # being too low. If rpc_count increases, more network roundtrips have |
440 | 1104 | # become necessary for this use case. Please do not adjust this number | 1103 | # become necessary for this use case. Please do not adjust this number |
441 | 1105 | # upwards without agreement from bzr's network support maintainers. | 1104 | # upwards without agreement from bzr's network support maintainers. |
445 | 1106 | self.assertLength(21, self.hpss_calls) | 1105 | self.assertLength(15, self.hpss_calls) |
446 | 1107 | self.expectFailure("per-file graph access requires VFS", | 1106 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
444 | 1108 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
447 | 1109 | 1107 | ||
448 | === modified file 'bzrlib/tests/blackbox/test_ls.py' | |||
449 | --- bzrlib/tests/blackbox/test_ls.py 2011-12-14 12:15:44 +0000 | |||
450 | +++ bzrlib/tests/blackbox/test_ls.py 2011-12-14 21:28:31 +0000 | |||
451 | @@ -262,6 +262,5 @@ | |||
452 | 262 | # being too low. If rpc_count increases, more network roundtrips have | 262 | # being too low. If rpc_count increases, more network roundtrips have |
453 | 263 | # become necessary for this use case. Please do not adjust this number | 263 | # become necessary for this use case. Please do not adjust this number |
454 | 264 | # upwards without agreement from bzr's network support maintainers. | 264 | # upwards without agreement from bzr's network support maintainers. |
458 | 265 | self.assertLength(15, self.hpss_calls) | 265 | self.assertLength(6, self.hpss_calls) |
459 | 266 | self.expectFailure("inventories can only be accessed over VFS", | 266 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
457 | 267 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
460 | 268 | 267 | ||
461 | === modified file 'bzrlib/tests/blackbox/test_sign_my_commits.py' | |||
462 | --- bzrlib/tests/blackbox/test_sign_my_commits.py 2011-12-14 18:17:43 +0000 | |||
463 | +++ bzrlib/tests/blackbox/test_sign_my_commits.py 2011-12-14 21:28:31 +0000 | |||
464 | @@ -165,9 +165,8 @@ | |||
465 | 165 | # being too low. If rpc_count increases, more network roundtrips have | 165 | # being too low. If rpc_count increases, more network roundtrips have |
466 | 166 | # become necessary for this use case. Please do not adjust this number | 166 | # become necessary for this use case. Please do not adjust this number |
467 | 167 | # upwards without agreement from bzr's network support maintainers. | 167 | # upwards without agreement from bzr's network support maintainers. |
471 | 168 | self.assertLength(51, self.hpss_calls) | 168 | self.assertLength(15, self.hpss_calls) |
472 | 169 | self.expectFailure("signing commits requires VFS access", | 169 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
470 | 170 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
473 | 171 | 170 | ||
474 | 172 | def test_verify_commits(self): | 171 | def test_verify_commits(self): |
475 | 173 | self.setup_smart_server_with_call_log() | 172 | self.setup_smart_server_with_call_log() |
476 | @@ -185,12 +184,5 @@ | |||
477 | 185 | # being too low. If rpc_count increases, more network roundtrips have | 184 | # being too low. If rpc_count increases, more network roundtrips have |
478 | 186 | # become necessary for this use case. Please do not adjust this number | 185 | # become necessary for this use case. Please do not adjust this number |
479 | 187 | # upwards without agreement from bzr's network support maintainers. | 186 | # upwards without agreement from bzr's network support maintainers. |
489 | 188 | 187 | self.assertLength(10, self.hpss_calls) | |
490 | 189 | # The number of readv requests seems to vary depending on the generated | 188 | self.assertThat(self.hpss_calls, ContainsNoVfsCalls) |
482 | 190 | # repository and how well it compresses, so allow for a bit of | ||
483 | 191 | # variation: | ||
484 | 192 | if len(self.hpss_calls) not in (18, 19): | ||
485 | 193 | self.fail("Incorrect length: wanted 18 or 19, got %d for %r" % ( | ||
486 | 194 | len(self.hpss_calls), self.hpss_calls)) | ||
487 | 195 | self.expectFailure("verifying commits requires VFS access", | ||
488 | 196 | self.assertThat, self.hpss_calls, ContainsNoVfsCalls) | ||
491 | 197 | 189 | ||
492 | === modified file 'bzrlib/tests/per_interbranch/test_push.py' | |||
493 | --- bzrlib/tests/per_interbranch/test_push.py 2011-10-15 01:09:01 +0000 | |||
494 | +++ bzrlib/tests/per_interbranch/test_push.py 2011-12-14 21:28:31 +0000 | |||
495 | @@ -279,10 +279,10 @@ | |||
496 | 279 | # remote graph any further. | 279 | # remote graph any further. |
497 | 280 | bzr_core_trace = Equals( | 280 | bzr_core_trace = Equals( |
498 | 281 | ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19', | 281 | ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19', |
500 | 282 | 'get', 'Branch.set_last_revision_info', 'Branch.unlock']) | 282 | 'Branch.set_last_revision_info', 'Branch.unlock']) |
501 | 283 | bzr_loom_trace = Equals( | 283 | bzr_loom_trace = Equals( |
502 | 284 | ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19', | 284 | ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19', |
504 | 285 | 'get', 'Branch.set_last_revision_info', 'get', 'Branch.unlock']) | 285 | 'Branch.set_last_revision_info', 'get', 'Branch.unlock']) |
505 | 286 | self.assertThat(calls_after_insert_stream, | 286 | self.assertThat(calls_after_insert_stream, |
506 | 287 | MatchesAny(bzr_core_trace, bzr_loom_trace)) | 287 | MatchesAny(bzr_core_trace, bzr_loom_trace)) |
507 | 288 | 288 | ||
508 | 289 | 289 | ||
509 | === modified file 'bzrlib/tests/per_repository_chk/test_supported.py' | |||
510 | --- bzrlib/tests/per_repository_chk/test_supported.py 2009-09-09 18:52:56 +0000 | |||
511 | +++ bzrlib/tests/per_repository_chk/test_supported.py 2011-12-14 21:28:31 +0000 | |||
512 | @@ -22,7 +22,9 @@ | |||
513 | 22 | osutils, | 22 | osutils, |
514 | 23 | repository, | 23 | repository, |
515 | 24 | ) | 24 | ) |
516 | 25 | from bzrlib.remote import RemoteRepository | ||
517 | 25 | from bzrlib.versionedfile import VersionedFiles | 26 | from bzrlib.versionedfile import VersionedFiles |
518 | 27 | from bzrlib.tests import TestNotApplicable | ||
519 | 26 | from bzrlib.tests.per_repository_chk import TestCaseWithRepositoryCHK | 28 | from bzrlib.tests.per_repository_chk import TestCaseWithRepositoryCHK |
520 | 27 | 29 | ||
521 | 28 | 30 | ||
522 | @@ -221,7 +223,9 @@ | |||
523 | 221 | # check our setup: B-id and C-id should have identical chk root keys. | 223 | # check our setup: B-id and C-id should have identical chk root keys. |
524 | 222 | inv_b = b.repository.get_inventory('B-id') | 224 | inv_b = b.repository.get_inventory('B-id') |
525 | 223 | inv_c = b.repository.get_inventory('C-id') | 225 | inv_c = b.repository.get_inventory('C-id') |
527 | 224 | self.assertEqual(inv_b.id_to_entry.key(), inv_c.id_to_entry.key()) | 226 | if not isinstance(repo, RemoteRepository): |
528 | 227 | # Remote repositories always return plain inventories | ||
529 | 228 | self.assertEqual(inv_b.id_to_entry.key(), inv_c.id_to_entry.key()) | ||
530 | 225 | # Now, manually insert objects for a stacked repo with only revision | 229 | # Now, manually insert objects for a stacked repo with only revision |
531 | 226 | # C-id: | 230 | # C-id: |
532 | 227 | # We need ('revisions', 'C-id'), ('inventories', 'C-id'), | 231 | # We need ('revisions', 'C-id'), ('inventories', 'C-id'), |
533 | @@ -250,6 +254,9 @@ | |||
534 | 250 | for a parent inventory of a new revision is missing. | 254 | for a parent inventory of a new revision is missing. |
535 | 251 | """ | 255 | """ |
536 | 252 | repo = self.make_repository('damaged-repo') | 256 | repo = self.make_repository('damaged-repo') |
537 | 257 | if isinstance(repo, RemoteRepository): | ||
538 | 258 | raise TestNotApplicable( | ||
539 | 259 | "Unable to obtain CHKInventory from remote repo") | ||
540 | 253 | b = self.make_branch_with_multiple_chk_nodes() | 260 | b = self.make_branch_with_multiple_chk_nodes() |
541 | 254 | src_repo = b.repository | 261 | src_repo = b.repository |
542 | 255 | src_repo.lock_read() | 262 | src_repo.lock_read() |
543 | @@ -293,6 +300,9 @@ | |||
544 | 293 | for a parent inventory of a new revision is missing. | 300 | for a parent inventory of a new revision is missing. |
545 | 294 | """ | 301 | """ |
546 | 295 | repo = self.make_repository('damaged-repo') | 302 | repo = self.make_repository('damaged-repo') |
547 | 303 | if isinstance(repo, RemoteRepository): | ||
548 | 304 | raise TestNotApplicable( | ||
549 | 305 | "Unable to obtain CHKInventory from remote repo") | ||
550 | 296 | b = self.make_branch_with_multiple_chk_nodes() | 306 | b = self.make_branch_with_multiple_chk_nodes() |
551 | 297 | b.lock_read() | 307 | b.lock_read() |
552 | 298 | self.addCleanup(b.unlock) | 308 | self.addCleanup(b.unlock) |
553 | 299 | 309 | ||
554 | === modified file 'bzrlib/tests/per_repository_vf/test_add_inventory_by_delta.py' | |||
555 | --- bzrlib/tests/per_repository_vf/test_add_inventory_by_delta.py 2011-05-02 22:39:15 +0000 | |||
556 | +++ bzrlib/tests/per_repository_vf/test_add_inventory_by_delta.py 2011-12-14 21:28:31 +0000 | |||
557 | @@ -109,4 +109,4 @@ | |||
558 | 109 | else: | 109 | else: |
559 | 110 | repo_delta.commit_write_group() | 110 | repo_delta.commit_write_group() |
560 | 111 | self.assertEqual(add_validator, delta_validator) | 111 | self.assertEqual(add_validator, delta_validator) |
562 | 112 | self.assertEqual(new_inv, inv) | 112 | self.assertEqual(list(new_inv.iter_entries()), list(inv.iter_entries())) |
563 | 113 | 113 | ||
564 | === modified file 'bzrlib/tests/test_remote.py' | |||
565 | --- bzrlib/tests/test_remote.py 2011-12-14 12:20:36 +0000 | |||
566 | +++ bzrlib/tests/test_remote.py 2011-12-14 21:28:31 +0000 | |||
567 | @@ -69,6 +69,7 @@ | |||
568 | 69 | from bzrlib.smart.repository import ( | 69 | from bzrlib.smart.repository import ( |
569 | 70 | SmartServerRepositoryGetParentMap, | 70 | SmartServerRepositoryGetParentMap, |
570 | 71 | SmartServerRepositoryGetStream_1_19, | 71 | SmartServerRepositoryGetStream_1_19, |
571 | 72 | _stream_to_byte_stream, | ||
572 | 72 | ) | 73 | ) |
573 | 73 | from bzrlib.symbol_versioning import deprecated_in | 74 | from bzrlib.symbol_versioning import deprecated_in |
574 | 74 | from bzrlib.tests import ( | 75 | from bzrlib.tests import ( |
575 | @@ -4233,3 +4234,43 @@ | |||
576 | 4233 | 'Repository.unlock', ('quack/', 'token', 'False'), | 4234 | 'Repository.unlock', ('quack/', 'token', 'False'), |
577 | 4234 | 'success', ('ok', )) | 4235 | 'success', ('ok', )) |
578 | 4235 | repo.pack(['hinta', 'hintb']) | 4236 | repo.pack(['hinta', 'hintb']) |
579 | 4237 | |||
580 | 4238 | |||
581 | 4239 | class TestRepositoryIterInventories(TestRemoteRepository): | ||
582 | 4240 | """Test Repository.iter_inventories.""" | ||
583 | 4241 | |||
584 | 4242 | def _serialize_inv_delta(self, old_name, new_name, delta): | ||
585 | 4243 | serializer = inventory_delta.InventoryDeltaSerializer(True, False) | ||
586 | 4244 | return "".join(serializer.delta_to_lines(old_name, new_name, delta)) | ||
587 | 4245 | |||
588 | 4246 | def test_single_empty(self): | ||
589 | 4247 | transport_path = 'quack' | ||
590 | 4248 | repo, client = self.setup_fake_client_and_repository(transport_path) | ||
591 | 4249 | fmt = bzrdir.format_registry.get('2a')().repository_format | ||
592 | 4250 | repo._format = fmt | ||
593 | 4251 | stream = [('inventory-deltas', [ | ||
594 | 4252 | versionedfile.FulltextContentFactory('somerevid', None, None, | ||
595 | 4253 | self._serialize_inv_delta('null:', 'somerevid', []))])] | ||
596 | 4254 | client.add_expected_call( | ||
597 | 4255 | 'VersionedFileRepository.get_inventories', ('quack/', 'unordered'), | ||
598 | 4256 | 'success', ('ok', ), | ||
599 | 4257 | _stream_to_byte_stream(stream, fmt)) | ||
600 | 4258 | ret = list(repo.iter_inventories(["somerevid"])) | ||
601 | 4259 | self.assertLength(1, ret) | ||
602 | 4260 | inv = ret[0] | ||
603 | 4261 | self.assertEquals("somerevid", inv.revision_id) | ||
604 | 4262 | |||
605 | 4263 | def test_empty(self): | ||
606 | 4264 | transport_path = 'quack' | ||
607 | 4265 | repo, client = self.setup_fake_client_and_repository(transport_path) | ||
608 | 4266 | ret = list(repo.iter_inventories([])) | ||
609 | 4267 | self.assertEquals(ret, []) | ||
610 | 4268 | |||
611 | 4269 | def test_missing(self): | ||
612 | 4270 | transport_path = 'quack' | ||
613 | 4271 | repo, client = self.setup_fake_client_and_repository(transport_path) | ||
614 | 4272 | client.add_expected_call( | ||
615 | 4273 | 'VersionedFileRepository.get_inventories', ('quack/', 'unordered'), | ||
616 | 4274 | 'success', ('ok', ), iter([])) | ||
617 | 4275 | self.assertRaises(errors.NoSuchRevision, list, repo.iter_inventories( | ||
618 | 4276 | ["somerevid"])) | ||
619 | 4236 | 4277 | ||
620 | === modified file 'bzrlib/tests/test_smart.py' | |||
621 | --- bzrlib/tests/test_smart.py 2011-12-14 12:20:36 +0000 | |||
622 | +++ bzrlib/tests/test_smart.py 2011-12-14 21:28:31 +0000 | |||
623 | @@ -32,6 +32,7 @@ | |||
624 | 32 | bzrdir, | 32 | bzrdir, |
625 | 33 | errors, | 33 | errors, |
626 | 34 | gpg, | 34 | gpg, |
627 | 35 | inventory_delta, | ||
628 | 35 | tests, | 36 | tests, |
629 | 36 | transport, | 37 | transport, |
630 | 37 | urlutils, | 38 | urlutils, |
631 | @@ -2565,6 +2566,8 @@ | |||
632 | 2565 | smart_repo.SmartServerRepositoryAbortWriteGroup) | 2566 | smart_repo.SmartServerRepositoryAbortWriteGroup) |
633 | 2566 | self.assertHandlerEqual('VersionedFileRepository.get_serializer_format', | 2567 | self.assertHandlerEqual('VersionedFileRepository.get_serializer_format', |
634 | 2567 | smart_repo.SmartServerRepositoryGetSerializerFormat) | 2568 | smart_repo.SmartServerRepositoryGetSerializerFormat) |
635 | 2569 | self.assertHandlerEqual('VersionedFileRepository.get_inventories', | ||
636 | 2570 | smart_repo.SmartServerRepositoryGetInventories) | ||
637 | 2568 | self.assertHandlerEqual('Transport.is_readonly', | 2571 | self.assertHandlerEqual('Transport.is_readonly', |
638 | 2569 | smart_req.SmartServerIsReadonly) | 2572 | smart_req.SmartServerIsReadonly) |
639 | 2570 | 2573 | ||
640 | @@ -2630,3 +2633,48 @@ | |||
641 | 2630 | smart_req.SuccessfulSmartServerResponse(('ok', ), ), | 2633 | smart_req.SuccessfulSmartServerResponse(('ok', ), ), |
642 | 2631 | request.do_body('')) | 2634 | request.do_body('')) |
643 | 2632 | 2635 | ||
644 | 2636 | |||
645 | 2637 | class TestSmartServerRepositoryGetInventories(tests.TestCaseWithTransport): | ||
646 | 2638 | |||
647 | 2639 | def _get_serialized_inventory_delta(self, repository, base_revid, revid): | ||
648 | 2640 | base_inv = repository.revision_tree(base_revid).inventory | ||
649 | 2641 | inv = repository.revision_tree(revid).inventory | ||
650 | 2642 | inv_delta = inv._make_delta(base_inv) | ||
651 | 2643 | serializer = inventory_delta.InventoryDeltaSerializer(True, False) | ||
652 | 2644 | return "".join(serializer.delta_to_lines(base_revid, revid, inv_delta)) | ||
653 | 2645 | |||
654 | 2646 | def test_single(self): | ||
655 | 2647 | backing = self.get_transport() | ||
656 | 2648 | request = smart_repo.SmartServerRepositoryGetInventories(backing) | ||
657 | 2649 | t = self.make_branch_and_tree('.', format='2a') | ||
658 | 2650 | self.addCleanup(t.lock_write().unlock) | ||
659 | 2651 | self.build_tree_contents([("file", "somecontents")]) | ||
660 | 2652 | t.add(["file"], ["thefileid"]) | ||
661 | 2653 | t.commit(rev_id='somerev', message="add file") | ||
662 | 2654 | self.assertIs(None, request.execute('', 'unordered')) | ||
663 | 2655 | response = request.do_body("somerev\n") | ||
664 | 2656 | self.assertTrue(response.is_successful()) | ||
665 | 2657 | self.assertEquals(response.args, ("ok", )) | ||
666 | 2658 | stream = [('inventory-deltas', [ | ||
667 | 2659 | versionedfile.FulltextContentFactory('somerev', None, None, | ||
668 | 2660 | self._get_serialized_inventory_delta( | ||
669 | 2661 | t.branch.repository, 'null:', 'somerev'))])] | ||
670 | 2662 | fmt = bzrdir.format_registry.get('2a')().repository_format | ||
671 | 2663 | self.assertEquals( | ||
672 | 2664 | "".join(response.body_stream), | ||
673 | 2665 | "".join(smart_repo._stream_to_byte_stream(stream, fmt))) | ||
674 | 2666 | |||
675 | 2667 | def test_empty(self): | ||
676 | 2668 | backing = self.get_transport() | ||
677 | 2669 | request = smart_repo.SmartServerRepositoryGetInventories(backing) | ||
678 | 2670 | t = self.make_branch_and_tree('.', format='2a') | ||
679 | 2671 | self.addCleanup(t.lock_write().unlock) | ||
680 | 2672 | self.build_tree_contents([("file", "somecontents")]) | ||
681 | 2673 | t.add(["file"], ["thefileid"]) | ||
682 | 2674 | t.commit(rev_id='somerev', message="add file") | ||
683 | 2675 | self.assertIs(None, request.execute('', 'unordered')) | ||
684 | 2676 | response = request.do_body("") | ||
685 | 2677 | self.assertTrue(response.is_successful()) | ||
686 | 2678 | self.assertEquals(response.args, ("ok", )) | ||
687 | 2679 | self.assertEquals("".join(response.body_stream), | ||
688 | 2680 | "Bazaar pack format 1 (introduced in 0.18)\nB54\n\nBazaar repository format 2a (needs bzr 1.16 or later)\nE") | ||
689 | 2633 | 2681 | ||
690 | === modified file 'doc/en/release-notes/bzr-2.5.txt' | |||
691 | --- doc/en/release-notes/bzr-2.5.txt 2011-12-14 20:39:58 +0000 | |||
692 | +++ doc/en/release-notes/bzr-2.5.txt 2011-12-14 21:28:31 +0000 | |||
693 | @@ -95,6 +95,10 @@ | |||
694 | 95 | 95 | ||
695 | 96 | * New HPSS call ``BzrDir.checkout_metadir``. (Jelmer Vernooij, #894459) | 96 | * New HPSS call ``BzrDir.checkout_metadir``. (Jelmer Vernooij, #894459) |
696 | 97 | 97 | ||
697 | 98 | * New HPSS call ``VersionedFileRepository.get_inventories``, | ||
698 | 99 | speeding up various commands including ``bzr export``, | ||
699 | 100 | ``bzr checkout`` and ``bzr cat``. (Jelmer Vernooij, #608640) | ||
700 | 101 | |||
701 | 98 | Testing | 102 | Testing |
702 | 99 | ******* | 103 | ******* |
703 | 100 | 104 | ||
704 | @@ -156,6 +160,9 @@ | |||
705 | 156 | * Plugins can now register additional "location aliases". | 160 | * Plugins can now register additional "location aliases". |
706 | 157 | (Jelmer Vernooij) | 161 | (Jelmer Vernooij) |
707 | 158 | 162 | ||
708 | 163 | * ``bzr status`` no longer shows shelves if files are specified. | ||
709 | 164 | (Francis Devereux) | ||
710 | 165 | |||
711 | 159 | * Revision specifiers will now only browse as much history as they | 166 | * Revision specifiers will now only browse as much history as they |
712 | 160 | need to, rather than grabbing the whole history unnecessarily in some | 167 | need to, rather than grabbing the whole history unnecessarily in some |
713 | 161 | cases. (Jelmer Vernooij) | 168 | cases. (Jelmer Vernooij) |
714 | @@ -281,8 +288,14 @@ | |||
715 | 281 | ``Repository.get_revision_signature_text``. | 288 | ``Repository.get_revision_signature_text``. |
716 | 282 | (Jelmer Vernooij) | 289 | (Jelmer Vernooij) |
717 | 283 | 290 | ||
718 | 291 | * Add HPSS call for ``Branch.get_checkout_format``. (Jelmer Vernooij, #894459) | ||
719 | 292 | |||
720 | 284 | * Add HPSS call for ``Repository.pack``. (Jelmer Vernooij, #894461) | 293 | * Add HPSS call for ``Repository.pack``. (Jelmer Vernooij, #894461) |
721 | 285 | 294 | ||
722 | 295 | * Add HPSS calls for ``Repository.iter_files_bytes``, speeding up | ||
723 | 296 | several commands including ``bzr export`` and ``bzr co --lightweight``. | ||
724 | 297 | (Jelmer Vernooij, #608640) | ||
725 | 298 | |||
726 | 286 | * Custom HPSS error handlers can now be installed in the smart server client | 299 | * Custom HPSS error handlers can now be installed in the smart server client |
727 | 287 | using the ``error_translators`` and ``no_context_error_translators`` | 300 | using the ``error_translators`` and ``no_context_error_translators`` |
728 | 288 | registries. (Jelmer Vernooij) | 301 | registries. (Jelmer Vernooij) |
Timings when checking out bzr.dev:
~/src/bzr/ hpss-get- inventories/ bzr co --lightweight bzr://people. samba.org/ bzr.dev 2.25s user 0.46s system 15% cpu 17.277 total hpss-get- inventories/ bzr export /tmp/bzr.dev2 bzr://people. samba.org/ bzr.dev 1.54s user 0.40s system 12% cpu 15.435 total
~/src/bzr/
versus against an older bzr server:
~/src/bzr/ hpss-get- inventories/ bzr co --lightweight bzr://people. samba.org/ bzr.dev 4.91s user 1.03s system 10% cpu 58.807 total hpss-get- inventories/ bzr export /tmp/bzr.dev5 bzr://people. samba.org/ bzr.dev 4.35s user 0.98s system 9% cpu 58.357 total
~/src/bzr/