Merge lp:~jelmer/bzr/hpss-get-inventories into lp:bzr

Proposed by Jelmer Vernooij
Status: Superseded
Proposed branch: lp:~jelmer/bzr/hpss-get-inventories
Merge into: lp:bzr
Prerequisite: lp:~jelmer/bzr/hpss-_get-checkout-format
Diff against target: 969 lines (+466/-182)
17 files modified
bzrlib/remote.py (+207/-50)
bzrlib/smart/branch.py (+0/-19)
bzrlib/smart/bzrdir.py (+19/-0)
bzrlib/smart/repository.py (+58/-1)
bzrlib/smart/request.py (+6/-3)
bzrlib/tests/blackbox/test_annotate.py (+1/-1)
bzrlib/tests/blackbox/test_branch.py (+1/-1)
bzrlib/tests/blackbox/test_cat.py (+2/-3)
bzrlib/tests/blackbox/test_checkout.py (+3/-8)
bzrlib/tests/blackbox/test_export.py (+2/-3)
bzrlib/tests/blackbox/test_log.py (+4/-6)
bzrlib/tests/blackbox/test_ls.py (+2/-3)
bzrlib/tests/blackbox/test_sign_my_commits.py (+3/-10)
bzrlib/tests/per_interbranch/test_push.py (+2/-2)
bzrlib/tests/test_remote.py (+78/-41)
bzrlib/tests/test_smart.py (+68/-31)
doc/en/release-notes/bzr-2.5.txt (+10/-0)
To merge this branch: bzr merge lp:~jelmer/bzr/hpss-get-inventories
Reviewer Review Type Date Requested Status
Vincent Ladeuil Approve
Review via email: mp+83837@code.launchpad.net

This proposal has been superseded by a proposal from 2011-12-11.

Commit message

Add HPSS call for ``Repository.iter_inventories``.

Description of the change

Add a HPSS call for ``Repository.iter_inventories``.

This massively reduces the number of roundtrips for various commands, and causes a fair number to no longer use VFS calls at all when used against a modern remote server.

In the process I removed the HistoryMissing exception that was thrown in the private Repository._get_inventory_xml method, and never caught anywhere except to be converted to a NoSuchRevision exception. The private Repository._iter_inventories no longer directly raises NoSuchRevision but instead indicates when it can't find an inventory so that it is possible to fetch some inventories from one repository and others from another (such as a fallback repository).

Repository.get_deltas_for_revisions() now has its own implementation for remote repositories, which means the client won't use the VFS to get at the inventories to generate the deltas from.

To post a comment you must log in.
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Timings when checking out bzr.dev:

~/src/bzr/hpss-get-inventories/bzr co --lightweight bzr://people.samba.org/bzr.dev 2.25s user 0.46s system 15% cpu 17.277 total
~/src/bzr/hpss-get-inventories/bzr export /tmp/bzr.dev2 bzr://people.samba.org/bzr.dev 1.54s user 0.40s system 12% cpu 15.435 total

versus against an older bzr server:

~/src/bzr/hpss-get-inventories/bzr co --lightweight bzr://people.samba.org/bzr.dev 4.91s user 1.03s system 10% cpu 58.807 total
~/src/bzr/hpss-get-inventories/bzr export /tmp/bzr.dev5 bzr://people.samba.org/bzr.dev 4.35s user 0.98s system 9% cpu 58.357 total

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

For comparison (though I should it's a different server):

apt-get source bzr 2.79s user 0.54s system 27% cpu 12.338 total

so we're getting closer.

Revision history for this message
Andrew Bennetts (spiv) wrote :

Jelmer Vernooij wrote:
> Add a HPSS call for ``Repository.iter_inventories``.

I am a bit concerned that by adding verbs like this, with their own ad hoc
record stream-like wire format, that we're growing the maintenance burden
unnecessarily, and not reusing improvements everywhere we could.

I'm glad this is using inventory-deltas. But if it's worth zlib compressing
them here, shouldn't we do that for inventory-deltas from get_record_stream too?

Also, could you implement this via the get_record_stream RPC with a new
parameter that means “just inventories (rather than rev+inv+texts+chks+sigs for
given keys)”?

On one hand it might feel a bit ugly to make one RPC do so many things,
get_record_stream, do so many things, but on the other I think it provides a
useful pressure to keep the interface minimal and consistent (e.g. whether
records are zlib-compressed).

Adding this iter_inventories RPC might be the right thing (although putting
“iter” in the name of an RPC feels weird to me, that's surely a property of the
Python API rather than a property of the RPC), but I'd like hear what you think
about the tradeoffs of doing that vs. extending get_record_stream.

-Andrew.

Revision history for this message
Martin Pool (mbp) wrote :

On 30 November 2011 09:37, Andrew Bennetts
<email address hidden> wrote:
> Jelmer Vernooij wrote:
>> Add a HPSS call for ``Repository.iter_inventories``.
>
> I am a bit concerned that by adding verbs like this, with their own ad hoc
> record stream-like wire format, that we're growing the maintenance burden
> unnecessarily, and not reusing improvements everywhere we could.

+1

--
Martin

Revision history for this message
Vincent Ladeuil (vila) wrote :

>> I am a bit concerned that by adding verbs like this, with their own ad hoc
>> record stream-like wire format, that we're growing the maintenance burden
>> unnecessarily, and not reusing improvements everywhere we could.

> +1

I kind of had the same feeling when jelmer started adding a bunch of verbs that were basically replacing a vfs roundtrip by a smart request roundtrip.

BUT

If we stop maintaining these verbs on the server side, the clients will fallback to vfs.

So we introduce different/better verbs, we can remove the old ones in both the server and the client and all but the newest clients can still fallback to vfs.

The net effect is at least to get to a point where the most recent client/server do not use vfs at all.

This sounds like a good incremental step to me.

'reusing improvements everywhere we could' is still valuable but doesn't to block progress.

review: Approve
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

Hi Andrew,

> Jelmer Vernooij wrote:
> > Add a HPSS call for ``Repository.iter_inventories``.
> I am a bit concerned that by adding verbs like this, with their own ad hoc
> record stream-like wire format, that we're growing the maintenance burden
> unnecessarily, and not reusing improvements everywhere we could.
>
> I'm glad this is using inventory-deltas. But if it's worth zlib compressing
> them here, shouldn't we do that for inventory-deltas from get_record_stream
> too?
It's a pretty big inventory delta in this case - for something like gcc it matters for the delta between null: and the initial revision that was requested. I haven't investigated whether it would help for record streams too, but I imagine it would have less of an effect.

> Also, could you implement this via the get_record_stream RPC with a new
> parameter that means “just inventories (rather than rev+inv+texts+chks+sigs
> for
> given keys)”?
>
> On one hand it might feel a bit ugly to make one RPC do so many things,
> get_record_stream, do so many things, but on the other I think it provides a
> useful pressure to keep the interface minimal and consistent (e.g. whether
> records are zlib-compressed).
>
> Adding this iter_inventories RPC might be the right thing (although putting
> “iter” in the name of an RPC feels weird to me, that's surely a property of
> the Python API rather than a property of the RPC), but I'd like hear what you
> think about the tradeoffs of doing that vs. extending get_record_stream.

With regard to the name: Most of the other calls seem to be named after the equivalent method in the client / server, that's why I went with Repository.iter_inventories. I'm not particularly tied to that name, something like "Repository.get_inventories" or "Repository.stream_inventories" seems reasonable to me too.

I'm not sure whether this should be part of get_record_stream. I have a hard time understanding the get_record_stream code as it is, so I'd rather not make it even more complex by adding more parameters - for example, get_record_streams seem to be dependent on the repository on-disk format to an extent - the inventory stream is not, as it's using inventory deltas. In other words, adding another verb was simpler, while keeping it all understandable for a mere mortal like me. :-)

Either way, I agree we should be reusing more code between the work I've done recently and the existing record stream
 calls. But it seems to me that would best be done by refactoring so they e.g. use the same code for sending a stream of blobs of indeterminate length, rather than by all being a part of the same verb.

What do you think?

Cheers,

Jelmer

Revision history for this message
Andrew Bennetts (spiv) wrote :
Download full text (4.5 KiB)

Jelmer Vernooij wrote:

> > I'm glad this is using inventory-deltas. But if it's worth zlib compressing
> > them here, shouldn't we do that for inventory-deltas from get_record_stream
> > too?
> It's a pretty big inventory delta in this case - for something like gcc it
> matters for the delta between null: and the initial revision that was
> requested. I haven't investigated whether it would help for record streams
> too, but I imagine it would have less of an effect.

Well, I know that there are already cases that send deltas from null: —
something to do with stacking perhaps? So reusing this improvement globally
would be nice.

> > Also, could you implement this via the get_record_stream RPC with a new
> > parameter that means “just inventories (rather than rev+inv+texts+chks+sigs
> > for
> > given keys)”?
> >
> > On one hand it might feel a bit ugly to make one RPC do so many things,
> > get_record_stream, do so many things, but on the other I think it provides a
> > useful pressure to keep the interface minimal and consistent (e.g. whether
> > records are zlib-compressed).
> >
> > Adding this iter_inventories RPC might be the right thing (although putting
> > “iter” in the name of an RPC feels weird to me, that's surely a property of
> > the Python API rather than a property of the RPC), but I'd like hear what you
> > think about the tradeoffs of doing that vs. extending get_record_stream.
>
> With regard to the name: Most of the other calls seem to be named after the
> equivalent method in the client / server, that's why I went with
> Repository.iter_inventories. I'm not particularly tied to that name, something
> like "Repository.get_inventories" or "Repository.stream_inventories" seems
> reasonable to me too.

I think of using the name of the Python API as a good guide, not a strict rule.
Certainly there's no insert_stream_locked method on Repository…

“Repository.get_inventories” sounds fine to me.

> I'm not sure whether this should be part of get_record_stream. I have a hard
> time understanding the get_record_stream code as it is, so I'd rather not make
> it even more complex by adding more parameters - for example,
> get_record_streams seem to be dependent on the repository on-disk format to an
> extent - the inventory stream is not, as it's using inventory deltas. In other
> words, adding another verb was simpler, while keeping it all understandable
> for a mere mortal like me. :-)
>
> Either way, I agree we should be reusing more code between the work I've done
> recently and the existing record stream calls. But it seems to me that would
> best be done by refactoring so they e.g. use the same code for sending a
> stream of blobs of indeterminate length, rather than by all being a part of
> the same verb.

Well, the path to reuse we've developed *is* record streams. I'm quite willing
to believe that get_record_stream's parameters aren't a convenient way to
express all needs. But I'd really like it the thing that was returned by a new
verb was a true record stream — something you could pass directly to
insert_record_stream. The idea is that record streams should be the One True
Way we have to say “here is a stream of da...

Read more...

Revision history for this message
Jelmer Vernooij (jelmer) wrote :
Download full text (6.7 KiB)

Am 05/12/11 11:31, schrieb Andrew Bennetts:
> Jelmer Vernooij wrote:
> …
>>> I'm glad this is using inventory-deltas. But if it's worth zlib compressing
>>> them here, shouldn't we do that for inventory-deltas from get_record_stream
>>> too?
>> It's a pretty big inventory delta in this case - for something like gcc it
>> matters for the delta between null: and the initial revision that was
>> requested. I haven't investigated whether it would help for record streams
>> too, but I imagine it would have less of an effect.
> Well, I know that there are already cases that send deltas from null: —
> something to do with stacking perhaps? So reusing this improvement globally
> would be nice.
I guess the best way to do this would be to add a zlib pack record kind
for network streams? I.e. a new entry in
NetworkRecordStream._kind_factory, and something equivalent on the
server side?
>>> Also, could you implement this via the get_record_stream RPC with a new
>>> parameter that means “just inventories (rather than rev+inv+texts+chks+sigs
>>> for
>>> given keys)”?
>>>
>>> On one hand it might feel a bit ugly to make one RPC do so many things,
>>> get_record_stream, do so many things, but on the other I think it provides a
>>> useful pressure to keep the interface minimal and consistent (e.g. whether
>>> records are zlib-compressed).
>>>
>>> Adding this iter_inventories RPC might be the right thing (although putting
>>> “iter” in the name of an RPC feels weird to me, that's surely a property of
>>> the Python API rather than a property of the RPC), but I'd like hear what you
>>> think about the tradeoffs of doing that vs. extending get_record_stream.
>> With regard to the name: Most of the other calls seem to be named after the
>> equivalent method in the client / server, that's why I went with
>> Repository.iter_inventories. I'm not particularly tied to that name, something
>> like "Repository.get_inventories" or "Repository.stream_inventories" seems
>> reasonable to me too.
> I think of using the name of the Python API as a good guide, not a strict rule.
> Certainly there's no insert_stream_locked method on Repository…
>
> “Repository.get_inventories” sounds fine to me.
I've renamed it.
>
>> I'm not sure whether this should be part of get_record_stream. I have a hard
>> time understanding the get_record_stream code as it is, so I'd rather not make
>> it even more complex by adding more parameters - for example,
>> get_record_streams seem to be dependent on the repository on-disk format to an
>> extent - the inventory stream is not, as it's using inventory deltas. In other
>> words, adding another verb was simpler, while keeping it all understandable
>> for a mere mortal like me. :-)
>>
>> Either way, I agree we should be reusing more code between the work I've done
>> recently and the existing record stream calls. But it seems to me that would
>> best be done by refactoring so they e.g. use the same code for sending a
>> stream of blobs of indeterminate length, rather than by all being a part of
>> the same verb.
> Well, the path to reuse we've developed *is* record streams. I'm quite willing
> to believe that get_record_stream's parameters aren'...

Read more...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bzrlib/remote.py'
--- bzrlib/remote.py 2011-12-11 16:36:30 +0000
+++ bzrlib/remote.py 2011-12-11 16:36:30 +0000
@@ -27,6 +27,7 @@
27 errors,27 errors,
28 gpg,28 gpg,
29 graph,29 graph,
30 inventory_delta,
30 lock,31 lock,
31 lockdir,32 lockdir,
32 osutils,33 osutils,
@@ -490,6 +491,45 @@
490 self._next_open_branch_result = None491 self._next_open_branch_result = None
491 return _mod_bzrdir.BzrDir.break_lock(self)492 return _mod_bzrdir.BzrDir.break_lock(self)
492493
494 def _vfs_checkout_metadir(self):
495 self._ensure_real()
496 return self._real_bzrdir.checkout_metadir()
497
498 def checkout_metadir(self):
499 medium = self._client._medium
500 if medium._is_remote_before((2, 5)):
501 return self._vfs_checkout_metadir()
502 path = self._path_for_remote_call(self._client)
503 try:
504 response = self._client.call('BzrDir.checkout_metadir',
505 path)
506 except errors.UnknownSmartMethod:
507 medium._remember_remote_is_before((2, 5))
508 return self._vfs_checkout_metadir()
509 if len(response) != 3:
510 raise errors.UnexpectedSmartServerResponse(response)
511 control_name, repo_name, branch_name = response
512 try:
513 format = controldir.network_format_registry.get(control_name)
514 except KeyError:
515 raise errors.UnknownFormatError(kind='control', format=control_name)
516 if repo_name:
517 try:
518 repo_format = _mod_repository.network_format_registry.get(
519 repo_name)
520 except KeyError:
521 raise errors.UnknownFormatError(kind='repository',
522 format=repo_name)
523 format.repository_format = repo_format
524 if branch_name:
525 try:
526 format.set_branch_format(
527 branch.network_format_registry.get(branch_name))
528 except KeyError:
529 raise errors.UnknownFormatError(kind='branch',
530 format=branch_name)
531 return format
532
493 def _vfs_cloning_metadir(self, require_stacking=False):533 def _vfs_cloning_metadir(self, require_stacking=False):
494 self._ensure_real()534 self._ensure_real()
495 return self._real_bzrdir.cloning_metadir(535 return self._real_bzrdir.cloning_metadir(
@@ -1788,9 +1828,122 @@
1788 def get_inventory(self, revision_id):1828 def get_inventory(self, revision_id):
1789 return list(self.iter_inventories([revision_id]))[0]1829 return list(self.iter_inventories([revision_id]))[0]
17901830
1831 def _iter_inventories_rpc(self, revision_ids, ordering):
1832 if ordering is None:
1833 ordering = 'unordered'
1834 path = self.bzrdir._path_for_remote_call(self._client)
1835 body = "\n".join(revision_ids)
1836 response_tuple, response_handler = (
1837 self._call_with_body_bytes_expecting_body(
1838 "VersionedFileRepository.get_inventories",
1839 (path, ordering), body))
1840 if response_tuple[0] != "ok":
1841 raise errors.UnexpectedSmartServerResponse(response_tuple)
1842 deserializer = inventory_delta.InventoryDeltaDeserializer()
1843 byte_stream = response_handler.read_streamed_body()
1844 decoded = smart_repo._byte_stream_to_stream(byte_stream)
1845 if decoded is None:
1846 # no results whatsoever
1847 return
1848 src_format, stream = decoded
1849 if src_format.network_name() != self._format.network_name():
1850 raise AssertionError(
1851 "Mismatched RemoteRepository and stream src %r, %r" % (
1852 src_format.network_name(), self._format.network_name()))
1853 # ignore the src format, it's not really relevant
1854 prev_inv = Inventory(root_id=None,
1855 revision_id=_mod_revision.NULL_REVISION)
1856 # there should be just one substream, with inventory deltas
1857 substream_kind, substream = stream.next()
1858 for record in substream:
1859 (parent_id, new_id, versioned_root, tree_references, invdelta) = (
1860 deserializer.parse_text_bytes(record.get_bytes_as("fulltext")))
1861 if parent_id != prev_inv.revision_id:
1862 raise AssertionError("invalid base %r != %r" % (parent_id,
1863 prev_inv.revision_id))
1864 inv = prev_inv.create_by_apply_delta(invdelta, new_id)
1865 yield inv, inv.revision_id
1866 prev_inv = inv
1867
1868 def _iter_inventories_vfs(self, revision_ids, ordering=None):
1869 self._ensure_real()
1870 return self._real_repository._iter_inventories(revision_ids, ordering)
1871
1791 def iter_inventories(self, revision_ids, ordering=None):1872 def iter_inventories(self, revision_ids, ordering=None):
1792 self._ensure_real()1873 """Get many inventories by revision_ids.
1793 return self._real_repository.iter_inventories(revision_ids, ordering)1874
1875 This will buffer some or all of the texts used in constructing the
1876 inventories in memory, but will only parse a single inventory at a
1877 time.
1878
1879 :param revision_ids: The expected revision ids of the inventories.
1880 :param ordering: optional ordering, e.g. 'topological'. If not
1881 specified, the order of revision_ids will be preserved (by
1882 buffering if necessary).
1883 :return: An iterator of inventories.
1884 """
1885 if ((None in revision_ids)
1886 or (_mod_revision.NULL_REVISION in revision_ids)):
1887 raise ValueError('cannot get null revision inventory')
1888 for inv, revid in self._iter_inventories(revision_ids, ordering):
1889 if inv is None:
1890 raise errors.NoSuchRevision(self, revid)
1891 yield inv
1892
1893 def _iter_inventories(self, revision_ids, ordering=None):
1894 if len(revision_ids) == 0:
1895 return
1896 missing = set(revision_ids)
1897 if ordering is None:
1898 order_as_requested = True
1899 invs = {}
1900 order = list(revision_ids)
1901 order.reverse()
1902 next_revid = order.pop()
1903 else:
1904 order_as_requested = False
1905 if ordering != 'unordered' and self._fallback_repositories:
1906 raise ValueError('unsupported ordering %r' % ordering)
1907 iter_inv_fns = [self._iter_inventories_rpc] + [
1908 fallback._iter_inventories for fallback in
1909 self._fallback_repositories]
1910 try:
1911 for iter_inv in iter_inv_fns:
1912 request = [revid for revid in revision_ids if revid in missing]
1913 for inv, revid in iter_inv(request, ordering):
1914 if inv is None:
1915 continue
1916 missing.remove(inv.revision_id)
1917 if ordering != 'unordered':
1918 invs[revid] = inv
1919 else:
1920 yield inv, revid
1921 if order_as_requested:
1922 # Yield as many results as we can while preserving order.
1923 while next_revid in invs:
1924 inv = invs.pop(next_revid)
1925 yield inv, inv.revision_id
1926 try:
1927 next_revid = order.pop()
1928 except IndexError:
1929 # We still want to fully consume the stream, just
1930 # in case it is not actually finished at this point
1931 next_revid = None
1932 break
1933 except errors.UnknownSmartMethod:
1934 for inv, revid in self._iter_inventories_vfs(revision_ids, ordering):
1935 yield inv, revid
1936 return
1937 # Report missing
1938 if order_as_requested:
1939 if next_revid is not None:
1940 yield None, next_revid
1941 while order:
1942 revid = order.pop()
1943 yield invs.get(revid), revid
1944 else:
1945 while missing:
1946 yield None, missing.pop()
17941947
1795 @needs_read_lock1948 @needs_read_lock
1796 def get_revision(self, revision_id):1949 def get_revision(self, revision_id):
@@ -2149,6 +2302,8 @@
21492302
2150 @needs_read_lock2303 @needs_read_lock
2151 def _get_inventory_xml(self, revision_id):2304 def _get_inventory_xml(self, revision_id):
2305 # This call is used by older working tree formats,
2306 # which stored a serialized basis inventory.
2152 self._ensure_real()2307 self._ensure_real()
2153 return self._real_repository._get_inventory_xml(revision_id)2308 return self._real_repository._get_inventory_xml(revision_id)
21542309
@@ -2171,11 +2326,58 @@
2171 revids.update(set(fallback.all_revision_ids()))2326 revids.update(set(fallback.all_revision_ids()))
2172 return list(revids)2327 return list(revids)
21732328
2329 def _filtered_revision_trees(self, revision_ids, file_ids):
2330 """Return Tree for a revision on this branch with only some files.
2331
2332 :param revision_ids: a sequence of revision-ids;
2333 a revision-id may not be None or 'null:'
2334 :param file_ids: if not None, the result is filtered
2335 so that only those file-ids, their parents and their
2336 children are included.
2337 """
2338 inventories = self.iter_inventories(revision_ids)
2339 for inv in inventories:
2340 # Should we introduce a FilteredRevisionTree class rather
2341 # than pre-filter the inventory here?
2342 filtered_inv = inv.filter(file_ids)
2343 yield InventoryRevisionTree(self, filtered_inv, filtered_inv.revision_id)
2344
2174 @needs_read_lock2345 @needs_read_lock
2175 def get_deltas_for_revisions(self, revisions, specific_fileids=None):2346 def get_deltas_for_revisions(self, revisions, specific_fileids=None):
2176 self._ensure_real()2347 medium = self._client._medium
2177 return self._real_repository.get_deltas_for_revisions(revisions,2348 if medium._is_remote_before((1, 2)):
2178 specific_fileids=specific_fileids)2349 self._ensure_real()
2350 for delta in self._real_repository.get_deltas_for_revisions(
2351 revisions, specific_fileids):
2352 yield delta
2353 return
2354 # Get the revision-ids of interest
2355 required_trees = set()
2356 for revision in revisions:
2357 required_trees.add(revision.revision_id)
2358 required_trees.update(revision.parent_ids[:1])
2359
2360 # Get the matching filtered trees. Note that it's more
2361 # efficient to pass filtered trees to changes_from() rather
2362 # than doing the filtering afterwards. changes_from() could
2363 # arguably do the filtering itself but it's path-based, not
2364 # file-id based, so filtering before or afterwards is
2365 # currently easier.
2366 if specific_fileids is None:
2367 trees = dict((t.get_revision_id(), t) for
2368 t in self.revision_trees(required_trees))
2369 else:
2370 trees = dict((t.get_revision_id(), t) for
2371 t in self._filtered_revision_trees(required_trees,
2372 specific_fileids))
2373
2374 # Calculate the deltas
2375 for revision in revisions:
2376 if not revision.parent_ids:
2377 old_tree = self.revision_tree(_mod_revision.NULL_REVISION)
2378 else:
2379 old_tree = trees[revision.parent_ids[0]]
2380 yield trees[revision.revision_id].changes_from(old_tree)
21792381
2180 @needs_read_lock2382 @needs_read_lock
2181 def get_revision_delta(self, revision_id, specific_fileids=None):2383 def get_revision_delta(self, revision_id, specific_fileids=None):
@@ -3157,51 +3359,6 @@
3157 self.bzrdir, self._client)3359 self.bzrdir, self._client)
3158 return self._control_files3360 return self._control_files
31593361
3160 def _get_checkout_format_vfs(self, lightweight=False):
3161 self._ensure_real()
3162 if lightweight:
3163 format = RemoteBzrDirFormat()
3164 self.bzrdir._format._supply_sub_formats_to(format)
3165 format.workingtree_format = self._real_branch._get_checkout_format(
3166 lightweight=lightweight).workingtree_format
3167 return format
3168 else:
3169 return self._real_branch._get_checkout_format(lightweight=False)
3170
3171 def _get_checkout_format(self, lightweight=False):
3172 medium = self._client._medium
3173 if medium._is_remote_before((2, 5)):
3174 return self._get_checkout_format_vfs(lightweight)
3175 try:
3176 response = self._client.call('Branch.get_checkout_format',
3177 self._remote_path(), lightweight)
3178 except errors.UnknownSmartMethod:
3179 medium._remember_remote_is_before((2, 5))
3180 return self._get_checkout_format_vfs(lightweight)
3181 if len(response) != 3:
3182 raise errors.UnexpectedSmartServerResponse(response)
3183 control_name, repo_name, branch_name = response
3184 try:
3185 format = controldir.network_format_registry.get(control_name)
3186 except KeyError:
3187 raise errors.UnknownFormatError(kind='control', format=control_name)
3188 if repo_name:
3189 try:
3190 repo_format = _mod_repository.network_format_registry.get(
3191 repo_name)
3192 except KeyError:
3193 raise errors.UnknownFormatError(kind='repository',
3194 format=repo_name)
3195 format.repository_format = repo_format
3196 if branch_name:
3197 try:
3198 format.set_branch_format(
3199 branch.network_format_registry.get(branch_name))
3200 except KeyError:
3201 raise errors.UnknownFormatError(kind='branch',
3202 format=branch_name)
3203 return format
3204
3205 def get_physical_lock_status(self):3362 def get_physical_lock_status(self):
3206 """See Branch.get_physical_lock_status()."""3363 """See Branch.get_physical_lock_status()."""
3207 try:3364 try:
32083365
=== modified file 'bzrlib/smart/branch.py'
--- bzrlib/smart/branch.py 2011-12-11 16:36:30 +0000
+++ bzrlib/smart/branch.py 2011-11-25 14:04:12 +0000
@@ -446,22 +446,3 @@
446 return SuccessfulSmartServerResponse(('yes',))446 return SuccessfulSmartServerResponse(('yes',))
447 else:447 else:
448 return SuccessfulSmartServerResponse(('no',))448 return SuccessfulSmartServerResponse(('no',))
449
450
451class SmartServerBranchRequestGetCheckoutFormat(SmartServerBranchRequest):
452 """Get the format to use for checkouts of a branch.
453
454 New in 2.5.
455 """
456
457 def do_with_branch(self, branch, lightweight):
458 control_format = branch._get_checkout_format(lightweight)
459 control_name = control_format.network_name()
460 if not control_format.fixed_components:
461 branch_name = control_format.get_branch_format().network_name()
462 repo_name = control_format.repository_format.network_name()
463 else:
464 branch_name = ''
465 repo_name = ''
466 return SuccessfulSmartServerResponse(
467 (control_name, repo_name, branch_name))
468449
=== modified file 'bzrlib/smart/bzrdir.py'
--- bzrlib/smart/bzrdir.py 2011-11-21 14:44:03 +0000
+++ bzrlib/smart/bzrdir.py 2011-12-11 16:36:30 +0000
@@ -208,6 +208,25 @@
208 branch_name))208 branch_name))
209209
210210
211class SmartServerBzrDirRequestCheckoutMetaDir(SmartServerRequestBzrDir):
212 """Get the format to use for checkouts.
213
214 New in 2.5.
215 """
216
217 def do_bzrdir_request(self):
218 control_format = self._bzrdir.checkout_metadir()
219 control_name = control_format.network_name()
220 if not control_format.fixed_components:
221 branch_name = control_format.get_branch_format().network_name()
222 repo_name = control_format.repository_format.network_name()
223 else:
224 branch_name = ''
225 repo_name = ''
226 return SuccessfulSmartServerResponse(
227 (control_name, repo_name, branch_name))
228
229
211class SmartServerRequestCreateBranch(SmartServerRequestBzrDir):230class SmartServerRequestCreateBranch(SmartServerRequestBzrDir):
212231
213 def do(self, path, network_name):232 def do(self, path, network_name):
214233
=== modified file 'bzrlib/smart/repository.py'
--- bzrlib/smart/repository.py 2011-12-05 15:16:52 +0000
+++ bzrlib/smart/repository.py 2011-12-11 16:36:30 +0000
@@ -14,7 +14,7 @@
14# along with this program; if not, write to the Free Software14# along with this program; if not, write to the Free Software
15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
1616
17"""Server-side repository related request implmentations."""17"""Server-side repository related request implementations."""
1818
19import bz219import bz2
20import os20import os
@@ -28,6 +28,8 @@
28 bencode,28 bencode,
29 errors,29 errors,
30 estimate_compressed_size,30 estimate_compressed_size,
31 inventory as _mod_inventory,
32 inventory_delta,
31 osutils,33 osutils,
32 pack,34 pack,
33 trace,35 trace,
@@ -43,6 +45,7 @@
43from bzrlib.repository import _strip_NULL_ghosts, network_format_registry45from bzrlib.repository import _strip_NULL_ghosts, network_format_registry
44from bzrlib import revision as _mod_revision46from bzrlib import revision as _mod_revision
45from bzrlib.versionedfile import (47from bzrlib.versionedfile import (
48 ChunkedContentFactory,
46 NetworkRecordStream,49 NetworkRecordStream,
47 record_to_fulltext_bytes,50 record_to_fulltext_bytes,
48 )51 )
@@ -1220,3 +1223,57 @@
1220 yield zlib.compress(record.get_bytes_as('fulltext'))1223 yield zlib.compress(record.get_bytes_as('fulltext'))
1221 finally:1224 finally:
1222 self._repository.unlock()1225 self._repository.unlock()
1226
1227
1228class SmartServerRepositoryGetInventories(SmartServerRepositoryRequest):
1229 """Get the inventory deltas for a set of revision ids.
1230
1231 This accepts a list of revision ids, and then sends a chain
1232 of deltas for the inventories of those revisions. The first
1233 revision will be empty.
1234
1235 The server writes back zlibbed serialized inventory deltas,
1236 in the ordering specified. The base for each delta is the
1237 inventory generated by the previous delta.
1238
1239 New in 2.5.
1240 """
1241
1242 def _inventory_delta_stream(self, repository, ordering, revids):
1243 prev_inv = _mod_inventory.Inventory(root_id=None,
1244 revision_id=_mod_revision.NULL_REVISION)
1245 serializer = inventory_delta.InventoryDeltaSerializer(
1246 repository.supports_rich_root(),
1247 repository._format.supports_tree_reference)
1248 repository.lock_read()
1249 try:
1250 for inv, revid in repository._iter_inventories(revids, ordering):
1251 if inv is None:
1252 continue
1253 inv_delta = inv._make_delta(prev_inv)
1254 lines = serializer.delta_to_lines(
1255 prev_inv.revision_id, inv.revision_id, inv_delta)
1256 yield ChunkedContentFactory(inv.revision_id, None, None, lines)
1257 prev_inv = inv
1258 finally:
1259 repository.unlock()
1260
1261 def body_stream(self, repository, ordering, revids):
1262 substream = self._inventory_delta_stream(repository,
1263 ordering, revids)
1264 return _stream_to_byte_stream([('inventory-delta', substream)],
1265 repository._format)
1266
1267 def do_body(self, body_bytes):
1268 return SuccessfulSmartServerResponse(('ok', ),
1269 body_stream=self.body_stream(self._repository, self._ordering,
1270 body_bytes.splitlines()))
1271
1272 def do_repository_request(self, repository, ordering):
1273 if ordering == 'unordered':
1274 # inventory deltas for a topologically sorted stream
1275 # are likely to be smaller
1276 ordering = 'topological'
1277 self._ordering = ordering
1278 # Signal that we want a body
1279 return None
12231280
=== modified file 'bzrlib/smart/request.py'
--- bzrlib/smart/request.py 2011-12-11 16:36:30 +0000
+++ bzrlib/smart/request.py 2011-12-11 16:36:30 +0000
@@ -547,9 +547,6 @@
547 'Branch.get_stacked_on_url', 'bzrlib.smart.branch',547 'Branch.get_stacked_on_url', 'bzrlib.smart.branch',
548 'SmartServerBranchRequestGetStackedOnURL', info='read')548 'SmartServerBranchRequestGetStackedOnURL', info='read')
549request_handlers.register_lazy(549request_handlers.register_lazy(
550 'Branch.get_checkout_format', 'bzrlib.smart.branch',
551 'SmartServerBranchRequestGetCheckoutFormat', info='read')
552request_handlers.register_lazy(
553 'Branch.get_physical_lock_status', 'bzrlib.smart.branch',550 'Branch.get_physical_lock_status', 'bzrlib.smart.branch',
554 'SmartServerBranchRequestGetPhysicalLockStatus', info='read')551 'SmartServerBranchRequestGetPhysicalLockStatus', info='read')
555request_handlers.register_lazy(552request_handlers.register_lazy(
@@ -586,6 +583,9 @@
586 'Branch.revision_id_to_revno', 'bzrlib.smart.branch',583 'Branch.revision_id_to_revno', 'bzrlib.smart.branch',
587 'SmartServerBranchRequestRevisionIdToRevno', info='read')584 'SmartServerBranchRequestRevisionIdToRevno', info='read')
588request_handlers.register_lazy(585request_handlers.register_lazy(
586 'BzrDir.checkout_metadir', 'bzrlib.smart.bzrdir',
587 'SmartServerBzrDirRequestCheckoutMetaDir', info='read')
588request_handlers.register_lazy(
589 'BzrDir.cloning_metadir', 'bzrlib.smart.bzrdir',589 'BzrDir.cloning_metadir', 'bzrlib.smart.bzrdir',
590 'SmartServerBzrDirRequestCloningMetaDir', info='read')590 'SmartServerBzrDirRequestCloningMetaDir', info='read')
591request_handlers.register_lazy(591request_handlers.register_lazy(
@@ -754,6 +754,9 @@
754 'VersionedFileRepository.get_serializer_format', 'bzrlib.smart.repository',754 'VersionedFileRepository.get_serializer_format', 'bzrlib.smart.repository',
755 'SmartServerRepositoryGetSerializerFormat', info='read')755 'SmartServerRepositoryGetSerializerFormat', info='read')
756request_handlers.register_lazy(756request_handlers.register_lazy(
757 'VersionedFileRepository.get_inventories', 'bzrlib.smart.repository',
758 'SmartServerRepositoryGetInventories', info='read')
759request_handlers.register_lazy(
757 'Repository.tarball', 'bzrlib.smart.repository',760 'Repository.tarball', 'bzrlib.smart.repository',
758 'SmartServerRepositoryTarball', info='read')761 'SmartServerRepositoryTarball', info='read')
759request_handlers.register_lazy(762request_handlers.register_lazy(
760763
=== modified file 'bzrlib/tests/blackbox/test_annotate.py'
--- bzrlib/tests/blackbox/test_annotate.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_annotate.py 2011-12-11 16:36:30 +0000
@@ -326,6 +326,6 @@
326 # being too low. If rpc_count increases, more network roundtrips have326 # being too low. If rpc_count increases, more network roundtrips have
327 # become necessary for this use case. Please do not adjust this number327 # become necessary for this use case. Please do not adjust this number
328 # upwards without agreement from bzr's network support maintainers.328 # upwards without agreement from bzr's network support maintainers.
329 self.assertLength(19, self.hpss_calls)329 self.assertLength(15, self.hpss_calls)
330 self.expectFailure("annotate accesses inventories, which require VFS access",330 self.expectFailure("annotate accesses inventories, which require VFS access",
331 self.assertThat, self.hpss_calls, NoVfsCalls)331 self.assertThat, self.hpss_calls, NoVfsCalls)
332332
=== modified file 'bzrlib/tests/blackbox/test_branch.py'
--- bzrlib/tests/blackbox/test_branch.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_branch.py 2011-12-11 16:36:30 +0000
@@ -483,7 +483,7 @@
483 # being too low. If rpc_count increases, more network roundtrips have483 # being too low. If rpc_count increases, more network roundtrips have
484 # become necessary for this use case. Please do not adjust this number484 # become necessary for this use case. Please do not adjust this number
485 # upwards without agreement from bzr's network support maintainers.485 # upwards without agreement from bzr's network support maintainers.
486 self.assertLength(40, self.hpss_calls)486 self.assertLength(33, self.hpss_calls)
487 self.expectFailure("branching to the same branch requires VFS access",487 self.expectFailure("branching to the same branch requires VFS access",
488 self.assertThat, self.hpss_calls, NoVfsCalls)488 self.assertThat, self.hpss_calls, NoVfsCalls)
489489
490490
=== modified file 'bzrlib/tests/blackbox/test_cat.py'
--- bzrlib/tests/blackbox/test_cat.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_cat.py 2011-12-11 16:36:30 +0000
@@ -239,6 +239,5 @@
239 # being too low. If rpc_count increases, more network roundtrips have239 # being too low. If rpc_count increases, more network roundtrips have
240 # become necessary for this use case. Please do not adjust this number240 # become necessary for this use case. Please do not adjust this number
241 # upwards without agreement from bzr's network support maintainers.241 # upwards without agreement from bzr's network support maintainers.
242 self.assertLength(16, self.hpss_calls)242 self.assertLength(9, self.hpss_calls)
243 self.expectFailure("cat still uses VFS calls",243 self.assertThat(self.hpss_calls, NoVfsCalls)
244 self.assertThat, self.hpss_calls, NoVfsCalls)
245244
=== modified file 'bzrlib/tests/blackbox/test_checkout.py'
--- bzrlib/tests/blackbox/test_checkout.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_checkout.py 2011-12-11 16:36:30 +0000
@@ -179,8 +179,7 @@
179 for count in range(9):179 for count in range(9):
180 t.commit(message='commit %d' % count)180 t.commit(message='commit %d' % count)
181 self.reset_smart_call_log()181 self.reset_smart_call_log()
182 out, err = self.run_bzr(['checkout', self.get_url('from'),182 out, err = self.run_bzr(['checkout', self.get_url('from'), 'target'])
183 'target'])
184 # This figure represent the amount of work to perform this use case. It183 # This figure represent the amount of work to perform this use case. It
185 # is entirely ok to reduce this number if a test fails due to rpc_count184 # is entirely ok to reduce this number if a test fails due to rpc_count
186 # being too low. If rpc_count increases, more network roundtrips have185 # being too low. If rpc_count increases, more network roundtrips have
@@ -202,9 +201,5 @@
202 # being too low. If rpc_count increases, more network roundtrips have201 # being too low. If rpc_count increases, more network roundtrips have
203 # become necessary for this use case. Please do not adjust this number202 # become necessary for this use case. Please do not adjust this number
204 # upwards without agreement from bzr's network support maintainers.203 # upwards without agreement from bzr's network support maintainers.
205 if len(self.hpss_calls) < 28 or len(self.hpss_calls) > 40:204 self.assertLength(15, self.hpss_calls)
206 self.fail(205 self.assertThat(self.hpss_calls, NoVfsCalls)
207 "Incorrect length: wanted between 28 and 40, got %d for %r" % (
208 len(self.hpss_calls), self.hpss_calls))
209 self.expectFailure("lightweight checkouts require VFS calls",
210 self.assertThat, self.hpss_calls, NoVfsCalls)
211206
=== modified file 'bzrlib/tests/blackbox/test_export.py'
--- bzrlib/tests/blackbox/test_export.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_export.py 2011-12-11 16:36:30 +0000
@@ -448,6 +448,5 @@
448 # being too low. If rpc_count increases, more network roundtrips have448 # being too low. If rpc_count increases, more network roundtrips have
449 # become necessary for this use case. Please do not adjust this number449 # become necessary for this use case. Please do not adjust this number
450 # upwards without agreement from bzr's network support maintainers.450 # upwards without agreement from bzr's network support maintainers.
451 self.assertLength(16, self.hpss_calls)451 self.assertLength(7, self.hpss_calls)
452 self.expectFailure("export requires inventory access which requires VFS",452 self.assertThat(self.hpss_calls, NoVfsCalls)
453 self.assertThat, self.hpss_calls, NoVfsCalls)
454453
=== modified file 'bzrlib/tests/blackbox/test_log.py'
--- bzrlib/tests/blackbox/test_log.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_log.py 2011-12-11 16:36:30 +0000
@@ -1086,9 +1086,8 @@
1086 # being too low. If rpc_count increases, more network roundtrips have1086 # being too low. If rpc_count increases, more network roundtrips have
1087 # become necessary for this use case. Please do not adjust this number1087 # become necessary for this use case. Please do not adjust this number
1088 # upwards without agreement from bzr's network support maintainers.1088 # upwards without agreement from bzr's network support maintainers.
1089 self.assertLength(19, self.hpss_calls)1089 self.assertLength(11, self.hpss_calls)
1090 self.expectFailure("verbose log accesses inventories, which require VFS",1090 self.assertThat(self.hpss_calls, NoVfsCalls)
1091 self.assertThat, self.hpss_calls, NoVfsCalls)
10921091
1093 def test_per_file(self):1092 def test_per_file(self):
1094 self.setup_smart_server_with_call_log()1093 self.setup_smart_server_with_call_log()
@@ -1103,6 +1102,5 @@
1103 # being too low. If rpc_count increases, more network roundtrips have1102 # being too low. If rpc_count increases, more network roundtrips have
1104 # become necessary for this use case. Please do not adjust this number1103 # become necessary for this use case. Please do not adjust this number
1105 # upwards without agreement from bzr's network support maintainers.1104 # upwards without agreement from bzr's network support maintainers.
1106 self.assertLength(21, self.hpss_calls)1105 self.assertLength(15, self.hpss_calls)
1107 self.expectFailure("per-file graph access requires VFS",1106 self.assertThat(self.hpss_calls, NoVfsCalls)
1108 self.assertThat, self.hpss_calls, NoVfsCalls)
11091107
=== modified file 'bzrlib/tests/blackbox/test_ls.py'
--- bzrlib/tests/blackbox/test_ls.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_ls.py 2011-12-11 16:36:30 +0000
@@ -262,6 +262,5 @@
262 # being too low. If rpc_count increases, more network roundtrips have262 # being too low. If rpc_count increases, more network roundtrips have
263 # become necessary for this use case. Please do not adjust this number263 # become necessary for this use case. Please do not adjust this number
264 # upwards without agreement from bzr's network support maintainers.264 # upwards without agreement from bzr's network support maintainers.
265 self.assertLength(15, self.hpss_calls)265 self.assertLength(6, self.hpss_calls)
266 self.expectFailure("inventories can only be accessed over VFS",266 self.assertThat(self.hpss_calls, NoVfsCalls)
267 self.assertThat, self.hpss_calls, NoVfsCalls)
268267
=== modified file 'bzrlib/tests/blackbox/test_sign_my_commits.py'
--- bzrlib/tests/blackbox/test_sign_my_commits.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/blackbox/test_sign_my_commits.py 2011-12-11 16:36:30 +0000
@@ -165,7 +165,7 @@
165 # being too low. If rpc_count increases, more network roundtrips have165 # being too low. If rpc_count increases, more network roundtrips have
166 # become necessary for this use case. Please do not adjust this number166 # become necessary for this use case. Please do not adjust this number
167 # upwards without agreement from bzr's network support maintainers.167 # upwards without agreement from bzr's network support maintainers.
168 self.assertLength(54, self.hpss_calls)168 self.assertLength(50, self.hpss_calls)
169 self.expectFailure("signing commits requires VFS access",169 self.expectFailure("signing commits requires VFS access",
170 self.assertThat, self.hpss_calls, NoVfsCalls)170 self.assertThat, self.hpss_calls, NoVfsCalls)
171171
@@ -185,12 +185,5 @@
185 # being too low. If rpc_count increases, more network roundtrips have185 # being too low. If rpc_count increases, more network roundtrips have
186 # become necessary for this use case. Please do not adjust this number186 # become necessary for this use case. Please do not adjust this number
187 # upwards without agreement from bzr's network support maintainers.187 # upwards without agreement from bzr's network support maintainers.
188188 self.assertLength(10, self.hpss_calls)
189 # The number of readv requests seems to vary depending on the generated189 self.assertThat(self.hpss_calls, NoVfsCalls)
190 # repository and how well it compresses, so allow for a bit of
191 # variation:
192 if len(self.hpss_calls) not in (18, 19):
193 self.fail("Incorrect length: wanted 18 or 19, got %d for %r" % (
194 len(self.hpss_calls), self.hpss_calls))
195 self.expectFailure("verifying commits requires VFS access",
196 self.assertThat, self.hpss_calls, NoVfsCalls)
197190
=== modified file 'bzrlib/tests/per_interbranch/test_push.py'
--- bzrlib/tests/per_interbranch/test_push.py 2011-10-15 01:09:01 +0000
+++ bzrlib/tests/per_interbranch/test_push.py 2011-12-11 16:36:30 +0000
@@ -279,10 +279,10 @@
279 # remote graph any further.279 # remote graph any further.
280 bzr_core_trace = Equals(280 bzr_core_trace = Equals(
281 ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19',281 ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19',
282 'get', 'Branch.set_last_revision_info', 'Branch.unlock'])282 'Branch.set_last_revision_info', 'Branch.unlock'])
283 bzr_loom_trace = Equals(283 bzr_loom_trace = Equals(
284 ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19',284 ['Repository.insert_stream_1.19', 'Repository.insert_stream_1.19',
285 'get', 'Branch.set_last_revision_info', 'get', 'Branch.unlock'])285 'Branch.set_last_revision_info', 'get', 'Branch.unlock'])
286 self.assertThat(calls_after_insert_stream,286 self.assertThat(calls_after_insert_stream,
287 MatchesAny(bzr_core_trace, bzr_loom_trace))287 MatchesAny(bzr_core_trace, bzr_loom_trace))
288288
289289
=== modified file 'bzrlib/tests/test_remote.py'
--- bzrlib/tests/test_remote.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/test_remote.py 2011-12-11 16:36:30 +0000
@@ -69,6 +69,7 @@
69from bzrlib.smart.repository import (69from bzrlib.smart.repository import (
70 SmartServerRepositoryGetParentMap,70 SmartServerRepositoryGetParentMap,
71 SmartServerRepositoryGetStream_1_19,71 SmartServerRepositoryGetStream_1_19,
72 _stream_to_byte_stream,
72 )73 )
73from bzrlib.symbol_versioning import deprecated_in74from bzrlib.symbol_versioning import deprecated_in
74from bzrlib.tests import (75from bzrlib.tests import (
@@ -503,6 +504,43 @@
503 self.assertRaises(errors.UnknownFormatError, a_bzrdir.cloning_metadir)504 self.assertRaises(errors.UnknownFormatError, a_bzrdir.cloning_metadir)
504505
505506
507class TestBzrDirCheckoutMetaDir(TestRemote):
508
509 def test__get_checkout_format(self):
510 transport = MemoryTransport()
511 client = FakeClient(transport.base)
512 reference_bzrdir_format = bzrdir.format_registry.get('default')()
513 control_name = reference_bzrdir_format.network_name()
514 client.add_expected_call(
515 'BzrDir.checkout_metadir', ('quack/', ),
516 'success', (control_name, '', ''))
517 transport.mkdir('quack')
518 transport = transport.clone('quack')
519 a_bzrdir = RemoteBzrDir(transport, RemoteBzrDirFormat(),
520 _client=client)
521 result = a_bzrdir.checkout_metadir()
522 # We should have got a reference control dir with default branch and
523 # repository formats.
524 self.assertEqual(bzrdir.BzrDirMetaFormat1, type(result))
525 self.assertEqual(None, result._repository_format)
526 self.assertEqual(None, result._branch_format)
527 self.assertFinished(client)
528
529 def test_unknown_format(self):
530 transport = MemoryTransport()
531 client = FakeClient(transport.base)
532 client.add_expected_call(
533 'BzrDir.checkout_metadir', ('quack/',),
534 'success', ('dontknow', '', ''))
535 transport.mkdir('quack')
536 transport = transport.clone('quack')
537 a_bzrdir = RemoteBzrDir(transport, RemoteBzrDirFormat(),
538 _client=client)
539 self.assertRaises(errors.UnknownFormatError,
540 a_bzrdir.checkout_metadir)
541 self.assertFinished(client)
542
543
506class TestBzrDirDestroyBranch(TestRemote):544class TestBzrDirDestroyBranch(TestRemote):
507545
508 def test_destroy_default(self):546 def test_destroy_default(self):
@@ -1102,47 +1140,6 @@
1102 self.assertFinished(client)1140 self.assertFinished(client)
11031141
11041142
1105class TestBranchGetCheckoutFormat(RemoteBranchTestCase):
1106
1107 def test__get_checkout_format(self):
1108 transport = MemoryTransport()
1109 client = FakeClient(transport.base)
1110 reference_bzrdir_format = bzrdir.format_registry.get('default')()
1111 control_name = reference_bzrdir_format.network_name()
1112 client.add_expected_call(
1113 'Branch.get_stacked_on_url', ('quack/',),
1114 'error', ('NotStacked',))
1115 client.add_expected_call(
1116 'Branch.get_checkout_format', ('quack/', False),
1117 'success', (control_name, '', ''))
1118 transport.mkdir('quack')
1119 transport = transport.clone('quack')
1120 branch = self.make_remote_branch(transport, client)
1121 result = branch._get_checkout_format()
1122 # We should have got a reference control dir with default branch and
1123 # repository formats.
1124 self.assertEqual(bzrdir.BzrDirMetaFormat1, type(result))
1125 self.assertEqual(None, result._repository_format)
1126 self.assertEqual(None, result._branch_format)
1127 self.assertFinished(client)
1128
1129 def test_unknown_format(self):
1130 transport = MemoryTransport()
1131 client = FakeClient(transport.base)
1132 client.add_expected_call(
1133 'Branch.get_stacked_on_url', ('quack/',),
1134 'error', ('NotStacked',))
1135 client.add_expected_call(
1136 'Branch.get_checkout_format', ('quack/', False),
1137 'success', ('dontknow', '', ''))
1138 transport.mkdir('quack')
1139 transport = transport.clone('quack')
1140 branch = self.make_remote_branch(transport, client)
1141 self.assertRaises(errors.UnknownFormatError,
1142 branch._get_checkout_format)
1143 self.assertFinished(client)
1144
1145
1146class TestBranchGetPhysicalLockStatus(RemoteBranchTestCase):1143class TestBranchGetPhysicalLockStatus(RemoteBranchTestCase):
11471144
1148 def test_get_physical_lock_status_yes(self):1145 def test_get_physical_lock_status_yes(self):
@@ -4216,3 +4213,43 @@
4216 'Repository.unlock', ('quack/', 'token', 'False'),4213 'Repository.unlock', ('quack/', 'token', 'False'),
4217 'success', ('ok', ))4214 'success', ('ok', ))
4218 repo.pack(['hinta', 'hintb'])4215 repo.pack(['hinta', 'hintb'])
4216
4217
4218class TestRepositoryIterInventories(TestRemoteRepository):
4219 """Test Repository.iter_inventories."""
4220
4221 def _serialize_inv_delta(self, old_name, new_name, delta):
4222 serializer = inventory_delta.InventoryDeltaSerializer(True, False)
4223 return "".join(serializer.delta_to_lines(old_name, new_name, delta))
4224
4225 def test_single_empty(self):
4226 transport_path = 'quack'
4227 repo, client = self.setup_fake_client_and_repository(transport_path)
4228 fmt = bzrdir.format_registry.get('2a')().repository_format
4229 repo._format = fmt
4230 stream = [('inventory-delta', [
4231 versionedfile.FulltextContentFactory('somerevid', None, None,
4232 self._serialize_inv_delta('null:', 'somerevid', []))])]
4233 client.add_expected_call(
4234 'VersionedFileRepository.get_inventories', ('quack/', 'unordered'),
4235 'success', ('ok', ),
4236 _stream_to_byte_stream(stream, fmt))
4237 ret = list(repo.iter_inventories(["somerevid"]))
4238 self.assertLength(1, ret)
4239 inv = ret[0]
4240 self.assertEquals("somerevid", inv.revision_id)
4241
4242 def test_empty(self):
4243 transport_path = 'quack'
4244 repo, client = self.setup_fake_client_and_repository(transport_path)
4245 ret = list(repo.iter_inventories([]))
4246 self.assertEquals(ret, [])
4247
4248 def test_missing(self):
4249 transport_path = 'quack'
4250 repo, client = self.setup_fake_client_and_repository(transport_path)
4251 client.add_expected_call(
4252 'VersionedFileRepository.get_inventories', ('quack/', 'unordered'),
4253 'success', ('ok', ), iter([]))
4254 self.assertRaises(errors.NoSuchRevision, list, repo.iter_inventories(
4255 ["somerevid"]))
42194256
=== modified file 'bzrlib/tests/test_smart.py'
--- bzrlib/tests/test_smart.py 2011-12-11 16:36:30 +0000
+++ bzrlib/tests/test_smart.py 2011-12-11 16:36:30 +0000
@@ -32,6 +32,7 @@
32 bzrdir,32 bzrdir,
33 errors,33 errors,
34 gpg,34 gpg,
35 inventory_delta,
35 tests,36 tests,
36 transport,37 transport,
37 urlutils,38 urlutils,
@@ -224,6 +225,24 @@
224 self.assertEqual(expected, request.execute('', 'False'))225 self.assertEqual(expected, request.execute('', 'False'))
225226
226227
228class TestSmartServerBzrDirRequestCloningMetaDir(
229 tests.TestCaseWithMemoryTransport):
230 """Tests for BzrDir.checkout_metadir."""
231
232 def test_checkout_metadir(self):
233 backing = self.get_transport()
234 request = smart_dir.SmartServerBzrDirRequestCheckoutMetaDir(
235 backing)
236 branch = self.make_branch('.', format='2a')
237 response = request.execute('')
238 self.assertEqual(
239 smart_req.SmartServerResponse(
240 ('Bazaar-NG meta directory, format 1\n',
241 'Bazaar repository format 2a (needs bzr 1.16 or later)\n',
242 'Bazaar Branch Format 7 (needs bzr 1.6)\n')),
243 response)
244
245
227class TestSmartServerBzrDirRequestDestroyBranch(246class TestSmartServerBzrDirRequestDestroyBranch(
228 tests.TestCaseWithMemoryTransport):247 tests.TestCaseWithMemoryTransport):
229 """Tests for BzrDir.destroy_branch."""248 """Tests for BzrDir.destroy_branch."""
@@ -1458,35 +1477,6 @@
1458 smart_req.SmartServerResponse(('no',)), response)1477 smart_req.SmartServerResponse(('no',)), response)
14591478
14601479
1461class TestSmartServerBranchRequestGetCheckoutFormat(TestLockedBranch):
1462
1463 def test_lightweight(self):
1464 backing = self.get_transport()
1465 request = smart_branch.SmartServerBranchRequestGetCheckoutFormat(
1466 backing)
1467 branch = self.make_branch('.', format='2a')
1468 response = request.execute('', 'True')
1469 self.assertEqual(
1470 smart_req.SmartServerResponse(
1471 ('Bazaar-NG meta directory, format 1\n',
1472 'Bazaar repository format 2a (needs bzr 1.16 or later)\n',
1473 'Bazaar Branch Format 7 (needs bzr 1.6)\n')),
1474 response)
1475
1476 def test_heavyweight(self):
1477 backing = self.get_transport()
1478 request = smart_branch.SmartServerBranchRequestGetCheckoutFormat(
1479 backing)
1480 branch = self.make_branch('.', format='2a')
1481 response = request.execute('', 'False')
1482 self.assertEqual(
1483 smart_req.SmartServerResponse((
1484 'Bazaar-NG meta directory, format 1\n',
1485 'Bazaar repository format 2a (needs bzr 1.16 or later)\n',
1486 'Bazaar Branch Format 7 (needs bzr 1.6)\n')),
1487 response)
1488
1489
1490class TestSmartServerBranchRequestUnlock(TestLockedBranch):1480class TestSmartServerBranchRequestUnlock(TestLockedBranch):
14911481
1492 def setUp(self):1482 def setUp(self):
@@ -2456,8 +2446,6 @@
2456 smart_branch.SmartServerBranchPutConfigFile)2446 smart_branch.SmartServerBranchPutConfigFile)
2457 self.assertHandlerEqual('Branch.get_parent',2447 self.assertHandlerEqual('Branch.get_parent',
2458 smart_branch.SmartServerBranchGetParent)2448 smart_branch.SmartServerBranchGetParent)
2459 self.assertHandlerEqual('Branch.get_checkout_format',
2460 smart_branch.SmartServerBranchRequestGetCheckoutFormat)
2461 self.assertHandlerEqual('Branch.get_physical_lock_status',2449 self.assertHandlerEqual('Branch.get_physical_lock_status',
2462 smart_branch.SmartServerBranchRequestGetPhysicalLockStatus)2450 smart_branch.SmartServerBranchRequestGetPhysicalLockStatus)
2463 self.assertHandlerEqual('Branch.get_tags_bytes',2451 self.assertHandlerEqual('Branch.get_tags_bytes',
@@ -2492,6 +2480,8 @@
2492 smart_dir.SmartServerRequestInitializeBzrDir)2480 smart_dir.SmartServerRequestInitializeBzrDir)
2493 self.assertHandlerEqual('BzrDirFormat.initialize_ex_1.16',2481 self.assertHandlerEqual('BzrDirFormat.initialize_ex_1.16',
2494 smart_dir.SmartServerRequestBzrDirInitializeEx)2482 smart_dir.SmartServerRequestBzrDirInitializeEx)
2483 self.assertHandlerEqual('BzrDir.checkout_metadir',
2484 smart_dir.SmartServerBzrDirRequestCheckoutMetaDir)
2495 self.assertHandlerEqual('BzrDir.cloning_metadir',2485 self.assertHandlerEqual('BzrDir.cloning_metadir',
2496 smart_dir.SmartServerBzrDirRequestCloningMetaDir)2486 smart_dir.SmartServerBzrDirRequestCloningMetaDir)
2497 self.assertHandlerEqual('BzrDir.get_config_file',2487 self.assertHandlerEqual('BzrDir.get_config_file',
@@ -2558,6 +2548,8 @@
2558 smart_repo.SmartServerRepositoryAbortWriteGroup)2548 smart_repo.SmartServerRepositoryAbortWriteGroup)
2559 self.assertHandlerEqual('VersionedFileRepository.get_serializer_format',2549 self.assertHandlerEqual('VersionedFileRepository.get_serializer_format',
2560 smart_repo.SmartServerRepositoryGetSerializerFormat)2550 smart_repo.SmartServerRepositoryGetSerializerFormat)
2551 self.assertHandlerEqual('VersionedFileRepository.get_inventories',
2552 smart_repo.SmartServerRepositoryGetInventories)
2561 self.assertHandlerEqual('Transport.is_readonly',2553 self.assertHandlerEqual('Transport.is_readonly',
2562 smart_req.SmartServerIsReadonly)2554 smart_req.SmartServerIsReadonly)
25632555
@@ -2623,3 +2615,48 @@
2623 smart_req.SuccessfulSmartServerResponse(('ok', ), ),2615 smart_req.SuccessfulSmartServerResponse(('ok', ), ),
2624 request.do_body(''))2616 request.do_body(''))
26252617
2618
2619class TestSmartServerRepositoryGetInventories(tests.TestCaseWithTransport):
2620
2621 def _get_serialized_inventory_delta(self, repository, base_revid, revid):
2622 base_inv = repository.revision_tree(base_revid).inventory
2623 inv = repository.revision_tree(revid).inventory
2624 inv_delta = inv._make_delta(base_inv)
2625 serializer = inventory_delta.InventoryDeltaSerializer(True, False)
2626 return "".join(serializer.delta_to_lines(base_revid, revid, inv_delta))
2627
2628 def test_single(self):
2629 backing = self.get_transport()
2630 request = smart_repo.SmartServerRepositoryGetInventories(backing)
2631 t = self.make_branch_and_tree('.', format='2a')
2632 self.addCleanup(t.lock_write().unlock)
2633 self.build_tree_contents([("file", "somecontents")])
2634 t.add(["file"], ["thefileid"])
2635 t.commit(rev_id='somerev', message="add file")
2636 self.assertIs(None, request.execute('', 'unordered'))
2637 response = request.do_body("somerev\n")
2638 self.assertTrue(response.is_successful())
2639 self.assertEquals(response.args, ("ok", ))
2640 stream = [('inventory-delta', [
2641 versionedfile.FulltextContentFactory('somerev', None, None,
2642 self._get_serialized_inventory_delta(
2643 t.branch.repository, 'null:', 'somerev'))])]
2644 fmt = bzrdir.format_registry.get('2a')().repository_format
2645 self.assertEquals(
2646 "".join(response.body_stream),
2647 "".join(smart_repo._stream_to_byte_stream(stream, fmt)))
2648
2649 def test_empty(self):
2650 backing = self.get_transport()
2651 request = smart_repo.SmartServerRepositoryGetInventories(backing)
2652 t = self.make_branch_and_tree('.', format='2a')
2653 self.addCleanup(t.lock_write().unlock)
2654 self.build_tree_contents([("file", "somecontents")])
2655 t.add(["file"], ["thefileid"])
2656 t.commit(rev_id='somerev', message="add file")
2657 self.assertIs(None, request.execute('', 'unordered'))
2658 response = request.do_body("")
2659 self.assertTrue(response.is_successful())
2660 self.assertEquals(response.args, ("ok", ))
2661 self.assertEquals("".join(response.body_stream),
2662 "Bazaar pack format 1 (introduced in 0.18)\nB54\n\nBazaar repository format 2a (needs bzr 1.16 or later)\nE")
26262663
=== modified file 'doc/en/release-notes/bzr-2.5.txt'
--- doc/en/release-notes/bzr-2.5.txt 2011-12-11 16:36:30 +0000
+++ doc/en/release-notes/bzr-2.5.txt 2011-12-11 16:36:30 +0000
@@ -111,6 +111,9 @@
111* Plugins can now register additional "location aliases".111* Plugins can now register additional "location aliases".
112 (Jelmer Vernooij)112 (Jelmer Vernooij)
113113
114* ``bzr status`` no longer shows shelves if files are specified.
115 (Francis Devereux)
116
114* Revision specifiers will now only browse as much history as they117* Revision specifiers will now only browse as much history as they
115 need to, rather than grabbing the whole history unnecessarily in some118 need to, rather than grabbing the whole history unnecessarily in some
116 cases. (Jelmer Vernooij)119 cases. (Jelmer Vernooij)
@@ -236,6 +239,13 @@
236 ``Repository.get_revision_signature_text``.239 ``Repository.get_revision_signature_text``.
237 (Jelmer Vernooij)240 (Jelmer Vernooij)
238241
242* Add HPSS calls for ``Repository.iter_files_bytes`` and
243 ``VersionedFileRepository.get_inventories``, speeding up
244 several commands including ``bzr export`` and ``bzr co --lightweight``.
245 (Jelmer Vernooij, #608640)
246
247* Add HPSS call for ``Branch.get_checkout_format``. (Jelmer Vernooij, #894459)
248
239* Add HPSS call for ``Repository.pack``. (Jelmer Vernooij, #894461)249* Add HPSS call for ``Repository.pack``. (Jelmer Vernooij, #894461)
240250
241* Custom HPSS error handlers can now be installed in the smart server client251* Custom HPSS error handlers can now be installed in the smart server client