OverflowError calling fp.read on large repository

Can you please try upgrading to a later version of bzr, as this bug may already have been fixed. See http://bazaar-vcs.org/Download

Thanks for your answer. I apologize because I forget to mention that the version I'm using is the latest available from the official repositories for Ubuntu Hardy (LTS). In fact I was quite surprised to find that practically there wasn't any update since Hardy has been released!

I manually installed the latest version and (without recreating the repository!!) get the following:

$ ~/Desktop/bzr-1.9/bzr branch file:///home/anna/Pictures/ pic.bzr-repo
bzr: ERROR: exceptions.OverflowError: long int too large to convert to int

Traceback (most recent call last):
  File "/home/anna/Desktop/bzr-1.9/bzrlib/commands.py", line 893, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/commands.py", line 839, in run_bzr
    ret = run(*run_argv)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/commands.py", line 539, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/builtins.py", line 980, in run
    force_new_repo=standalone)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/bzrdir.py", line 1113, in sprout
    result_repo.fetch(source_repository, revision_id=revision_id)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repository.py", line 989, in fetch
    find_ghosts=find_ghosts)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repository.py", line 2849, in fetch
    return self._pack(self.source, self.target, revision_ids)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repository.py", line 2856, in _pack
    revision_ids).pack()
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repofmt/pack_repo.py", line 605, in pack
    return self._create_pack_from_packs()
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repofmt/pack_repo.py", line 740, in _create_pack_from_packs
    self._copy_text_texts()
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repofmt/pack_repo.py", line 704, in _copy_text_texts
    self.new_pack.text_index, readv_group_iter, total_items))
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repofmt/pack_repo.py", line 825, in _copy_nodes_graph
    write_index, output_lines, pb, readv_group_iter, total_items):
  File "/home/anna/Desktop/bzr-1.9/bzrlib/repofmt/pack_repo.py", line 848, in _do_copy_nodes_graph
    izip(reader.iter_records(), node_vector):
  File "/home/anna/Desktop/bzr-1.9/bzrlib/pack.py", line 272, in _iter_records
    for record in self._iter_record_objects():
  File "/home/anna/Desktop/bzr-1.9/bzrlib/pack.py", line 277, in _iter_record_objects
    record_kind = self.reader_func(1)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/pack.py", line 218, in reader_func
    return self._source.read(length)
  File "/home/anna/Desktop/bzr-1.9/bzrlib/pack.py", line 177, in read
    self._next()
  File "/home/anna/Desktop/bzr-1.9/bzrlib/pack.py", line 172, in _next
    length, data = self.readv_result.next()
  File "/home/anna/Desktop/bzr-1.9/bzrlib/transport/__init__.py", line 680, in _seek_and_read
    data = fp.read(c_offset.length)
OverflowError: long int too large to convert to int

bzr 1.9 on python 2.5.2 (linux2)
arguments: ['/home/anna/Desktop/...

We have fixed the code that does smart fetches to not fetch more than 5MB at a time, but IIRC we don't have any code that limits the local fetching.

So it seems we are trying to copy >2GB of data in one read + write, and that breaks fp.read().

As a side note, it appears that you are not using a shared repository, which means we have to copy the 20GB of history around. If you had done "bzr init-repo" at the beginning, we would only need to create new working-trees or very lightweight branches, etc.

As a quick workaround, you could do:

  bzr init-repo
  rm .bzr/repository -rf
  mv existing_branch/.bzr/repository .bzr
  touch .bzr/repository/shared-storage

Note that you only want to do this if you truly *don't* have a shared repository already, otherwise it deletes the existing shared repo.

Anyway, the simple fix is to pass "max_size=XXXX" to Transport._coalesce_offsets. At the moment we use "0" as the default value, which means unlimited. We could make 0 == 2GB, or more reasonably have a harder max-size of 10MB or so. 10MB should be a reasonable buffer size that doesn't hit all of available RAM, and also avoids enough round trips.

As an example, see the attached patch.

Merged for 1.11 -
http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C49347330.7000205%40arbash-meinel.com%3E