Cannot allocate memory with large manifest file since 0.7.03

Could someone try the new version and let me know if it helps? Thanks!

* Release tarball Install - https://launchpad.net/duplicity/+download
* Daily duplicity builds - https://launchpad.net/~duplicity-team/+archive/ubuntu/daily

NOTE: UNinstall duplicity first if it was installed via the distribution repository. For Ubuntu, that would be "sudo apt-get purge duplicity".

Just successfully ran a full backup with duplicity 0.7.14+bzr1335 after purging and installing from the daily PPA.

Thanks for testing!

...Ken

On Fri, Nov 3, 2017 at 11:15 AM, Julian Felix Rost <
<email address hidden>> wrote:

> Just successfully ran a full backup with duplicity 0.7.14+bzr1335 after
> purging and installing from the daily PPA.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1720159
>
> Title:
> Cannot allocate memory with large manifest file since 0.7.03
>
> Status in Duplicity:
> Fix Committed
>
> Bug description:
> Bug cause: https://bugs.launchpad.net/duplicity/+bug/1044715
> Question: https://answers.launchpad.net/duplicity/+question/658653
>
> Hi team
>
> I have used the duplicity for backup my server files from few months
> before. And It was working fine. Currently I have upgraded the
> duplicity new version(0.7.14), after upgraded the backup manifest
> cache file written in large size(near to GBs) due to bulk delete or
> changes in files.
>
> It was stopped the next backup process with this error - "OSError:
> [Errno 12] Cannot allocate memory".
>
> Please help me on this. Is there any way to solve this problem?
>
> Herewith I have placed the logged error information for your
> reference.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1548, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1534, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1383, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1515, in do_backup
> check_last_manifest(col_stats) # not needed for full backup
> File "/usr/bin/duplicity", line 1217, in check_last_manifest
> last_backup_set.check_manifests()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 208, in check_manifests
> remote_manifest = self.get_remote_manifest()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 241, in get_remote_manifest
> manifest_buffer = self.backend.get_data(self.remote_manifest_name)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py",
> line 677, in get_data
> fin = self.get_fileobj_read(filename, parseresults)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py",
> line 671, in get_fileobj_read
> return tdp.filtered_open_with_delete("rb")
> File "/usr/local/lib/python2.7/dist-packages/duplicity/dup_temp.py",
> line 120, in filtered_open_with_delete
> fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
> File "/usr/local/lib/python2.7/dist-packages/duplicity/path.py", line
> 779, in filtered_open
> return gpg.GPGFile(False, self, gpg_profile)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line
> 217, in __init__
> 'logger': self.logger_fp})
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 374, in run
> create_fhs, attach_fhs)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 414, in _attach_fork_exec
> process.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>...

Today an automatic backup had the following error stack trace. Still on build duplicity 0.7.14+bzr1335.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1559, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1545, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1394, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1473, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 729, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 558, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 521, in integrate_patch_iters
    for patch_seq in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 389, in yield_tuples
    setrorps(overflow, elems)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 378, in setrorps
    elems[i] = iter_list[i].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 107, in filter_path_iter
    for path in path_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 121, in difftar2path_iter
    tarinfo_list = [tar_iter.next()]
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 339, in next
    self.set_tarfile()
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 333, in set_tarfile
    self.current_fp = self.fileobj_iter.next()
  File "/usr/bin/duplicity", line 766, in get_fileobj_iter
    manifest.volume_info_dict[vol_num])
  File "/usr/bin/duplicity", line 809, in restore_get_enc_fileobj
    fileobj = tdp.filtered_open_with_delete("rb")
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 120, in filtered_open_with_delete
    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 779, in filtered_open
    return gpg.GPGFile(False, self, gpg_profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 224, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 420, in _attach_fork_exec
    process.thread.start()
  File "/usr/lib/python2.7/threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
 error: can't start new thread

That's https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1727653

See https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1727653/comments/17

Please add your name to that bug report to get notifications.

I have updated duplicity, but still getting memory error:
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 430, in write_multivol
    globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 385, in GPGWriteFile
    data = block_iter.next().data
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 520, in next
    result = self.process_continued()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 711, in process_continued
    return self.tarinfo2tarblock(index, ti, data)
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 490, in tarinfo2tarblock
    return TarBlock(index, "%s%s%s" % (headers, file_data, filler_data))
 MemoryError

Is there anything else that I can do?

It may be that you have too many incrementals. Did you start a new full
backup?

On Wed, Mar 28, 2018 at 12:58 AM, ausboffin <email address hidden>
wrote:

> I have updated duplicity, but still getting memory error:
> Traceback (innermost last):
> File "/usr/bin/duplicity", line 1555, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1541, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1393, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1523, in do_backup
> incremental_backup(sig_chain)
> File "/usr/bin/duplicity", line 668, in incremental_backup
> globals.backend)
> File "/usr/bin/duplicity", line 430, in write_multivol
> globals.volsize)
> File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 385, in
> GPGWriteFile
> data = block_iter.next().data
> File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 520,
> in next
> result = self.process_continued()
> File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 711,
> in process_continued
> return self.tarinfo2tarblock(index, ti, data)
> File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 490,
> in tarinfo2tarblock
> return TarBlock(index, "%s%s%s" % (headers, file_data, filler_data))
> MemoryError
>
> Is there anything else that I can do?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1720159
>
> Title:
> Cannot allocate memory with large manifest file since 0.7.03
>
> Status in Duplicity:
> Fix Released
>
> Bug description:
> Bug cause: https://bugs.launchpad.net/duplicity/+bug/1044715
> Question: https://answers.launchpad.net/duplicity/+question/658653
>
> Hi team
>
> I have used the duplicity for backup my server files from few months
> before. And It was working fine. Currently I have upgraded the
> duplicity new version(0.7.14), after upgraded the backup manifest
> cache file written in large size(near to GBs) due to bulk delete or
> changes in files.
>
> It was stopped the next backup process with this error - "OSError:
> [Errno 12] Cannot allocate memory".
>
> Please help me on this. Is there any way to solve this problem?
>
> Herewith I have placed the logged error information for your
> reference.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1548, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1534, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1383, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1515, in do_backup
> check_last_manifest(col_stats) # not needed for full backup
> File "/usr/bin/duplicity", line 1217, in check_last_manifest
> last_backup_set.check_manifests()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 208, in check_manifests
> remote_manifest = self.get_remote_manifest()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 241, in get_remote_manifest
> manifest_buffer = self.backend.get_data...

Thanks Kenneth,
I will try to do a new full backup. When I work out how to do that using the CLI. ;-)

I attempted a full backup via CLI and got the following error:

Cleanup of temporary directory /tmp/duplicity-0i2xBl-tempdir failed - this is probably a bug.
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 430, in write_multivol
    globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 381, in GPGWriteFile
    bytes_to_go = data_size - get_current_size()
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 373, in get_current_size
    return os.stat(filename).st_size
 OSError: [Errno 2] No such file or directory: '/tmp/duplicity-0i2xBl-tempdir/mktemp-VuWAbp-1544'

Should this be a new thread?

It's already a reported bug/nuisance. Your backup was finished at that
point.

On Sun, Apr 1, 2018 at 7:09 PM, ausboffin <email address hidden>
wrote:

> I attempted a full backup via CLI and got the following error:
>
> Cleanup of temporary directory /tmp/duplicity-0i2xBl-tempdir failed - this
> is probably a bug.
> Traceback (innermost last):
> File "/usr/bin/duplicity", line 1555, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1541, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1393, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1511, in do_backup
> full_backup(col_stats)
> File "/usr/bin/duplicity", line 572, in full_backup
> globals.backend)
> File "/usr/bin/duplicity", line 430, in write_multivol
> globals.volsize)
> File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 381, in
> GPGWriteFile
> bytes_to_go = data_size - get_current_size()
> File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 373, in
> get_current_size
> return os.stat(filename).st_size
> OSError: [Errno 2] No such file or directory: '/tmp/duplicity-0i2xBl-
> tempdir/mktemp-VuWAbp-1544'
>
> Should this be a new thread?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1720159
>
> Title:
> Cannot allocate memory with large manifest file since 0.7.03
>
> Status in Duplicity:
> Fix Released
>
> Bug description:
> Bug cause: https://bugs.launchpad.net/duplicity/+bug/1044715
> Question: https://answers.launchpad.net/duplicity/+question/658653
>
> Hi team
>
> I have used the duplicity for backup my server files from few months
> before. And It was working fine. Currently I have upgraded the
> duplicity new version(0.7.14), after upgraded the backup manifest
> cache file written in large size(near to GBs) due to bulk delete or
> changes in files.
>
> It was stopped the next backup process with this error - "OSError:
> [Errno 12] Cannot allocate memory".
>
> Please help me on this. Is there any way to solve this problem?
>
> Herewith I have placed the logged error information for your
> reference.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1548, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1534, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1383, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1515, in do_backup
> check_last_manifest(col_stats) # not needed for full backup
> File "/usr/bin/duplicity", line 1217, in check_last_manifest
> last_backup_set.check_manifests()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 208, in check_manifests
> remote_manifest = self.get_remote_manifest()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 241, in get_remote_manifest
> manifest_buffer = self.backend.get_data(self.remote_manifest_name)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/backend.py",
> line 677, in get_data
> fi...

Hello,

I use duplicity to backup my data. however I get always this message at end the backup

--------------[ Backup Statistics ]--------------
StartTime 1524741993.34 (Thu Apr 26 13:26:33 2018)
EndTime 1524746175.96 (Thu Apr 26 14:36:15 2018)
ElapsedTime 4182.62 (1 hour 9 minutes 42.62 seconds)
SourceFiles 3821570
SourceFileSize 191576839080 (178 GB)
NewFiles 2088902
NewFileSize 191562222070 (178 GB)
DeletedFiles 3
ChangedFiles 54
ChangedFileSize 1834439 (1.75 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2088959
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 199
-------------------------------------------------

Could you help me please to fix Error 199 ?

Thanks for your support

Best regards

Battant