Ubuntu
conduit package

conduit fails on large folders

Bug #297085 reported by Robin Sheat
6
Affects Status Importance Assigned to Milestone
Conduit
Won't Fix
High
conduit (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

Binary package hint: conduit

I'm attempting to synchronise one folder to a backup drive on the same machine. The folder contains quite a large number of directories and files (but not stupidly huge).

Conduit goes so far and then runs out of file handles.

Version: 0.3.14-0ubuntu1 on amd64 Intrepid.

From the console:

[Syncronization ][DEBUG ] 1WAY PUT: emusic (file:///home/robin/mp3/emusic/Celldweller/Celldweller/08 Symbiont.mp3) -----> emusic (Synchronization.py:489)
[datatypes.DataType ][DEBUG ] Getting Rid for file:///home/robin/mp3/emusic/Celldweller/Celldweller/08 Symbiont.mp3 (DataType.py:151)
[platform.FileGio ][INFO ] Standard Attributes: standard::content-type,standard::name,standard::type,standard::size,standard::display-name,standard::edit-name,standard::copy-name,standard::icon,standard::fast-content-type (FileGio.py:25)
[platform.FileGio ][INFO ] Time Attributes: time::modified,time::modified-usec,time::access,time::access-usec,time::changed,time::changed-usec (FileGio.py:25)
[TypeConverter ][DEBUG ] Convert file -> file using [('file', 'file', {})] (TypeConverter.py:209)
[TypeConverter ][DEBUG ] Skipping file -> file (TypeConverter.py:168)
[Syncronization ][CRITICAL] UNKNOWN CONVERSION ERROR
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/conduit/Synchronization.py", line 355, in _convert_data
  File "/var/lib/python-support/python2.5/conduit/TypeConverter.py", line 217, in convert
  File "/var/lib/python-support/python2.5/conduit/TypeConverter.py", line 95, in _retain_info_in_conversion
  File "/var/lib/python-support/python2.5/conduit/datatypes/File.py", line 301, in set_mtime
  File "/var/lib/python-support/python2.5/conduit/datatypes/File.py", line 222, in force_new_mtime
  File "/var/lib/python-support/python2.5/conduit/datatypes/File.py", line 100, in _is_tempfile
  File "/usr/lib/python2.5/tempfile.py", line 262, in gettempdir
  File "/usr/lib/python2.5/tempfile.py", line 209, in _get_default_tempdir
IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/home/robin']
 (Synchronization.py:363)
[Syncronization ][INFO ] Could not put data: Was None (Synchronization.py:345)
[Syncronization ][DEBUG ] 1WAY PUT: emusic (file:///home/robin/mp3/emusic/Celldweller/Celldweller/01 Cell #1.mp3) -----> emusic (Synchronization.py:489)
[datatypes.DataType ][DEBUG ] Getting Rid for file:///home/robin/mp3/emusic/Celldweller/Celldweller/01 Cell #1.mp3 (DataType.py:151)
[Syncronization ][CRITICAL] UNKNOWN SYNCHRONIZATION ERROR
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/conduit/Synchronization.py", line 737, in run
  File "/var/lib/python-support/python2.5/conduit/Synchronization.py", line 490, in one_way_sync
  File "/var/lib/python-support/python2.5/conduit/datatypes/DataType.py", line 154, in get_rid
  File "/var/lib/python-support/python2.5/conduit/datatypes/File.py", line 290, in get_mtime
  File "/var/lib/python-support/python2.5/conduit/platform/FileGio.py", line 85, in get_mtime
AttributeError: 'NoneType' object has no attribute 'get_attribute_uint64'
 (Synchronization.py:745)
[Syncronization ][DEBUG ] Syncworker state 3 (Synchronization.py:648)
[dataproviders.File ][WARNING] Error saving volume group file: 'File' object has no attribute 'is_on_removale_volume' (File.py:362)
[dataproviders.File ][WARNING] Error saving volume group file: 'File' object has no attribute 'is_on_removale_volume' (File.py:362)
[Syncronization ][DEBUG ] Deleting worker: <SyncWorker(Wrapper: emusic twoway (UID: FolderTwoWay-file:///home/robin/mp3/emusic) |--> [<conduit.ModuleWrapper.ModuleWrapper instance at 0x2da1ea8>], stopped)> (Synchronization.py:86)
/var/lib/python-support/python2.5/conduit/Main.py:185: GtkWarning: Error loading theme icon 'gtk-missing-image' for stock: Error opening file: Too many open files

Related branches

lp:ubuntu/karmic/conduit
Revision history for this message
John Stowers (nzjrs) wrote :

Dunno how this launchpad thing works, but this is a dupe of GNOME bugzilla #555305

Revision history for this message
mabawsa (mabawsa) wrote :

I have the same issue.
[Syncronization ][DEBUG ] FolderTwoWay-file:///home/dougie/Music Changes: Modified 0 items
[] (Synchronization.py:242)
[Syncronization ][DEBUG ] FolderTwoWay-file:///home/dougie/Music Changes: Deleted 0 items
[] (Synchronization.py:243)
[Syncronization ][DEBUG ] 1WAY PUT: Music (file:///home/dougie/Music/The Soft Boys/The Soft Boys 1976-1981 (2)/04 - Have A Heart Betty (I'm Not Fireproof) #1.m4a) -----> music (Synchronization.py:489)
[datatypes.DataType ][DEBUG ] Getting Rid for file:///home/dougie/Music/The Soft Boys/The Soft Boys 1976-1981 (2)/04 - Have A Heart Betty (I'm Not Fireproof) #1.m4a (DataType.py:151)
[Syncronization ][CRITICAL] UNKNOWN SYNCHRONIZATION ERROR
Traceback (most recent call last):
  File "/var/lib/python-support/python2.5/conduit/Synchronization.py", line 737, in run
    self.one_way_sync(self.source, sink)
  File "/var/lib/python-support/python2.5/conduit/Synchronization.py", line 490, in one_way_sync
    dataRid = data.get_rid()
  File "/var/lib/python-support/python2.5/conduit/datatypes/DataType.py", line 154, in get_rid
    mtime=self.get_mtime(),
  File "/var/lib/python-support/python2.5/conduit/datatypes/File.py", line 290, in get_mtime
    ts = self._file.get_mtime()
  File "/var/lib/python-support/python2.5/conduit/platform/FileGio.py", line 85, in get_mtime
    mtime = self.fileInfo.get_attribute_uint64('time::modified')
AttributeError: 'NoneType' object has no attribute 'get_attribute_uint64'
 (Synchronization.py:745)
[Syncronization ][DEBUG ] Syncworker state 3 (Synchronization.py:648)
[dataproviders.File ][WARNING] Error saving volume group file: 'File' object has no attribute 'is_on_removale_volume' (File.py:362)
[dataproviders.File ][WARNING] Error saving volume group file: 'File' object has no attribute 'is_on_removale_volume' (File.py:362)
[Syncronization ][DEBUG ] Deleting worker: <SyncWorker(Wrapper: Music twoway (UID: FolderTwoWay-file:///home/dougie/Music) |--> [<conduit.ModuleWrapper.ModuleWrapper instance at 0x2464830>], stopped)> (Synchronization.py:86)

Changed in conduit:
status: New → Confirmed
Revision history for this message
mabawsa (mabawsa) wrote :

Seems this is a a problem with "#" being in a file
http://bugzilla.gnome.org/show_bug.cgi?id=529301

I removed the # and everything worked as advertised.

Bug Watch Updater (bug-watch-updater)
Changed in conduit:
status: Unknown → Invalid
Julien Lavergne (gilir)
Changed in conduit:
status: Invalid → Unknown
Bug Watch Updater (bug-watch-updater)
Changed in conduit:
status: Unknown → Confirmed
Revision history for this message
Julien Lavergne (gilir) wrote :

There is a patch upstream. If anyone want to try it, I attach it to the bug report.

Changed in conduit (Ubuntu):
importance: Undecided → Medium
Revision history for this message
jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount) wrote :

Is that really the problem? When I one-way sync my 35 GB folder, it'll sync maybe four folders and then say that it succeeded.

Revision history for this message
Julien Lavergne (gilir) wrote :

I uploaded to my PPA (https://launchpad.net/~gilir/+archive/updates) a possible fix for this problem (conduit - 0.3.16-0ubuntu3). Please test with this package and report back if the problem is still here or not.

Revision history for this message
jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount) wrote :

It's syncing my library as I type. It works like a charm.

Wonderful! Thank you, Julien.

Revision history for this message
jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount) wrote :

Spoke too soon. I was syncing my 40 GB Music Library to my External device (Rockbox), and it was going smoothly until 40%. It just stopped there. I refreshed and re-synced, and it said that everything was OK.

Here's a terminal output...

Revision history for this message
jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount) wrote :

I noticed that the folder has a colon in the name. Could that be the problem? I looked through all the folders that had been synced up to that point, and I didn't see any with a ":" symbol.

Revision history for this message
Robin Sheat (eythian) wrote :

Colons will choke on FAT32 devices. It's a limitation (one of many) of that filesystem.

Revision history for this message
Julien Lavergne (gilir) wrote :

@Chauncellor
Thanks for the test, but I think you find another problem :)
I'll upload the fix to close this bug. If you have another problem, please open a new bug.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package conduit - 0.3.16-0ubuntu3

---------------
conduit (0.3.16-0ubuntu3) karmic; urgency=low

  * debian/patches
   - 07-programming-error-unicode.dpatch: Fix unicode convertion with python2.6
    (LP: #371175).
   - 08-escaping-caracters.dpatch: Add uri escaping to handle '#' characters
    (LP: #297085).

 -- Julien Lavergne <email address hidden> Sun, 30 Aug 2009 13:35:15 +0200

Changed in conduit (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount) wrote :

I mass renamed my library to suit FAT32 and all 42 GB has been synced well. I did have lots of conflicts, but I guess that's for another report.

Thank you, Julien :)

Bug Watch Updater (bug-watch-updater)
Changed in conduit:
importance: Unknown → High
Bug Watch Updater (bug-watch-updater)
Changed in conduit:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.