Initial library import slows to a crawl

OP can you try again with the latest version? There's been some changes to this functionality

Thanks for reporting the bug!

Turns out this was an issue with deduplication - it was using a deduplication run atop linked list instead of a dedicated auto-deduplicating data structure such as hash set or tree set.

I've switched the data structure for files to import to TreeSet; it should fix the issue. I'm still testing whether it fixed it because importing 35,000 tracks in the old implementation is indeed slow!

I also did an import under sysprof and now most time is spent in mpegaudioparse2 and wcsxfrm called from it, so there's not much we can do to speed up the import further.

Did this bug fix really get merged? Last night when i was importing my library it took about an hour to import ~2500 songs.

In 0.3.1 now when I change my music folder the interface freezes from the moment I click "Open" in the file dialog and while the import does seem to stop at some point (the cpu usage for noise goes down after 10~20 minutes) the interface remains frozen and unusable. When I force kill it and start it up again it's like the import didn't happen...
Also, gdb gives me this as a fatal error: [GLib-GObject] Read-only property 'read-only-view' on class 'GeeReadOnlyBidirSortedSet' has type 'GeeSortedSet' which is not equal to or more restrictive than the type 'GeeBidirSortedSet' of the property on the interface 'GeeBidirSortedSet'
Is that related? By the way I also get the "FileUtils.vala:229: Could not pre-scan music folder. Progress percentage may be off: Permission denied" warning that already has 2 duplicate bugs filed about it I think.

I just tried getting noise 0.3.1+r1909+pkg95~daily~ubuntu0.4.1 to scan my 24 GB music collection, and it has also slowed to a crawl, currently a bit below the 8 GB mark. Two things I've noticed:

1. I can hear disk activity each time it says it has scanned another song. It would probably help to batch the inserts into transactions (e.g. only commit the changes every N seconds, or every N files during the scan).

2. When I checked the schema of the sqlite3 database, there were no indexes. I'm guessing there is a SELECT query being done before inserting each file that results in a full table scan. This would be consistent with the process slowing down as the database gets larger. If you can add an index that can be used by this query (or queries), it will likely make a huge performance improvement.

The second option is probably the easiest one to fix to get rid of the slow down. The first would probably only make a noticeable difference after fixing the second.

Am I the only one still experiencing this extremely slow import process, even with all updates installed?