Merge lp:~jpakkane/mediascanner2/invalidatorfix into lp:mediascanner2

[Jussi Pakkanen]

Send invalidations signals during scans that take a long time so the dash gets updated.

Send invalidations signals during scans that take a long time so the dash gets updated.

[Marcus Tomlinson (marcustomlinson)]

+1

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:298
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/179/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-amd64-ci/16/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/16
        deb: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/16/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-i386-ci/16

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/179/rebuild

review: Needs Fixing (continuous-integration)

[Francis Ginther (fginther)]

The jenkins node for the amd64 build failed, I've restarted a new ci run.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:298
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/180/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-amd64-ci/17
    FAILURE: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/17/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-i386-ci/17

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/180/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:298
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/181/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-amd64-ci/18
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/18
        deb: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/18/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-i386-ci/18

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/181/rebuild

review: Approve (continuous-integration)

[Jussi Pakkanen (jpakkane)]

If you run this on the device and tail -F mediascanner's log ever now and then it prints out an error about the database being locked. This seems to only happen when the scope is querying the database so it has one reader and one writer. Sqlite documentation says that this should be fully supported but it for some reason it does not work. Will investigate deeper.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:299
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/182/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-amd64-ci/19
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/19
        deb: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/19/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-i386-ci/19

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/182/rebuild

review: Approve (continuous-integration)

[lp:~jpakkane/mediascanner2/invalidatorfix]

300. By Jussi Pakkanen on 2015-01-27

Use wal mode for the journal as it seems to work when there are multiple connections to the db.

[Marcus Tomlinson (marcustomlinson)]

+1

review: Approve

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:300
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/183/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-amd64-ci/20
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/20
        deb: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-armhf-ci/20/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-vivid-i386-ci/20

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/183/rebuild

review: Approve (continuous-integration)

=== modified file 'src/daemon/scannerdaemon.cc'
--- src/daemon/scannerdaemon.cc	2014-10-22 21:08:00 +0000
+++ src/daemon/scannerdaemon.cc	2015-01-27 13:35:55 +0000
@@ -211,9 +211,21 @@
 
 void ScannerDaemon::readFiles(MediaStore &store, const string &subdir, const MediaType type) {
     Scanner s(extractor.get(), subdir, type);
+    const int update_interval = 10; // How often to send invalidations.
+    struct timespec previous_update, current_time;
+    clock_gettime(CLOCK_MONOTONIC, &previous_update);
+    previous_update.tv_sec -= update_interval/2; // Send the first update sooner for better visual appeal.
     while(true) {
         try {
             auto d = s.next();
+            clock_gettime(CLOCK_MONOTONIC, &current_time);
+            while(g_main_context_pending(g_main_context_default())) {
+                g_main_context_iteration(g_main_context_default(), FALSE);
+            }
+            if(current_time.tv_sec - previous_update.tv_sec >= update_interval) {
+                invalidator.invalidate();
+                previous_update = current_time;
+            }
             // If the file is broken or unchanged, use fallback.
             if (store.is_broken_file(d.filename, d.etag)) {
                 fprintf(stderr, "Using fallback data for unscannable file %s.\n", d.filename.c_str());

=== modified file 'src/mediascanner/MediaStore.cc'
--- src/mediascanner/MediaStore.cc	2014-09-24 09:39:06 +0000
+++ src/mediascanner/MediaStore.cc	2015-01-27 13:35:55 +0000
@@ -42,6 +42,11 @@
 
 using namespace std;
 
+static int dummy_callback(void*, int , char**, char**) {
+    return 0;
+}
+
+
 namespace mediascanner {
 
 // Increment this whenever changing db schema.
@@ -398,6 +403,16 @@
     if(sqlite3_open_v2(filename.c_str(), &p->db, sqliteFlags, nullptr) != SQLITE_OK) {
         throw runtime_error(sqlite3_errmsg(p->db));
     }
+    if(access == MS_READ_WRITE) {
+        // The default journal mode does not seem to be safe but
+        // instead gives out database locked errors when writing
+        // and querying at the same time.
+        char *err;
+        sqlite3_exec(p->db, "PRAGMA journal_mode=WAL;", dummy_callback, nullptr, &err);
+        if(err) {
+            throw runtime_error(err);
+        }
+    }
     register_tokenizer(p->db);
     register_functions(p->db);
     int detectedSchemaVersion = getSchemaVersion(p->db);