Merge lp:~jamesh/mediascanner2/fix-tokenizer into lp:mediascanner2

[James Henstridge]

Update the FTS tokenizer to work with SQLite 3.8.5, and reenable the MediaStoreTest.query_short test. Bump libsqlite3-dev build dependency to 3.8.5 too.

Fix the full text search tokenizer to work with SQLite 3.8.5.

With the following change, sqlite3 stopped passing the wildcard suffix to the tokenizer, which broke our hack for detecting when we were tokenizing a wild card query:

  http://www.sqlite.org/cgi/src/info/e21bf7a2ade6373e94ea403c665f78e1ad22143f

So now always tokenize short words at the end of input.

This shouldn't require a rebuild of the index, since this last case in the tokenizer shouldn't have been hit when parsing actual file metadata (probably: it would have matched input that actually ended in '*'). The new version of the tokenizer could potentially cause extra words to be added to the index, but they are probably low quality (being short) so I wouldn't bother with a rebuild.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:250
http://jenkins.qa.ubuntu.com/job/mediascanner2-ci/102/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-utopic-amd64-ci/43
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-utopic-armhf-ci/43
        deb: http://jenkins.qa.ubuntu.com/job/mediascanner2-utopic-armhf-ci/43/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mediascanner2-utopic-i386-ci/43

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mediascanner2-ci/102/rebuild

review: Approve (continuous-integration)

[Jussi Pakkanen (jpakkane)]

Needs to close bug 1335281 but otherwise ok.

review: Approve

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2014-06-06 09:06:32 +0000
+++ CMakeLists.txt	2014-07-01 10:16:01 +0000
@@ -12,7 +12,7 @@
 include(FindPkgConfig)
 pkg_check_modules(MEDIASCANNER REQUIRED
   gio-2.0
-  sqlite3
+  sqlite3>=3.8.5
 )
 pkg_check_modules(GST gstreamer-1.0 gstreamer-pbutils-1.0 REQUIRED)
 pkg_check_modules(GLIB glib-2.0 REQUIRED)

=== modified file 'debian/control'
--- debian/control	2014-06-24 15:47:53 +0000
+++ debian/control	2014-07-01 10:16:01 +0000
@@ -15,7 +15,7 @@
                libgstreamer1.0-dev,
                libgtest-dev,
                libproperties-cpp-dev,
-               libsqlite3-dev,
+               libsqlite3-dev (>= 3.8.5),
                qt5-default,
                qtbase5-dev,
                qtbase5-dev-tools,

=== modified file 'src/mediascanner/mozilla/fts3_porter.c'
--- src/mediascanner/mozilla/fts3_porter.c	2014-05-14 13:22:54 +0000
+++ src/mediascanner/mozilla/fts3_porter.c	2014-07-01 10:16:01 +0000
@@ -1080,10 +1080,9 @@
     /* We emit a token if:
      *  - there are two ideograms together,
      *  - there are three chars or more,
-     *  - we think this is a query and wildcard magic is desired.
-     * We think is a wildcard query when we have at least one
-     *  character, our current offset is one shy of nInput and the
-     *  character at iOffset is '*'.
+     *  - we are at the end of the input and have at least one char.
+     * This final case is to cover wildcard prefix searches with short
+     * prefixes.
      */
     // It is possible we have no token to emit here if iPrevBigramOffset was not
     //  0 on entry and there was no second CJK character.  iPrevBigramOffset
@@ -1092,10 +1091,9 @@
         (numChars == 2 && state == BIGRAM_USE) ||
         // otherwise, drop two-letter words (considered stop-words)
         (numChars >=3) ||
-        // wildcard case:
+        // final word wildcard case:
         (numChars >= 1 &&
-         (c->iOffset == c->nInput - 1) &&
-         (z[c->iOffset] == '*'))) {
+         c->iOffset == c->nInput)) {
       /* figure out the number of bytes to copy/stem */
       int n = c->iOffset - iStartOffset;
       /* make sure there is enough buffer space */

=== modified file 'test/test_mediastore.cc'
--- test/test_mediastore.cc	2014-06-27 14:12:38 +0000
+++ test/test_mediastore.cc	2014-07-01 10:16:01 +0000
@@ -316,13 +316,6 @@
     EXPECT_EQ(result[1], audio2); // title has highest weighting
 }
 
-/* When SQLite updated from 3.8.2 to 3.8.5, the behavior for
- * short searches seems to have changed and the test queries
- * here return empty. This test is currently disabled to make
- * the test suite pass so a new version of Mediascanner
- * can enter the archive. Do try to re-enable this as
- * soon as possible.
-
 TEST_F(MediaStoreTest, query_short) {
     MediaFile audio1 = MediaFileBuilder("/path/foo5.ogg")
         .setType(AudioMedia)
@@ -346,7 +339,7 @@
     result = store.query("xy", AudioMedia);
     EXPECT_EQ(result.size(), 1);
 }
-*/
+
 TEST_F(MediaStoreTest, query_empty) {
     MediaFile audio1 = MediaFileBuilder("/path/foo5.ogg")
         .setType(AudioMedia)