Merge lp:~jpakkane/mediascanner2/rtmupdate into lp:mediascanner2/rtm-14.09

Proposed by Jussi Pakkanen on 2015-01-26
Status: Merged
Approved by: Pete Woods on 2015-01-28
Approved revision: 293
Merged at revision: 287
Proposed branch: lp:~jpakkane/mediascanner2/rtmupdate
Merge into: lp:mediascanner2/rtm-14.09
Diff against target: 408 lines (+177/-15)
12 files modified
CMakeLists.txt (+1/-0)
debian/changelog (+18/-0)
debian/control (+2/-0)
src/daemon/CMakeLists.txt (+4/-2)
src/daemon/MetadataExtractor.cc (+44/-7)
src/daemon/SubtreeWatcher.cc (+2/-0)
src/daemon/scannerdaemon.cc (+12/-0)
src/mediascanner/MediaStore.cc (+5/-1)
src/mediascanner/internal/sqliteutils.hh (+14/-1)
src/mediascanner/mozilla/fts3_porter.c (+23/-4)
test/basic.cc (+25/-0)
test/test_metadataextractor.cc (+27/-0)
To merge this branch: bzr merge lp:~jpakkane/mediascanner2/rtmupdate
Reviewer Review Type Date Requested Status
Pete Woods (community) Approve on 2015-01-28
dobey (community) Approve on 2015-01-27
Marcus Tomlinson (community) 2015-01-26 Approve on 2015-01-26
Review via email: mp+247583@code.launchpad.net

Commit message

Update rtm with cherry picked bug fixes from trunk. Fix RTM critical bug #1414566

Description of the change

For files that do not have EXIF timestamps we use creation date instead. Re-run queries that return SQLITE_BUSY.

To post a comment you must log in.
Marcus Tomlinson (marcustomlinson) wrote :

+1

review: Approve
dobey (dobey) wrote :

Looks ok to me.

review: Approve
Pete Woods (pete-woods) :
review: Approve
lp:~jpakkane/mediascanner2/rtmupdate updated on 2015-02-04
294. By Jussi Pakkanen on 2015-02-04

Fixed debian/changelog version numbers so build can proceed.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'CMakeLists.txt'
2--- CMakeLists.txt 2014-09-26 04:24:19 +0000
3+++ CMakeLists.txt 2015-02-04 15:06:18 +0000
4@@ -18,6 +18,7 @@
5 )
6 pkg_check_modules(GST gstreamer-1.0 gstreamer-pbutils-1.0 REQUIRED)
7 pkg_check_modules(GLIB glib-2.0 REQUIRED)
8+pkg_check_modules(PIXBUF gdk-pixbuf-2.0 REQUIRED)
9 pkg_check_modules(EXIF libexif REQUIRED)
10 pkg_check_modules(DBUSCPP dbus-cpp REQUIRED)
11 pkg_check_modules(APPARMOR libapparmor REQUIRED)
12
13=== modified file 'debian/changelog'
14--- debian/changelog 2014-10-01 06:59:46 +0000
15+++ debian/changelog 2015-02-04 15:06:18 +0000
16@@ -1,3 +1,21 @@
17+mediascanner2 (0.105+15.04.20150127.1-0ubuntu1) UNRELEASED; urgency=medium
18+
19+ * Retry sqlite queries that are busy. (LP: #1414566)
20+ * Use creation date as timestamp on pictures that do not have
21+ exif data.
22+ * New build-dependency on gdkpixbuf.
23+
24+ -- Jussi Pakkanen <jussi.pakkanen@ubuntu.com> Tue, 27 Jan 2015 17:09:24 +0200
25+
26+mediascanner2 (0.105+15.04.20141030.1~rtm-0ubuntu1) 14.09; urgency=low
27+
28+ [ James Henstridge ]
29+ * When a new directory is added and a new inotify watch is set up,
30+ call fileAdded() on any regular files already in the directory so
31+ they are not missed. (LP: #1379817)
32+
33+ -- Ubuntu daily release <ps-jenkins@lists.canonical.com> Thu, 30 Oct 2014 11:23:12 +0000
34+
35 mediascanner2 (0.105+14.10.20141001-0ubuntu1) utopic; urgency=low
36
37 [ James Henstridge ]
38
39=== modified file 'debian/control'
40--- debian/control 2014-09-26 04:24:19 +0000
41+++ debian/control 2015-02-04 15:06:18 +0000
42@@ -15,6 +15,7 @@
43 libglib2.0-dev,
44 libgstreamer-plugins-base1.0-dev,
45 libgstreamer1.0-dev,
46+ libgdk-pixbuf2.0-dev,
47 libgtest-dev,
48 libproperties-cpp-dev,
49 libsqlite3-dev (>= 3.8.5),
50@@ -30,6 +31,7 @@
51 # For running unit tests
52 gstreamer1.0-plugins-base,
53 gstreamer1.0-plugins-good,
54+ shared-mime-info,
55 Homepage: https://launchpad.net/mediascanner2
56 XS-Testsuite: autopkgtest
57 # if you don't have have commit access to this branch but would like to upload
58
59=== modified file 'src/daemon/CMakeLists.txt'
60--- src/daemon/CMakeLists.txt 2014-09-26 04:24:19 +0000
61+++ src/daemon/CMakeLists.txt 2015-02-04 15:06:18 +0000
62@@ -1,4 +1,5 @@
63-add_definitions(${MEDIASCANNER_CFLAGS} ${GST_CFLAGS} ${EXIF_CFLAGS} ${UDISKS_CFLAGS})
64+add_definitions(${MEDIASCANNER_CFLAGS} ${GST_CFLAGS} ${EXIF_CFLAGS} ${UDISKS_CFLAGS}
65+${PIXBUF_CFLAGS})
66 include_directories(..)
67
68 add_library(scannerstuff STATIC
69@@ -10,7 +11,8 @@
70 ../mediascanner/utils.cc
71 )
72
73-target_link_libraries(scannerstuff ${GST_LIBRARIES} ${EXIF_LDFLAGS} ${UDISKS_LDFLAGS})
74+target_link_libraries(scannerstuff ${GST_LIBRARIES} ${EXIF_LDFLAGS} ${UDISKS_LDFLAGS}
75+${PIXBUF_LDFLAGS})
76
77 add_executable(scannerdaemon
78 scannerdaemon.cc
79
80=== modified file 'src/daemon/MetadataExtractor.cc'
81--- src/daemon/MetadataExtractor.cc 2014-09-25 10:52:51 +0000
82+++ src/daemon/MetadataExtractor.cc 2015-02-04 15:06:18 +0000
83@@ -27,6 +27,7 @@
84 #include <gio/gio.h>
85 #include <gst/gst.h>
86 #include <gst/pbutils/pbutils.h>
87+#include <gdk-pixbuf/gdk-pixbuf.h>
88
89 #include <cstdio>
90 #include <ctime>
91@@ -35,8 +36,18 @@
92 #include <stdexcept>
93 #include <vector>
94
95+#include <unistd.h>
96+#include <sys/types.h>
97+#include <sys/stat.h>
98+
99 using namespace std;
100
101+namespace {
102+const char exif_date_template[] = "%Y:%m:%d %H:%M:%S";
103+const char iso8601_date_format[] = "%Y-%m-%dT%H:%M:%S";
104+const char iso8601_date_with_zone_format[] = "%Y-%m-%dT%H:%M:%S%z";
105+}
106+
107 namespace mediascanner {
108
109 struct MetadataExtractorPrivate {
110@@ -44,7 +55,8 @@
111 MetadataExtractorPrivate() : discoverer(nullptr, g_object_unref) {};
112
113 void extract_gst(const DetectedFile &d, MediaFileBuilder &mfb);
114- void extract_exif(const DetectedFile &d, MediaFileBuilder &mfb);
115+ bool extract_exif(const DetectedFile &d, MediaFileBuilder &mfb);
116+ void extract_pixbuf(const DetectedFile &d, MediaFileBuilder &mfb);
117 };
118
119 MetadataExtractor::MetadataExtractor(int seconds) {
120@@ -218,9 +230,6 @@
121 EXIF_TAG_DATE_TIME_ORIGINAL,
122 EXIF_TAG_DATE_TIME_DIGITIZED
123 };
124- static const char exif_date_template[] = "%Y:%m:%d %H:%M:%S";
125- static const char iso8601_date_format[] = "%Y-%m-%dT%H:%M:%S";
126- static const char iso8601_date_with_zone_format[] = "%Y-%m-%dT%H:%M:%S%z";
127 struct tm timeinfo;
128 bool have_date = false;
129
130@@ -356,7 +365,7 @@
131 mfb.setLongitude(longitude);
132 }
133
134-void MetadataExtractorPrivate::extract_exif(const DetectedFile &d, MediaFileBuilder &mfb) {
135+bool MetadataExtractorPrivate::extract_exif(const DetectedFile &d, MediaFileBuilder &mfb) {
136 std::unique_ptr<ExifLoader, void(*)(ExifLoader*)> loader(
137 exif_loader_new(), exif_loader_unref);
138 exif_loader_write_file(loader.get(), d.filename.c_str());
139@@ -366,7 +375,7 @@
140 loader.reset();
141
142 if (!data) {
143- throw runtime_error("Unable to load EXIF data from " + d.filename);
144+ return false;
145 }
146 exif_data_fix(data.get());
147 ExifByteOrder order = exif_data_get_byte_order(data.get());
148@@ -374,6 +383,32 @@
149 parse_exif_date(data.get(), order, mfb);
150 parse_exif_dimensions(data.get(), order, mfb);
151 parse_exif_location(data.get(), order, mfb);
152+ return true;
153+}
154+
155+void MetadataExtractorPrivate::extract_pixbuf(const DetectedFile &d, MediaFileBuilder &mfb) {
156+ gint width, height;
157+
158+ if(!gdk_pixbuf_get_file_info(d.filename.c_str(), &width, &height)) {
159+ string msg("Could not determine resolution of ");
160+ msg += d.filename;
161+ msg += ".";
162+ throw runtime_error(msg);
163+ }
164+
165+ struct stat info;
166+ if(stat(d.filename.c_str(), &info) == 0) {
167+ char buf[1024];
168+ struct tm ptm;
169+ localtime_r(&info.st_mtime, &ptm);
170+ if (strftime(buf, sizeof(buf), iso8601_date_format, &ptm) != 0) {
171+ mfb.setDate(buf);
172+ }
173+ }
174+
175+
176+ mfb.setWidth(width);
177+ mfb.setHeight(height);
178 }
179
180 MediaFile MetadataExtractor::extract(const DetectedFile &d) {
181@@ -385,7 +420,9 @@
182
183 switch (d.type) {
184 case ImageMedia:
185- p->extract_exif(d, mfb);
186+ if(!p->extract_exif(d, mfb)) {
187+ p->extract_pixbuf(d, mfb);
188+ }
189 break;
190 default:
191 p->extract_gst(d, mfb);
192
193=== modified file 'src/daemon/SubtreeWatcher.cc'
194--- src/daemon/SubtreeWatcher.cc 2014-09-24 09:39:06 +0000
195+++ src/daemon/SubtreeWatcher.cc 2015-02-04 15:06:18 +0000
196@@ -135,6 +135,8 @@
197 lstat(fullpath.c_str(), &statbuf);
198 if(S_ISDIR(statbuf.st_mode)) {
199 addDir(fullpath);
200+ } else if (S_ISREG(statbuf.st_mode)) {
201+ fileAdded(fullpath);
202 }
203 }
204 }
205
206=== modified file 'src/daemon/scannerdaemon.cc'
207--- src/daemon/scannerdaemon.cc 2014-09-30 11:50:44 +0000
208+++ src/daemon/scannerdaemon.cc 2015-02-04 15:06:18 +0000
209@@ -189,9 +189,21 @@
210
211 void ScannerDaemon::readFiles(MediaStore &store, const string &subdir, const MediaType type) {
212 Scanner s(extractor.get(), subdir, type);
213+ const int update_interval = 10; // How often to send invalidations.
214+ struct timespec previous_update, current_time;
215+ clock_gettime(CLOCK_MONOTONIC, &previous_update);
216+ previous_update.tv_sec -= update_interval/2; // Send the first update sooner for better visual appeal.
217 while(true) {
218 try {
219 auto d = s.next();
220+ clock_gettime(CLOCK_MONOTONIC, &current_time);
221+ while(g_main_context_pending(g_main_context_default())) {
222+ g_main_context_iteration(g_main_context_default(), FALSE);
223+ }
224+ if(current_time.tv_sec - previous_update.tv_sec >= update_interval) {
225+ invalidator.invalidate();
226+ previous_update = current_time;
227+ }
228 // If the file is broken or unchanged, use fallback.
229 if (store.is_broken_file(d.filename, d.etag)) {
230 fprintf(stderr, "Using fallback data for unscannable file %s.\n", d.filename.c_str());
231
232=== modified file 'src/mediascanner/MediaStore.cc'
233--- src/mediascanner/MediaStore.cc 2014-09-24 09:39:06 +0000
234+++ src/mediascanner/MediaStore.cc 2015-02-04 15:06:18 +0000
235@@ -410,7 +410,11 @@
236 archiveItems(retireprefix);
237 } else {
238 if(detectedSchemaVersion != schemaVersion) {
239- throw runtime_error("Tried to open a db with an unsupported schema version.");
240+ std::string msg("Tried to open a db with schema version ");
241+ msg += std::to_string(detectedSchemaVersion);
242+ msg += ", while supported version is ";
243+ msg += std::to_string(schemaVersion) + ".";
244+ throw runtime_error(msg);
245 }
246 }
247 }
248
249=== modified file 'src/mediascanner/internal/sqliteutils.hh'
250--- src/mediascanner/internal/sqliteutils.hh 2014-07-16 08:52:59 +0000
251+++ src/mediascanner/internal/sqliteutils.hh 2015-02-04 15:06:18 +0000
252@@ -71,7 +71,20 @@
253 }
254
255 bool step() {
256- rc = sqlite3_step(statement);
257+ // Sqlite docs list a few cases where you need to to a rollback
258+ // if a calling step fails. We don't match those cases but if
259+ // we change queries that may start to happen.
260+ // https://sqlite.org/c3ref/step.html
261+ //
262+ // The proper fix would probably be to move to a WAL log, but
263+ // it seems to require write access to the mediastore dir
264+ // even for readers, which is problematic for confined apps.
265+ int retry_count=0;
266+ const int max_retries = 100;
267+ do {
268+ rc = sqlite3_step(statement);
269+ retry_count++;
270+ } while(rc == SQLITE_BUSY && retry_count < max_retries);
271 switch (rc) {
272 case SQLITE_DONE:
273 return false;
274
275=== modified file 'src/mediascanner/mozilla/fts3_porter.c'
276--- src/mediascanner/mozilla/fts3_porter.c 2014-07-01 09:59:07 +0000
277+++ src/mediascanner/mozilla/fts3_porter.c 2015-02-04 15:06:18 +0000
278@@ -934,6 +934,18 @@
279 return 0; /* no need to break; nothing to emit */
280 }
281
282+static inline int tokenizer_hides_wildcards(void) {
283+ static int first_run = 1;
284+ static int hides_wildcard;
285+
286+ if (first_run) {
287+ int version = sqlite3_libversion_number();
288+ hides_wildcard = version > 3008004 && version < 3008007;
289+ first_run = 0;
290+ }
291+ return hides_wildcard;
292+}
293+
294 /**
295 * Generate a new token. There are basically three types of token we can
296 * generate:
297@@ -1080,9 +1092,14 @@
298 /* We emit a token if:
299 * - there are two ideograms together,
300 * - there are three chars or more,
301- * - we are at the end of the input and have at least one char.
302- * This final case is to cover wildcard prefix searches with short
303- * prefixes.
304+ * - we think this is a query and wildcard magic is desired.
305+ * We think is a wildcard query when we have at least one
306+ * character, our current offset is one shy of nInput and the
307+ * character at iOffset is '*'.
308+ *
309+ * Sqlite 3.8.5 ... 3.8.6 changed how the tokenizer is called such
310+ * that we can't detect wildcards. In that case, we accept any
311+ * short word at the end of the input.
312 */
313 // It is possible we have no token to emit here if iPrevBigramOffset was not
314 // 0 on entry and there was no second CJK character. iPrevBigramOffset
315@@ -1093,7 +1110,9 @@
316 (numChars >=3) ||
317 // final word wildcard case:
318 (numChars >= 1 &&
319- c->iOffset == c->nInput)) {
320+ (tokenizer_hides_wildcards() ?
321+ (c->iOffset == c->nInput) :
322+ (c->iOffset == c->nInput - 1 && z[c->iOffset] == '*')))) {
323 /* figure out the number of bytes to copy/stem */
324 int n = c->iOffset - iStartOffset;
325 /* make sure there is enough buffer space */
326
327=== modified file 'test/basic.cc'
328--- test/basic.cc 2014-09-24 09:39:06 +0000
329+++ test/basic.cc 2015-02-04 15:06:18 +0000
330@@ -230,6 +230,31 @@
331 ASSERT_THROW(s.next(), StopIteration);
332 }
333
334+TEST(Mediascanner, scan_files_found_in_new_dir) {
335+ string testdir = TEST_DIR "/testdir";
336+ string subdir = testdir + "/subdir";
337+ string testfile = SOURCE_DIR "/media/testfile.ogg";
338+ string outfile = subdir + "/testfile.ogg";
339+ clear_dir(testdir);
340+ ASSERT_GE(mkdir(testdir.c_str(), S_IRUSR | S_IWUSR | S_IXUSR), 0);
341+
342+ MediaStore store(":memory:", MS_READ_WRITE);
343+ MetadataExtractor extractor;
344+ InvalidationSender invalidator;
345+ SubtreeWatcher watcher(store, extractor, invalidator);
346+ watcher.addDir(testdir);
347+ ASSERT_EQ(watcher.directoryCount(), 1);
348+ ASSERT_EQ(store.size(), 0);
349+
350+ // Create a new directory and a file inside that directory before
351+ // the watcher has a chance to set up an inotify watch.
352+ ASSERT_GE(mkdir(subdir.c_str(), S_IRUSR | S_IWUSR | S_IXUSR), 0);
353+ copy_file(testfile, outfile);
354+ iterate_main_loop();
355+ ASSERT_EQ(watcher.directoryCount(), 2);
356+ ASSERT_EQ(store.size(), 1);
357+}
358+
359 int main(int argc, char **argv) {
360 gst_init (&argc, &argv);
361 ::testing::InitGoogleTest(&argc, argv);
362
363=== added file 'test/media/image3.png'
364Binary files test/media/image3.png 1970-01-01 00:00:00 +0000 and test/media/image3.png 2015-02-04 15:06:18 +0000 differ
365=== modified file 'test/test_metadataextractor.cc'
366--- test/test_metadataextractor.cc 2014-06-06 09:01:07 +0000
367+++ test/test_metadataextractor.cc 2015-02-04 15:06:18 +0000
368@@ -22,6 +22,7 @@
369
370 #include "test_config.h"
371
372+#include<algorithm>
373 #include<stdexcept>
374 #include<cstdio>
375 #include<string>
376@@ -132,6 +133,32 @@
377 EXPECT_DOUBLE_EQ(153.1727346, file.getLongitude());
378 }
379
380+// PNG files don't have exif entries, so test we work with those, too.
381+TEST_F(MetadataExtractorTest, png_file) {
382+ MetadataExtractor e;
383+ MediaFile file = e.extract(e.detect(SOURCE_DIR "/media/image3.png"));
384+
385+ EXPECT_EQ(ImageMedia, file.getType());
386+ EXPECT_EQ(640, file.getWidth());
387+ EXPECT_EQ(400, file.getHeight());
388+ // The time stamp on the test file can be anything. We can't guarantee what it is,
389+ // so just inspect the format.
390+ auto timestr = file.getDate();
391+ EXPECT_EQ(timestr.size(), 19);
392+ EXPECT_EQ(timestr.find('T'), 10);
393+
394+ // These can't go inside EXPECT_EQ because it is a macro and mixing templates
395+ // with macros makes things explode.
396+ auto dashes = std::count_if(timestr.begin(), timestr.end(), [](char c) { return c == '-';});
397+ auto colons = std::count_if(timestr.begin(), timestr.end(), [](char c) { return c == ':';});
398+ EXPECT_EQ(dashes, 2);
399+ EXPECT_EQ(colons, 2);
400+
401+ EXPECT_DOUBLE_EQ(0, file.getLatitude());
402+ EXPECT_DOUBLE_EQ(0, file.getLongitude());
403+
404+}
405+
406 int main(int argc, char **argv) {
407 gst_init (&argc, &argv);
408 ::testing::InitGoogleTest(&argc, argv);

Subscribers

People subscribed via source and target branches