Merge lp:~michihenning/thumbnailer/add-preview-image-extraction into lp:thumbnailer/devel

Proposed by Michi Henning
Status: Merged
Approved by: James Henstridge
Approved revision: 284
Merged at revision: 286
Proposed branch: lp:~michihenning/thumbnailer/add-preview-image-extraction
Merge into: lp:thumbnailer/devel
Diff against target: 251 lines (+127/-25)
5 files modified
debian/control (+2/-0)
src/thumbnailer.cpp (+1/-1)
src/vs-thumb/thumbnailextractor.cpp (+64/-24)
tests/stress/stress_test.cpp (+16/-0)
tests/vs-thumb/vs-thumb_test.cpp (+44/-0)
To merge this branch: bzr merge lp:~michihenning/thumbnailer/add-preview-image-extraction
Reviewer Review Type Date Requested Status
James Henstridge Approve
PS Jenkins bot (community) continuous-integration Approve
Review via email: mp+271063@code.launchpad.net

Commit message

vs-thumb now looks for both image and preview image tags to extract artwork.

Description of the change

vs-thumb now looks for both image and preview image tags to extract artwork.

To post a comment you must log in.
Revision history for this message
James Henstridge (jamesh) wrote :

This looks it works by accident. A file can contain multiple images, so the code in question is iterating through them to find one that represents the cover. It stops when it hits the last image.

If there are no IMAGE tags in the file, it will ask for the 0th PREVIEW_IMAGE and generally do the right thing.

If there is a single image in the file, the first iteration of the loop will find that image, and on the second iteration it will get an error and look for PREVIEW_IMAGE at index 1. So if there is a single PREVIEW_IMAGE tag, we'll ignore it.

I think we want to iterate through all IMAGE tags and then all PREVIEW_IMAGE tags here.

review: Needs Fixing
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
James Henstridge (jamesh) wrote :

I think I've discovered the image type handling bug. Since you're modifying the code in question, I'll just note it here rather than creating a conflicting MP.

The following code:

        auto caps = gst_sample_get_caps(sample);
        auto structure = gst_caps_get_structure(caps, 0);

Needs to be changed to:

        auto structure = gst_sample_get_info(sample);

It seems the sample code I based this on was doing it wrong.

Revision history for this message
Michi Henning (michihenning) wrote :

Bloody awesome, thank you! Works like a charm, we are getting coverage on the cover art branch now.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
283. By Michi Henning

Added missing gstreamer initialization to stress test.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
284. By Michi Henning

Merged devel.

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
James Henstridge (jamesh) wrote :

This mostly looks good now. I've left a few inline comments about areas that could be improved though.

Revision history for this message
Michi Henning (michihenning) wrote :

Not happy about that either. It's awkward. I'm planning to re-factor vs-thumb anyway, so we get better diagnostics with time stamps. That'll be a good time to do something about the image extraction too, so could we let this slide for now? Functionally, it's correct, I believe.

Revision history for this message
James Henstridge (jamesh) wrote :

Lets merge this.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'debian/control'
--- debian/control 2015-09-15 03:22:34 +0000
+++ debian/control 2015-09-20 10:12:45 +0000
@@ -7,6 +7,8 @@
7 cmake-extras (>= 0.4),7 cmake-extras (>= 0.4),
8 debhelper (>= 9),8 debhelper (>= 9),
9 google-mock,9 google-mock,
10 gstreamer1.0-fluendo-mp3,
11 gstreamer1.0-plugins-bad-faad,
10 gstreamer1.0-plugins-good,12 gstreamer1.0-plugins-good,
11 libapparmor-dev,13 libapparmor-dev,
12 libboost-filesystem-dev,14 libboost-filesystem-dev,
1315
=== modified file 'src/thumbnailer.cpp'
--- src/thumbnailer.cpp 2015-09-15 09:37:17 +0000
+++ src/thumbnailer.cpp 2015-09-20 10:12:45 +0000
@@ -629,7 +629,6 @@
629 Image full_size(string(raw_data.data(), raw_data.size()));629 Image full_size(string(raw_data.data(), raw_data.size()));
630 return RequestBase::ImageData(full_size, RequestBase::CachePolicy::cache_fullsize, Location::remote);630 return RequestBase::ImageData(full_size, RequestBase::CachePolicy::cache_fullsize, Location::remote);
631 }631 }
632 qDebug() << "common_fetch(): Request failed:" << artreply->error_string();
633 if (artreply->not_found_error())632 if (artreply->not_found_error())
634 {633 {
635 return RequestBase::ImageData(RequestBase::FetchStatus::not_found, Location::remote);634 return RequestBase::ImageData(RequestBase::FetchStatus::not_found, Location::remote);
@@ -638,6 +637,7 @@
638 {637 {
639 return RequestBase::ImageData(RequestBase::FetchStatus::no_network, Location::remote);638 return RequestBase::ImageData(RequestBase::FetchStatus::no_network, Location::remote);
640 }639 }
640 qCritical() << "common_fetch(): Request failed:" << artreply->error_string();
641 return RequestBase::ImageData(RequestBase::FetchStatus::error, Location::remote);641 return RequestBase::ImageData(RequestBase::FetchStatus::error, Location::remote);
642}642}
643643
644644
=== modified file 'src/vs-thumb/thumbnailextractor.cpp'
--- src/vs-thumb/thumbnailextractor.cpp 2015-09-10 06:58:42 +0000
+++ src/vs-thumb/thumbnailextractor.cpp 2015-09-20 10:12:45 +0000
@@ -171,14 +171,59 @@
171171
172struct ThumbnailExtractor::Private172struct ThumbnailExtractor::Private
173{173{
174 void find_audio_cover(GstTagList* tags, char const* tag_name);
175
176 enum ImageType { cover, other };
177
174 gobj_ptr<GstElement> playbin;178 gobj_ptr<GstElement> playbin;
175 gint64 duration = -1;179 gint64 duration = -1;
176180
177 std::unique_ptr<GstSample, decltype(&gst_sample_unref)> sample{nullptr, gst_sample_unref};181 std::unique_ptr<GstSample, decltype(&gst_sample_unref)> sample{nullptr, gst_sample_unref};
178 GdkPixbufRotation sample_rotation = GDK_PIXBUF_ROTATE_NONE;182 GdkPixbufRotation sample_rotation = GDK_PIXBUF_ROTATE_NONE;
179 bool sample_raw = true;183 bool sample_raw = true;
184 ImageType image_type;
180};185};
181186
187// Look for an image with the specified tag. If we find a cover image, image_type is set to cover,
188// and sample points at the image. If we find some other (non-cover) image, image_type is set to other,
189// and sample points at the image. Otherwise, if we can't find any image at all, sample is set to nullptr.
190
191void ThumbnailExtractor::Private::find_audio_cover(GstTagList* tags, const char* tag_name)
192{
193 image_type = ThumbnailExtractor::Private::other;
194
195 bool found_cover = false;
196 for (int i = 0; !found_cover; i++)
197 {
198 GstSample* s;
199 if (!gst_tag_list_get_sample_index(tags, tag_name, i, &s))
200 {
201 break;
202 }
203 assert(s);
204 // Check the type of this image
205 int type = GST_TAG_IMAGE_TYPE_UNDEFINED;
206 auto structure = gst_sample_get_info(s);
207 if (structure)
208 {
209 gst_structure_get_enum(structure, "image-type", GST_TYPE_TAG_IMAGE_TYPE, &type);
210 }
211 if (type == GST_TAG_IMAGE_TYPE_FRONT_COVER)
212 {
213 sample.reset(gst_sample_ref(s));
214 found_cover = true;
215 image_type = ThumbnailExtractor::Private::cover;
216 }
217 else if (type == GST_TAG_IMAGE_TYPE_UNDEFINED && !sample)
218 {
219 // Save the first unknown image tag, but continue scanning
220 // in case there is one marked as the cover.
221 sample.reset(gst_sample_ref(s));
222 }
223 gst_sample_unref(s);
224 }
225}
226
182/* GstPlayFlags flags from playbin.227/* GstPlayFlags flags from playbin.
183 *228 *
184 * GStreamer does not install headers for the enums of individual229 * GStreamer does not install headers for the enums of individual
@@ -294,6 +339,7 @@
294 g_signal_emit_by_name(p->playbin.get(), "get-video-tags", 0, &tags);339 g_signal_emit_by_name(p->playbin.get(), "get-video-tags", 0, &tags);
295 if (tags)340 if (tags)
296 {341 {
342 // TODO: The "flip-rotate-*" transforms defined in gst/gsttaglist.h need to be added here.
297 char* orientation = nullptr;343 char* orientation = nullptr;
298 if (gst_tag_list_get_string_index(tags, GST_TAG_IMAGE_ORIENTATION, 0, &orientation) && orientation != nullptr)344 if (gst_tag_list_get_string_index(tags, GST_TAG_IMAGE_ORIENTATION, 0, &orientation) && orientation != nullptr)
299 {345 {
@@ -326,32 +372,26 @@
326 p->sample.reset();372 p->sample.reset();
327 p->sample_rotation = GDK_PIXBUF_ROTATE_NONE;373 p->sample_rotation = GDK_PIXBUF_ROTATE_NONE;
328 p->sample_raw = false;374 p->sample_raw = false;
329 bool found_cover = false;375
330 for (int i = 0; !found_cover; i++)376 p->find_audio_cover(tags, GST_TAG_IMAGE);
377 if (!p->sample || p->image_type == ThumbnailExtractor::Private::other)
331 {378 {
332 GstSample* sample;379 // Save whatever the first look-up returned (possibly nothing).
333 if (!gst_tag_list_get_sample_index(tags, GST_TAG_IMAGE, i, &sample))380 auto provisional_sample = std::move(p->sample);
334 {381
335 break;382 // We didn't find a full-size image, or we found an image that is not marked as the cover.
336 }383 // Try to find a preview image instead.
337 // Check the type of this image384 p->find_audio_cover(tags, GST_TAG_PREVIEW_IMAGE);
338 auto caps = gst_sample_get_caps(sample);385 if (p->image_type == ThumbnailExtractor::Private::other && provisional_sample)
339 auto structure = gst_caps_get_structure(caps, 0);386 {
340 int type = GST_TAG_IMAGE_TYPE_UNDEFINED;387 // We found a preview image that is *not* a cover, and the
341 gst_structure_get_enum(structure, "image-type", GST_TYPE_TAG_IMAGE_TYPE, &type);388 // first look-up for a normal image returned something
342 if (type == GST_TAG_IMAGE_TYPE_FRONT_COVER)389 // full-size that is *also* not a cover. Return the image
343 {390 // from the first look-up in preference to this one.
344 p->sample.reset(gst_sample_ref(sample));391 p->sample = std::move(provisional_sample);
345 found_cover = true;392 }
346 }
347 else if (type == GST_TAG_IMAGE_TYPE_UNDEFINED && !p->sample)
348 {
349 // Save the first unknown image tag, but continue scanning
350 // in case there is one marked as the cover.
351 p->sample.reset(gst_sample_ref(sample));
352 }
353 gst_sample_unref(sample);
354 }393 }
394
355 gst_tag_list_unref(tags);395 gst_tag_list_unref(tags);
356 return bool(p->sample);396 return bool(p->sample);
357}397}
358398
=== modified file 'tests/stress/stress_test.cpp'
--- tests/stress/stress_test.cpp 2015-07-31 06:23:59 +0000
+++ tests/stress/stress_test.cpp 2015-09-20 10:12:45 +0000
@@ -23,6 +23,19 @@
23#include <utils/dbusserver.h>23#include <utils/dbusserver.h>
24#include <utils/supports_decoder.h>24#include <utils/supports_decoder.h>
2525
26#pragma GCC diagnostic push
27#pragma GCC diagnostic ignored "-Wold-style-cast"
28#pragma GCC diagnostic ignored "-Wcast-qual"
29#pragma GCC diagnostic ignored "-Wcast-align"
30#ifdef __clang__
31#pragma clang diagnostic ignored "-Wparentheses-equality"
32#endif
33#include <gst/gst.h>
34#ifdef __clang__
35#pragma clang diagnostic pop
36#endif
37#pragma GCC diagnostic pop
38
26#include <boost/filesystem.hpp>39#include <boost/filesystem.hpp>
27#include <gtest/gtest.h>40#include <gtest/gtest.h>
28#include <QSignalSpy>41#include <QSignalSpy>
@@ -296,6 +309,9 @@
296309
297int main(int argc, char** argv)310int main(int argc, char** argv)
298{311{
312 // Need this because we are using a static test fixture and can't rely an global constructor order.
313 gst_init(&argc, &argv);
314
299 QCoreApplication app(argc, argv);315 QCoreApplication app(argc, argv);
300316
301 setenv("GSETTINGS_BACKEND", "memory", true);317 setenv("GSETTINGS_BACKEND", "memory", true);
302318
=== modified file 'tests/vs-thumb/vs-thumb_test.cpp'
--- tests/vs-thumb/vs-thumb_test.cpp 2015-09-10 06:58:42 +0000
+++ tests/vs-thumb/vs-thumb_test.cpp 2015-09-20 10:12:45 +0000
@@ -50,6 +50,8 @@
50const char MP4_LANDSCAPE_TEST_FILE[] = TESTDATADIR "/gegl-landscape.mp4";50const char MP4_LANDSCAPE_TEST_FILE[] = TESTDATADIR "/gegl-landscape.mp4";
51const char MP4_PORTRAIT_TEST_FILE[] = TESTDATADIR "/gegl-portrait.mp4";51const char MP4_PORTRAIT_TEST_FILE[] = TESTDATADIR "/gegl-portrait.mp4";
52const char VORBIS_TEST_FILE[] = TESTDATADIR "/testsong.ogg";52const char VORBIS_TEST_FILE[] = TESTDATADIR "/testsong.ogg";
53const char AAC_TEST_FILE[] = TESTDATADIR "/testsong.m4a";
54const char MP3_TEST_FILE[] = TESTDATADIR "/testsong.mp3";
5355
54class ExtractorTest : public ::testing::Test56class ExtractorTest : public ::testing::Test
55{57{
@@ -181,6 +183,48 @@
181 EXPECT_EQ(gdk_pixbuf_get_height(image.get()), 200);183 EXPECT_EQ(gdk_pixbuf_get_height(image.get()), 200);
182}184}
183185
186TEST_F(ExtractorTest, extract_aac_cover_art)
187{
188 if (!supports_decoder("audio/mpeg"))
189 {
190 fprintf(stderr, "No support for AAC decoder\n");
191 return;
192 }
193
194 ThumbnailExtractor extractor;
195
196 std::string outfile = tempdir + "/out.jpg";
197 extractor.set_uri(filename_to_uri(AAC_TEST_FILE));
198 ASSERT_FALSE(extractor.has_video());
199 ASSERT_TRUE(extractor.extract_audio_cover_art());
200 extractor.save_screenshot(outfile);
201
202 auto image = load_image(outfile);
203 EXPECT_EQ(gdk_pixbuf_get_width(image.get()), 200);
204 EXPECT_EQ(gdk_pixbuf_get_height(image.get()), 200);
205}
206
207TEST_F(ExtractorTest, extract_mp3_cover_art)
208{
209 if (!supports_decoder("audio/mpeg"))
210 {
211 fprintf(stderr, "No support for MP3 decoder\n");
212 return;
213 }
214
215 ThumbnailExtractor extractor;
216
217 std::string outfile = tempdir + "/out.jpg";
218 extractor.set_uri(filename_to_uri(MP3_TEST_FILE));
219 ASSERT_FALSE(extractor.has_video());
220 ASSERT_TRUE(extractor.extract_audio_cover_art());
221 extractor.save_screenshot(outfile);
222
223 auto image = load_image(outfile);
224 EXPECT_EQ(gdk_pixbuf_get_width(image.get()), 200);
225 EXPECT_EQ(gdk_pixbuf_get_height(image.get()), 200);
226}
227
184TEST_F(ExtractorTest, file_not_found)228TEST_F(ExtractorTest, file_not_found)
185{229{
186 ThumbnailExtractor extractor;230 ThumbnailExtractor extractor;

Subscribers

People subscribed via source and target branches

to all changes: