Merge lp:~alfonsosanchezbeato/media-hub/get-selected-streams into lp:media-hub

[Alfonso Sanchez-Beato]

Gather information about missing codecs and selected streams from gstreamer
so we have more information on whether we should play the video or not (LP: #1417950).

Gather information about missing codecs and selected streams from gstreamer
so we have more information on whether we should play the video or not (LP: #1417950).

[Jim Hodapp (jhodapp)]

A few suggestions below. I like the general direction of this approach. Are you planning on adding anything else to it or is this it?

review: Needs Fixing (code)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Comments addressed. @Jim, yes, for the moment this is it until we modify media-hub's DBus interface so we can transmit more information to the client like:

* Available tracks with type and description
* Information on which tracks can be played and which not due to missing codecs
* A way to select the tracks to play
* An intermediate state in which we can access to this information but we are still not playing (gstreamer state for the stream would be "paused"). Maybe we can move to that state when media is set instead of moving from the initial gstreamer state to playing when the DBus play() method is invoked.

[Jim Hodapp (jhodapp)]

See my comments below.

review: Needs Fixing (code)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Comments addressed

[Jim Hodapp (jhodapp)]

Looks great, thanks. Let me know when you have this in a silo and I'll help test this well.

review: Approve (code)

[lp:~alfonsosanchezbeato/media-hub/get-selected-streams]

149. By Alfonso Sanchez-Beato on 2015-08-10

Gather information about missing codecs and selected streams from gstreamer
so we have more information on whether we should play the video or not.

=== modified file 'debian/changelog'
--- debian/changelog	2015-07-24 14:54:21 +0000
+++ debian/changelog	2015-08-10 13:09:04 +0000
@@ -1,3 +1,4 @@
+<<<<<<< TREE
 media-hub (3.1.0+15.10.20150724-0ubuntu1) wily; urgency=medium
 
   [ Alfonso Sanchez-Beato (email Canonical) ]
@@ -7,6 +8,16 @@
  -- CI Train Bot <[e-mail]>  Fri, 24 Jul 2015 14:54:21 +0000
 
 media-hub (3.1.0+15.10.20150710-0ubuntu1) wily; urgency=medium
+=======
+media-hub (3.1.0+15.04.20150810-0ubuntu1) UNRELEASED; urgency=medium
+
+  * Gather information about missing codecs and selected streams from gstreamer
+    so we have more information on whether we should play the video or not.
+
+ -- Alfonso Sanchez-Beato (email Canonical) <[e-mail]>  Mon, 10 Aug 2015 15:04:18 +0200
+
+media-hub (3.1.0+15.04.20150710-0ubuntu1) vivid; urgency=medium
+>>>>>>> MERGE-SOURCE
 
   [ CI Train Bot ]
   * New rebuild forced.
@@ -17,7 +28,7 @@
 
  -- CI Train Bot <[e-mail]>  Fri, 10 Jul 2015 18:12:21 +0000
 
-media-hub (3.1.0+15.10.20150604-0ubuntu1) wily; urgency=medium
+media-hub (3.1.0+15.04.20150604-0ubuntu1) vivid; urgency=medium
 
   [ CI Train Bot ]
   * New rebuild forced.
@@ -28,7 +39,7 @@
 
  -- CI Train Bot <[e-mail]>  Thu, 04 Jun 2015 14:18:22 +0000
 
-media-hub (3.1.0+15.10.20150601-0ubuntu1) wily; urgency=medium
+media-hub (3.1.0+15.04.20150601-0ubuntu1) vivid; urgency=medium
 
   [ CI Train Bot ]
   * New rebuild forced.
@@ -39,7 +50,7 @@
 
  -- CI Train Bot <[e-mail]>  Mon, 01 Jun 2015 16:31:53 +0000
 
-media-hub (3.1.0+15.10.20150527.1-0ubuntu1) wily; urgency=medium
+media-hub (3.1.0+15.04.20150527.1-0ubuntu1) vivid; urgency=medium
 
   [ CI Train Bot ]
   * New rebuild forced.
@@ -50,7 +61,7 @@
 
  -- CI Train Bot <[e-mail]>  Wed, 27 May 2015 18:38:16 +0000
 
-media-hub (3.1.0+15.10.20150522-0ubuntu1) wily; urgency=medium
+media-hub (3.1.0+15.04.20150522-0ubuntu1) vivid; urgency=medium
 
   [ Jim Hodapp ]
   * Fix issues with not reporting failed decoding error to the client.

=== modified file 'debian/control'
--- debian/control	2015-03-12 11:38:32 +0000
+++ debian/control	2015-08-10 13:09:04 +0000
@@ -25,6 +25,7 @@
                libprocess-cpp-dev,
                libproperties-cpp-dev,
                libgstreamer1.0-dev,
+               libgstreamer-plugins-base1.0-dev,
                pkg-config,
                libpulse-dev,
                qtbase5-dev,

=== modified file 'src/core/media/CMakeLists.txt'
--- src/core/media/CMakeLists.txt	2015-02-24 15:22:19 +0000
+++ src/core/media/CMakeLists.txt	2015-08-10 13:09:04 +0000
@@ -1,4 +1,5 @@
 pkg_check_modules(PC_GSTREAMER_1_0 REQUIRED gstreamer-1.0)
+pkg_check_modules(PC_GSTREAMER_PBUTILS_1_0 REQUIRED gstreamer-pbutils-1.0)
 pkg_check_modules(PC_PULSE_AUDIO REQUIRED libpulse)
 include_directories(${PC_GSTREAMER_1_0_INCLUDE_DIRS} ${HYBRIS_MEDIA_CFLAGS} ${PC_PULSE_AUDIO_INCLUDE_DIRS})
 
@@ -126,6 +127,7 @@
   ${DBUS_CPP_LDFLAGS}
   ${GLog_LIBRARY}
   ${PC_GSTREAMER_1_0_LIBRARIES}
+  ${PC_GSTREAMER_PBUTILS_1_0_LIBRARIES}
   ${PROCESS_CPP_LDFLAGS}
   ${GIO_LIBRARIES}
   ${HYBRIS_MEDIA_LIBRARIES}

=== modified file 'src/core/media/gstreamer/engine.cpp'
--- src/core/media/gstreamer/engine.cpp	2015-06-01 16:30:59 +0000
+++ src/core/media/gstreamer/engine.cpp	2015-08-10 13:09:04 +0000
@@ -67,8 +67,22 @@
     {
         if (p.second == "playbin")
         {
-            std::cout << "State changed on playbin: " << p.first.new_state << std::endl;
-            playback_status_changed(gst_state_to_player_status(p.first));
+            std::cout << "State changed on playbin: "
+                      << gst_element_state_get_name(p.first.new_state) << std::endl;
+            const auto status = gst_state_to_player_status(p.first);
+            /*
+             * When state moves to "paused" the pipeline is already set. We check that we
+             * have streams to play.
+             */
+            if (status == media::Player::PlaybackStatus::paused &&
+                    !playbin.can_play_streams()) {
+                std::cerr << "** Cannot play: some codecs are missing" << std::endl;
+                playbin.reset();
+                const media::Player::Error e = media::Player::Error::format_error;
+                error(e);
+            } else {
+                playback_status_changed(status);
+            }
         }
     }
 
@@ -149,7 +163,8 @@
                 break;
             case GST_STREAM_ERROR_CODEC_NOT_FOUND:
                 std::cerr << "** Encountered a GST_STREAM_ERROR_CODEC_NOT_FOUND" << std::endl;
-                ret_error = media::Player::Error::format_error;
+                // Missing codecs are handled later, when state switches to "paused"
+                ret_error = media::Player::Error::no_error;
                 break;
             case GST_STREAM_ERROR_DECODE:
                 std::cerr << "** Encountered a GST_STREAM_ERROR_DECODE" << std::endl;
@@ -163,8 +178,10 @@
             }
         }
 
-        std::cout << "Resetting playbin pipeline after unrecoverable error" << std::endl;
-        playbin.reset();
+        if (ret_error != media::Player::Error::no_error) {
+            std::cerr << "Resetting playbin pipeline after unrecoverable error" << std::endl;
+            playbin.reset();
+        }
         return ret_error;
     }
 

=== modified file 'src/core/media/gstreamer/playbin.cpp'
--- src/core/media/gstreamer/playbin.cpp	2015-07-24 14:54:19 +0000
+++ src/core/media/gstreamer/playbin.cpp	2015-08-10 13:09:04 +0000
@@ -1,5 +1,5 @@
 /*
- * Copyright © 2013 Canonical Ltd.
+ * Copyright © 2013-2015 Canonical Ltd.
  *
  * This program is free software: you can redistribute it and/or modify it
  * under the terms of the GNU Lesser General Public License version 3,
@@ -20,6 +20,8 @@
 
 #include <core/media/gstreamer/engine.h>
 
+#include <gst/pbutils/missing-plugins.h>
+
 #if defined(MEDIA_HUB_HAVE_HYBRIS_MEDIA_COMPAT_LAYER)
 #include <hybris/media/surface_texture_client_hybris.h>
 #include <hybris/media/media_codec_layer.h>
@@ -112,7 +114,11 @@
         core::ubuntu::media::video::Width{0}},
       player_lifetime(media::Player::Lifetime::normal),
       about_to_finish_handler_id(0),
-      source_setup_handler_id(0)
+      source_setup_handler_id(0),
+      is_missing_audio_codec(false),
+      is_missing_video_codec(false),
+      audio_stream_id(-1),
+      video_stream_id(-1)
 {
     if (!pipeline)
         throw std::runtime_error("Could not create pipeline for playbin.");
@@ -177,6 +183,44 @@
         std::cout << "Failed to reset the pipeline state. Client reconnect may not function properly." << std::endl;
     }
     file_type = MEDIA_FILE_TYPE_NONE;
+    is_missing_audio_codec = false;
+    is_missing_video_codec = false;
+    audio_stream_id = -1;
+    video_stream_id = -1;
+}
+
+void gstreamer::Playbin::process_message_element(GstMessage *message)
+{
+    if (!gst_is_missing_plugin_message(message))
+      return;
+
+    gchar *desc = gst_missing_plugin_message_get_description(message);
+    std::cerr << "Missing plugin: " << desc << std::endl;
+    g_free(desc);
+
+    const GstStructure *msg_data = gst_message_get_structure(message);
+    if (g_strcmp0("decoder", gst_structure_get_string(msg_data, "type")) != 0)
+        return;
+
+    GstCaps *caps;
+    if (!gst_structure_get(msg_data, "detail", GST_TYPE_CAPS, &caps, NULL)) {
+        std::cerr << __PRETTY_FUNCTION__ << ": No detail" << std::endl;
+        return;
+    }
+
+    GstStructure *caps_data = gst_caps_get_structure(caps, 0);
+    if (!caps_data) {
+        std::cerr << __PRETTY_FUNCTION__ << ": No caps data" << std::endl;
+        return;
+    }
+
+    const gchar *mime = gst_structure_get_name(caps_data);
+    if (strstr(mime, "audio"))
+        is_missing_audio_codec = true;
+    else if (strstr(mime, "video"))
+        is_missing_video_codec = true;
+
+    std::cerr << "Missing decoder for " << mime << std::endl;
 }
 
 void gstreamer::Playbin::on_new_message_async(const Bus::Message& message)
@@ -193,8 +237,15 @@
         signals.on_info(message.detail.error_warning_info);
         break;
     case GST_MESSAGE_STATE_CHANGED:
+        if (message.source == "playbin") {
+            g_object_get(G_OBJECT(pipeline), "current-audio", &audio_stream_id, NULL);
+            g_object_get(G_OBJECT(pipeline), "current-video", &video_stream_id, NULL);
+        }
         signals.on_state_changed(std::make_pair(message.detail.state_changed, message.source));
         break;
+    case GST_MESSAGE_ELEMENT:
+        process_message_element(message.message);
+        break;
     case GST_MESSAGE_TAG:
         {
             gchar *orientation;
@@ -598,3 +649,22 @@
 {
     return file_type;
 }
+
+bool gstreamer::Playbin::can_play_streams() const
+{
+    /*
+     * We do not consider that we can play the video when
+     * 1. No audio stream selected due to missing decoder
+     * 2. No video stream selected due to missing decoder
+     * 3. No stream selected at all
+     * Note that if there are several, say, audio streams, we will play the file
+     * provided that we can decode just one of them, even if there are missing
+     * audio codecs. We will also play files with only one type of stream.
+     */
+    if ((is_missing_audio_codec && audio_stream_id == -1) ||
+            (is_missing_video_codec && video_stream_id == -1) ||
+            (audio_stream_id == -1 && video_stream_id == -1))
+        return false;
+    else
+        return true;
+}

=== modified file 'src/core/media/gstreamer/playbin.h'
--- src/core/media/gstreamer/playbin.h	2015-06-12 18:14:05 +0000
+++ src/core/media/gstreamer/playbin.h	2015-08-10 13:09:04 +0000
@@ -72,6 +72,7 @@
 
     void on_new_message(const Bus::Message& message);
     void on_new_message_async(const Bus::Message& message);
+    void process_message_element(GstMessage *message);
 
     gstreamer::Bus& message_bus();
 
@@ -110,6 +111,8 @@
 
     MediaFileType media_file_type() const;
 
+    bool can_play_streams() const;
+
     GstElement* pipeline;
     gstreamer::Bus bus;
     MediaFileType file_type;
@@ -137,6 +140,10 @@
         core::Signal<core::ubuntu::media::video::Dimensions> on_video_dimensions_changed;
         core::Signal<void> client_disconnected;
     } signals;
+    bool is_missing_audio_codec;
+    bool is_missing_video_codec;
+    gint audio_stream_id;
+    gint video_stream_id;
 };
 }