Merge lp:~afrantzis/mir/fix-1189770 into lp:mir

[Alexandros Frantzis]

server: Gracefully handle exceptions thrown from server threads

server: Gracefully handle exceptions thrown from server threads

This MP solves issues with incomplete server shutdown in case an exception is thrown from a server component thread. It achieves this by catching and storing exceptions that are still unhandled at thread boundaries, then terminating the server gracefully and finally rethrowing the unhandled exception from the mir::run_mir() level (because we still need to fail, e.g., for apport to catch the problem). This MP adds graceful termination for the input and compositor threads.

I am not completely happy with the indirect dependence of 3rd-party android Thread class on mir functionality, but I guess it will do for now.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:1584
http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/1433/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-trusty-i386-build/1751
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-trusty-amd64-build/1749
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-trusty-touch/1321
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/1165
        deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/1165/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/1170
        deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/1170/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/1324
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-trusty-armhf/1324/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/1219
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/6516

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1433/rebuild

review: Approve (continuous-integration)

[Robert Carr (robertcarr)]

LGTM

review: Approve

[Kevin DuBois (kdub)]

okay, although it does seem like a future maintainer/improver would have to figure out the terminate_with_current_exception linking. "I guess it will do for now." is how I felt when searching for a different way too, so +1.

review: Approve

[Alberto Aguirre (albaguirre)]

Perhaps we can explore a custom terminate_handler (std::set_terminate)? But that probably won't help much...

Otherwise...this....will...do for now.

review: Approve

[Daniel van Vugt (vanvugt)]

Unfortunately this is a regression on bug 1237332, and worse because core files are not longer produced at all.

Not regressing on bug 1237332 is more important than resolving bug 1189770 right now. The ability to get crash dumps from users is critical. Or else you can never fully see what's going wrong and never fix it.

review: Needs Fixing

[Alexandros Frantzis (afrantzis)]

> Unfortunately this is a regression on bug 1237332, and worse because core files are not longer produced at all.

We are rethrowing the exception from mir::run_mir() so if that's not caught we will abort() and get a core.

The way I see it, it makes sense to rethrow the exception from mir::run_mir(), since we can gracefully terminate instead, and we should allow our users to do the same. The current situation where we abort() in such cases is just a side effect of an implementation detail (i.e. that we use threads).

I agree that the core file produced (if produced) in such situations is not useful since we are capturing state after teardown. We could use "gcore" to capture the state in mir::terminate_with_exception() and somehow pass that information to interested parties. One way would be to read the command or file pattern from /proc/sys/kernel/core_pattern, and send the gcore output to that. Unfortunately, there are some problems there: we may need root privileges to write or pipe the gcore output to the core_pattern target and also we can't find some core_pattern parameters from userspace (like %P, although this only seem to matter when running in a container).

[Alexandros Frantzis (afrantzis)]

> Unfortunately, there are some problems there: we may need root privileges to write or pipe the
> gcore output to the core_pattern target

Talked to pitti, and at least for the apport case this is fixable.

[Daniel van Vugt (vanvugt)]

I think there are too many serious problems with rethrowing. You lose the original stack trace and therefore critically lose the memory contents and access to the variables leading to the exception. So you can't debug your cores and bugs don't get fixed. And too many users continue to indefinitely experience the same crashes as we experienced in the public testing of XMir.

Plus, all theory aside, when I manually tested this branch no core files came out of crashes. And I think it's naive to assume you can manually craft a core file (of yourself) as reliably and useful as the kernel will produce for you.

But the real solution could be very simple... If we carefully convert unrecoverable fatal exceptions into clean abort()'s with clean core and stack dumping (bug 1285084), then a change like this one could be fine. But we're not there yet.

[Alan Griffiths (alan-griffiths)]

At the heart of the above disagreement is the idea that there can be "unrecoverable fatal exceptions" - that is just bad design. (Even Java's questionable "Throwable" hierarchy makes unrecoverable "Error"s something different to "Exception"s.)

I think the requirement for an orderly shutdown and "correct" behavior of the executable beats that of having a core from the throw site. But it is close and I'd very much like to have both.

At yesterday's "standup" we discussed introduced a "core_and_throw" option to some throw sites - but I don't see that as a prerequisite for fixing lp:1189770.

[Alan Griffiths (alan-griffiths)]

134 +std::exception_ptr termination_exception;

AFAICS this can be accessed on multiple threads and therefore needs synchronization.

review: Needs Fixing

[Daniel van Vugt (vanvugt)]

Not being able to analyse crashes is an absolutely blocking issue because it prevents resolution of critical bugs. We must ensure we don't go backwards in this area.

"core and throw" is a naive idea. I've seen many attempts to do that on many platforms over the years but nothing ever beats letting the kernel produce clean, complete and accurate cores for you. Most attempts to dump cores of self or stack trace of self make accurate debugging more difficult. You should not do it.

This proposal could be OK only after fatal errors have been converted to abort/raise rather than exceptions.

review: Needs Fixing

[Alexandros Frantzis (afrantzis)]

> 134 +std::exception_ptr termination_exception;
>
> AFAICS this can be accessed on multiple threads and therefore needs
> synchronization.

Fixed, and also changed it so that only the first terminate_with_current_exception request is handled. The rationale is that we can get multiple requests for two reasons:

1. Code in two or more threads throwing at the same time (possibly because of the same underlying reason).
2. Code in one thread throwing, then during shutdown another thread throws.

In case (1) it doesn't matter which exception we catch (which comes first depends on timing anyway). In case (2) we want to catch the original exception more than we want to catch the shutdown exception.

Ideally we would want to record all, and that's possible by storing all exceptions and throwing an exception with all of the original exceptions nested, but that's not critical, so leaving it for the future, unless the team prefers to have it now.

[Alan Griffiths (alan-griffiths)]

141 +std::exception_ptr termination_exception;
142 +std::mutex termination_exception_mutex;

termination_exception is accessed in run_mir() without locking the mutex.

review: Needs Fixing

[Alexandros Frantzis (afrantzis)]

> termination_exception is accessed in run_mir() without locking the mutex.

I was thinking that our internal threads would have either not started or finished in the two points we access termination_exception in run_mir(). I guess an external user could be using mir::terminate_with_exception(), although I can't think of a valid use case for it. In any case, fixed.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:1585
http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/1499/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-android-utopic-i386-build/19/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-clang-utopic-amd64-build/20/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-utopic-touch/20/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-amd64-ci/16/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-armhf-ci/16/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-armhf/26/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1499/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:1587
http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/1502/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-utopic-i386-build/23
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-utopic-amd64-build/24
    ABORTED: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-utopic-touch/24/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-amd64-ci/21/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-armhf-ci/19/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-armhf/30/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1502/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:1587
http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-ci/1503/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-utopic-i386-build/25
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-utopic-amd64-build/26
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-utopic-touch/26
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-amd64-ci/22
        deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-amd64-ci/22/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-armhf-ci/20
        deb: http://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-utopic-armhf-ci/20/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-armhf/32
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-armhf/32/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/1296
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/6717

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1503/rebuild

review: Approve (continuous-integration)

[Alan Griffiths (alan-griffiths)]

LGTM

review: Approve

[Cemil Azizoglu (cemil-azizoglu)]

> I think the requirement for an orderly shutdown and "correct" behavior of the
> executable beats that of having a core from the throw site. But it is close
> and I'd very much like to have both.

+1 indeed!

IMHO for a user not being able to recover her display is several orders of magnitude worse than a frustrated development team having a hard time root-causing bugs due to lack of sane core dumps :-).

But let's not frustrate ourselves either. Even though programmer-assembled dumps may be difficult, we should put in that functionality. Soon after this one gets merged.

review: Approve

=== modified file '3rd_party/android-deps/std/Thread.h'
--- 3rd_party/android-deps/std/Thread.h	2013-05-06 22:22:39 +0000
+++ 3rd_party/android-deps/std/Thread.h	2014-05-02 15:53:48 +0000
@@ -32,6 +32,11 @@
 #include <pthread.h>
 #endif
 
+namespace mir
+{
+void terminate_with_current_exception();
+}
+
 namespace mir_input
 {
 class Thread : virtual public RefBase
@@ -56,8 +61,15 @@
 
         thread = std::thread([this]() -> void
             {
-                if (auto result = readyToRun()) status.store(result);
-                else while (!exitPending() && threadLoop());
+                try
+                {
+                    if (auto result = readyToRun()) status.store(result);
+                    else while (!exitPending() && threadLoop());
+                }
+                catch (...)
+                {
+                    mir::terminate_with_current_exception();
+                }
             });
 
 #ifdef HAVE_PTHREADS
 
=== modified file 'include/server/mir/run_mir.h'
--- include/server/mir/run_mir.h	2013-04-25 16:52:27 +0000
+++ include/server/mir/run_mir.h	2014-05-02 15:53:48 +0000
@@ -39,6 +39,8 @@
     std::function<void(DisplayServer&)> init);
 
 void report_exception(std::ostream& out);
+
+void terminate_with_current_exception();
 }
 
 
 
=== modified file 'include/test/mir_test/fake_event_hub.h'
--- include/test/mir_test/fake_event_hub.h	2013-10-01 17:29:49 +0000
+++ include/test/mir_test/fake_event_hub.h	2014-05-02 15:53:48 +0000
@@ -142,6 +142,8 @@
     FakeDevice* getDevice(int32_t deviceId);
     size_t eventsQueueSize() const;
 
+    void throw_exception_in_next_get_events();
+
     // list of RawEvents available for consumption via getEvents
     std::mutex guard;
     std::list<droidinput::RawEvent> events_available;
@@ -155,6 +157,7 @@
 
 private:
     const KeyInfo* getKey(const FakeDevice* device, int32_t scanCode, int32_t usageCode) const;
+    bool throw_in_get_events = false;
 
 };
 }
 
=== modified file 'src/server/compositor/multi_threaded_compositor.cpp'
--- src/server/compositor/multi_threaded_compositor.cpp	2014-04-30 04:46:52 +0000
+++ src/server/compositor/multi_threaded_compositor.cpp	2014-05-02 15:53:48 +0000
@@ -26,6 +26,7 @@
 #include "mir/scene/legacy_scene_change_notification.h"
 #include "mir/scene/surface_observer.h"
 #include "mir/scene/surface.h"
+#include "mir/run_mir.h"
 
 #include <thread>
 #include <condition_variable>
@@ -161,6 +162,7 @@
     }
 
     void operator()() noexcept // noexcept is important! (LP: #1237332)
+    try
     {
         /*
          * Make the buffer the current rendering target, and release
@@ -180,6 +182,10 @@
 
         run_compositing_loop([&] { return display_buffer_compositor->composite();});
     }
+    catch (...)
+    {
+        mir::terminate_with_current_exception();
+    }
 
 private:
     std::shared_ptr<mc::DisplayBufferCompositorFactory> const display_buffer_compositor_factory;
@@ -196,6 +202,7 @@
     }
 
     void operator()() noexcept // noexcept is important! (LP: #1237332)
+    try
     {
         run_compositing_loop(
             [this]
@@ -215,6 +222,10 @@
                 return false;
             });
     }
+    catch (...)
+    {
+        mir::terminate_with_current_exception();
+    }
 
 private:
     void wait_until_next_fake_vsync()
 
=== modified file 'src/server/run_mir.cpp'
--- src/server/run_mir.cpp	2013-10-21 08:17:25 +0000
+++ src/server/run_mir.cpp	2014-05-02 15:53:48 +0000
@@ -23,6 +23,8 @@
 #include "mir/frontend/connector.h"
 #include "mir/raii.h"
 
+#include <exception>
+#include <mutex>
 #include <csignal>
 #include <cstdlib>
 #include <cassert>
@@ -32,6 +34,8 @@
 auto const intercepted = { SIGQUIT, SIGABRT, SIGFPE, SIGSEGV, SIGBUS };
 
 std::weak_ptr<mir::frontend::Connector> weak_connector;
+std::exception_ptr termination_exception;
+std::mutex termination_exception_mutex;
 
 extern "C" void delete_endpoint()
 {
@@ -58,6 +62,10 @@
 void mir::run_mir(ServerConfiguration& config, std::function<void(DisplayServer&)> init)
 {
     DisplayServer* server_ptr{nullptr};
+    {
+        std::lock_guard<std::mutex> lock{termination_exception_mutex};
+        termination_exception = nullptr;
+    }
     auto main_loop = config.the_main_loop();
 
     main_loop->register_signal_handler(
@@ -88,4 +96,18 @@
 
     init(server);
     server.run();
+
+    std::lock_guard<std::mutex> lock{termination_exception_mutex};
+    if (termination_exception)
+        std::rethrow_exception(termination_exception);
+}
+
+void mir::terminate_with_current_exception()
+{
+    std::lock_guard<std::mutex> lock{termination_exception_mutex};
+    if (!termination_exception)
+    {
+        termination_exception = std::current_exception();
+        raise(SIGTERM);
+    }
 }
 
=== modified file 'tests/acceptance-tests/test_server_shutdown.cpp'
--- tests/acceptance-tests/test_server_shutdown.cpp	2014-03-06 06:05:17 +0000
+++ tests/acceptance-tests/test_server_shutdown.cpp	2014-05-02 15:53:48 +0000
@@ -19,10 +19,14 @@
 #include "mir_toolkit/mir_client_library.h"
 #include "mir/compositor/renderer.h"
 #include "mir/compositor/renderer_factory.h"
+#include "mir/compositor/display_buffer_compositor.h"
+#include "mir/compositor/display_buffer_compositor_factory.h"
 #include "mir/input/composite_event_filter.h"
+#include "mir/run_mir.h"
 
 #include "mir_test_framework/display_server_test_fixture.h"
 #include "mir_test/fake_event_hub_input_configuration.h"
+#include "mir_test/fake_event_hub.h"
 #include "mir_test_framework/cross_process_sync.h"
 #include "mir_test_doubles/stub_renderer.h"
 
@@ -55,6 +59,25 @@
     }
 };
 
+class ExceptionThrowingDisplayBufferCompositorFactory : public mc::DisplayBufferCompositorFactory
+{
+public:
+    std::unique_ptr<mc::DisplayBufferCompositor>
+        create_compositor_for(mg::DisplayBuffer&) override
+    {
+        struct ExceptionThrowingDisplayBufferCompositor : mc::DisplayBufferCompositor
+        {
+            bool composite() override
+            {
+                throw std::runtime_error("ExceptionThrowingDisplayBufferCompositor");
+            }
+        };
+
+        return std::unique_ptr<mc::DisplayBufferCompositor>(
+            new ExceptionThrowingDisplayBufferCompositor{});
+    }
+};
+
 void null_surface_callback(MirSurface*, void*)
 {
 }
@@ -94,6 +117,46 @@
     std::string const flag_file;
 };
 
+struct FakeEventHubServerConfig : TestingServerConfiguration
+{
+    std::shared_ptr<mi::InputConfiguration> the_input_configuration() override
+    {
+        if (!input_configuration)
+        {
+            input_configuration =
+                std::make_shared<mtd::FakeEventHubInputConfiguration>(
+                    the_composite_event_filter(),
+                    the_input_region(),
+                    std::shared_ptr<mi::CursorListener>(),
+                    the_input_report());
+        }
+
+        return input_configuration;
+    }
+
+    std::shared_ptr<mi::InputManager> the_input_manager() override
+    {
+        return DefaultServerConfiguration::the_input_manager();
+    }
+
+    std::shared_ptr<mir::shell::InputTargeter> the_input_targeter() override
+    {
+        return DefaultServerConfiguration::the_input_targeter();
+    }
+    std::shared_ptr<mir::scene::InputRegistrar> the_input_registrar() override
+    {
+        return DefaultServerConfiguration::the_input_registrar();
+    }
+
+    mia::FakeEventHub* the_fake_event_hub()
+    {
+        the_input_configuration();
+        return input_configuration->the_fake_event_hub();
+    }
+
+    std::shared_ptr<mtd::FakeEventHubInputConfiguration> input_configuration;
+};
+
 }
 
 using ServerShutdown = BespokeDisplayServerTestFixture;
@@ -194,42 +257,7 @@
     Flag resources_freed_success{"resources_free_success_7e9c69fc.tmp"};
     Flag resources_freed_failure{"resources_free_failure_7e9c69fc.tmp"};
 
-    /* Use the real input manager, but with a fake event hub */
-    struct ServerConfig : TestingServerConfiguration
-    {
-        std::shared_ptr<mi::InputConfiguration> the_input_configuration() override
-        {
-            if (!input_configuration)
-            {
-                input_configuration =
-                    std::make_shared<mtd::FakeEventHubInputConfiguration>(
-                        the_composite_event_filter(),
-                        the_input_region(),
-                        std::shared_ptr<mi::CursorListener>(),
-                        the_input_report());
-            }
-
-            return input_configuration;
-        }
-
-        std::shared_ptr<mi::InputManager> the_input_manager() override
-        {
-            return DefaultServerConfiguration::the_input_manager();
-        }
-
-        std::shared_ptr<mir::shell::InputTargeter> the_input_targeter() override
-        {
-            return DefaultServerConfiguration::the_input_targeter();
-        }
-        std::shared_ptr<mir::scene::InputRegistrar> the_input_registrar() override
-        {
-            return DefaultServerConfiguration::the_input_registrar();
-        }
-
-        std::shared_ptr<mi::InputConfiguration> input_configuration;
-    };
-
-    auto server_config = std::make_shared<ServerConfig>();
+    auto server_config = std::make_shared<FakeEventHubServerConfig>();
     launch_server_process(*server_config);
 
     struct ClientConfig : TestingClientConfiguration
@@ -427,3 +455,74 @@
 INSTANTIATE_TEST_CASE_P(ServerShutdown,
     OnSignal,
     ::testing::Values(SIGQUIT, SIGABRT, SIGFPE, SIGSEGV, SIGBUS));
+
+TEST(ServerShutdownWithThreadException,
+     server_releases_resources_on_abnormal_input_thread_termination)
+{
+    auto server_config = std::make_shared<FakeEventHubServerConfig>();
+    auto fake_event_hub = server_config->the_fake_event_hub();
+
+    std::thread server{
+        [&]
+        {
+            EXPECT_THROW(
+                mir::run_mir(*server_config, [](mir::DisplayServer&){}),
+                std::runtime_error);
+        }};
+
+    fake_event_hub->throw_exception_in_next_get_events();
+    server.join();
+
+    std::weak_ptr<mir::graphics::Display> display = server_config->the_display();
+    std::weak_ptr<mir::compositor::Compositor> compositor = server_config->the_compositor();
+    std::weak_ptr<mir::frontend::Connector> connector = server_config->the_connector();
+    std::weak_ptr<mir::input::InputManager> input_manager = server_config->the_input_manager();
+
+    server_config.reset();
+
+    EXPECT_EQ(0, display.use_count());
+    EXPECT_EQ(0, compositor.use_count());
+    EXPECT_EQ(0, connector.use_count());
+    EXPECT_EQ(0, input_manager.use_count());
+}
+
+TEST(ServerShutdownWithThreadException,
+     server_releases_resources_on_abnormal_compositor_thread_termination)
+{
+    struct ServerConfig : TestingServerConfiguration
+    {
+        std::shared_ptr<mc::DisplayBufferCompositorFactory>
+            the_display_buffer_compositor_factory() override
+        {
+            return display_buffer_compositor_factory(
+                [this]()
+                {
+                    return std::make_shared<ExceptionThrowingDisplayBufferCompositorFactory>();
+                });
+        }
+    };
+
+    auto server_config = std::make_shared<ServerConfig>();
+
+    std::thread server{
+        [&]
+        {
+            EXPECT_THROW(
+                mir::run_mir(*server_config, [](mir::DisplayServer&){}),
+                std::runtime_error);
+        }};
+
+    server.join();
+
+    std::weak_ptr<mir::graphics::Display> display = server_config->the_display();
+    std::weak_ptr<mir::compositor::Compositor> compositor = server_config->the_compositor();
+    std::weak_ptr<mir::frontend::Connector> connector = server_config->the_connector();
+    std::weak_ptr<mir::input::InputManager> input_manager = server_config->the_input_manager();
+
+    server_config.reset();
+
+    EXPECT_EQ(0, display.use_count());
+    EXPECT_EQ(0, compositor.use_count());
+    EXPECT_EQ(0, connector.use_count());
+    EXPECT_EQ(0, input_manager.use_count());
+}
 
=== modified file 'tests/mir_test_doubles/fake_event_hub.cpp'
--- tests/mir_test_doubles/fake_event_hub.cpp	2013-10-02 11:04:00 +0000
+++ tests/mir_test_doubles/fake_event_hub.cpp	2014-05-02 15:53:48 +0000
@@ -221,6 +221,10 @@
     (void) timeoutMillis;
     {
         std::lock_guard<std::mutex> lg(guard);
+
+        if (throw_in_get_events)
+            throw std::runtime_error("FakeEventHub::getEvents() exception");
+
         for (size_t i = 0; i < bufferSize && events_available.size() > 0; ++i)
         {
             buffer[i] = events_available.front();
@@ -688,3 +692,9 @@
 {
     return events_available.size();
 }
+
+void FakeEventHub::throw_exception_in_next_get_events()
+{
+    std::lock_guard<std::mutex> lg(guard);
+    throw_in_get_events = true;
+}