Merge lp:~afrantzis/mir/fix-1576760-increase-timeout into lp:mir

[Alexandros Frantzis]

tests: Increase timeouts to accommodate valgrind slowness

tests: Increase timeouts to accommodate valgrind slowness

Use the existing 'timeout' variable which was set in the code, but not used until now.

[Mir CI Bot (mir-ci-bot)]

PASSED: Continuous integration, rev:3567
https://mir-jenkins.ubuntu.com/job/mir-ci/1216/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/1397
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1448
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1439
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1439
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1411
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1411/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1411
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1411/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1411
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1411/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1411
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1411/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1411
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1411/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/1216/rebuild

review: Approve (continuous-integration)

[Cemil Azizoglu (cemil-azizoglu)]

I suspect these timeouts are overly conservative. If these tests start failing they may take a really long time to run since the timeout value is set to 10 at the moment. We should probably change the timeout to something like 2 or 3, and the (two) instances that require more than that can be hard-coded to have 5 or 10.

review: Needs Fixing

[Alexandros Frantzis (afrantzis)]

> I suspect these timeouts are overly conservative

valgrind under high load (which we can't control since our CI infrastructure is shared) can be *extremely* slow.

> If these tests start failing they may take a really long time to run since the timeout value
> is set to 10 at the moment

Can't say I am really concerned with this, since it's an extremely rare event and will add in the absolute worst case 5 minutes of total time.

> We should probably change the timeout to something like 2 or 3, and the (two) instances
> that require more than that can be hard-coded to have 5 or 10.

The reason the two instances have 5 and 10 respectively (which are arbitrary values), is because they failed at some point in the past due to valgrind slowness and high CI load, so we had to increase their timeouts.

Now NestedServer.when_monitor_plugged_in_client_is_notified_of_new_display_configuration happened to fail, so we need to increase its timeout too. In the future another test will fail for the same reason.

I tried setting the timeout to 3 seconds, but I got some failures locally under valgrind + sufficient load in various NestedServer.* tests. I couldn't get failures locally with 5 seconds, so I think it's a good start, although only time will tell if it will be enough for CI.

[Mir CI Bot (mir-ci-bot)]

PASSED: Continuous integration, rev:3568
https://mir-jenkins.ubuntu.com/job/mir-ci/1220/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/1403
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1454
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1445
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1445
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1417
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1417/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1417
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1417/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1417
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1417/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1417
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1417/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1417
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1417/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/1220/rebuild

review: Approve (continuous-integration)

[Cemil Azizoglu (cemil-azizoglu)]

ok

review: Approve

=== modified file 'tests/acceptance-tests/test_nested_mir.cpp'
--- tests/acceptance-tests/test_nested_mir.cpp	2016-06-02 08:19:38 +0000
+++ tests/acceptance-tests/test_nested_mir.cpp	2016-07-01 10:10:31 +0000
@@ -122,7 +122,7 @@
     {{640, 0}, {1920, 1080}}
 };
 
-std::chrono::seconds const timeout{10};
+std::chrono::seconds const timeout{5};
 
 // We can't rely on the test environment to have X cursors, so we provide some of our own
 auto const cursor_names = {
@@ -405,7 +405,7 @@
 
         mir_wait_for(mir_connection_apply_display_config(connection, configuration));
 
-        initial_condition.wait_for(std::chrono::seconds{1});
+        initial_condition.wait_for(timeout);
         mir_display_config_destroy(configuration);
 
         Mock::VerifyAndClearExpectations(&display);
@@ -598,7 +598,7 @@
     trigger_lifecycle_event(mir_lifecycle_state_resumed);
     trigger_lifecycle_event(mir_lifecycle_state_will_suspend);
 
-    events_processed.wait_for(std::chrono::seconds{5});
+    events_processed.wait_for(timeout);
 }
 
 TEST_F(NestedServer, client_may_connect_to_nested_server_and_create_surface)
@@ -613,7 +613,7 @@
             return mir_surface_get_visibility(surface) == mir_surface_visibility_exposed
                 && mir_surface_get_focus(surface) == mir_surface_focused;
         },
-        std::chrono::seconds{10});
+        timeout);
 
     EXPECT_TRUE(became_exposed_and_focused);
 }
@@ -644,7 +644,7 @@
 
         mir_buffer_stream_swap_buffers_sync(mir_surface_get_buffer_stream(surface));
 
-        wait.wait_for(std::chrono::seconds{1});
+        wait.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_session_mediator_report.get());
     }
 
@@ -659,7 +659,7 @@
         mir_surface_release_sync(surface);
         mir_connection_release(connection);
 
-        wait.wait_for(std::chrono::seconds{1});
+        wait.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_session_mediator_report.get());
     }
 
@@ -685,7 +685,7 @@
     client.update_display_configuration(
         [](MirDisplayConfiguration* config) { config->outputs->used = false; });
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
 
@@ -714,7 +714,7 @@
 
         mir_wait_for(mir_connection_apply_display_config(client.connection, configuration));
 
-        config_reported.wait_for(std::chrono::seconds{3});
+        config_reported.wait_for(timeout);
         Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
     }
 
@@ -744,7 +744,7 @@
         mir_wait_for(mir_surface_configure_cursor(client.surface, conf));
         mir_cursor_configuration_destroy(conf);
 
-        condition.wait_for(std::chrono::seconds{1});
+        condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_cursor.get());
     }
 
@@ -756,7 +756,7 @@
 
         mir_buffer_stream_swap_buffers_sync(client.buffer_stream);
 
-        condition.wait_for(std::chrono::seconds{1});
+        condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_cursor.get());
     }
 }
@@ -786,7 +786,7 @@
         mir_wait_for(mir_surface_configure_cursor(client.surface, cursor));
         mir_cursor_configuration_destroy(cursor);
 
-        condition.wait_for(std::chrono::seconds{1});
+        condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_cursor.get());
     }
 }
@@ -814,7 +814,7 @@
         mir_wait_for(mir_surface_configure_cursor(client.surface, conf));
         mir_cursor_configuration_destroy(conf);
 
-        condition.wait_for(std::chrono::seconds{1});
+        condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(mock_cursor.get());
     }
 
@@ -867,7 +867,7 @@
 
     ClientWithAPaintedSurface client(nested_mir);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 
     EXPECT_TRUE(condition.raised());
@@ -889,7 +889,7 @@
 
     server.the_display_configuration_controller()->set_base_configuration(new_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     Mock::VerifyAndClearExpectations(config_policy.get());
     EXPECT_TRUE(condition.raised());
 }
@@ -914,7 +914,7 @@
                 [](MirDisplayConfiguration* config) { config->outputs->used = false; });
 
             // Wait for initial config to be applied
-            initial_condition.wait_for(std::chrono::seconds{1});
+            initial_condition.wait_for(timeout);
 
             Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
             ASSERT_TRUE(initial_condition.raised());
@@ -924,7 +924,7 @@
             .WillRepeatedly(InvokeWithoutArgs([&] { condition.raise(); }));
     }
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
@@ -952,7 +952,7 @@
         nested_mir.server.the_display_configuration_controller()->set_base_configuration(new_config);
 
         // Wait for initial config to be applied
-        initial_condition.wait_for(std::chrono::seconds{1});
+        initial_condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
         ASSERT_TRUE(initial_condition.raised());
 
@@ -960,7 +960,7 @@
             .WillRepeatedly(InvokeWithoutArgs([&] { condition.raise(); }));
     }
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
 }
 
@@ -979,7 +979,7 @@
 
     display.emit_configuration_change_event(new_config);
 
-    client_config_changed.wait_for(std::chrono::seconds{1});
+    client_config_changed.wait_for(timeout);
 
     ASSERT_TRUE(client_config_changed.raised());
 
@@ -1008,7 +1008,7 @@
         nested_mir.server.the_display_configuration_controller()->set_base_configuration(initial_config);
 
         // Wait for initial config to be applied
-        initial_condition.wait_for(std::chrono::seconds{1});
+        initial_condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
         ASSERT_TRUE(initial_condition.raised());
     }
@@ -1024,7 +1024,7 @@
 
     display.emit_configuration_change_event(expect_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
@@ -1054,7 +1054,7 @@
 
     display.emit_configuration_change_event(expect_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
@@ -1074,7 +1074,7 @@
 
     display.emit_configuration_change_event(new_config);
 
-    client_config_changed.wait_for(std::chrono::seconds{1});
+    client_config_changed.wait_for(timeout);
 
     ASSERT_TRUE(client_config_changed.raised());
 
@@ -1109,7 +1109,7 @@
         nested_mir.server.the_display_configuration_controller()->set_base_configuration(initial_config);
 
         // Wait for initial config to be applied
-        initial_condition.wait_for(std::chrono::seconds{1});
+        initial_condition.wait_for(timeout);
         Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
         ASSERT_TRUE(initial_condition.raised());
     }
@@ -1131,7 +1131,7 @@
 
     display.emit_configuration_change_event(new_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
@@ -1167,7 +1167,7 @@
 
     display.emit_configuration_change_event(new_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
     EXPECT_TRUE(condition.raised());
     Mock::VerifyAndClearExpectations(the_mock_display_configuration_report().get());
 }
@@ -1192,7 +1192,7 @@
 
     display.emit_configuration_change_event(new_config);
 
-    condition.wait_for(std::chrono::seconds{1});
+    condition.wait_for(timeout);
 
     EXPECT_TRUE(condition.raised());
 
@@ -1235,14 +1235,14 @@
 
     display.emit_configuration_change_event(new_hw_config);
 
-    client_config_changed.wait_for(std::chrono::seconds{1});
+    client_config_changed.wait_for(timeout);
     if (client_config_changed.raised())
     {
         client.update_display_configuration(
             [](MirDisplayConfiguration* config) { config->outputs->orientation = mir_orientation_inverted; });
     }
 
-    host_config_change.wait_for(std::chrono::seconds{1});
+    host_config_change.wait_for(timeout);
 
     EXPECT_TRUE(host_config_change.raised());
     Mock::VerifyAndClearExpectations(&display);