Merge lp:~vanvugt/mir/fix-1509291 into lp:mir

[Daniel van Vugt]

Make ClientLatency tests less prone to spurious failures (LP: #1509291)
  * Widened the error margin; and
  * Replaced sleeps with precise waits.

It even works under valgrind for me now (see LP: #1522031)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:3186
http://jenkins.qa.ubuntu.com/job/mir-ci/5807/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5255
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4161
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5204
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-xenial-touch/130/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/135
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/135/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/135
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/135/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5204
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5204/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7729
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/25922
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-xenial-armhf/128/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/5807/rebuild

review: Approve (continuous-integration)

[Alberto Aguirre (albaguirre)]

LGTM.

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/mir-autolanding/2261/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5272
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4178/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5221
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-autolanding/95
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-autolanding/95/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-autolanding/95
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-autolanding/95/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5221
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5221/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7743
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/25971

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:3187
http://jenkins.qa.ubuntu.com/job/mir-ci/5817/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5275
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4181
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5224
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-xenial-touch/136/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/145
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/145/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/145
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/145/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5224
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5224/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7746
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/25974
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-xenial-armhf/134/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/5817/rebuild

review: Approve (continuous-integration)

[Cemil Azizoglu (cemil-azizoglu)]

Good

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/mir-autolanding/2263/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5282
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4188
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5231/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-autolanding/97
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-autolanding/97/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-autolanding/97
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-autolanding/97/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5231
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5231/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7753/console
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/25990

review: Needs Fixing (continuous-integration)

[Alberto Aguirre (albaguirre)]

^--CI network hiccup. Let's try again.

[PS Jenkins bot (ps-jenkins)]

review: Approve (continuous-integration)

=== modified file 'tests/acceptance-tests/test_latency.cpp'
--- tests/acceptance-tests/test_latency.cpp	2015-10-20 03:30:00 +0000
+++ tests/acceptance-tests/test_latency.cpp	2015-12-11 07:19:47 +0000
@@ -30,6 +30,9 @@
 #include <gmock/gmock.h>
 #include <deque>
 
+#include <mutex>
+#include <condition_variable>
+
 namespace mtf = mir_test_framework;
 namespace mtd = mir::test::doubles;
 namespace mt = mir::test;
@@ -47,6 +50,7 @@
     {
         std::lock_guard<std::mutex> lock{mutex};
         post_count++;
+        posted.notify_one();
     }
 
     void record_submission(uint32_t submission_id)
@@ -74,8 +78,21 @@
         return latency;
     }
 
+    bool wait_for_posts(unsigned int count, std::chrono::milliseconds timeout)
+    {
+        std::unique_lock<std::mutex> lock(mutex);
+        auto const deadline = std::chrono::system_clock::now() + timeout;
+        while (post_count < count)
+        {
+            if (posted.wait_until(lock, deadline) == std::cv_status::timeout)
+                return false;
+        }
+        return true;
+    }
+
 private:
     std::mutex mutex;
+    std::condition_variable posted;
     unsigned int post_count{0};
 
     // Note that a buffer_id may appear twice in the list as the client is
@@ -190,6 +207,12 @@
     Stats stats;
     TimeTrackingDisplay display{stats};
     unsigned int test_submissions{100};
+
+    // We still have a margin for error here. The client and server will
+    // be scheduled somewhat unpredictably which affects results. Also
+    // affecting results will be the first few frames before the buffer
+    // quere is full (during which there will be no buffer latency).
+    float const error_margin = 0.4f;
 };
 }
 
@@ -204,9 +227,8 @@
         mir_buffer_stream_swap_buffers_sync(stream);
     }
 
-    // Wait for the compositor to finish rendering all those frames,
-    // or else we'll be missing some samples and get a spurious average.
-    std::this_thread::sleep_for(std::chrono::milliseconds(500));
+    ASSERT_TRUE(stats.wait_for_posts(test_submissions,
+                                     std::chrono::seconds(60)));
 
     unsigned int const expected_client_buffers = 3;
 
@@ -219,12 +241,6 @@
 
     auto observed_latency = display.group.average_latency();
 
-    // We still have a margin for error here. The client and server will
-    // be scheduled somewhat unpredictably which affects results. Also
-    // affecting results will be the first few frames before the buffer
-    // quere is full (during which there will be no buffer latency).
-    float const error_margin = 0.1f;
-
     EXPECT_THAT(observed_latency, Gt(expected_min_latency-error_margin));
     EXPECT_THAT(observed_latency, Lt(expected_max_latency+error_margin));
 }
@@ -248,13 +264,12 @@
         mir_buffer_stream_swap_buffers_sync(stream);
     }
 
-    // Wait for the compositor to finish rendering all those frames,
-    // or else we'll be missing some samples and get a spurious average.
-    std::this_thread::sleep_for(std::chrono::milliseconds(500));
+    ASSERT_TRUE(stats.wait_for_posts(test_submissions,
+                                     std::chrono::seconds(60)));
 
     // As the client is producing frames slower than the compositor consumes
     // them, the buffer queue never fills. So latency is low;
     float const observed_latency = display.group.average_latency();
     EXPECT_THAT(observed_latency, Ge(0.0f));
-    EXPECT_THAT(observed_latency, Lt(1.1f));
+    EXPECT_THAT(observed_latency, Le(1.0f + error_margin));
 }