Merge lp:~vanvugt/mir/compositor-test-in-c-minor into lp:mir

[Daniel van Vugt]

Add a compositor performance test, which is actually a regression test
for LP: #1563287.

This is the same test as before:
  https://code.launchpad.net/~vanvugt/mir/compositor-test/+merge/292608
but rewritten from shell script into C/Gtest.

And now you can see the compositor test running in CI. Just not easily...
 1. Click on the job that has 'arch=cross-armhf'
 2. Click 'Console Output'
 3. Scroll to bottom and click 'device-runtests-mir ...'
 4. Click 'krillin'
 5. Click 'Console Output'
 6. Click 'Full Log'
 7. Search in page 'Mir Performance Tests'

[Mir CI Bot (mir-ci-bot)]

PASSED: Continuous integration, rev:3507
https://mir-jenkins.ubuntu.com/job/mir-ci/952/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/1019
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1065
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1056
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1056
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1029
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1029/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1029
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1029/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1029
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1029/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1029
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1029/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1029
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1029/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/952/rebuild

review: Approve (continuous-integration)

[Mir CI Bot (mir-ci-bot)]

FAILED: Continuous integration, rev:3509
https://mir-jenkins.ubuntu.com/job/mir-ci/956/
Executed test runs:
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-mir/1023/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1069
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1060
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1060
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1033
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1033/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1033
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1033/artifact/output/*zip*/output.zip
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1033/console
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1033/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1033
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1033/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1033
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1033/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/956/rebuild

review: Needs Fixing (continuous-integration)

[Daniel van Vugt (vanvugt)]

Harmless phone failure. Try again...

03:34:13 Slave went offline during the build
03:34:13 ERROR: Connection was broken: java.io.EOFException

[Mir CI Bot (mir-ci-bot)]

PASSED: Continuous integration, rev:3510
https://mir-jenkins.ubuntu.com/job/mir-ci/958/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/1027
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1073
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1064
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1064
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1037
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1037/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1037
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1037/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1037
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1037/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1037
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1037/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1037
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1037/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/958/rebuild

review: Approve (continuous-integration)

[Kevin DuBois (kdub)]

Would prefer this to the bash.

+ EXPECT_GE(compositor_fps, 58.0f);
iirc, we've had 55fps devices before. (and krillin is 66fps). Would be better to check the display configuration

+ bin_dir+"/mir_demo_server --compositor-report=log -f "+mir_sock;
Would prefer to override the compositor report, so then we would rely on our API instead of log parsing. We could also probably use our existing test infrastructure for starting servers? (although, we probably need some hygiene by now to get the acceptance tests to run against the real drivers easily.)

So I guess (given the difficulties of using our test infrastructure + server API to check this), alright by me, after making the compositor fps check a bit more robust.

review: Approve

[Daniel van Vugt (vanvugt)]

55Hz: I don't recall 55Hz devices. We can lower the threshold if required. When the bug occurs on krillin through you see a drop from 66FPS to 9-20FPS.

Overriding the compositor report is inappropriately cumbersome, but also is not possible for any tests that might have more than one test due to Android bug 1546912. This is why all-but-one of the glmark2 tests are disabled on Android right now too.

[Mir CI Bot (mir-ci-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://mir-jenkins.ubuntu.com/job/mir-autolanding/257/
Executed test runs:
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-mir/1036/console
    None: https://mir-jenkins.ubuntu.com/job/generic-land-mp/277/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/1083
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/1074
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/1074
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/1046/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1046
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/1046/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1046
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/1046/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1046
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/1046/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1046
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/1046/artifact/output/*zip*/output.zip

review: Needs Fixing (continuous-integration)

[Daniel van Vugt (vanvugt)]

Just bug 1576690. Try again.

[Mir CI Bot (mir-ci-bot)]

review: Approve (continuous-integration)

=== modified file 'debian/mir-test-tools.install'
--- debian/mir-test-tools.install	2016-05-03 04:36:33 +0000
+++ debian/mir-test-tools.install	2016-05-09 08:40:31 +0000
@@ -6,6 +6,7 @@
 usr/bin/mir_integration_tests*
 usr/bin/mir_performance_tests
 usr/bin/mir_glmark2_performance_test
+usr/bin/mir_compositor_performance_test
 usr/bin/mir_privileged_tests
 usr/bin/mir_test_reload_protobuf
 usr/bin/mir_test_client_*

=== modified file 'tests/performance-tests/CMakeLists.txt'
--- tests/performance-tests/CMakeLists.txt	2016-04-27 09:21:20 +0000
+++ tests/performance-tests/CMakeLists.txt	2016-05-09 08:40:31 +0000
@@ -16,3 +16,10 @@
   mirserver
   ${GTEST_BOTH_LIBRARIES}
 )
+
+mir_add_wrapped_executable(mir_compositor_performance_test
+  test_compositor.cpp
+)
+target_link_libraries(mir_compositor_performance_test
+  ${GTEST_BOTH_LIBRARIES}
+)

=== modified file 'tests/performance-tests/performance_tests.sh'
--- tests/performance-tests/performance_tests.sh	2016-04-22 05:09:40 +0000
+++ tests/performance-tests/performance_tests.sh	2016-05-09 08:40:31 +0000
@@ -30,7 +30,8 @@
 
 tests_run=0
 failures=0
-for perf_test in mir_glmark2_performance_test
+for perf_test in mir_glmark2_performance_test \
+                 mir_compositor_performance_test
 do
     title_banner "Mir Performance Tests now running: $perf_test"
     tests_run=$(($tests_run + 1))

=== added file 'tests/performance-tests/test_compositor.cpp'
--- tests/performance-tests/test_compositor.cpp	1970-01-01 00:00:00 +0000
+++ tests/performance-tests/test_compositor.cpp	2016-05-09 08:40:31 +0000
@@ -0,0 +1,222 @@
+/*
+ * Copyright © 2016 Canonical Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3,
+ * as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Daniel van Vugt <[e-mail]>
+ */
+
+#include <gtest/gtest.h>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <unistd.h>
+#include <string>
+#include <thread>
+#include <initializer_list>
+
+using namespace std::literals::chrono_literals;
+
+namespace
+{
+
+std::string mir_bin_dir()
+{
+    char path[256];
+    auto len = readlink("/proc/self/exe", path, sizeof(path)-1);
+    if (len < 0)
+        len = 0;
+    path[len] = '\0';
+    if (auto slash = strrchr(path, '/'))
+        *slash = '\0';
+    return path;
+}
+
+void kill_nicely(pid_t pid)
+{
+    if (kill(pid, SIGTERM) < 0)
+        return;
+    int const timeout = 5;
+    int status, count = 0;
+    while (0 == waitpid(pid, &status, WNOHANG) && count < timeout)
+    {
+        sleep(1);
+        ++count;
+    }
+    kill(pid, SIGKILL);
+}
+
+int exec_cmd(char const* cmd)
+{
+    auto buf = strdup(cmd);
+    size_t argc = 1;
+    char* argv[256] = {buf};
+    char *c = buf;
+    while (*c)
+    {
+        if (*c == ' ')
+        {
+            *c = '\0';
+            if (argc < (sizeof(argv)/sizeof(argv[0]) - 1))
+            {
+                argv[argc] = c + 1;
+                ++argc;
+            }
+        }
+        ++c;
+    }
+    argv[argc] = NULL;
+    return execv(argv[0], argv);
+}
+
+FILE* popen_with_pid(char const* cmd, pid_t& pid)
+{
+    int pipefd[2];
+    if (pipe(pipefd))
+        return NULL;
+
+    int const& pipe_out = pipefd[0];
+    int const& pipe_in = pipefd[1];
+
+    pid = fork();
+    if (pid < 0)
+    {
+        close(pipe_in);
+        close(pipe_out);
+        return NULL;
+    }
+    else if (pid == 0)
+    {
+        close(pipe_out);
+        dup2(pipe_in, 1);  // Child stdout goes into pipe_in
+        close(pipe_in);
+        exec_cmd(cmd);
+        exit(errno);
+    }
+    else
+    {
+        close(pipe_in);
+    }
+
+    return fdopen(pipe_out, "r");
+}
+
+bool spawn_and_forget(char const* cmd)
+{
+    int pid = fork();
+    if (pid == 0)
+    {
+        // Silence stdout/stderr
+        close(1);
+        close(2);
+        exec_cmd(cmd);
+        exit(errno);
+    }
+    return (pid > 0);
+}
+
+bool spawn_and_forget(std::string const& cmd)
+{
+    return spawn_and_forget(cmd.c_str());
+}
+
+bool wait_for_file(char const* path, std::chrono::seconds timeout)
+{
+    struct stat s;
+    int count = 0, max = timeout.count();
+    int ret;
+    while ((ret = stat(path, &s)) < 0 && errno == ENOENT && count < max)
+    {
+        ++count;
+        std::this_thread::sleep_for(1s);
+    }
+    return ret == 0;
+}
+
+struct CompositorPerformance : testing::Test
+{
+    void SetUp() override
+    {
+        compositor_fps = compositor_render_time = -1.0f;
+
+        auto const mir_sock = "/tmp/mir_test_socket_"+std::to_string(getpid());
+        auto const server_cmd =
+            bin_dir+"/mir_demo_server --compositor-report=log -f "+mir_sock;
+    
+        server_output = popen_with_pid(server_cmd.c_str(), server_pid);
+        ASSERT_TRUE(server_output) << server_cmd;
+        ASSERT_TRUE(wait_for_file(mir_sock.c_str(), 5s)) << server_cmd;
+        setenv("MIR_SOCKET", mir_sock.c_str(), 1);
+    }
+
+    void TearDown() override
+    {
+        kill_nicely(server_pid);
+        fclose(server_output);
+    }
+
+    void spawn_clients(std::initializer_list<std::string> clients)
+    {
+        for (auto& client : clients)
+        {
+            spawn_and_forget(bin_dir+"/"+client);
+            std::this_thread::sleep_for(100ms);
+        }
+    }
+
+    void wait_for_server(std::chrono::seconds timeout)
+    {
+        pid_t pid = server_pid;
+        std::thread killer([timeout,pid]()
+        {
+            std::this_thread::sleep_for(timeout);
+            kill_nicely(pid);
+        });
+        killer.detach();
+
+        char line[256];
+        while (fgets(line, sizeof(line), server_output))
+        {
+            if (char const* perf = strstr(line, "averaged "))
+            {
+                float fps, render_time;
+                if (2 == sscanf(perf, "averaged %f FPS, %f ms/frame",
+                                &fps, &render_time))
+                {
+                    compositor_fps = fps;
+                    compositor_render_time = render_time;
+                }
+            }
+        }
+    }
+
+    std::string const bin_dir{mir_bin_dir()};
+    pid_t server_pid = 0;
+    FILE* server_output;
+    float compositor_fps, compositor_render_time;
+};
+
+} // anonymous namespace
+
+TEST_F(CompositorPerformance, regression_test_1563287)
+{
+    spawn_clients({"mir_demo_client_flicker",
+                   "mir_demo_client_egltriangle -b0.5 -f",
+                   "mir_demo_client_progressbar",
+                   "mir_demo_client_scroll",
+                   "mir_demo_client_egltriangle -b0.5",
+                   "mir_demo_client_multiwin"});
+    wait_for_server(10s);
+    EXPECT_GE(compositor_fps, 58.0f);
+    EXPECT_LT(compositor_render_time, 17.0f);
+}