Merge lp:~vanvugt/mir/generalise-performance-tests into lp:mir

[Daniel van Vugt]

Generalise performance tests:
  * Rename mir_performance_tests to mir_glmark2_performance_test to
    show it is just a client performance test; and
  * Add a replacement script 'mir_performance_tests' that executes
    the old test but also will launch other performance tests in
    future.

[Mir CI Bot (mir-ci-bot)]

FAILED: Continuous integration, rev:3473
https://mir-jenkins.ubuntu.com/job/mir-ci/886/
Executed test runs:
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-mir/918/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/957
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/948
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/948
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/928/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/928
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/928/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/928
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/928/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/928
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/928/artifact/output/*zip*/output.zip
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/928/console

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/886/rebuild

review: Needs Fixing (continuous-integration)

[Mir CI Bot (mir-ci-bot)]

FAILED: Continuous integration, rev:3474
https://mir-jenkins.ubuntu.com/job/mir-ci/888/
Executed test runs:
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-mir/920/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/959
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/950
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/950
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/930
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/930/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/930
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/930/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/930
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/930/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/930
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/930/artifact/output/*zip*/output.zip
    FAILURE: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/930/console

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/888/rebuild

review: Needs Fixing (continuous-integration)

[Mir CI Bot (mir-ci-bot)]

PASSED: Continuous integration, rev:3475
https://mir-jenkins.ubuntu.com/job/mir-ci/889/
Executed test runs:
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-mir/921
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/960
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/951
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/951
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/931
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=clang,platform=mesa,release=vivid+overlay/931/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/931
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=amd64,compiler=gcc,platform=mesa,release=xenial/931/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/931
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=cross-armhf,compiler=gcc,platform=android,release=vivid+overlay/931/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/931
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=android,release=vivid+overlay/931/artifact/output/*zip*/output.zip
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/931
        deb: https://mir-jenkins.ubuntu.com/job/build-2-binpkg-mir/arch=i386,compiler=gcc,platform=mesa,release=xenial/931/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/889/rebuild

review: Approve (continuous-integration)

[Cemil Azizoglu (cemil-azizoglu)]

lgtm

review: Approve

[Alan Griffiths (alan-griffiths)]

*Needs Discussion*

I would have expected additional performance tests to be added to mir_performance_tests. I.e. not placed in separate executables. Googletest provides a better infrastructure for collating and/or selecting tests than this script.

Do we really want to do something different?

review: Needs Information

[Kevin DuBois (kdub)]

IMO, its a bit cleaner and more cohesive to roll things into gtest, but I don't mind a shell script either. I suppose abstain, as I'd rather it still be called mir_performance_test (we don't need to say .sh)

review: Abstain

[Daniel van Vugt (vanvugt)]

gtest is not appropriate for a couple of reasons:

  1. It's not a convenient launchpad for non-C/C++ external performance tests.
  2. It doesn't work at all for the Android platform: bug 1546912

[Alan Griffiths (alan-griffiths)]

By my reading, bug 1546912 only limits starting the server in-process. That is inconvenient, but hardly "gtest doesn't work at all".

I still question whether we really need performance tests written in c++, python and bash.

review: Needs Information

=== modified file 'debian/mir-test-tools.install'
--- debian/mir-test-tools.install	2016-03-29 17:02:43 +0000
+++ debian/mir-test-tools.install	2016-04-22 05:11:19 +0000
@@ -5,6 +5,7 @@
 usr/bin/mir_umock_acceptance_tests
 usr/bin/mir_integration_tests*
 usr/bin/mir_performance_tests
+usr/bin/mir_glmark2_performance_test
 usr/bin/mir_privileged_tests
 usr/bin/mir_test_reload_protobuf
 usr/bin/mir_test_client_*

=== modified file 'tests/performance-tests/CMakeLists.txt'
--- tests/performance-tests/CMakeLists.txt	2015-06-17 05:20:42 +0000
+++ tests/performance-tests/CMakeLists.txt	2016-04-22 05:11:19 +0000
@@ -1,25 +1,18 @@
-set(
-  PERFORMANCE_TESTS_SOURCES
+add_custom_target(mir_performance_tests ALL
+  cp ${CMAKE_CURRENT_SOURCE_DIR}/performance_tests.sh ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/mir_performance_tests
+)
+install(PROGRAMS ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/mir_performance_tests
+  DESTINATION ${CMAKE_INSTALL_BINDIR}
+)
 
+mir_add_wrapped_executable(mir_glmark2_performance_test
   test_glmark2-es2-mir.cpp
 )
-
-mir_add_wrapped_executable(
-  mir_performance_tests
-
-  ${PERFORMANCE_TESTS_SOURCES}
-)
-
-add_dependencies(mir_performance_tests GMock)
-
-target_link_libraries(
-  mir_performance_tests
-
+add_dependencies(mir_glmark2_performance_test GMock)
+target_link_libraries(mir_glmark2_performance_test
   mir-test-static
   mir-test-framework-static
   mir-test-doubles-static
   mirserver
-
   ${GTEST_BOTH_LIBRARIES}
 )
-

=== added file 'tests/performance-tests/performance_tests.sh'
--- tests/performance-tests/performance_tests.sh	1970-01-01 00:00:00 +0000
+++ tests/performance-tests/performance_tests.sh	2016-04-22 05:11:19 +0000
@@ -0,0 +1,42 @@
+#!/bin/sh
+#
+# Copyright © 2016 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Author: Daniel van Vugt <[e-mail]>
+#
+
+### This is a wrapper script that runs all Mir's performance tests ###
+
+bin_dir=`dirname $0`
+
+title_banner()
+{
+    echo "======================================================================="
+    echo "|  $*"
+    echo "======================================================================="
+}
+
+tests_run=0
+failures=0
+for perf_test in mir_glmark2_performance_test
+do
+    title_banner "Mir Performance Tests now running: $perf_test"
+    tests_run=$(($tests_run + 1))
+    $bin_dir/$perf_test || failures=$(($failures + 1))
+done
+
+passes=$(($tests_run - $failures))
+title_banner "Mir Performance Tests ran $tests_run tests: $passes passed, $failures failed."
+return $failures