Mir

Merge lp:~mir-team/mir/more-performance-tests into lp:mir

more-performance-tests
Merge into development-branch

Proposed by Daniel van Vugt on 2016-02-12

Status:

Merged

Approved by:

Cemil Azizoglu on 2016-02-19

Approved revision:

no longer in the source branch.

Merged at revision:

3325

Proposed branch:

lp:~mir-team/mir/more-performance-tests

Merge into:

lp:mir

Diff against target:

135 lines (+94/-10)

1 file modified

tests/performance-tests/test_glmark2-es2-mir.cpp (+94/-10)

To merge this branch:

bzr merge lp:~mir-team/mir/more-performance-tests

Related bugs:

Bug #1369763: [performance] Android overlays don't honour eglSwapInterval 0	Medium	Won't Fix
Bug #1546912: The Android graphics platform causes a Mir server to crash if it is stopped and started again in the same process	Medium	Won't Fix

Link a bug report

Reviewer	Review Type	Date Requested	Status
Cemil Azizoglu (community)			Approve on 2016-02-19
Alexandros Frantzis (community)			Approve on 2016-02-19
PS Jenkins bot (community)	continuous-integration		Approve on 2016-02-18
Mir CI Bot	continuous-integration		Approve on 2016-02-18
Alan Griffiths		2016-02-12	Needs Information on 2016-02-16
Review via email: mp+285837@code.launchpad.net

Commit message

Add a bunch more automated performance tests covering more use cases.

I was recently pleased to find that Jenkins had detected a performance
regression in my code. However then I was worried when I realized that
Jenkins should not have failed at all, and it was just by luck of Android
bug 1369763 that CI failed.

So the failure was useful to me, but also should not have happened. Time
to expand the tests to remove the element of luck, so that such performance regressions always cause a failure...

Revision history for this message

Mir CI Bot (mir-ci-bot) wrote on 2016-02-12:

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/291/rebuild

review: Approve (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2016-02-12:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/6277/rebuild

review: Approve (continuous-integration)

Revision history for this message

Cemil Azizoglu (cemil-azizoglu) wrote on 2016-02-13:

Was confused about why this was MPed. It makes sense after seeing 'description'.

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2016-02-16:

+ * FIXME: These all fail because the test server doesn't start more than once:
+ * "std::exception::what: Exception while creating graphics platform
+ * Exiting Mir! Reason: Nested Mir and Host Mir cannot use the same
+ * socket file to accept connections!"

What is this really?

1. We have loads of test cases that start up and close down a server in the acceptance tests. So a server can be run multiple times.

2. That error is about configuring --host and --file to the same (not about starting a server multiple times).

review: Needs Information

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2016-02-16:

> + * FIXME: These all fail because the test server doesn't start more than
> once:
> + * "std::exception::what: Exception while creating graphics platform
> + * Exiting Mir! Reason: Nested Mir and Host Mir cannot use the same
> + * socket file to accept connections!"
>
> What is this really?

Oh, that's lp:1290345

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2016-02-16:

> > + * FIXME: These all fail because the test server doesn't start more than
> > once:
> > + * "std::exception::what: Exception while creating graphics platform
> > + * Exiting Mir! Reason: Nested Mir and Host Mir cannot use the same
> > + * socket file to accept connections!"
> >
> > What is this really?
>
> Oh, that's lp:1290345

Right, I'm MP'd a fix for lp:1290345 - you only need part1 for the problem seen here to be resolved.

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2016-02-16:

Is the name of the output file significant anywhere? IIRC we were collating results somewhere.

review: Needs Information

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2016-02-17:

Good question. Although if and when we get these multiple results we might want to collate the multiple results from multiple files.

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2016-02-17:

FAILED: Continuous integration, rev:3316
http://jenkins.qa.ubuntu.com/job/mir-ci/6309/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5947
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4854
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5903/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/633
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/633/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/633
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/633/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5900
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5900/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/8288/console
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/27689

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/6309/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Mir CI Bot (mir-ci-bot) wrote on 2016-02-17:

FAILED: Continuous integration, rev:3316
https://mir-jenkins.ubuntu.com/job/mir-ci/321/
Executed test runs:
    ABORTED: https://mir-jenkins.ubuntu.com/job/build/110/console
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-0-fetch/116
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=vivid+overlay/112
    SUCCESS: https://mir-jenkins.ubuntu.com/job/build-1-sourcepkg/release=xenial/112

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/321/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2016-02-17:

FAILED: Continuous integration, rev:3316
http://jenkins.qa.ubuntu.com/job/mir-ci/6310/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/5949
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/4856
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/5905/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/634
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-amd64-ci/634/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/634
        deb: http://jenkins.qa.ubuntu.com/job/mir-xenial-i386-ci/634/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5902
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/5902/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/8290/console
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/27692

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/6310/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2016-02-17:

armhf has issues. No idea what the problem is yet but desktop and valgrind on desktop has no problem. Only armhf:

Error in `mir_performance_tests': free(): invalid pointer: 0xb5f02b08 ***

Revision history for this message

Alexandros Frantzis (afrantzis) wrote on 2016-02-17:

The mir glmark2 performance tests are taking too long by default (5min * N, N is currently 6).

Suggestions:

1. Reduce the amount of time spent in each benchmark scene. This can be achieved with a "-b :duration=2" option (apply duration of 2 seconds to all benchmark scenes)

2. Cherrypick only a few benchmark scenes to run by default. This is again controlled with "-b" command line arguments.

3. Both of the above

review: Needs Fixing

Revision history for this message

Cemil Azizoglu (cemil-azizoglu) wrote on 2016-02-17:

> The mir glmark2 performance tests are taking too long by default (5min * N, N
> is currently 6).
>
> Suggestions:
>
> 1. Reduce the amount of time spent in each benchmark scene. This can be
> achieved with a "-b :duration=2" option (apply duration of 2 seconds to all
> benchmark scenes)
>
> 2. Cherrypick only a few benchmark scenes to run by default. This is again
> controlled with "-b" command line arguments.
>
> 3. Both of the above

review: Needs Fixing

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2016-02-18:

Already started on that path :)

34 + char const* selection = getenv("MIR_GLMARK2_TEST_QUICK") ?
35 + "-b build " : "";

Not sure where we will end up as the default behaviour though.

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2016-02-18:

OK, the tests are now fast and don't crash.

Sadly most of them can't be run on Android due to bugs in our Android driver.

Revision history for this message

Mir CI Bot (mir-ci-bot) wrote on 2016-02-18:

Click here to trigger a rebuild:
https://mir-jenkins.ubuntu.com/job/mir-ci/336/rebuild

review: Approve (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2016-02-18:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/6325/rebuild

review: Approve (continuous-integration)

Revision history for this message

Alexandros Frantzis (afrantzis) wrote on 2016-02-19:

OK as a first step.

review: Approve

Revision history for this message

Cemil Azizoglu (cemil-azizoglu) wrote on 2016-02-19:

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Emanuele Antonio Faraone

Gerry Boland

Mir development team

 === modified file 'tests/performance-tests/test_glmark2-es2-mir.cpp'
 --- tests/performance-tests/test_glmark2-es2-mir.cpp	2016-01-29 08:18:22 +0000
 +++ tests/performance-tests/test_glmark2-es2-mir.cpp	2016-02-18 08:39:23 +0000
@@ -1,3 +1,19 @@
++/*
++ * Copyright © 2014-2016 Canonical Ltd.
++ *
++ * This program is free software: you can redistribute it and/or modify it
++ * under the terms of the GNU General Public License version 3,
++ * as published by the Free Software Foundation.
++ *
++ * This program is distributed in the hope that it will be useful,
++ * but WITHOUT ANY WARRANTY; without even the implied warranty of
++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++ * GNU General Public License for more details.
++ *
++ * You should have received a copy of the GNU General Public License
++ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
++ */
++
  #include "mir_test_framework/async_server_runner.h"
  #include "mir/test/popen.h"
@@ -27,11 +43,26 @@
+     }
      enum ResultFileType {raw, json};
--    virtual void run_glmark2(char const* output_filename, ResultFileType file_type)
++    virtual int run_glmark2(char const* args)
+     {
--        auto const cmd = "MIR_SOCKET=" + new_connection() + " glmark2-es2-mir --fullscreen";
++        ResultFileType file_type = raw; // Should this still be selectable?
++
++        auto const cmd = "MIR_SOCKET=" + new_connection()
++                       + " glmark2-es2-mir -b build "
++                       + args;
          mir::test::Popen p(cmd);
--
++
++        const ::testing::TestInfo* const test_info =
++          ::testing::UnitTest::GetInstance()->current_test_info();
++
++        char output_filename[256];
++        snprintf(output_filename, sizeof(output_filename) - 1,
++                 "/tmp/%s_%s.log",
++                 test_info->test_case_name(), test_info->name());
++
++        printf("Saving GLMark2 detailed results to: %s\n", output_filename);
++        // ^ Although I would vote to just print them to stdout instead
++
          std::string line;
          std::ofstream glmark2_output;
          int score = -1;
@@ -50,9 +81,6 @@
+             }
+         }
--        auto const minimum_acceptable_score = 52;
--        EXPECT_THAT(score, ::testing::Ge(minimum_acceptable_score));
--
          if (file_type == json)
+         {
              std::string json =  "{";
@@ -62,11 +90,67 @@
                  json += "}";
              glmark2_output << json;
+         }
++
++        return score;
+     }
  };
--TEST_F(GLMark2Test, benchmark_fullscreen_default)
--{
--    run_glmark2("/tmp/glmark2_fullscreen_default.results", raw);
--}
++TEST_F(GLMark2Test, fullscreen_default)
++{
++    EXPECT_THAT(run_glmark2("--fullscreen"), ::testing::Ge(56));
++}
++
++#ifdef ANDROID
++TEST_F(GLMark2Test, DISABLED_windowed_default) // Android bug LP: #1546912
++#else
++TEST_F(GLMark2Test, windowed_default)
++#endif
++{
++    EXPECT_THAT(run_glmark2(""), ::testing::Ge(56));
++}
++
++#ifdef ANDROID
++TEST_F(GLMark2Test, DISABLED_fullscreen_interval1) // Android bug LP: #1546912
++#else
++TEST_F(GLMark2Test, fullscreen_interval1)
++#endif
++{
++    add_to_environment("MIR_CLIENT_FORCE_SWAP_INTERVAL", "1");
++    // Our devices seem to range 57-67Hz
++    EXPECT_NEAR(60, run_glmark2("--fullscreen"), 10);
++}
++
++#ifdef ANDROID
++TEST_F(GLMark2Test, DISABLED_windowed_interval1) // Android bug LP: #1546912
++#else
++TEST_F(GLMark2Test, windowed_interval1)
++#endif
++{
++    add_to_environment("MIR_CLIENT_FORCE_SWAP_INTERVAL", "1");
++    // Our devices seem to range 57-67Hz
++    EXPECT_NEAR(60, run_glmark2(""), 10);
++}
++
++#ifdef ANDROID
++TEST_F(GLMark2Test, DISABLED_fullscreen_interval0) // Android bug LP: #1369763
++                                                   // and LP: #1546912
++#else
++TEST_F(GLMark2Test, fullscreen_interval0)
++#endif
++{
++    add_to_environment("MIR_CLIENT_FORCE_SWAP_INTERVAL", "0");
++    EXPECT_THAT(run_glmark2("--fullscreen"), ::testing::Ge(100));
++}
++
++#ifdef ANDROID
++TEST_F(GLMark2Test, DISABLED_windowed_interval0) // Android bug LP: #1369763
++                                                 // and LP: #1546912
++#else
++TEST_F(GLMark2Test, windowed_interval0)
++#endif
++{
++    add_to_environment("MIR_CLIENT_FORCE_SWAP_INTERVAL", "0");
++    EXPECT_THAT(run_glmark2(""), ::testing::Ge(100));
++}
++
+ }

Mir

Merge lp:~mir-team/mir/more-performance-tests into lp:mir

Commit message

Description of the change

Preview Diff

Subscribers