Mir

Merge lp:~josharenson/mir/install_glmark2 into lp:mir

install_glmark2
Merge into development-branch

Proposed by Josh Arenson on 2014-04-30

Status:	Superseded
Proposed branch:	lp:~josharenson/mir/install_glmark2
Merge into:	lp:mir
Diff against target:	335 lines (+230/-4) 10 files modified CMakeLists.txt (+1/-1) debian/control (+1/-0) debian/mir-test-tools.install (+1/-0) include/test/mir_test/popen.h (+59/-0) tests/CMakeLists.txt (+5/-0) tests/mir_test/CMakeLists.txt (+2/-1) tests/mir_test/popen.cpp (+65/-0) tests/mir_test_framework/in_process_server.cpp (+6/-2) tests/performance-tests/CMakeLists.txt (+28/-0) tests/performance-tests/test_glmark2-es2-mir.cpp (+62/-0)
To merge this branch:	bzr merge lp:~josharenson/mir/install_glmark2
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Alan Griffiths		2014-04-30	Needs Fixing on 2014-05-01
PS Jenkins bot (community)	continuous-integration	2014-04-30	Needs Fixing on 2014-05-01
Robert Carr		2014-04-30	Pending
Alberto Aguirre		2014-04-30	Pending
Daniel van Vugt		2014-04-30	Pending
Alexandros Frantzis		2014-04-30	Pending
Review via email: mp+217849@code.launchpad.net

This proposal supersedes a proposal from 2014-04-18.

This proposal has been superseded by a proposal from 2014-05-05.

Commit message

* Adds "performance tests" as a test category.
* Implements one test that runs glmark2, saving results to a local file.

Revision history for this message

Robert Carr (robertcarr) wrote on 2014-04-21: Posted in a previous version of this proposal

We don't need to link mir_performance_test against so much right?

review: Needs Fixing

Revision history for this message

Josh Arenson (josharenson) wrote on 2014-04-21: Posted in a previous version of this proposal

> We don't need to link mir_performance_test against so much right?
Absolutely.

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2014-04-22: Posted in a previous version of this proposal

Where should we source glmark2-es2-mir?

$ apt-cache search glmark2-es2-mir
$

review: Needs Information

Revision history for this message

Josh Arenson (josharenson) wrote on 2014-04-22: Posted in a previous version of this proposal

> Where should we source glmark2-es2-mir?
>
> $ apt-cache search glmark2-es2-mir
> $
glmark2-es2-mir (the version that actually runs) is currently located in ppa:mir-team/staging. I have also successfully built it from source. Currently, the Jenkins job adds the ppa, and installs the package as part of the environment set up.

Revision history for this message

Robert Carr (robertcarr) wrote on 2014-04-22: Posted in a previous version of this proposal

Why doesnt jenkins even say hello to this branch?

review: Needs Information

Revision history for this message

Josh Arenson (josharenson) wrote on 2014-04-22: Posted in a previous version of this proposal

> Why doesnt jenkins even say hello to this branch?
According to fginther, there is an outage.

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-04-23: Posted in a previous version of this proposal

FAILED: Continuous integration, rev:1572
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~josharenson/mir/install_glmark2/+merge/216504/+edit-commit-message

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1388/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Alexandros Frantzis (afrantzis) wrote on 2014-04-24: Posted in a previous version of this proposal

46 +find_program(GLMARK2_ES2_MIR_FOUND glmark2-es2-mir)
47 +
48 +if(NOT GLMARK2_ES2_MIR_FOUND)
49 + message("warning: glmark2-es2-mir not found, performance tests _may_ fail.")
50 +endif()

Since we are not running the program automatically when building, I am not sure how useful it is to print this message at build time.

77 +#include <stdlib.h>

We also need <stdio.h> for popen, getline etc (we shouldn't depend on other headers bringing it in).

Some style issues (see http://unity.ubuntu.com/mir/cppguide/index.html):

85 +class GLMark2Test : public ::testing::Test
86 +{
...

Mir uses 4 space indentations.

88 + enum ResultFileType {RAW, JSON};

Lowercase identifiers for enum values.

103 + while((read = getline(&line, &len, in)) != -1)
105 + if(file_type == RAW)
107 + else if(file_type == JSON)
109 + if(boost::regex_match(line, matches, re_glmark2_score))

Spaces between keyword (if,while) and opening parenthesis.

105 + if(file_type == RAW)
106 + glmark2_output << line;

Braces around statement, since the 'else' clause also has braces.

94 + const char *cmd
93 + FILE *in;
...

'*' goes with type in our C++ code and const goes after the type. So: "FILE* in" and "char const* cmd"

98 + std::ofstream glmark2_output;

No need to define all variables in the beginning of the code block if it is more convenient (or necessary for RAII purposes) not to do so. So, for example it is allowed to have "std::ofstream glmark2_output{output_filename};" after the ASSERT.

123 + glmark2_output.close();

No need to manually close the file, it's closed on std::ofstream destruction.

review: Needs Fixing

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2014-04-28: Posted in a previous version of this proposal

I have no opinion, but Jenkins won't ever land this till you add a commit message up the top.

review: Needs Fixing

Revision history for this message

Josh Arenson (josharenson) wrote on 2014-04-28: Posted in a previous version of this proposal

> I have no opinion, but Jenkins won't ever land this till you add a commit
> message up the top.
Thanks for the heads up. I'm working on a refactor and will add the message when done.

Revision history for this message

Alberto Aguirre (albaguirre) wrote on 2014-04-29: Posted in a previous version of this proposal

> > I have no opinion, but Jenkins won't ever land this till you add a commit
> > message up the top.
> Thanks for the heads up. I'm working on a refactor and will add the message
> when done.

You should mark this as "work in progress" so that others don't try to review code that is about to change.

Revision history for this message

Daniel van Vugt (vanvugt) on 2014-04-29: Posted in a previous version of this proposal

review: Abstain

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-04-30: Posted in a previous version of this proposal

FAILED: Continuous integration, rev:1577
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~josharenson/mir/install_glmark2/+merge/216504/+edit-commit-message

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1439/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Alexandros Frantzis (afrantzis) wrote on 2014-04-30: Posted in a previous version of this proposal

119 + /*HACK to wait for the server to start*/

You can override the mir::DefaultServerConfiguration::the_server_status_listener() and provide you own implementation to wait for server start.

/*HACK to stop the mir server*/

You can get the main loop object from mir::DefaultServerConfiguration::the_main_loop() and call the stop() method.

review: Needs Fixing

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2014-04-30: Posted in a previous version of this proposal

> 119 + /*HACK to wait for the server to start*/
>
> You can override the
> mir::DefaultServerConfiguration::the_server_status_listener() and provide you
> own implementation to wait for server start.
>
> /*HACK to stop the mir server*/
>
> You can get the main loop object from
> mir::DefaultServerConfiguration::the_main_loop() and call the stop() method.

Even simpler: just use the pre-existing InProcessServer test fixture to start the server before entering the test body and close it afterwards.

review: Needs Fixing

Revision history for this message

Alexandros Frantzis (afrantzis) wrote on 2014-04-30: Posted in a previous version of this proposal

> > 119 + /*HACK to wait for the server to start*/
> >
> > You can override the
> > mir::DefaultServerConfiguration::the_server_status_listener() and provide
> you
> > own implementation to wait for server start.
> >
> > /*HACK to stop the mir server*/
> >
> > You can get the main loop object from
> > mir::DefaultServerConfiguration::the_main_loop() and call the stop() method.
>
> Even simpler: just use the pre-existing InProcessServer test fixture to start
> the server before entering the test body and close it afterwards.

Even better, +1

Revision history for this message

Alberto Aguirre (albaguirre) wrote on 2014-04-30: Posted in a previous version of this proposal

~~~
111 + FILE* in;
112 + char* line = NULL;
113 + size_t len = 0;
114 + ssize_t read;

123 + ASSERT_TRUE((in = popen(cmd, "r")));
126 + while ((read = getline(&line, &len, in)) != -1)
151 + free(line);
152 + fclose(in);
~~~

I'd like to see that in a separate class and put under mir_test as it could be useful if we need to run other
benchmark programs OR use pstreams library which already does that for you.

~~~
virtual void RunGLMark2(char const* output_filename, ResultFileType file_type)
~~~
RunGLMark2 not in the style described at http://unity.ubuntu.com/mir/cppguide/index.html#Function_Names

~~~
106 + enum ResultFileType {raw, json};
~~~

Use enum class.

review: Needs Fixing

Revision history for this message

Alberto Aguirre (albaguirre) wrote on 2014-04-30: Posted in a previous version of this proposal

Maybe this will be useful:
https://code.launchpad.net/~mir-team/mir/popen-cpp-wrapper/+merge/217831

Popen p{"glmark2-es2-mir --fullscreen"};
std::string line;
while(p.get_line(line))
{
<do something with line>
}

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-05-01: Posted in a previous version of this proposal

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1457/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-05-01:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-ci/1458/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2014-05-01:

My earlier comment hasn't been addressed:

Even simpler: just use the pre-existing InProcessServer test fixture to start the server before entering the test body and close it afterwards.

review: Needs Fixing

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Emanuele Antonio Faraone

Gerry Boland

Josh Arenson

Mir development team

 === modified file 'CMakeLists.txt'
 --- CMakeLists.txt	2014-05-01 08:44:30 +0000
 +++ CMakeLists.txt	2014-05-05 21:25:21 +0000
@@ -76,7 +76,7 @@
  include_directories(include/shared)
  # Check for boost
--find_package(Boost 1.48.0 COMPONENTS chrono date_time filesystem system thread program_options regex REQUIRED)
++find_package(Boost 1.48.0 COMPONENTS chrono date_time filesystem system thread program_options regex iostreams REQUIRED)
  include_directories (
    ${Boost_INCLUDE_DIRS}
+ )
 === modified file 'debian/control'
 --- debian/control	2014-05-01 08:49:17 +0000
 +++ debian/control	2014-05-05 21:25:21 +0000
@@ -18,6 +18,7 @@
                 libboost-thread-dev,
                 libboost-regex-dev,
                 libboost-filesystem-dev,
++               libboost-iostreams-dev,
                 protobuf-compiler,
                 libdrm-dev,
                 libegl1-mesa-dev,
 === modified file 'debian/mir-test-tools.install'
 --- debian/mir-test-tools.install	2014-03-06 06:05:17 +0000
 +++ debian/mir-test-tools.install	2014-05-05 21:25:21 +0000
@@ -2,5 +2,6 @@
  usr/bin/mir_unit_tests
  usr/bin/mir_acceptance_tests
  usr/bin/mir_integration_tests
++usr/bin/mir_performance_tests
  usr/lib/*/mir/tools/libmirclientlttng.so
  usr/lib/*/mir/tools/libmirserverlttng.so
 === added file 'include/test/mir_test/popen.h'
 --- include/test/mir_test/popen.h	1970-01-01 00:00:00 +0000
 +++ include/test/mir_test/popen.h	2014-05-05 21:25:21 +0000
@@ -0,0 +1,59 @@
++/*
++ * Copyright © 2014 Canonical Ltd.
++ *
++ * This program is free software: you can redistribute it and/or modify it
++ * under the terms of the GNU General Public License version 3,
++ * as published by the Free Software Foundation.
++ *
++ * This program is distributed in the hope that it will be useful,
++ * but WITHOUT ANY WARRANTY; without even the implied warranty of
++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++ * GNU General Public License for more details.
++ *
++ * You should have received a copy of the GNU General Public License
++ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
++ *
++ * Authored by: Alberto Aguirre <alberto.aguirre@canonical.com>
++ */
++
++
++#ifndef MIR_TEST_POPEN_H_
++#define MIR_TEST_POPEN_H_
++
++#include <cstdio>
++#include <memory>
++#include <string>
++#include <streambuf>
++#include <istream>
++
++namespace mir
++{
++namespace test
++{
++/**
++ *  Popen - A popen c++ wrapper
++ */
++class Popen
++{
++public:
++    Popen(std::string const& cmd);
++
++    /**
++     * Read a line from the output of the executed command
++     * returns false if there is nothing more to read
++     */
++    bool get_line(std::string& line);
++
++private:
++    Popen() = delete;
++    Popen(Popen const&) = delete;
++    Popen& operator=(Popen const&) = delete;
++    std::unique_ptr<std::FILE, void(*)(FILE* f)> raw_stream;
++    std::unique_ptr<std::streambuf> stream_buffer;
++    std::istream stream;
++};
++
++}
++}
++
++#endif
 === modified file 'tests/CMakeLists.txt'
 --- tests/CMakeLists.txt	2014-03-26 05:48:59 +0000
 +++ tests/CMakeLists.txt	2014-05-05 21:25:21 +0000
@@ -26,6 +26,7 @@
  option(MIR_BUILD_ACCEPTANCE_TESTS "Build acceptance tests" ON)
  option(MIR_BUILD_INTEGRATION_TESTS "Build integration tests" ON)
++option(MIR_BUILD_PERFORMANCE_TESTS "Build performance tests" ON)
  option(MIR_BUILD_UNIT_TESTS "Build unit tests" ON)
  if (MIR_BUILD_ACCEPTANCE_TESTS)
@@ -36,6 +37,10 @@
    add_subdirectory(integration-tests/)
  endif (MIR_BUILD_INTEGRATION_TESTS)
++if (MIR_BUILD_PERFORMANCE_TESTS)
++    add_subdirectory(performance-tests/)
++endif(MIR_BUILD_PERFORMANCE_TESTS)
++
  if (MIR_BUILD_UNIT_TESTS)
    add_subdirectory(unit-tests/)
  endif (MIR_BUILD_UNIT_TESTS)
 === modified file 'tests/mir_test/CMakeLists.txt'
 --- tests/mir_test/CMakeLists.txt	2014-04-15 05:31:19 +0000
 +++ tests/mir_test/CMakeLists.txt	2014-05-05 21:25:21 +0000
@@ -5,4 +5,5 @@
    display_config_matchers.cpp
    cross_process_action.cpp
    spin_wait.cpp
--)
++  popen.cpp
++)
 \ No newline at end of file
 === added file 'tests/mir_test/popen.cpp'
 --- tests/mir_test/popen.cpp	1970-01-01 00:00:00 +0000
 +++ tests/mir_test/popen.cpp	2014-05-05 21:25:21 +0000
@@ -0,0 +1,65 @@
++/*
++ * Copyright © 2014 Canonical Ltd.
++ *
++ * This program is free software: you can redistribute it and/or modify it
++ * under the terms of the GNU General Public License version 3,
++ * as published by the Free Software Foundation.
++ *
++ * This program is distributed in the hope that it will be useful,
++ * but WITHOUT ANY WARRANTY; without even the implied warranty of
++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++ * GNU General Public License for more details.
++ *
++ * You should have received a copy of the GNU General Public License
++ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
++ *
++ * Authored by: Alberto Aguirre <alberto.aguirre@canonical.com>
++ */
++
++#include <mir_test/popen.h>
++
++#include <boost/throw_exception.hpp>
++#include <boost/exception/errinfo_errno.hpp>
++#include <boost/iostreams/device/file_descriptor.hpp>
++#include <boost/iostreams/stream_buffer.hpp>
++
++#include <stdexcept>
++
++namespace mt = mir::test;
++namespace io = boost::iostreams;
++
++namespace
++{
++std::unique_ptr<std::streambuf> create_stream_buffer(FILE* raw_stream)
++{
++    if (raw_stream == nullptr)
++    {
++        BOOST_THROW_EXCEPTION(
++            boost::enable_error_info(std::runtime_error("popen failed"))
++                << boost::errinfo_errno(errno));
++    }
++
++    int fd = fileno(raw_stream);
++    if (fd == -1)
++    {
++        BOOST_THROW_EXCEPTION(
++            boost::enable_error_info(std::runtime_error("invalid file stream"))
++                << boost::errinfo_errno(errno));
++    }
++    auto raw = new io::stream_buffer<io::file_descriptor_source>{
++        fd, io::never_close_handle};
++    return std::unique_ptr<std::streambuf>(raw);
++}
++}
++
++mt::Popen::Popen(std::string const& cmd)
++    : raw_stream{popen(cmd.c_str(), "r"), [](FILE* f){ pclose(f); }},
++      stream_buffer{create_stream_buffer(raw_stream.get())},
++      stream{stream_buffer.get()}
++{
++}
++
++bool mt::Popen::get_line(std::string& line)
++{
++    return std::getline(stream, line);
++}
 === modified file 'tests/mir_test_framework/in_process_server.cpp'
 --- tests/mir_test_framework/in_process_server.cpp	2014-03-06 06:05:17 +0000
 +++ tests/mir_test_framework/in_process_server.cpp	2014-05-05 21:25:21 +0000
@@ -25,6 +25,7 @@
  #include <gtest/gtest.h>
++#include <boost/algorithm/string.hpp>
  #include <condition_variable>
  #include <mutex>
@@ -37,12 +38,15 @@
  mtf::InProcessServer::InProcessServer() :
      old_env(getenv(env_no_file))
--{
--    if (!old_env) setenv(env_no_file, "", true);
++{
+ }
  void mtf::InProcessServer::SetUp()
+ {
++    if (!old_env || boost::iequals(old_env, "false"))
++    {
++        unsetenv(env_no_file);
++    }
      display_server = start_mir_server();
      ASSERT_TRUE(display_server);
+ }
 === added directory 'tests/performance-tests'
 === added file 'tests/performance-tests/CMakeLists.txt'
 --- tests/performance-tests/CMakeLists.txt	1970-01-01 00:00:00 +0000
 +++ tests/performance-tests/CMakeLists.txt	2014-05-05 21:25:21 +0000
@@ -0,0 +1,28 @@
++set(
++  PERFORMANCE_TESTS_SOURCES
++
++  test_glmark2-es2-mir.cpp
++)
++
++add_executable(
++  mir_performance_tests
++
++  ${PERFORMANCE_TESTS_SOURCES}
++)
++
++target_link_libraries(
++  mir_performance_tests
++
++  mir-test
++  mir-test-framework
++
++  ${Boost_LIBRARIES}
++  ${GTEST_BOTH_LIBRARIES}
++  ${GMOCK_LIBRARY}
++  ${GMOCK_MAIN_LIBRARY}
++)
++
++install(
++  TARGETS mir_performance_tests
++  RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
++)
 === added file 'tests/performance-tests/test_glmark2-es2-mir.cpp'
 --- tests/performance-tests/test_glmark2-es2-mir.cpp	1970-01-01 00:00:00 +0000
 +++ tests/performance-tests/test_glmark2-es2-mir.cpp	2014-05-05 21:25:21 +0000
@@ -0,0 +1,62 @@
++#include "mir/default_server_configuration.h"
++#include "mir_test/popen.h"
++#include "mir_test_framework/in_process_server.h"
++
++#include <gtest/gtest.h>
++
++#include <stdlib.h>
++#include <stdio.h>
++
++#include <boost/regex.hpp>
++#include <iostream>
++#include <fstream>
++#include <string>
++
++class GLMark2Test : public mir_test_framework::InProcessServer
++{
++public:
++    char const* argv[1] = {""};
++    int argc = sizeof(argv) / sizeof(argv[0]);
++    mir::DefaultServerConfiguration config{argc, argv};
++
++    mir::DefaultServerConfiguration& server_config() override {return config;};
++protected:
++    enum ResultFileType {raw, json};
++    virtual void RunGLMark2(char const* output_filename, ResultFileType file_type)
++    {
++        std::string const cmd = "glmark2-es2-mir --fullscreen";
++        mir::test::Popen p(cmd);
++
++        boost::cmatch matches;
++        boost::regex re_glmark2_score(".*glmark2\\s+Score:\\s+(\\d+).*");
++        std::string line;
++        std::ofstream glmark2_output;
++        glmark2_output.open(output_filename);
++        while (p.get_line(line))
++        {
++            if (file_type == raw)
++            {
++                glmark2_output << line;
++            }
++            else if (file_type == json)
++            {
++                if (boost::regex_match(line.c_str(), matches, re_glmark2_score))
++                {
++                    std::string json =  "{";
++                        json += "'benchmark_name':'glmark2-es2-mir'";
++                        json += ",";
++                        json += "'score':'" + matches[1] + "'";
++                        json += "}";
++                    glmark2_output << json;
++                    break;
++                }
++            }
++        }
++    }
++};
++
++TEST_F(GLMark2Test, benchmark_fullscreen_default)
++{
++    unsetenv("MIR_SERVER_NO_FILE");
++    RunGLMark2("glmark2_fullscreen_default.json", json);
++}

Mir

Merge lp:~josharenson/mir/install_glmark2 into lp:mir

Commit message

Description of the change

Preview Diff

Subscribers