Mir

CI test failure in multiple tests

Bug #1401364 reported by Daniel van Vugt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mir
Fix Released
High
Mir development team
Mir 0.10.0
mir (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Multiple tests fail in a similar fashion:

https://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/352/consoleFull

For example:

ClientLibraryThread.handles_no_signals fails due to memory leak (valgrind)

8: [ RUN ] ClientLibraryThread.handles_no_signals
8: ==10098==
8: ==10098== HEAP SUMMARY:
8: ==10098== in use at exit: 10,826 bytes in 83 blocks
8: ==10098== total heap usage: 254,821 allocs, 254,738 frees, 16,598,040 bytes allocated
8: ==10098==
8: ==10098== 92 (64 direct, 28 indirect) bytes in 1 blocks are definitely lost in loss record 63 of 83
8: ==10098== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
8: ==10098== by 0x5288C3C: allocate (new_allocator.h:104)
8: ==10098== by 0x5288C3C: allocate (alloc_traits.h:357)
8: ==10098== by 0x5288C3C: __shared_count<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:616)
8: ==10098== by 0x5288C3C: __shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:1090)
8: ==10098== by 0x5288C3C: shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:316)
8: ==10098== by 0x5288C3C: allocate_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:588)
8: ==10098== by 0x5288C3C: make_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:604)
8: ==10098== by 0x5288C3C: _M_make_routine<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (thread:196)
8: ==10098== by 0x5288C3C: thread<mir::client::rpc::StreamSocketTransport::init()::<lambda()> > (thread:138)
8: ==10098== by 0x5288C3C: mir::client::rpc::StreamSocketTransport::init() (stream_socket_transport.cpp:188)
8: ==10098== by 0x5289A62: mir::client::rpc::StreamSocketTransport::StreamSocketTransport(mir::Fd const&) (stream_socket_transport.cpp:41)
8: ==10098== by 0x527B894: mir::client::rpc::make_rpc_channel(std::string const&, std::shared_ptr<mir::client::SurfaceMap> const&, std::shared_ptr<mir::client::DisplayConfiguration> const&, std::shared_ptr<mir::client::rpc::RpcReport> const&, std::shared_ptr<mir::client::LifecycleControl> const&, std::shared_ptr<mir::client::EventSink> const&) (make_socket_rpc_channel.cpp:55)
8: ==10098== by 0x5271DB3: operator() (default_connection_configuration.cpp:92)
8: ==10098== by 0x5271DB3: std::_Function_handler<std::shared_ptr<google::protobuf::RpcChannel> (), mir::client::DefaultConnectionConfiguration::the_rpc_channel()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (functional:2025)
8: ==10098== by 0x5272648: operator() (functional:2439)
8: ==10098== by 0x5272648: operator() (cached_ptr.h:41)
8: ==10098== by 0x5272648: mir::client::DefaultConnectionConfiguration::the_rpc_channel() (default_connection_configuration.cpp:92)
8: ==10098== by 0x525DE95: MirConnection::MirConnection(mir::client::ConnectionConfiguration&) (mir_connection.cpp:108)
8: ==10098== by 0x52625EE: connect (mir_connection_api.cpp:78)
8: ==10098== by 0x52625EE: mir_connect (mir_connection_api.cpp:143)
8: ==10098== by 0x5CB7D1: ClientLibrary_creates_surface_Test::TestBody() (test_client_library.cpp:180)
8: ==10098== by 0x7AB7A2: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2078)
8: ==10098== by 0x7AB7A2: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
8: ==10098== by 0x7A1403: testing::Test::Run() (gtest.cc:2151)
8: ==10098== by 0x7A1527: testing::TestInfo::Run() [clone .part.372] (gtest.cc:2326)
8: ==10098==
8: ==10098== LEAK SUMMARY:
8: ==10098== definitely lost: 64 bytes in 1 blocks
8: ==10098== indirectly lost: 28 bytes in 2 blocks
8: ==10098== possibly lost: 426 bytes in 10 blocks
8: ==10098== still reachable: 10,308 bytes in 70 blocks
8: ==10098== suppressed: 0 bytes in 0 blocks
8: ==10098== Reachable blocks (those to which a pointer was found) are not shown.
8: ==10098== To see them, rerun with: --leak-check=full --show-leak-kinds=all
8: ==10098==
8: ==10098== For counts of detected and suppressed errors, rerun with: -v
8: ==10098== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
8: /tmp/buildd/mir-0.9.0+15.04.20141125bzr2141pkg0vivid331/tests/mir_test_framework/interprocess_client_server_test.cpp:90: Failure
8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)
8: ==10090==
8: ==10090== HEAP SUMMARY:
8: ==10090== in use at exit: 11,727 bytes in 86 blocks
8: ==10090== total heap usage: 258,445 allocs, 258,359 frees, 16,749,998 bytes allocated
8: ==10090==
8: ==10090== 92 (64 direct, 28 indirect) bytes in 1 blocks are definitely lost in loss record 64 of 86
8: ==10090== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
8: ==10090== by 0x5288C3C: allocate (new_allocator.h:104)
8: ==10090== by 0x5288C3C: allocate (alloc_traits.h:357)
8: ==10090== by 0x5288C3C: __shared_count<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:616)
8: ==10090== by 0x5288C3C: __shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:1090)
8: ==10090== by 0x5288C3C: shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:316)
8: ==10090== by 0x5288C3C: allocate_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:588)
8: ==10090== by 0x5288C3C: make_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:604)
8: ==10090== by 0x5288C3C: _M_make_routine<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (thread:196)
8: ==10090== by 0x5288C3C: thread<mir::client::rpc::StreamSocketTransport::init()::<lambda()> > (thread:138)
8: ==10090== by 0x5288C3C: mir::client::rpc::StreamSocketTransport::init() (stream_socket_transport.cpp:188)
8: ==10090== by 0x5289A62: mir::client::rpc::StreamSocketTransport::StreamSocketTransport(mir::Fd const&) (stream_socket_transport.cpp:41)
8: ==10090== by 0x527B894: mir::client::rpc::make_rpc_channel(std::string const&, std::shared_ptr<mir::client::SurfaceMap> const&, std::shared_ptr<mir::client::DisplayConfiguration> const&, std::shared_ptr<mir::client::rpc::RpcReport> const&, std::shared_ptr<mir::client::LifecycleControl> const&, std::shared_ptr<mir::client::EventSink> const&) (make_socket_rpc_channel.cpp:55)
8: ==10090== by 0x5271DB3: operator() (default_connection_configuration.cpp:92)
8: ==10090== by 0x5271DB3: std::_Function_handler<std::shared_ptr<google::protobuf::RpcChannel> (), mir::client::DefaultConnectionConfiguration::the_rpc_channel()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (functional:2025)
8: ==10090== by 0x5272648: operator() (functional:2439)
8: ==10090== by 0x5272648: operator() (cached_ptr.h:41)
8: ==10090== by 0x5272648: mir::client::DefaultConnectionConfiguration::the_rpc_channel() (default_connection_configuration.cpp:92)
8: ==10090== by 0x525DE95: MirConnection::MirConnection(mir::client::ConnectionConfiguration&) (mir_connection.cpp:108)
8: ==10090== by 0x52625EE: connect (mir_connection_api.cpp:78)
8: ==10090== by 0x52625EE: mir_connect (mir_connection_api.cpp:143)
8: ==10090== by 0x5CB7D1: ClientLibrary_creates_surface_Test::TestBody() (test_client_library.cpp:180)
8: ==10090== by 0x7AB7A2: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2078)
8: ==10090== by 0x7AB7A2: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
8: ==10090== by 0x7A1403: testing::Test::Run() (gtest.cc:2151)
8: ==10090== by 0x7A1527: testing::TestInfo::Run() [clone .part.372] (gtest.cc:2326)
8: ==10090==
8: ==10090== LEAK SUMMARY:
8: ==10090== definitely lost: 64 bytes in 1 blocks
8: ==10090== indirectly lost: 28 bytes in 2 blocks
8: ==10090== possibly lost: 1,071 bytes in 11 blocks
8: ==10090== still reachable: 10,564 bytes in 72 blocks
8: ==10090== suppressed: 0 bytes in 0 blocks
8: ==10090== Reachable blocks (those to which a pointer was found) are not shown.
8: ==10090== To see them, rerun with: --leak-check=full --show-leak-kinds=all
8: ==10090==
8: ==10090== For counts of detected and suppressed errors, rerun with: -v
8: ==10090== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
8: /tmp/buildd/mir-0.9.0+15.04.20141125bzr2141pkg0vivid331/tests/mir_test_framework/interprocess_client_server_test.cpp:153: Failure
8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)
8: [ FAILED ] ClientLibraryThread.handles_no_signals (707 ms)

See original description

Related branches

lp:~raof/mir/fix-all-the-CI
PS Jenkins bot (community): Needs Fixing (continuous-integration)
Daniel van Vugt: Approve
Mir development team: Pending requested
Diff: 31 lines (+5/-2)
2 files modified
src/client/mir_surface_api.cpp (+4/-2)
tests/acceptance-tests/test_client_library.cpp (+1/-0)
lp:ubuntu/vivid-proposed/mir
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Weird. I can't reproduce this locally.

Daniel van Vugt (vanvugt)
Changed in mir:
status: New → Triaged
milestone: none → 0.10.0
Daniel van Vugt (vanvugt)
tags: added: ci-blocker
Alexandros Frantzis (afrantzis)
Changed in mir:
assignee: nobody → Alexandros Frantzis (afrantzis)
Daniel van Vugt (vanvugt)
Changed in mir:
status: Triaged → In Progress
Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

Note that this is happening in many tests not only ClientLibraryThread.*

https://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/352/consoleFull

It seems the memory leak is only a consequence of something else failing:

8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)

summary: - ClientLibraryThread.handles_no_signals fails due to memory leak
- (valgrind)
+ CI test failure in multiple tests
description: updated
Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

> It seems the memory leak is only a consequence of something else failing

Actually, it's not clear what is happening. I can't reproduce locally either.

Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

Changed to critical as it's blocking CI.

Changed in mir:
importance: High → Critical
Daniel van Vugt (vanvugt)
Changed in mir:
assignee: Alexandros Frantzis (afrantzis) → Mir development team (mir-team)
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Test case (from Alan?)

valgrind --error-exitcode=1 --trace-children=yes --leak-check=full --show-leak-kinds=definite --errors-for-leak-kinds=definite bin/mir_acceptance_tests --gtest_filter=ClientLibraryErrorsDeathTest*:ClientLibraryThread*

Revision history for this message
Chris Halse Rogers (raof) wrote :

It's a (cunningly hidden) leak in ClientLibrary.surface_create.

Cunningly hidden in that it only manifests after a call to fork(), and only in the child.
So most people running the testsuite at home won't notice, because we run each test
in a separate process by default.

CI runs everything in one process, so hits this.

To duplicate that environment, you need to run the tests in a build with
cmake -DDISABLE_GTEST_TEST_DISCOVERY=ON -DENABLE_MEMCHECK_OPTION=ON

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

... and then:

make test ARGS="-V" | tee output.txt

Daniel van Vugt (vanvugt)
Changed in mir:
status: In Progress → Fix Committed
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Reverted to High priority. It don't think this ever qualified for Critical as it never affected any users, nor prevented us from working.

Changed in mir:
importance: Critical → High
Daniel van Vugt (vanvugt)
Changed in mir:
status: Fix Committed → Fix Released
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

mir (0.10.0+15.04.20150107.2-0ubuntu1) vivid; urgency=medium

Changed in mir (Ubuntu):
importance: Undecided → High
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.