Mir test fails with protobuf3: Protobuf-can-be-reloaded (SEGFAULT)

Bug #1619616 reported by Matthias Klose
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mir
Invalid
High
Unassigned
mir (Ubuntu)
Invalid
High
Unassigned
protobuf (Ubuntu)
Fix Released
High
Alan Griffiths

Bug Description

builds with 3.0.0-7, then has one failing test:

test 19
      Start 19: Protobuf-can-be-reloaded

19: Test command: /<<BUILDDIR>>/mir-0.24.0+16.10.20160815.3/obj-powerpc64le-linux-gnu/bin/mir_test_reload_protobuf
19: Test timeout computed to be: 9.99988e+06
19/19 Test #19: Protobuf-can-be-reloaded ..........................***Exception: SegFault 0.17 sec

95% tests passed, 1 tests failed out of 19

Total Test time (real) = 125.73 sec

The following tests FAILED:
  19 - Protobuf-can-be-reloaded (SEGFAULT)
Errors while running CTest
Makefile:85: recipe for target 'test' failed
make[2]: *** [test] Error 8

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The reason this matters:

The principle downstream from Mir (Unity8) loads Mir as one of a number of plugins, several of which use protobuf. That means that we're prone to being loaded, unloaded and reloaded and don't have control of this.

We've encountered problems in this scenario before that have been traced to protobuf so we have a specific test. But other Unity8 plugins may encounter the same issue.

As seen in comment #25 the problem here is entirely down to a change in the behavior of libmirprotobuf-lite - it no longer unloads on dlclose().

This is only a problem because there is code (see patch in comment #20) that assumes that the library will not be used after a call to google::protobuf::ShutdownProtobufLibrary() unless it is reloaded.

Tags: ftbfs
Matthias Klose (doko)
tags: added: ftbfs
Changed in mir (Ubuntu):
status: New → Confirmed
importance: Undecided → Critical
milestone: none → ubuntu-16.09
Changed in mir (Ubuntu):
assignee: nobody → Alan Griffiths (alan-griffiths)
Revision history for this message
Matthias Klose (doko) wrote :

maybe wait for https://launchpad.net/ubuntu/+source/protobuf/3.0.0-7

  * Added patch by Srivats P to restore New*Callback into google::protobuf
    namespace. Required for ostinato (#835435). (Closes: #836200).

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

sensible

Revision history for this message
Matthias Klose (doko) wrote :

builds with 3.0.0-7, then has one failing test:

test 19
      Start 19: Protobuf-can-be-reloaded

19: Test command: /<<BUILDDIR>>/mir-0.24.0+16.10.20160815.3/obj-powerpc64le-linux-gnu/bin/mir_test_reload_protobuf
19: Test timeout computed to be: 9.99988e+06
19/19 Test #19: Protobuf-can-be-reloaded ..........................***Exception: SegFault 0.17 sec

95% tests passed, 1 tests failed out of 19

Total Test time (real) = 125.73 sec

The following tests FAILED:
  19 - Protobuf-can-be-reloaded (SEGFAULT)
Errors while running CTest
Makefile:85: recipe for target 'test' failed
make[2]: *** [test] Error 8

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Only on powerpc64? I don't have one of those.

Revision history for this message
Matthias Klose (doko) wrote :

same on amd64

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :
Download full text (3.8 KiB)

This is during the second dlopen(libmirprotobuf...) and in the initialization code. So it seems similar to the problems discussed in lp:1391976. But the workaround linked there is clearly no longer adequate.

(gdb) bt
#0 google::protobuf::internal::Mutex::Lock (this=this@entry=0x0) at google/protobuf/stubs/common.cc:366
#1 0x00007ffff710c503 in google::protobuf::internal::MutexLock::MutexLock (mu=0x0, this=<synthetic pointer>)
    at ./google/protobuf/stubs/mutex.h:84
#2 google::protobuf::internal::OnShutdown (func=<optimised out>) at google/protobuf/stubs/common.cc:425
#3 0x00007ffff73b5e47 in mir::protobuf::protobuf_AddDesc_mir_5fprotobuf_2eproto ()
    at /home/alan/display_server/mir/build/src/protobuf/mir_protobuf.pb.cc:171
#4 0x00007ffff73b6321 in mir::protobuf::StaticDescriptorInitializer_mir_5fprotobuf_2eproto::StaticDescriptorInitializer_mir_5fprotobuf_2eproto() [clone .lto_priv.1] (this=0x7ffff75ee959 <mir::protobuf::static_descriptor_initializer_mir_5fprotobuf_2eproto_>)
    at /home/alan/display_server/mir/build/src/protobuf/mir_protobuf.pb.cc:184
#5 0x00007ffff736a01d in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
    at /home/alan/display_server/mir/build/src/protobuf/mir_protobuf.pb.cc:186
#6 0x00007ffff736bec9 in _GLOBAL__sub_I_mir_protobuf.pb.cc(void) ()
    at /home/alan/display_server/mir/build/src/protobuf/mir_protobuf.pb.cc:25335
#7 0x00007ffff7de87ba in call_init (l=<optimised out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffdef8, env=env@entry=0x7fffffffdf08)
    at dl-init.c:72
#8 0x00007ffff7de88cb in call_init (env=0x7fffffffdf08, argv=0x7fffffffdef8, argc=1, l=<optimised out>) at dl-init.c:30
#9 _dl_init (main_map=main_map@entry=0x5555557564a0, argc=1, argv=0x7fffffffdef8, env=0x7fffffffdf08) at dl-init.c:120
#10 0x00007ffff7ded856 in dl_open_worker (a=a@entry=0x7fffffffdb80) at dl-open.c:575
#11 0x00007ffff7de8664 in _dl_catch_error (objname=objname@entry=0x7fffffffdb70, errstring=errstring@entry=0x7fffffffdb78,
    mallocedp=mallocedp@entry=0x7fffffffdb6f, operate=operate@entry=0x7ffff7ded500 <dl_open_worker>, args=args@entry=0x7fffffffdb80)
    at dl-error.c:187
#12 0x00007ffff7dece39 in _dl_open (file=0x555555554ac4 "libmirprotobuf.so.3", mode=-2147483646,
    caller_dlopen=0x5555555549d9 <main+105>, nsid=-2, argc=<optimised out>, argv=<optimised out>, env=0x7fffffffdf08) at dl-open.c:660
#13 0x00007ffff7bd4ee9 in dlopen_doit (a=a@entry=0x7fffffffddb0) at dlopen.c:66
#14 0x00007ffff7de8664 in _dl_catch_error (objname=0x555555756430, errstring=0x555555756438, mallocedp=0x555555756428,
    operate=0x7ffff7bd4e90 <dlopen_doit>, args=0x7fffffffddb0) at dl-error.c:187
#15 0x00007ffff7bd5571 in _dlerror_run (operate=operate@entry=0x7ffff7bd4e90 <dlopen_doit>, args=args@entry=0x7fffffffddb0)
    at dlerror.c:163
#16 0x00007ffff7bd4f82 in __dlopen (file=<optimised out>, mode=<optimised out>) at dlopen.c:87
#17 0x00005555555549d9 in main (argc=1, argv=0x7fffffffdef8) at /home/alan/display_server/mir/tests/loader-tests/test_reload.c:36
(gdb) f 3
#3 0x00007ffff73b5e47 in mir::protobuf::protobuf_AddDesc_mir_5fprotobuf_2eproto ()
    at /home/alan/display_server/mir/build/sr...

Read more...

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

So, a workaround is to apply the same hack found in lp:~vanvugt/mir/test-1391976/+merge/262669.

But we obviously haven't needed that, so something related has changed in protobuf. I suspect we were carrying a patch that fixed the problem there.

Unfortunately, I'm at EOW now, so I haven't time to chase down the history.

Changed in mir (Ubuntu):
assignee: Alan Griffiths (alan-griffiths) → nobody
Changed in mir:
importance: Undecided → Critical
importance: Critical → High
Changed in mir (Ubuntu):
importance: Critical → High
Changed in mir:
milestone: none → 0.25.0
summary: - mir fails to build with protobuf3
+ Mir test fails with protobuf3: Protobuf-can-be-reloaded (SEGFAULT)
description: updated
Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

It looks very much as though protobuf3 reverts this change:

  $ bzr log -c 295 lp:protobuf

I've not yet proved that's the whole problem, but it would explain it.

Changed in mir (Ubuntu):
assignee: nobody → Alan Griffiths (alan-griffiths)
Changed in mir:
assignee: nobody → Alan Griffiths (alan-griffiths)
status: New → In Progress
Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Here's the fix to +branch/ubuntu/wily-proposed/protobuf/ that fixes the problem with reloading modules that depend on protobuf.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Ooops!! that's based on the wrong branch

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Patch for the correct branch

Revision history for this message
Matthias Klose (doko) wrote :

the test still fails when built using protobuf 3.0.0-7ubuntu2

Revision history for this message
Matthias Klose (doko) wrote :

hmm, and the hack found in lp:~vanvugt/mir/test-1391976/+merge/262669 was never applied in the protobuf package, if you look at protobuf's publishing history at
https://launchpad.net/ubuntu/+source/protobuf/+publishinghistory

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Two problems with that statement:
  (1) That's just a test and not a fix or workaround; and
  (2) It's for Mir, not to be "applied in the protobuf package".

The relevant workaround for Mir is actually:
  https://code.launchpad.net/~vanvugt/mir/fix-1391976/+merge/262678
but we'll hold off reapplying that while Alan is proposing proper fixes for protobuf itself.

Changed in protobuf (Ubuntu):
assignee: nobody → Alan Griffiths (alan-griffiths)
importance: Undecided → High
status: New → In Progress
Changed in mir:
status: In Progress → Confirmed
assignee: Alan Griffiths (alan-griffiths) → nobody
Changed in mir (Ubuntu):
assignee: Alan Griffiths (alan-griffiths) → nobody
Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

There was a hack put into Mir around the end of June 2015 as linked by Daniel (https://code.launchpad.net/~vanvugt/mir/fix-1391976/+merge/262678). As the comment suggests this was pending "a proper fix will come from libprotobuf itself but this way we don't need to wait for that".

This "hack" was removed at the end of July 2015. I don't see a corresponding change to protobuf. So I suspect I have been chasing the wrong fix. I found some differences that look related and tried patching them locally. I still have to figure out why patching things seems to work locally - especially with the typo in the patch file. Clearly I'm doing something wrong. :(

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

I'm very suspicious of my laptop running yakkety - even with "proposed" purged it is now crashing every few minutes (which makes keeping track of things a PITA).

The root cause is probably a failure in protobuf to re-initalise static data when clients call google::protobuf::ShutdownProtobufLibrary(). This leads to a failure when if the whole .so isn't unloaded before the next "GOOGLE_PROTOBUF_VERIFY_VERSION;"

Which is why the change I picked up looks relevant.

The "hack" mentioned above did appear to workaround the problem for Mir (i.e. the test passes). (But it will probably recur with other Qt plugins that use protobuf.)

So I'm going to reinstall and start over.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

OK, I have rebuild laptop and have a less flaky system. This (updated) patch is only a proof-of-concept as it clearly leaks resources on program exit in this form.

OTOH it does make the Mir test pass, so I have found the right problem.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

A simpler fix that removes the assumption that the shared library will be unloaded between uses

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

But I don't yet see how 2.6.1 avoided this issue.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Oops. The patch in #18 is for google/protobuf/stubs/common.cc:

415d414
< shutdown_functions = new vector<void (*)()>;
425a425,426
> if (internal::shutdown_functions == NULL)
> shutdown_functions = new vector<void (*)()>;
446,447d446
< delete internal::shutdown_functions_mutex;
< internal::shutdown_functions_mutex = NULL;

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

The reason 2.6.1 works is that libprotobuf-lite-so.9 is unloaded each iteration. But with the move to 3.0.0 that no longer happens.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :
Download full text (5.1 KiB)

This shows the problem, but not the underlying cause:

$ gdb bin/mir_test_reload_protobufGNU gdb (Ubuntu 7.11.90.20160906-0ubuntu1) 7.11.90.20160906-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/mir_test_reload_protobuf...done.
(gdb) b /home/alan/display_server/mir/tests/loader-tests/test_reload.c:36
No source file named /home/alan/display_server/mir/tests/loader-tests/test_reload.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (/home/alan/display_server/mir/tests/loader-tests/test_reload.c:36) pending.
(gdb) r
Starting program: /home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/mir_test_reload_protobuf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
MIR_CLIENT_PLATFORM_PATH=/home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/../lib/client-modules/
MIR_SERVER_PLATFORM_PATH=/home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/../lib/server-modules/
LD_LIBRARY_PATH=/home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/../lib
exec=/home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/mir_test_reload_protobuf.bin
process 2335 is executing new program: /home/alan/.CLion2016.2/system/cmake/generated/mir-9ec817d/9ec817d/Debug/bin/mir_test_reload_protobuf.bin
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0x7fffffffde58) at /home/alan/display_server/mir/tests/loader-tests/test_reload.c:36
36 dl = dlopen(libname, RTLD_NOW);
(gdb) info share
From To Syms Read Shared Object Library
0x00007ffff7dd8aa0 0x00007ffff7df6130 Yes /lib64/ld-linux-x86-64.so.2
0x00007ffff7bd4d80 0x00007ffff7bd598e Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007ffff79bba60 0x00007ffff79c8e01 Yes /lib/x86_64-linux-gnu/libpthread.so.0
0x00007ffff760e9c0 0x00007ffff775f0c3 Yes /lib/x86_64-linux-gnu/libc.so.6
(gdb) n
37 if (dl)
(gdb) info share
From To Syms Read Shared Object Library
0x00007ffff7dd8aa0 0x00007ffff7df6130 Yes /lib64/ld-linux-x86-64.so.2
0x00007ffff7bd4d80 0x00007ffff7bd598e Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007ffff79bba60 0x00007ffff79c8e01 Yes /lib/x86_64-linux-gnu/libpthread.so.0
0x00007ffff760e9c0 0x00007ffff775f0c3 ...

Read more...

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

In any case, if protobuf choses to remain resident it is only polite to work when the startup/shutdown cycle is repeated. (And that's what the patch achieves.)

Changed in mir:
status: Confirmed → Invalid
Changed in mir (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Invalid for the Mir project. Sounds like it's just another protobuf bug.

Hopefully we won't need to workaround it in Mir, but that's not completely out of the question. Especially if protobuf3 hits distro without first getting fixed we won't have much choice.

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Narrowing the problem: it isn't any of the Mir code, or the generated code, or explicit calls into libprotobuf-lite.

The following program loads and unloads libprotobuf-lite.so from 2.6.1 successfully. With 3.0.0 it fails to unload.

#include <stdio.h>
#include <dlfcn.h>

int main(int argc, char** argv)
{
    char const* libname = argc <= 1 ? "libprotobuf-lite.so" : argv[1];

    void* dl = NULL;

    dl = dlopen(libname, RTLD_NOW);
    if (!dl)
    {
        printf("dlopen() error: (%s)\n", dlerror());
        return 1;
    }

    printf("library loaded\n");
    dlclose(dl);

    dl = dlopen(libname, RTLD_NOW|RTLD_NOLOAD);

    if (dl)
    {
        printf("library didn't unload: (%s)\n", libname);
        dlclose(dl);
        return 1;
    }

    printf("library unloaded\n");

    return 0;
}

Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Hmm the library publishes a load of symbols that seem unnecessary at best:

$ readelf -Ws /usr/local/lib/libprotobuf-lite.so | grep -v " UND " | grep -v 6google8protobuf | wc -l
123

2.6.1 seems far more reasonable:

$ readelf -Ws /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.9 | grep -v " UND " | grep -v 6google8protobuf | wc -l
10

I'm especially suspicious of the following leading to entanglement with libstdc++:

$ readelf -Ws /usr/local/lib/libprotobuf-lite.so.10 | grep -v " UND " | grep -v 6google8protobuf | grep _ZNS
   402: 0000000000023080 474 FUNC WEAK DEFAULT 12 _ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE19_M_emplace_back_auxIJS5_EEEvDpOT_
   552: 000000000001b2c0 220 FUNC WEAK DEFAULT 12 _ZNSt6vectorIPFvvESaIS1_EE19_M_emplace_back_auxIJRKS1_EEEvDpOT_
   700: 0000000000023260 117 FUNC WEAK DEFAULT 12 _ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE12emplace_backIJS5_EEEvDpOT_
    39: 0000000000023310 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.isra.25
    40: 0000000000023310 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.20
    46: 000000000001c4a0 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.isra.29
    47: 000000000001c4a0 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.23
    72: 0000000000019740 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.isra.42
    73: 0000000000019740 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.36
    78: 000000000001b3a0 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.isra.30
    87: 000000000001e820 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPcEEvT_S7_St20forward_iterator_tag.isra.45
    88: 000000000001e820 194 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.39
   115: 000000000002ea80 174 FUNC LOCAL DEFAULT 12 _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.21.constprop.32
   466: 0000000000023260 117 FUNC WEAK DEFAULT 12 _ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE12emplace_backIJS5_EEEvDpOT_
   776: 000000000001b2c0 220 FUNC WEAK DEFAULT 12 _ZNSt6vectorIPFvvESaIS1_EE19_M_emplace_back_auxIJRKS1_EEEvDpOT_
   777: 0000000000023080 474 FUNC WEAK DEFAULT 12 _ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE19_M_emplace_back_auxIJS5_EEEvDpOT_

description: updated
Revision history for this message
Alan Griffiths (alan-griffiths) wrote :

Same patch as mentioned in #18 and #20 formatted better

Changed in mir:
milestone: 0.25.0 → none
Changed in mir (Ubuntu):
milestone: ubuntu-16.09 → none
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

protobuf (3.0.0-7ubuntu3) yakkety; urgency=medium

  [ Alan Griffiths ]
  * Fix reloading of the shared libraries with dlopen. LP: #1619616.

 -- Matthias Klose <email address hidden> Thu, 08 Sep 2016 14:19:44 +0200

Changed in protobuf (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package protobuf - 3.0.0-7ubuntu3

---------------
protobuf (3.0.0-7ubuntu3) yakkety; urgency=medium

  [ Alan Griffiths ]
  * Fix reloading of the shared libraries with dlopen. LP: #1619616.

 -- Matthias Klose <email address hidden> Thu, 08 Sep 2016 14:19:44 +0200

Changed in protobuf (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Gianfranco Costamagna (costamagnagianfranco) wrote :

why this patch is not upstream? I don't see it in the google protobuf github repo
https://github.com/google/protobuf/blob/master/src/google/protobuf/stubs/common.cc

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Good question. Alan seems to be on top of this problem. Maybe he can try to upstream the fix...?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.