MongoDB Memory corruption

------- Comment From <email address hidden> 2016-11-09 10:52 EDT-------
Hello Canonical,
Sending this bug to you for awareness and advice.

For what it's worth, GCC is just a placeholder package for now. The compiler and libraries aren't directly implicated, at least as of now. The origin of the stack corruption remains unknown.

Andrew, according to the valgrind community, the error

  Memcheck: mc_machine.c:329 (get_otrack_shadow_offset_wrk): the 'impossible' happened.

is probably due to the lock elision code accessing a hardware register that valgrind doesn't know about, so there isn't a shadow register to consult. Our valgrind guys will look into that aspect of things. In the meantime, they tell us that you can circumvent this problem by not using --track-origins. Could you please give this a try?

I had another thought this evening. In case this is a threading problem, have you tried building with Clang and using ThreadSanitizer? Support for this was added to ppc64el in 2015.

The following are reproduction instructions for the behavior that we are observing on Ubuntu 16.04 ppc64le. Note that we have run this same test on RHEL 7.1 ppc64le, and we do not observe any stack corruption. Note also that building and running this repro may depend on certain system libraries (SSL, etc) or python libraries being available on the system. Please install as needed. The particular commit here is fairly recent, just one that I happen to know demonstrates the issue.

- git clone https://github.com/mongodb/mongo.git
- cd mongo
- git checkout 3220495083b0d678578a76591f54ee1d7a5ec5df
- git apply acm.nov9.patch
- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j$(echo "$(grep -c processor /proc/cpuinfo)/2" | bc) ./mongo ./mongod ./mongos
- ulimit -c unlimited && python buildscripts/resmoke.py --suites=concurrency_sharded --storageEngine=wiredTiger --excludeWithAnyTags=requires_mmapv1 --dbpathPrefix=... --repeat=500 --continueOnFailure

Note that you should provide an actual argument for the --dbpathPrefix argument in the last step, as this is where the running database instances will store data.

You will need to leave this running for several hours, perhaps overnight. In our runs, we find that about 1% of the repeated runs of the test fail, dropping a core.

The core files are typically (but not always!) associated with crashes of the mongos binary inside one of the several mongo::bsonExtractXXX functions, where we find our hand-rolled stack canary to be corrupted. A typical stack trace of a crashing thread looks like:

$ gdb ./mongos core.2016-11-09T23:11:56+00:00
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./mongos...done.
[New LWP 3821]
...
[New LWP 3736]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".
Core was generated by `/home/acm/opt/src/mongo/mongos --configdb test-configRS/ubuntu1604-ppc-dev.pic.'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00003fff779ff21c in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x3fff5d98f140 (LWP 3821))]
(gdb) bt
#0 0x00003fff779ff21c in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00003fff77a01894 in __GI_abort () at abort.c:89
#2 ...

Here is the patch for the above comment

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Bill -

I will try again with valgrind without --track-origins=yes and post any interesting findings. Re ThreadSanitizer, we have tried before without success. The last time we tried, it didn't work because clang TSAN didn't support exceptions. Perhaps that has changed? We really like the sanitizers (we run ASAN and UBSAN in our CI loop), but our experience has been that there is a significant period of scrubbing out false positives and adding suppressions before meaningful signal can be extracted. I'm happy to add it to the list of things to try though.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Hi Andrew -- not sure about Clang TSAN supporting exceptions at this point; it is probable that we don't have a solution there as I would expect that to require target support, and I've not heard of that happening for POWER. That said, I've been less connected to the Clang community for the last year or so, so anything's possible.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

The attachment "Apply to 3220495083b0d678578a76591f54ee1d7a5ec5df" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Overnight, I ran this test case on both an Ubuntu 16.04 ppc64le system and a RHEL 7.1 ppc64le system.

The test ran 219 times on Ubuntu, with 15 cores, for a failure rate of around 5%. Most of the time corruption was detected in the Canary ctor (before doing other work), but a few times in the dtor:

$ grep "10000[12]" resmoke.log
[js_test:fsm_all_sharded_replication] 2016-11-09T21:06:42.140+0000 s40019| 2016-11-09T21:06:42.140+0000 I - [conn59] Fatal Assertion 100002 at src/mongo/bson/util/bson_extract.cpp 50
[js_test:fsm_all_sharded_replication] 2016-11-09T23:11:56.413+0000 s40019| 2016-11-09T23:11:56.412+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T03:49:43.577+0000 s40019| 2016-11-10T03:49:43.576+0000 I - [conn47] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T04:54:20.748+0000 s40019| 2016-11-10T04:54:20.745+0000 I - [conn28] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T04:56:43.929+0000 s40020| 2016-11-10T04:56:43.929+0000 I - [conn30] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T07:26:14.013+0000 s40019| 2016-11-10T07:26:14.013+0000 I - [conn3] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T07:52:45.732+0000 s40019| 2016-11-10T07:52:45.698+0000 I - [conn1] Fatal Assertion 100002 at src/mongo/bson/util/bson_extract.cpp 50
[js_test:fsm_all_sharded_replication] 2016-11-10T07:54:45.606+0000 s40020| 2016-11-10T07:54:45.606+0000 I - [conn30] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T08:28:04.278+0000 s40020| 2016-11-10T08:28:04.277+0000 I - [conn13] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T09:28:24.221+0000 s40020| 2016-11-10T09:28:24.221+0000 I - [conn32] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T09:41:14.880+0000 s40019| 2016-11-10T09:41:14.800+0000 I - [conn48] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T10:35:58.143+0000 s40019| 2016-11-10T10:35:58.118+0000 I - [conn2] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T11:06:01.027+0000 s40019| 2016-11-10T11:06:01.027+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T12:37:43.100+0000 s40019| 2016-11-10T12:37:43.100+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46
[js_test:fsm_all_sharded_replication] 2016-11-10T13:31:31.912+0000 s40020| 2016-11-10T13:31:31.912+0000 I - [conn1] Fatal Assertion 100001 at src/mongo/bson/util/bson_extract.cpp 46

The test ran 227 times ...

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

I tried valgrind as suggested above.

By adding --show-mismatched-frees=no and removing --track-origins=yes I was able to get the process to start up without a lot of false positives. However, the server process fails to open its listening socket, because valgrind reports an unsupported syscall:

[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019| --17827-- WARNING: unhandled ppc64le-linux syscall: 326
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019| --17827-- You may be able to write your own handler.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.496+0000 s40019| --17827-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.497+0000 s40019| --17827-- Nevertheless we consider this a bug. Please report
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.497+0000 s40019| --17827-- it at http://valgrind.org/support/bug_reports.html.
[js_test:fsm_all_sharded_replication] 2016-11-10T14:24:04.509+0000 s40019| 2016-11-10T14:24:04.509+0000 I - [mongosMain] Assertion: 15863:listen(): invalid socket? Function not implemented src/mongo/util/net/listen.cpp 184

The code around listen.cpp:184 looks like:

    182 SOCKET sock = ::socket(me.getType(), SOCK_STREAM, 0);
    183 ScopeGuard socketGuard = MakeGuard(&closesocket, sock);
    184 massert(15863,
    185 str::stream() << "listen(): invalid socket? " << errnoWithDescription(),
    186 sock >= 0);

It looks as if valgrind on ppc64le doesn't support the socket syscall? Note that I happened to be subscribed to the valgrind-developers list so I saw the other post there, and I've followed up with this same information already.

That discussion is here: https://sourceforge.net/p/valgrind/mailman/message/35483420/

In any event, it doesn't look like valgrind is going to be able to help here.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

OK, I upgraded valgrind to 3.12 on the power machine and I can now get it to run meaningfully. We are seeing many error reports of the following form:

[js_test:fsm_all_sharded_replication] 2016-11-10T16:19:58.396+0000 s40019| ==34604== Thread 50:
[js_test:fsm_all_sharded_replication] 2016-11-10T16:19:58.396+0000 s40019| ==34604== Invalid read of size 2
[js_test:fsm_all_sharded_replication] 2016-11-10T16:19:58.396+0000 s40019| ==34604== at 0x4F2AD20: __lll_unlock_elision (elision-unlock.c:36)
[js_test:fsm_all_sharded_replication] 2016-11-10T16:19:58.396+0000 s40019| ==34604== by 0x4F1DB07: __pthread_mutex_unlock_usercnt (pthread_mutex_unlock.c:64)
[js_test:fsm_all_sharded_replication] 2016-11-10T16:19:58.396+0000 s40019| ==34604== by 0x4F1DB07: pthread_mutex_unlock (pthread_mutex_unlock.c:314)

Or

[js_test:fsm_all_sharded_replication] 2016-11-10T16:20:43.998+0000 s40019| ==34604== Invalid write of size 2
[js_test:fsm_all_sharded_replication] 2016-11-10T16:20:43.998+0000 s40019| ==34604== at 0x4F2AD30: __lll_unlock_elision (elision-unlock.c:37)
[js_test:fsm_all_sharded_replication] 2016-11-10T16:20:43.998+0000 s40019| ==34604== by 0x4F1DB07: __pthread_mutex_unlock_usercnt (pthread_mutex_unlock.c:64)
[js_test:fsm_all_sharded_replication] 2016-11-10T16:20:43.998+0000 s40019| ==34604== by 0x4F1DB07: pthread_mutex_unlock (pthread_mutex_unlock.c:314)
[js_test:fsm_all_sharded_replication] 2016-11-10T16:20:43.999+0000 s40019| ==34604== by 0xD803C7: operator()<const mongo::executor::TaskExecutor::RemoteCommandCallbackArgs&, long unsigned int&, void> (functional:600)

In all cases, the invalid write appears to be a write into a freed block. Frequently, the address appears to be aligned 'Address 0x...e'. So, this is very interesting.

Another engineer and I took a close look at one of these instances, and we do not believe there is any way that the mutex could be accessed after it was deleted.

Is there a way we can disable the libc lock elision code? An environment variable or other similar setting? We would like to see if we still see these sorts of reports after disabling lock elision. If so, then it would almost certainly be a logic error in our code that we are just missing. On the other hand, if the valgrind reports go away when we disable lock elision, then it would be evidence that lock elision might be at fault for the stack corruption we are observing, at which point I would re-try our original repro.

Hi Andrew,

That indeed looks suspicious. I've been talking with our libc team. It appears that the existing patch that provides for disabling lock elision dynamically isn't present in the libc on Ubuntu 16.04, which is very unfortunate. They are thinking about other possible solutions.

This seems a very possible culprit as lock elision is not present on the other configurations where you've seen success, to my knowledge.

We may need to get a special test build of glibc with --disable-lock-elision for you to test with. I will ask the Ubuntu toolchain team whether this is a possibility.

Thanks for persevering through the valgrind concerns to get to this clue!

Bill

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

The following might override the HTM lock elision. Can someone try it to see if it works?

bergner@ampere:~$ cat pthread_mutex_lock.c
#include <pthread.h>

#define PTHREAD_MUTEX_NO_ELISION_NP 512
extern int __pthread_mutex_lock (pthread_mutex_t *);

int
pthread_mutex_lock (pthread_mutex_t *mutex)
{
  mutex->__data.__kind |= PTHREAD_MUTEX_NO_ELISION_NP;
  return __pthread_mutex_lock (mutex);
}
bergner@ampere:~$ gcc -c -fPIC pthread_mutex_lock.c
bergner@ampere:~$ gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1 pthread_mutex_lock.o
bergner@ampere:~$ LD_PRELOAD=./libfoo.so.1 ./a.out

...replacing ./a.out with the binary you want to run without lock elision.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

I have the libfoo.so.1 interposer running, I will let it run overnight and report back tomorrow with any interesting findings.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

I don't think the interposition is working, or I'm doing something wrong.

I changed pthread_mutex_lock.c to the following:

$ cat pthread_mutex_lock.c
#include <pthread.h>
#include <stdlib.h>

#define PTHREAD_MUTEX_NO_ELISION_NP 512
extern int __pthread_mutex_lock (pthread_mutex_t *);

int
pthread_mutex_lock (pthread_mutex_t *mutex)
{
  abort();
  mutex->__data.__kind |= PTHREAD_MUTEX_NO_ELISION_NP;
  return __pthread_mutex_lock (mutex);
}

$ gcc -c -fPIC pthread_mutex_lock.c
$ gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1

I then wrote a small C++ program:

$ cat use_foo.cpp
#include <mutex>

int main(int argc, char* argv[]) {
    std::mutex m;
    std::lock_guard<std::mutex> guard(m);
    return EXIT_SUCCESS;
}

Compiled it to a.out:

$ g++ -std=c++11 -pthread ./use_foo.cpp

When run, it does *not* terminate in abort:

LD_PRELOAD=$(realpath ./libfoo.so.1 ) ./a.out ; echo $?

Under gdb, I can see that the libfoo.so.1 library is loaded:

$ LD_PRELOAD=$(realpath ./libfoo.so.1 ) gdb ./a.out
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...(no debugging symbols found)...done.
(gdb) sta
Temporary breakpoint 1 at 0x10000a24
Starting program: /home/acm/opt/src/1640518/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, 0x0000000010000a24 in main ()
(gdb) info inferior
  Num Description Executable
* 1 process 14046 /home/acm/opt/src/1640518/a.out
(gdb) !lsof -p 14046 | grep lib
a.out 14046 amorrow mem REG 8,2 74328 2621621 /lib/powerpc64le-linux-gnu/libgcc_s.so.1
a.out 14046 amorrow mem REG 8,2 856616 2622143 /lib/powerpc64le-linux-gnu/libm-2.23.so
a.out 14046 amorrow mem REG 8,2 1851512 2622139 /lib/powerpc64le-linux-gnu/libc-2.23.so
a.out 14046 amorrow mem REG 8,2 171632 2622098 /lib/powerpc64le-linux-gnu/libpthread-2.23.so
a.out 14046 amorrow mem REG 8,2 2042040 2752992 /usr/lib/powerpc64le-linux-gnu/libstdc++.so.6.0.21
a.out 14046 amorrow mem REG 8,2 69136 6182484 /home/acm/opt/src/1640518/libfoo.so.1
a.out 14046 amorrow mem REG 8,2 268976 2622140 /lib/powerpc64le-linux-gnu/ld-2.23.so

Setting a breakpoint in pthread_mutex_lock lands me in __GI___pthread_mutex_lock. Perhaps the interposition symbol needs to be changed?

$ LD_PRELOAD=$(realpath ./libfoo.so.1 ) gdb ./a.out
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 20...

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

------- Comment From <email address hidden> 2016-11-10 19:54 EDT-------
(In reply to comment #49)

Something doesn't look right in your commands. They all have "LD_PRELOAD=" with nothing after the '=':
Comment #49 has:
> $ LD_PRELOAD= gdb ./a.out

That should be:
$ LD_PRELOAD=/path-to.../libfoo.so.1 gdb ./a.out

Also, if you use LD_PRELOAD (without export) for the gdb command, I wouldn't expect it to carry forward to the inferior. Although, as you say, gdb shows the library loaded.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

When I add the abort and use your C++ test case, I see the abort:

bergner@ampere:~$ cat pthread_mutex_lock.c
#include <stdlib.h>
#include <pthread.h>

#define PTHREAD_MUTEX_NO_ELISION_NP 512
extern int __pthread_mutex_lock (pthread_mutex_t *);

int
pthread_mutex_lock (pthread_mutex_t *mutex)
{
  abort();
  mutex->__data.__kind |= PTHREAD_MUTEX_NO_ELISION_NP;
  return __pthread_mutex_lock (mutex);
}
bergner@ampere:~$ gcc -fPIC -c pthread_mutex_lock.c
bergner@ampere:~$ gcc -shared -Wl,-soname,libbar.so.1 -o libbar.so.1 pthread_mutex_lock.o
bergner@ampere:~$ cat foo.cpp
#include <mutex>

int main(int argc, char* argv[]) {
    std::mutex m;
    std::lock_guard<std::mutex> guard(m);
    return EXIT_SUCCESS;
}
bergner@ampere:~$ g++ -std=c++11 -pthread foo.cpp
bergner@ampere:~$ ./a.out
bergner@ampere:~$ LD_PRELOAD=./libbar.so.1 ./a.out
Aborted

gdb shows the abort is from the shim library too:

bergner@ampere:~$ gdb -q ./a.out
Reading symbols from ./a.out...(no debugging symbols found)...done.
(gdb) set environment LD_PRELOAD=./libbar.so.1
(gdb) run
Starting program: /home/bergner/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc64le-linux-gnu/libthread_db.so.1".

Program received signal SIGABRT, Aborted.
0x00003fffb7b3f27c in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00003fffb7b3f27c in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00003fffb7b418f4 in __GI_abort () at abort.c:89
#2 0x00003fffb7f507e4 in pthread_mutex_lock () from ./libbar.so.1
#3 0x0000000010000954 in __gthread_mutex_lock(pthread_mutex_t*) ()
#4 0x0000000010000b10 in std::mutex::lock() ()
#5 0x0000000010000be8 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()
#6 0x0000000010000a80 in main ()
(gdb)

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

------- Comment From <email address hidden> 2016-11-10 20:44 EDT-------
(In reply to comment #51)
> (In reply to comment #49)
>
> Something doesn't look right in your commands. They all have "LD_PRELOAD="
> with nothing after the '=':
> Comment #49 has:
> > $ LD_PRELOAD= gdb ./a.out

It's a mirroring problem to our bugzilla. The command looks correct in the Launchpad comment. I recommend reading/adding comments in Launchpad rather than from our bugzilla.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

A test build of glibc with lock elision disabled is in progress here:

https://launchpad.net/~adconrad/+archive/ubuntu/nole/+packages

That said, the above trace looks suspiciously like a double-unlock. That breaks pthread rules, but the software implementation has historically let you do it anyway (or, rather, it fails to abort there because it would be expensive to do so, but your code is still buggy, prone to hanging, etc).

There's a thread on debian-devel right now about this same issue[1] as it related to ghostscript, where people noticed that with hardware lock elision, the world exploded, and without, it seemed to work, maybe, ish (though, as I said, the ghostscript bug was likely the cause of some mysterious hangs).

[1] https://lists.debian.org/debian-devel/2016/11/msg00210.html

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

from "man pthread_mutex_unlock":
       If the mutex type is PTHREAD_MUTEX_ERRORCHECK, then error checking
       shall be provided. If a thread attempts to relock a mutex that it has
       already locked, an error shall be returned. If a thread attempts to
       unlock a mutex that it has not locked or a mutex which is unlocked, an
       error shall be returned.

That might be something to try.

Per the blog post mentioned from the thread in #35, this sort of problem should also manifest on a Broadwell or Skylake processor. Andrew, have you tried running on such machines?

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

From that debian thread:

"Per logs from message #15 on bug #842796:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842796#15

SIGSEGV on __lll_unlock_elision is a signature (IME with very high
confidence) of an attempt to unlock an already unlocked lock while
running under hardware lock elision.

Well, unlocking an already unlocked lock is a pthreads API rule
violation, and it is going to crash the process on something that
implements hardware lock elision."

So I think we have some pretty good evidence of an application problem. I think that using Paul Clarke's suggestion may be necessary for you to figure out where the double-unlock is occurring. I'm not confident that valgrind will spot this.

We're going to continue trying to reproduce on our side and disable TLE to confirm that this segv goes away. Hard to know if this is related to the original reported problem, of course, but perhaps losing TLE will allow valgrind to find that if it's a separate issue.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

First, I'm not sure what I was doing wrong yesterday, but I now have the LD_PRELOAD lock-elision-disablement running. And, when running under valgrind, we no longer see the reports from valgrind. I'm now running without valgrind to see whether we still observe stack corruption.

A few comments on the most recent comments.

Re #40: Bill, we have never actually seen a natural SEGV in __lll_unlock_elision in our tests - instead we see random stack corruption, per my description in #5. We saw some complaints about __lll_lock_elision when running under valgrind, but some comments in this thread have made me question whether valgrind actually understands what is happening there.

Re #38: We were thinking the same thing. I will ask around to see if we have any appropriate machines.

Re #37: I don't think it is an option for us as almost all of our locks are wrapped pthread locks via std::mutex.

Re #35: Almost all of our lock management is via C++ RAII types (std::unique_lock, etc), and we run these tests across many many systems. I'm almost certain that a double unlock would have been caught by now, or would cause crashes elsewhere. In particular I would expect that the windows debug runtime would alert us. However, I will see if there is some easy way we can check for this.

I'm running the original repro right now with HLE disabled. Usually it takes a few hours before we see a stack corruption event, so I will follow up later today with more results.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

------- Comment From <email address hidden> 2016-11-11 14:14 EDT-------
I've been able to set up and reproduce this on a p8 16.04 system here. Seems to be the identical signature, 2 bytes at address 0xXXXXXe in the middle of the canary block set to 0x00.

That is good news that you have been able to reproduce the issue. I'm currently running the reproducer with the LD_PRELOAD disable-lock-elision hack in place, without valgrind, and I'm currently at 55 runs with no crashes. I will let it run overnight.

Also, per the earlier comment about double unlocking: I confirmed with one of the other engineers here that the Windows Debug CRT does check for double unlocks, and this test suite runs in our CI loop in that configuration without issue.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos
- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

I'll note that the LD_PRELOAD interposer library is only needed for binaries that are already compiled and you want to override the pthread_mutex_lock() routine. If you can recompile your source, then you can place the interposer directly into your source and there is no need for LD_PRELOADing anything. Ie, you could just add the following to any of your source files:

#define PTHREAD_MUTEX_NO_ELISION_NP (512)

extern int __pthread_mutex_lock (pthread_mutex_t *);
int
pthread_mutex_lock (pthread_mutex_t *mutex)
{
  mutex->__data.__kind |= PTHREAD_MUTEX_NO_ELISION_NP;
  return __pthread_mutex_lock (mutex);
}

...and possibly wrap the above in:

#ifdef __powerpc__
...
#endif

so it's only used on POWER?

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Question: Is there any magic I can do to this test case:

python buildscripts/resmoke.py --suites=concurrency_sharded --storageEngine=wiredTiger --excludeWithAnyTags=requires_mmapv1 --dbpathPrefix=... --repeat=500 --continueOnFailure

that would allow me to run multiple copies on the same machine?

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Found it, looks like the --basePort option to resmoke is what I want.

Arron, re #50, yes, you can run as may copies as you want simultaneously, as long as:

1) The --dbpathPrefix argument points to distinct paths. So resmoke.py ... --dbpathPrefix=/var/tmp/run1 and resmoke.py ... --dbpathPrefix=/var/tmp/run2, etc.

2) You specify disjoint "port ranges" with the --basePort argument to resmoke.py. I don't happen to know how deep in to each port range one instance of the test needs to go, so I usually separate them by 10k but that is probably way too generous. I'll bet 2k is sufficient.

Does that answer your question?

Peter, re #47, yes, that is certainly true. However, I'm actually finding it advantageous to load it via LD_PRELOAD exactly because I don't need to recompile. So I can toggle back and forth between lock elision on/off without needing to recompile.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Ahh, if you're never actually seeing a SEGV in __lll_unlock_elision, this may indeed be something far more subtle than a double unlock and, in fact, I'd be even MORE interested in having you run the same batch of testing on an otherwise identical (ie: Ubuntu 16.04, blah blah blah) Skylake system.

It may well be a subtle glibc bug on powerpc, or a whackadoo silicon bug, or a more generic glibc bug or still a mongo bug, but the more data we can scrape up (I mean, unless you accidentally stumble on the bug in the process and fix it, then yay), the better.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Andrew,
  Yes, that is working nicely with separate DB dirs and basePort I'm running multiple copies on one machine. Thanks!

Adam I agree on all points.

So far, my repro running with the LD_PRELOAD hack is at 118 iterations with no crashes and going strong. Given that we had an ~5% repro rate without the LD_PRELOAD hack, this is looking very encouraging, but I'm going to let it run all weekend just to be sure.

As for skylake, we are definitely going to work on tracking one down next week.

And of course, our default stance has always been that it is more likely our bug than anything else, but you never know! We will continue to run experiments and gather data.

Thanks again for your help.

This is the other thing I am trying. I've modified the Canary object to use a 128k stack zone and then use mprotect to mark the aligned 64k page that's in the middle of it read-only. When the destructor is called, it changes it back to read-write. This should cause any write to this region to get a segv, and give us an idea of what is writing on the stack in the resulting coredump.

One other thing, if you use the mprotect thing, it may be necessary to bump up the value of /proc/sys/vm/max_map_count, depending on how many of these Canary objects get constructed.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

An engineer on our side did some Canary+mprotect experiments as well, but I don't happen to have details on what the approach/results were right now. I'll ask them to update this ticket with any interesting findings they may have.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Here's another interesting data point. The original bug description specifies that the memory corruption is not seen on Ubuntu 15. Per https://bugzilla.linux.ibm.com/show_bug.cgi?id=117535, however, transactional lock elision has been enabled by default since Ubuntu 15.04 (glibc 2.21). Yet on 16.04, the use of TLE is sufficient to cause the stack corruption.

This seems to deepen the mystery more than it illuminates it. Have there been changes to TLE between the releases that could be at fault? Is another unknown component involved?

I'm told this morning that so far no failures are observed using the mprotect canary, the working theory being that the syscall disturbs the timing too much. Otherwise our results are consistent with yours on 16.04: failures with TLE enabled, no failures with the LD_PRELOAD workaround.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Regarding Ubuntu 15, I think that was a miscommunication somewhere along the line.

The only versions of Ubuntu that we build for are the LTS releases (12.04, 14.04, and 16.04), and the only one of those we have ever built on POWER is 16.04. Other than Ubuntu 16.04, the only other POWER distro we target is RHEL 7.1. So what we are seeing is I think still consistent with either a latent locking bug in MongoDB exposed by the new-to-us lock elision support, or one of the other sorts of issues as mentioned by Adam above.

Also, my test run using the disabled lock elision just reached 500 successful runs. I'm going to switch lock elision back on and make sure I start seeing corruption again, but I'm nearly certain that I will.

Thanks for the update on the mprotect experiment. It would have been great if that had worked. I think we will be able to share some details on our similar experiments on Monday.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

An update on my experiments:
* 500 runs no failures with TLE disabled
* 500 runs no failures TLE enabled but mprotect() syscall in Canary constructor/destructor
* 500 runs 11 failed with TLE enabled so about 2% fail rate
* Tried switching SMT off and interestingly got 200 runs no fails with TLE enabled.

This suggests to me that the timing of this race condition is rather tight. A lot of the fails are in the checksum right after the memset in the Canary constructor, which means the other guy comes back and writes the stack after memset wrote it but before the checksum read it. Also if strace's syscall timing is to be believed, 14% of the time mprotect() is under 8 microseconds, and even that seems to be sufficient to prevent the problem. Finally, by forcing other pthreads to always be on a different processor core (by disabling SMT) we also apparently eliminate this.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Hello, could bugproxy please be silenced, and/or prevented from reposting the same python command line over and over again? It seems there are sometimes attachments/comments that get amplified and re-posted with every other comment.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

We'll try to get to the bottom of the bugproxy oddity. It seems to be echoing a chunk out of comment #5 for no reason that I can see.

- python ./buildscripts/scons.py CC=/usr/bin/gcc CXX=/usr/bin/g++ CCFLAGS="-mcpu=power8 -mtune=power8 -mcmodel=medium" --ssl --implicit-cache --build-fast-and-loose -j ./mongo ./mongod ./mongos

Given the results of Aaron's experiments over the weekend, here's a summary of what we think we're seeing:
 - There is an unsafe interaction between two threads.
 - This interaction can only be observed in a very small time window, and then relatively rarely.
 - The interaction is observable only when TLE is enabled. We posit this is because TLE improves lock/unlock speed, so that the observable time window applies.
 - The time window is sufficiently small that a relatively fast syscall to mprotect is sufficient to disrupt the timing.
 - If all threads are forced to run on separate processors (SMT=1), this is sufficient to disrupt the timing.

We believe this is an application problem. Further "evidence" against a problem in TLE itself is that TLE has been enabled for ppc64el on Ubuntu for over 18 months without any similar reports.

Unfortunately, I don't think we have any way to directly debug the problem, due to the narrow time window. We have considered hardware watchpoints. There is a DAWR (data address watch register) for each thread, but using it in this case seems impractical. GDB's implementation of hardware watchpoints in a multithreaded environment is such that, when a hardware watchpoint is set, it is set to the same address for all threads. So even if we were to script the setting of a hardware watchpoint under GDB, the time required for GDB to set up the watchpoint address on the fly would surely exceed the critical time window. You could try it, but I wouldn't expect much.

Further debugging seems to require application knowledge or a code crawl of some kind. The setting of two flag bytes is the only clue we have. To my knowledge we have never seen only a single byte clobbered in the canary, and the two bytes appear to be aligned on a 16-bit boundary. So the code doing the clobbering very likely contains a store-halfword (sth or, less likely, sthx or sthu) instruction. You could examine the disassembly of the application for occurrences of sth to narrow the field of search.

It could be two individual stores, in which case you'd be looking for two instructions very close together of the form:

  stb r<x>,<n>(r<b>)
  stb r<y>,<n+1>(r<b>)

Example:

  stb r5,0(r9)
  stb r6,1(r9)

Obviously this is a huge application so this doesn't help much in and of itself, but perhaps if you've already narrowed the problem down somewhat, this could be helpful.

I am running low on ideas for how we can help you debug the problem. We'll continue discussing it; if any better thoughts arise, we'll be sure to let you know.

Bill

Well, let me back that off a little. We're going to look into the TLE code a little more. The various __lll_*_elision routines are handed a pointer to short that they update, which certainly looks suspicious. So it's certainly possible that something in the pthreads implementation that calls this code is providing a bad pointer to TLE.

However, we would expect to see the same problem on x86 or s390 if that is the case, unless there is some POWER-specific code in the pthreads implementation. So again a Skylake experiment would be helpful.

Ulrich Weigand made an interesting comment on the glibc code in our internal bug (no longer mirrored), so I'm mirroring it by hand here.

--- Comment #77 from Ulrich Weigand <email address hidden> ---
According to comment #45, we see an invalid read at __lll_unlock_elision
(elision-unlock.c:36) and and invalid write at the next line. Looking at the
code, those are:

int
__lll_unlock_elision (int *lock, short *adapt_count, int pshared)
{
  /* When the lock was free we're in a transaction. */
  if (*lock == 0)
    __libc_tend (0);
  else
    {
      lll_unlock ((*lock), pshared);

      /* Update the adapt count AFTER completing the critical section.
         Doing this here prevents unneeded stalling when entering
         a critical section. Saving about 8% runtime on P8. */
      if (*adapt_count > 0)
        (*adapt_count)--;
    }
  return 0;
}

exactly the two lines accessing *adapt_count (which is a short).

What seems suspicious here is the fact that these lines happen (deliberately)
*outside* of the critical section protected by the mutex. Now this should
indeed be normally OK. However, I understand it is allowed to use that
critical section to also protect the life time of the memory object holding the
mutex itself. If this is done, I'm wondering whether those two lines outside
the critical section may now access that *memory* outside of its protected life
time ...

Just as a theoretical example, assume you have thread 1 executing a function
that allocates a mutex on its stack, locks the mutex, and passes its address to
a thread 2. Thread 2 now does now work and then unlocks the mutex. Thread 1,
in the meantime, attempts to lock the mutex again, which will block until it
was unlocked by thread 2. After it got the lock, thread 1 now immediately
unlocks the mutex again and then the function returns.

Now in this scenario, the cross-thread lock-unlock pair cannot use TM, so it
will fall back to regular locks, which means the unlock in thread 2 will
perform the *adapt_count update. On the other hand, the second lock-unlock
pair in thread 1 might just happen to go through with TM, and thus not touch
*adapt_count at all. This means there is no synchronization between the update
of *adapt_count in thread 2, and the re-use of the stack memory underlying the
mutex in thread 1, which could lead to the *adapt_count update being seen
*after* that reuse in thread 1, clobbering memory ...

Am I missing something here? Of course, I'm not sure if MongoDB shows that
exact pattern. But I guess different patterns might show similar races too.

One simple test might be to move the *adapt_count update to before the
lll_unlock and check whether the issue still reproduces. Not sure about
performance impacts, however.

I think it is very likely that we are doing the sort of stack-based mutex pattern described above, or something similar. In particular, I'd expect that we certainly have states where we wait on a stack mutex, and then immediately unwind and destroy the mutex after we unblock.

I'm working on getting access to a Skylake machine - we will definitely do that test if we can get access.

So, I rebuilt the glibc 2.23 from the 16.04 sources and modified the values written to the adapt_count parm in the lock elision code. It's a short and the original code may store values 0, 1, 2, 3. We were seeing either 1 (canary hit in constructor) or 0 (canary hit in destructor). I changed it to use the values 0x3333, 0x2222, 0x1111, and 0. And I just saw the constructor canary hit with the expected values 0x11, 0x11 in the changed bytes. So, this is a race condition in the lock elision code with mutex located on the stack and being reused quickly by another hardware thread on the same processor core.

Aaron thank you very much for running that experiment and confirming that this is an issue in libc. I think the component should probably be updated?

Also, would you like us to try to continue to repro on a Skylake machine, or is this all architecture neutral code and therefore the POWER repro is conclusive?

Hi Andrew,

This is a POWER-specific "optimization" that dates to last December (so it in fact wouldn't show up in Ubuntu 15.04 or 15.10, it appears). The decrement used to be attached to the lock rather than the unlock and it was apparently moved to the present location because it showed a good performance improvement. Aaron is running some experiments with moving it ahead of the unlock rather than following it, which we believe will remove the problem. That may or may not be the final version of the fix, but it looks like the simplest solution.

The only reason for you to test on a Skylake machine would be for your own peace of mind, in case any TLE-specific issues are hiding in the x86 implementation. This specific problem, at least, is a POWER issue.

Hi Bill -

Thanks for the update, and for clarifying that this is POWER 16.04 only. We are very happy to be at a root cause for this issue - it had us pretty worried! We really appreciate all the help from everyone involved here.

Will there be an upstream glibc bug associated with this ticket that we can follow?

Hi Andrew,

Aaron has just opened https://sourceware.org/bugzilla/show_bug.cgi?id=20822. An odd confluence of vacations, world holidays, and family leave has conspired to take all of our glibc experts out of the office tomorrow, so we may be a little delayed with testing and submitting the final fix, thought the rest of us will do what we can with it. We are likewise very relieved to have proof of a cause! And we're certainly sorry for all the trouble this has caused you.

Our team will work with the Ubuntu toolchain guys to get a solution in the field as soon as possible. Thanks again for your patience and professionalism!

Bill

Hi Bill -

Thanks for the glibc bug link.

Totally understand about people being out, not a problem. However, I'm not very familiar with the development process for upstream glibc fixes to make their way into an LTS release.

Do you have a rough estimate of the timeline for that landing in something that we can pull in via apt-get update? Are we talking days, weeks, months? My expectation right now is that we are going to continue forward with issuing the MongoDB 3.4 GA on POWER Ubuntu 16.04, and relnote this issue, probably stating that MongoDB 3.4 should not be run in production on POWER Ubuntu 16.04 unless glibc has been upgraded to version X or newer.

We might even add a server startup warning, if it will be possible for us to detect at runtime that we are running with an affected glibc. I'd imagine that might prove difficult if this comes in as an Ubuntu patch to the existing libc version though. Thoughts on that?

Andrew

Hi Andrew -

I don't work directly in the glibc community, so I'm not completely familiar with their policies. However, the first step is to get a bug approved upstream so that it can be backported to the 2.23 release (in this case). Adam Conrad at Canonical has volunteered to help us shepherd the patch along, so we hope to be able to expedite it.

We have two possible patches to fix the problem; we are trying to determine which will have the least performance impact. I am hopeful that we can get a decision on that tomorrow so the patch can be put on the libc-alpha list and, we hope, approved without too much difficulty.

I will have to defer to Canonical folks (Adam, Matthias) on the process for getting the fix available, but I'm sure this isn't their first rodeo and this shouldn't take too long. My best guess is that the whole process may get done within a week, if all goes well with the community, and a little longer if not. We are definitely not talking months.

Patch submission is here: https://sourceware.org/ml/libc-alpha/2016-11/msg00568.html

Hi Andrew,

Canonical's plans for handling this in the short term are described here: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1642390. We will continue to work the POWER patch, but the SRU described there is all you should need to track for your relnote.

Bill

Howdy, I'm the originator of:

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1641241

(which is dup'd to this bug)

I tested the new ("ubuntu5") libc6 packages from xenial-proposed. They prevent the crash with TensorFlow, and I have not noticed any other problems.

Will update:

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1642390

Status changed to 'Confirmed' because the bug affects multiple users.

This bug was fixed in the package glibc - 2.24-9ubuntu2

---------------
glibc (2.24-9ubuntu2) zesty; urgency=medium

  * debian/patches/any/cvs-resolv-internal-qtype.diff: Revert to avoid
    failure in name resolution on upgrades from yakkety (LP: #1674532)

 -- Adam Conrad <email address hidden> Tue, 21 Mar 2017 15:27:15 -0600