assert hit in eglibc (valloc) with libtorrent v0.15

Ah ah. I have had the same one since I switched to libtorrent v0.15. However, libtorrent author thinks the problem come from eglibc (Debian and Ubuntu recently switched from glibc to eglibc). I will forward this to him so that he sees I'm not the only one :)

Here is the backtrace with line numbers:

qbittorrent: malloc.c:3929: __libc_valloc: Assertion `!p || ((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x2) || ar_ptr == (((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x4) ? ((heap_info *)((unsigned long)(((mchunkptr)((char*)(p) - 2*(sizeof(size_t))))) & ~((2 * (512 * 1024))-1)))->ar_ptr : &main_arena)' failed.

(gdb) bt 25
#0 0x00ca0422 in __kernel_vsyscall ()
#1 0x006514d1 in *__GI_raise (sig=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0x00654932 in *__GI_abort () at abort.c:92
#3 0x0064a648 in *__GI___assert_fail (
    assertion=0x74bfb0 "!p || ((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x2) || ar_ptr == (((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x4) ? ((heap_info *)((unsigned long)(((mchunkptr)((char*)("...,
    file=0x747efe "malloc.c", line=3929, function=0x748250 "__libc_valloc")
    at assert.c:81
#4 0x006966bf in __libc_valloc (bytes=262152) at malloc.c:3928
#5 0x001fcfcd in libtorrent::page_aligned_allocator::malloc (bytes=262152)
    at allocator.cpp:50
#6 0x0022682a in boost::pool<libtorrent::page_aligned_allocator>::ordered_malloc_need_resize (this=0x867cffc) at /usr/include/boost/pool/pool.hpp:461
#7 0x0021f2f6 in boost::pool<libtorrent::page_aligned_allocator>::ordered_malloc (this=0x867ce88, category=0x3cf8d3 "receive buffer")
    at /usr/include/boost/pool/pool.hpp:232
#8 libtorrent::disk_buffer_pool::allocate_buffer (this=0x867ce88,
    category=0x3cf8d3 "receive buffer") at disk_io_thread.cpp:115
#9 0x002ddd99 in libtorrent::aux::session_impl::allocate_disk_buffer (
    this=0x867cd20, category=0x3cf8d3 "receive buffer")
    at session_impl.cpp:2902
---Type <return> to continue, or q <return> to quit---
#10 0x002ae0a9 in libtorrent::peer_connection::allocate_disk_receive_buffer (
    this=0xb5e2fd60, disk_buffer_size=16384) at peer_connection.cpp:3265
#11 0x0020a555 in libtorrent::bt_peer_connection::on_piece (this=0xb5e2fd60,
    received=1) at bt_peer_connection.cpp:1129
#12 0x00206778 in libtorrent::bt_peer_connection::dispatch_message (
    this=0xb5e2fd60, received=1) at bt_peer_connection.cpp:1580
#13 0x0020b638 in libtorrent::bt_peer_connection::on_receive (this=0xb5e2fd60,
    error=..., bytes_transferred=1) at bt_peer_connection.cpp:2924
#14 0x002b631f in libtorrent::peer_connection::on_receive_data (
    this=0xb5e2fd60, error=..., bytes_transferred=5)
    at peer_connection.cpp:4303
#15 0x002bcda2 in boost::asio::detail::handler_queue::handler_wrapper<boost::asio::detail::binder2<libtorrent::peer_connection::allocating_handler<boost::_bi::bind_t<void, boost::_mfi::mf2<void, libtorrent::peer_connection, boost::system::error_code const&, unsigned int>, boost::_bi::list3<boost::_bi::value<boost::intrusive_ptr<libtorrent::peer_connection> >, boost::arg<1>, boost::arg<2> > >, 256u>, boost::system::error_code, unsigned int> >::do_call(boost::asio::detail::handler_queue::handler*) () from /usr/lib/libtorrent-rasterbar.so.6
#16 0x0025890c in boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false> >::run(boost::system::error_code&) ()
   from /usr/lib/libtorrent-rasterbar.s...

same problem under rc3 and lasted libtorrent build (4069)

debug attached

sorry , apparently it's not the same debug but maybe same origine problem ?)

ilos> yes, it is the same ASSERT hit in malloc.c. What distro and what version of the distro are you using?

i'm using ubuntu 9.10

Ok, so we all are using eglibc instead of the usual glibc. I'm waiting for input from eglibc maintainer.

The ASSERT that is hit was imported recently in eglibc from glibc with the following changelog: (public_vALLOc): Try other arenas in case _int_valloc fails.

Commit Diff is here:
http://www.eglibc.org/cgi-bin/viewcvs.cgi/fsf/trunk/libc/malloc/malloc.c?rev=6938&view=diff&r1=6938&r2=6937&p1=fsf/trunk/libc/malloc/malloc.c&p2=/fsf/trunk/libc/malloc/malloc.c

However, since the hit only seems to happen in eglibc, I made a diff for this function (public_vALLOc) and got only this:
3845,3846c3750,3751
< force_reg (__memalign_hook);
< if (__builtin_expect (hook != NULL, 0))
---
> __memalign_hook;
> if (hook != NULL)

These lines in questions (in the diff) were changed later in eglibc svn by:
http://www.eglibc.org/cgi-bin/viewcvs.cgi/fsf/trunk/libc/malloc/malloc.c?rev=7881&r1=7832&r2=7881 :
- if (hook != NULL)
+ if (__builtin_expect (hook != NULL, 0))

with changelog: Add branch prediction for use of the hooks.

and

http://www.eglibc.org/cgi-bin/viewcvs.cgi/fsf/trunk/libc/malloc/malloc.c?rev=8305&r1=8254&r2=8305 :
- __memalign_hook;
+ force_reg (__memalign_hook);

with changelog: Force hook value into register.

In my last post, I meant that these lines were corrected in glibc (and provided the revisions). This changes have not been reflected in eglibc svn (yet?)

  /* Force a value to be in a register and stop the compiler referring
       to the source (mostly memory location) again. */
    #define force_reg(val) \
      ({ __typeof (val) _v; asm ("" : "=r" (_v) : "0" (val)); _v; })

I have the same problem with deluge svn version and libtorrent 0.15:

gdb --args python /usr/local/bin/deluged -d
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Leyendo símbolos desde /usr/bin/python...(no debugging symbols found)...hecho.
(gdb) run
Starting program: /usr/bin/python /usr/local/bin/deluged -d
[Thread debugging using libthread_db enabled]
[Nuevo Thread 0xb6fb3b70 (LWP 11848)]
[Nuevo Thread 0xb67b2b70 (LWP 11849)]
[Nuevo Thread 0xb5fb1b70 (LWP 11850)]
[Nuevo Thread 0xb55ffb70 (LWP 11851)]
[Thread 0xb55ffb70 (LWP 11851) terminado]
[ERROR ] 20:01:33 init:50 Unable to enable plugin!
[ERROR ] 20:01:33 init:51 'Core' object has no attribute '_timer'
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/deluge-1.2.1-py2.6.egg/deluge/plugins/init.py", line 48, in enable
    self.plugin.enable()
  File "build/bdist.linux-i686/egg/freespace/core.py", line 76, in enable
    if not self._timer:
AttributeError: 'Core' object has no attribute '_timer'
python: malloc.c:3929: __libc_valloc: Afirmación `!p || ((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x2) || ar_ptr == (((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x4) ? ((heap_info *)((unsigned long)(((mchunkptr)((char*)(p) - 2*(sizeof(size_t))))) & ~((2 * (512 * 1024))-1)))->ar_ptr : &main_arena)' fallida.

Program received signal SIGABRT, Aborted.
[Cambiando a Thread 0xb67b2b70 (LWP 11849)]
0xffffe430 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe430 in __kernel_vsyscall ()
#1 0xb7e564d1 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7e59932 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb7e4f648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6
#4 0xb7e9b6ef in valloc () from /lib/tls/i686/cmov/libc.so.6
#5 0xb72f66dd in libtorrent::page_aligned_allocator::malloc (bytes=262152) at allocator.cpp:78
#6 0xb7322c7a in boost::pool<libtorrent::page_aligned_allocator>::ordered_malloc_need_resize (this=0x84deb2c) at /usr/include/boost/pool/pool.hpp:461
#7 0xb7318b86 in boost::pool<libtorrent::page_aligned_allocator>::ordered_malloc (this=0x84de9a8, category=0xb74d3d1a "receive buffer")
    at /usr/include/boost/pool/pool.hpp:232
#8 libtorrent::disk_buffer_pool::allocate_buffer (this=0x84de9a8, category=0xb74d3d1a "receive buffer") at disk_io_thread.cpp:120
#9 0xb73d88f9 in libtorrent::aux::session_impl::allocate_disk_buffer (this=0x84de830, category=0xb74d3d1a "receive buffer") at session_impl.cpp:3399
#10 0xb73ab68a in libtorrent::peer_connection::allocate_disk_receive_buffer (this=0xb4be0c08, disk_buffer_size=16384) at peer_connection.cpp:3306
#11 0xb730566d in libtorrent::bt_peer_connection::on_piece (this=0xb4be0c08, received=1) at bt_peer_connection.cpp:1131
#12 0xb7301058 in libtorrent::bt_peer_connection::di...

Oh yes, there is a simple patch to make libtorrent v0.15 use posix_memalign() instead of valloc().

It works fine but this is only a workaround.

Thank you very much, Christophe Dumez.

Is this bug reported to eglibc developers? Where?

I tried to contact them by mail (without response). Libtorrent author also attempted (I don't think he got any answer either).

This bug report is marked as affecting eglibc so I think eglibc package maintainer also received notifications.

I have reported this error in libtorrent bugtracker: http://code.google.com/p/libtorrent/issues/detail?id=57 and I have sent one email to <email address hidden>.

I think it's bad enough that this error has not been resolved, taking into account the seriousness of the problem and the number of months that have passed since the first report.

Today, one of eglibc's developers answered my email:
"...
You will need to explain what you think the bug in EGLIBC is; the fact that a program crashes with an assertion failure in EGLIBC is not itself strong evidence of a defect in EGLIBC."

It seems that he has not clear if the problem is or not in eglibc...

Well, it could be in libtorrent too... But glibc users don't seem affected so...
In any case, it would - at least - be nice to know what this assert means. There is no useful comment near the assert in the code.

Simon, who did you contact? I sent mails to 4 developers and none of them answered.

Personally, I switched from Ubuntu to Fedora. I'm now using libtorrent v0.15 now without any problem nor patches :P

Farewell eglibc :)

It seems that I have had luck because they don't answer never if you aren't a customer: "I generally do not respond to off-list mail from non-customers.".
I've had to send the error report to this mailing list: <email address hidden> (http://www.eglibc.org/mailing_lists).

Anyway, I am surprised and outraged by the inaction on the part of the libtorrent developers, eglibc developers and Ubuntu developers. None do or say anything about this severe bug!

I think it's a priority to determine the source of the problem and fix it.
Why no other program or library has this problem with eglibc?
Is libtorrent v0.15 the only library that uses this function from eglibc?

Yes, I think that valloc() is not used that often. Also, When I tested valloc() in a separate program in Ubuntu, it worked fine. So the bug is not easy to reproduce outside libtorrent.

Note that I would not say that libtorrent author is unresponsive (although I think he is on holidays right now). He tried to contact eglibc mailing list about the meaning of the assert. He also patched his code so that we could switch easily from valloc() to posix_memalign(). This is the reason why my patch is so small.

And unlike eglibc developer, I do think that hitting an assert in eglibc is proof on a bug in eglibc. If they think that hitting this assert can be due to a bad use of valloc() by libtorrent, then they should point us in the right direction (the assert only has meaning for them).

I have similar problem, with glibc 2.11.1, libtorrent-rasterbar 0.15.0 with and without patch above. Qbittorrent version is 2.2.5.
My distro is PLD.

My deluge + libtorrent 0.15 works fine in karmic with Christophe Dumez's patch (thanks other time). But it crashes without patch.
@uzsolt, are you sure that your problem is relationed with this bug?

I got another report from an OpenSuse user recently saying that he hit the exact same assert with glibc 2.10.1 (and many OpenSuse patches).

It would be interesting insert debug logs here from users using glibc.

@Simon:
qbittorrent: malloc.c:3951: __libc_valloc: Assertion `!p || ((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x2) || ar_ptr == (((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x4) ? ((heap_info *)((unsigned long)(((mchunkptr)((char*)(p) - 2*(sizeof(size_t))))) & ~((2 * (512 * 1024))-1)))->ar_ptr : &main_arena)' failed.

*************************************************************
Catching SIGABRT, please report a bug at http://bug.qbittorrent.org
and provide the following backtrace:
stack trace:
  [0xb7748400]
  [0xb7748430]
  /lib/libc.so.6 : gsignal()+0x47 [0xb641a577]

As I see, the log message is similar to original.

I have what looks like the same problem.
Distro: Archlinux (i686)
libtorrent: 0.15.0
deluge: 1.2.3
glibc: 2.11.1

$ gdb --args python /usr/bin/deluged -l deluged-debug-7.log -L debug -d
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/bin/python /usr/bin/deluged -l deluged-debug-7.log -L debug -d
[Thread debugging using libthread_db enabled]
[New Thread 0xb6f3bb70 (LWP 7309)]
[New Thread 0xb673ab70 (LWP 7310)]
[New Thread 0xb5f39b70 (LWP 7313)]
[New Thread 0xb536cb70 (LWP 7314)]
[New Thread 0xb4b6bb70 (LWP 7315)]
python: malloc.c:3951: __libc_valloc: Assertion `!p || ((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x2) || ar_ptr == (((((mchunkptr)((char*)(p) - 2*(sizeof(size_t)))))->size & 0x4) ? ((heap_info *)((unsigned long)(((mchunkptr)((char*)(p) - 2*(sizeof(size_t))))) & ~((2 * (512 * 1024))-1)))->ar_ptr : &main_arena)' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb6f3bb70 (LWP 7309)]
0xb7fe1424 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7fe1424 in __kernel_vsyscall ()
#1 0xb7d33561 in raise () from /lib/libc.so.6
#2 0xb7d34d62 in abort () from /lib/libc.so.6
#3 0xb7d74d64 in __malloc_assert () from /lib/libc.so.6
#4 0xb7d7931b in valloc () from /lib/libc.so.6
#5 0xb723049d in libtorrent::page_aligned_allocator::malloc(unsigned int) () from /usr/lib/libtorrent-rasterbar.so.6
#6 0xb725cf8a in boost::pool<libtorrent::page_aligned_allocator>::ordered_malloc_need_resize() () from /usr/lib/libtorrent-rasterbar.so.6
#7 0xb7252b46 in libtorrent::disk_buffer_pool::allocate_buffer(char const*) () from /usr/lib/libtorrent-rasterbar.so.6
#8 0xb7253ece in libtorrent::disk_io_thread::read_into_piece(libtorrent::disk_io_thread::cached_piece_entry&, int, int, int, boost::unique_lock<boost::recursive_mutex>&) () from /usr/lib/libtorrent-rasterbar.so.6
#9 0xb7256d15 in libtorrent::disk_io_thread::copy_from_piece(std::_List_iterator<libtorrent::disk_io_thread::cached_piece_entry>, bool&, libtorrent::disk_io_job const&, boost::unique_lock<boost::recursive_mutex>&) () from /usr/lib/libtorrent-rasterbar.so.6
#10 0xb7257baa in libtorrent::disk_io_thread::try_read_from_cache(libtorrent::disk_io_job const&) ()
   from /usr/lib/libtorrent-rasterbar.so.6
#11 0xb72596a8 in libtorrent::disk_io_thread::operator()() () from /usr/lib/libtorrent-rasterbar.so.6
#12 0xb72604d3 in boost::detail::thread_data<boost::reference_wrapper<libtorrent::disk_io_thread> >::run() ()
   from /usr/lib/libtorrent-rasterbar.so.6
#13 0xb6fd0095 in thread_proxy () from /usr/lib/libboost_thread-mt.so.1.41.0
#14 0xb7e5594c in start_thread () from /lib/libpthread.so.0
#15 0xb7dd6eae in clone () from /lib/libc.so.6

Contents of deluged-debug-7...

@jack: I reported it on ArchLinux bug tracker: http://bugs.archlinux.org/task/19072

@Christophe Dumez: Nice. I've just finished building the patched libtorrent. I'll try it and report back.

@jack: oh, It is confirmed the patch works fine. However, it is likely that you will experience other crashes in libtorrent v0.15. There were several issues reported against v0.15.0 already.
If you really want stability, I suggest you use v0.14.10 until v0.15.x gets more mature.

Looks like it works.

It seems this is caused by a bug in eglibc. This bug has been fixed in v2.12 but Ubuntu Lucid ships v2.11.1.

2010-02-24 Ulrich Drepper <email address hidden>
 [BZ #5553]
 * malloc/malloc.c (public_vALLOc): Set ar_ptr when trying main_arena.
 (public_pVALLOc): Likewise.
 Patch by Petr Baudis.

Maybe this patch could be applied to the Ubuntu Lucid package?

Here is where the bug was originally reported and solved:
http://sourceware.org/bugzilla/show_bug.cgi?id=5553

In lucid-proposed, awaiting ubuntu-sru approval

I mean: In lucid-proposed unaccepted queue, awaiting ubuntu-sru approval

ACK from SRU team

Accepted eglibc into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Any tester experiencing this problem could give a try to the package in lucid-proposed and report here ?

Thanks in advance.

Where is the test case?

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".