Segfaults trying to send crash report

Bug #1850608 reported by Brian Foster
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
whoopsie (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

After this morning's upgrade to "whoopsie 0.2.62ubuntu0.2"
plus "libwhoopsie0 0.2.62ubuntu0.2"(fix to bug #1830865),
whoopsie repeatedly crashes with a segfault (in what seems to be libc?).
An excerpt from /var/log/syslog (slightly redacted):

Oct 30 08:33:06 [REDACTED] systemd[1]: Started crash report submission daemon.
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] Using lock path: /var/lock/whoopsie/lock
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] The default IPv4 route is: /org/freedesktop/NetworkManager/ActiveConnection/1
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] Not a paid data plan: /org/freedesktop/NetworkManager/ActiveConnection/1
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] Found usable connection: /org/freedesktop/NetworkManager/ActiveConnection/1
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] Parsing /var/crash/_usr_bin_kdeinit5.1000.crash.
Oct 30 08:33:06 [REDACTED] whoopsie[22414]: [08:33:06] Uploading /var/crash/_usr_bin_kdeinit5.1000.crash.
Oct 30 08:33:10 [REDACTED] kernel: [ 2129.099314] whoopsie[22414]: segfault at 7f4f878a4000 ip 00007f4f866fe5e1 sp 00007ffe348bed78 error 4 in libc-2.27.so[7f4f86591000+1e7000]
Oct 30 08:33:11 [REDACTED] systemd[1]: whoopsie.service: Main process exited, code=dumped, status=11/SEGV
Oct 30 08:33:11 [REDACTED] systemd[1]: whoopsie.service: Failed with result 'core-dump'.
Oct 30 08:33:11 [REDACTED] systemd[1]: whoopsie.service: Service hold-off time over, scheduling restart.
Oct 30 08:33:11 [REDACTED] systemd[1]: whoopsie.service: Scheduled restart job, restart counter is at 133.
Oct 30 08:33:11 [REDACTED] systemd[1]: Stopped crash report submission daemon.

The above sequence of logged events repeats endlessly,
with only(?) the times and whoospie PID changing.

The problem can be (temporarily) stopped by `service whoopsie stop';
using `service whoopsie start' resumes the above behaviour.
Rebooting does not "fix".

Manually running whoopsie shows the same behaviour:

$ sudo /bin/sh -c 'export CRASH_DB_URL=https://daisy.ubuntu.com; exec whoopsie -f'
[09:45:14] Using lock path: /var/lock/whoopsie/lock
[09:45:14] The default IPv4 route is: /org/freedesktop/NetworkManager/ActiveConnection/1
[09:45:14] Not a paid data plan: /org/freedesktop/NetworkManager/ActiveConnection/1
[09:45:14] Found usable connection: /org/freedesktop/NetworkManager/ActiveConnection/1
[09:45:14] Parsing /var/crash/_usr_bin_kdeinit5.1000.crash.
[09:45:14] Uploading /var/crash/_usr_bin_kdeinit5.1000.crash.
Segmentation fault
$
$ ls -la /var/crash/
total 33340
drwxrwsrwt 2 root whoopsie 4096 Oct 30 08:52 .
drwxr-xr-x 16 root root 4096 Sep 14 2018 ..
-rw-r--r-- 1 root whoopsie 281 Oct 30 08:55 kexec_cmd
-rwxrwxrwx 1 root whoopsie 0 Oct 30 08:22 .lock
-rw-r----- 1 blf whoopsie 17624101 Oct 30 08:53 _usr_bin_kdeinit5.1000.crash
-rw-rw-r-- 1 blf whoopsie 0 Oct 30 08:22 _usr_bin_kdeinit5.1000.upload
-rw-r----- 1 root whoopsie 398157 Oct 24 11:49 _usr_bin_sddm.0.crash
-rw-r----- 1 whoopsie whoopsie 15996866 Oct 30 08:22 _usr_bin_whoopsie.108.crash
-rw-r----- 1 root whoopsie 99379 Oct 22 15:07 _usr_sbin_inetd.0.crash
$

Because (as far as I know) the various `.crash' files MAY contain sensitive information,
I have not attached them to this (public) bug report.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: whoopsie 0.2.62ubuntu0.2
ProcVersionSignature: Ubuntu 4.15.0-66.75-generic 4.15.18
Uname: Linux 4.15.0-66-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.8
Architecture: amd64
Date: Wed Oct 30 09:28:01 2019
InstallationDate: Installed on 2016-10-07 (1118 days ago)
InstallationMedia: Kubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
RelatedPackageVersions: apport-noui N/A
SourcePackage: whoopsie
UpgradeStatus: Upgraded to bionic on 2018-08-18 (438 days ago)

Related branches

Revision history for this message
Brian Foster (blfoster) wrote :
Revision history for this message
Tiago Stürmer Daitx (tdaitx) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

This bug report has been modified to private. We would highly appreciate if you could upload your crash report here for analysis.

information type: Public → Private Security
Revision history for this message
Brian Foster (blfoster) wrote :

Attached is compressed tar(1) archive (snapshot) I took of /var/crash shortly after filing
this bug report.

I note that when I try the manual-run test now,
the daisy server seems to be off-line?:

$ sudo /bin/sh -c 'export CRASH_DB_URL=https://daisy.ubuntu.com; exec whoopsie -f'
[11:08:23] Using lock path: /var/lock/whoopsie/lock
[11:08:24] The default IPv4 route is: /org/freedesktop/NetworkManager/ActiveConnection/1
[11:08:24] Not a paid data plan: /org/freedesktop/NetworkManager/ActiveConnection/1
[11:08:24] Found usable connection: /org/freedesktop/NetworkManager/ActiveConnection/1
[11:08:24] Parsing /var/crash/_usr_bin_kdeinit5.1000.crash.
[11:08:24] Uploading /var/crash/_usr_bin_kdeinit5.1000.crash.
[11:08:27] Sent; server replied with: Couldn't connect to server
[11:08:27] Response code: 0
^C
$
$ ping daisy.ubuntu.com
PING daisy.ubuntu.com (162.213.33.132) 56(84) bytes of data.
^C
--- daisy.ubuntu.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3060ms
$

The above _suggests_ the problem only(?) happens if `whoopsie' can connect with the server?

Revision history for this message
Brian Foster (blfoster) wrote :

With some help from
https://askubuntu.com/questions/434431/how-can-i-read-a-crash-file-from-var-crash
I have generated as full of a stack backtrace as I can
(please note I have ALL(?) relevant -dbg / -dbgsym packages installed)
using apport-retrace(1) on the `_usr_bin_whoopsie.108.crash' file generated about
when all this started (and hence, I _presume_, is a capture of "the" problem).
It is attached (gzip(1) compressed), and shows libc::memcpy() segfaults after
a call from libcurl... see attached file for details.

I have no idea how to check/verify the contents of a `.crash' file, and so have
no idea what might be "wrong" or "odd" about the `kdeinit' file which seems to be
the input data for the `whoopsie' segfault.

Revision history for this message
Alex Murray (alexmurray) wrote :

Thanks - the crash report can be unpacked via apport-unpack:

$ gunzip _usr_bin_whoopsie.108.crash.gz
$ apport-unpack _usr_bin_whoopsie.108.crash lp1850608

Then the contents can be seen in the lp1850608 directory - looking at StackTrace in particular we see the following:

...
#11 0x0000563d8f963194 in upload_report (message_data=0x7f1dc0b60010 "r\276\003", message_len=140728898666098, s=0x7ffe5de79f00) at src/whoopsie.c:328
        curl = 0x563d9001f8e0
        result_code = CURLE_OK
        response_code = 0
        list = 0x563d90027190
        __func__ = "upload_report"
...

And that message_len looks suspiciously large

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package whoopsie - 0.2.66ubuntu0.2

---------------
whoopsie (0.2.66ubuntu0.2) eoan-security; urgency=medium

  * SECURITY REGRESSION: segfault when sending crash report (LP: #1850608)
    - lib/bson/bson.c: properly initialize value.

 -- Marc Deslauriers <email address hidden> Wed, 30 Oct 2019 08:58:38 -0400

Changed in whoopsie (Ubuntu):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package whoopsie - 0.2.52.5ubuntu0.3

---------------
whoopsie (0.2.52.5ubuntu0.3) xenial-security; urgency=medium

  * SECURITY REGRESSION: segfault when sending crash report (LP: #1850608)
    - lib/bson/bson.c: properly initialize value.

 -- Marc Deslauriers <email address hidden> Wed, 30 Oct 2019 09:03:35 -0400

Changed in whoopsie (Ubuntu):
status: New → Fix Released
Revision history for this message
Brian Foster (blfoster) wrote :

I suspect this is what is happening:

  whoopsie.c::parse_and_upload_report() calls
  whoopsie.c::bonsify() to get some length (a size_t) using
  bson.c::boson_size(), which calls
  platform.h::bson_little_endian32() to convert an unknown
   value (presumably 32-bits) of unknown endianness into an
   size_t (64-bits) with opaque endianness.

A look at bson_little_endian32() shows it transfers only 4-bytes.
However, a size_t is 8-bytes (on my 64-bit system),
and those bytes are being copied into an *uninitialized* size_t
(in bson_size()). End result is 4 of the 8 bytes returned will
be garbage, hence the bizarre length noted previously.

No fix is proposed, nor do I intend to propose one (sorry!).

Revision history for this message
Brian Foster (blfoster) wrote :

I have just (briefly) looked at the fix as-of "whoopsie 0.2.62ubuntu0.3",
and concur with with the change (which should correct the problem as per
the previous analysis). Whether or not it actually works I can_not_ say,
as the `daisy.ubuntu.com' server is currently not-responding (for what I
presume is very understandable reasons!).

Thank You for the prompt action on this problem!

Revision history for this message
Haw Loeung (hloeung) wrote : Re: [Bug 1850608] Re: Segfaults trying to send crash report

On Wed, Oct 30, 2019 at 06:46:30PM -0000, Brian Foster wrote:

> Whether or not it actually works I can_not_ say, as the
> `daisy.ubuntu.com' server is currently not-responding (for what I
> presume is very understandable reasons!).

The daisy.u.c service has been intentionally throttled to avoid
another DoS from crashing clients retrying. It should be restored to
full service shortly after some time for the latest whoopsie changes
to make it's way out.

Revision history for this message
Brian Foster (blfoster) wrote :

This morning, almost exactly 24hrs after the problem was noted,
my system, using the fixed whoopsie (0.2.62ubuntu0.3),
successfully sent all the pending crash reports (including,
of course, the previous whoopsie's own crash);
confirmed by checking /var/log/syslog.

information type: Private Security → Public
tags: added: id-5db9d1d5e211d16cb03620fe
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.