apport-unpack: ValueError: ['UserGroups'] has no binary content

I get this, too, with a .crash file from qemu that for some reason seems to be truncated after the USerGroups section !?

Attaching the .crash file causing this issue.

[apport-bug also has issues with this file.]

Status changed to 'Confirmed' because the bug affects multiple users.

nazgul@rivendell:~$ dpkg -la apport
Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
| Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
         Halb installiert/Trigger erWartet/Trigger anhängig
|/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht)
||/ Name Version Architektur Beschreibung
+++-=========================================-=========================-=========================-========================================================================================
ii apport 2.17.1-0ubuntu2 all automatically generate crash reports for debugging

I run into the same while debugging some crash of a daemon in Xenial. Installation is a server install. I cannot say whether the unpack fully completes. Fortunately the core file gets extracted at least.

Ubuntu 15.10, apport 2.19.1-0ubuntu5. Trying to unpack to a non-existing directory. Get

     Traceback (most recent call last):
      File "/usr/bin/apport-unpack", line 73, in <module>
        pr.extract_keys(f, bin_keys, dir)
      File "/usr/lib/python3/dist-packages/problem_report.py", line 253, in extract_keys
        [item for item, element in b64_block.items() if element is False])

    $ grep UserGroups /var/crash/_usr_bin_hhvm.33.crash | cat -A
    UserGroups: $
    $

This happens when a crash file is corrupt and does not have any data for a given key. What type of behavior would you expect or prefer in this case?

https://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/problem_report.py#L208

See the raised ValueError for how this occurs.

Umm, don't know, probably print a warning instead of the traceback. Not sure about the exit status - I guess a non zero one is a good idea?

[Expired for apport (Ubuntu) because there has been no activity for 60 days.]

[Expired for apport (Ubuntu Xenial) because there has been no activity for 60 days.]

To my way of thinking, an exception is for something the programmer did not foresee. Since this is now a known failure mode of Apport, I would expect - in an ideal world in which there was enough time to work on all known bugs - a clean exit with a clear error message.

I run into the same problem in trusty, any workaround to bypass it ?

$ sudo apport-unpack _usr_lib_erlang_erts-5.10.4_bin_beam.smp.108.27June2018.crash ./core
Traceback (most recent call last):
File "/usr/bin/apport-unpack", line 74, in <module>
pr.extract_keys(f, bin_keys, dir)
File "/usr/lib/python3/dist-packages/problem_report.py", line 270, in extract_keys
[item for item, element in b64_block.items() if element is False])
ValueError: ['UserGroups'] has no binary content

Then this code dump can not be used because ./core/CoreDump is truncated: expected core file size >= 20734648320, found: 5708352522.
Cannot access memory at address 0x7fe8f266e1a8
Cannot access memory at address 0x7fe8f266e1a0
(gdb) bt
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fe8eb9bd738:
#0 0x00007fe8f1345c37 in ?? ()
Cannot access memory at address 0x7fe8eb9bd738

I simply add a non-empty value into crash file then I can bypass this issue.

If apport-unpack expects a value and cannot have an empty field for UserGroups, that would mean the bug is in report.py. The thing that is generating the totally broken crash report.

report.py will leave the UserGroups field empty if the user is not part of any system groups (GIDs less thans 1000). Also report.py ignores the user's primary group if that is the only system group they are a part of.

It's surprising this has been this way for literally years, since 12.04 at least. I really don't understand why. Especially since most system users are only a part of one group, their own primary group. AND when you run apport-unpack on a crash report with an empty UserGroups field will generate a new crash report for apport-unpack. Like seriously...

P.S. this problem is going to be present in 20.04. So likely unchanged for another 2 years.

This bug was fixed in the package apport - 2.20.11-0ubuntu37

---------------
apport (2.20.11-0ubuntu37) groovy; urgency=medium

  * apport/report.py: If the user is not a part of any system groups then
    set UserGroups to an empty string. (LP: #1427600)

 -- Brian Murray <email address hidden> Tue, 02 Jun 2020 10:19:45 -0700

Unfortunately the fix you added by inserting an empty space string ' ' for the UserGroups field does not resolve the bug at all. This is because when reading the report in (problem_report.py) all fields which are empty get ignored, and a field which is all whitespace counts as empty. Prior to the fix the report already contained the UserGroups field, it just didn't have an empty space after it. So nothing really changed.

This is the new stacktrace when running apport-unpack with a crash report generated with the fixed report.py

Traceback (most recent call last):
  File "/usr/bin/apport-unpack", line 77, in <module>
    pr.extract_keys(f, bin_keys, dir)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 269, in extract_keys
    raise ValueError('%s has no binary content' %
ValueError: ['UserGroups'] has no binary content

The UserGroups field can't be empty. For example if I add a group in the UserGroups field, apport-unpack runs without errors. This is my current fix to not deal with the constant stacktrace from apport-unpack.

sed -i -E 's/(UserGroups:)/\1 nogroup/' /var/crash/_crashfile

An alternative fix for report.py would be if there are no UserGroups DO NOT add the UserGroups field at all. Adding it with an empty string or whitespace is the problem. If it's not present at all apport-unpack will not throw the error.

Hello Iain, or anyone else affected,

Accepted apport into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apport/2.20.11-0ubuntu27.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Latest fix filling in 'N/A' for absent groups is working, apport-unpack is no longer throwing stacktrace when prcoessing the report. I still don't understand why primary groups are ignored, but the important thing is the reports are processable without errors now.

This might be a totally separate issue, but will it be possible to backport the 'N/A' fix to bionic?

Reproduced by testing a user with a system group as their primary group (will have N/A in report), this is equivalent to the user being in no system groups (even though they are). Also a user which has an additional group which is a system group (will show that system group in report).

# system group to test with
sudo addgroup --system testgroup1

# create user with their primary group as the above system group (user doesn't have to be system user, just don't have to fill in gecos)
sudo adduser --system --ingroup testgroup1 testuser1
# create user with additional group as system group, their primary group in this case is nogroup
sudo adduser --system testuser2
sudo usermod -a -G testgroup1 testuser2

# run simple app as first user
sudo -u testuser1 /usr/bin/sleep 60000
# kill and create crash report
sudo pkill -SEGV -f '^/usr/bin/sleep 60000'
report="/var/crash/_usr_bin_sleep.$(id -u testuser1).crash"
# wait a bit while report is generated
sleep 2
# check resulting crash report, should print UserGroups: N/A
sudo grep -H 'UserGroups:' $report
# try apport-unpack
sudo apport-unpack $report /tmp/crash1

# reset and try with other user
rm "$report"
sudo -u testuser2 /usr/bin/sleep 60000
sudo pkill -SEGV -f '^/usr/bin/sleep 60000'
report="/var/crash/_usr_bin_sleep.$(id -u testuser2).crash"
sleep 2
# should print UserGroups: testgroup1
sudo grep -H 'UserGroups:' "$report"
sudo apport-unpack $report /tmp/crash2

I've verified the fix on Ubuntu 20.04 LTS.

bdmurray@clean-focal-amd64:~$ apt-cache policy apport
apport:
  Installed: 2.20.11-0ubuntu27.3
  Candidate: 2.20.11-0ubuntu27.3
  Version table:
 *** 2.20.11-0ubuntu27.3 500
        500 http://192.168.10.7/ubuntu focal-proposed/main amd64 Packages
        500 http://192.168.10.7/ubuntu focal-proposed/main i386 Packages
bdmurray@clean-focal-amd64:~$ sudo apport-unpack /var/crash/_usr_bin_vim.basic.1001.crash /tmp/junk2
No Traceback
bdmurray@clean-focal-amd64:~$ cat /tmp/junk2/UserGroups
N/A

On Sun, Jun 07, 2020 at 06:03:38AM -0000, Nicholas Schell wrote:
> Latest fix filling in 'N/A' for absent groups is working, apport-unpack
> is no longer throwing stacktrace when prcoessing the report. I still
> don't understand why primary groups are ignored, but the important thing
> is the reports are processable without errors now.

The reason primary groups are not included is because apport tries to
exclude some information for privacy reasons and that includes user and
group names.

> This might be a totally separate issue, but will it be possible to
> backport the 'N/A' fix to bionic?

Its possible but not very likely. Could you explain your use case for
using apport-unpack?

--
Brian Murray

We have an automated solution that unpacks the crash report and sends along the relevant information about the crash+stacktrace. Had to initially deal with the fact that apport-unpack was crashing (returning non-zero exit code). Just manually entering in UserGroups info to every crash report prior to processing it now.

I was mostly thinking of the other users which will likely continue to be using the 18.04 LTS for some time before they migrate to 20.04. This has been such a long standing bug and it only takes a few lines of code to fix it. I understand if it won't be backported, as it is very minor and people have been ok with it for the last 5 years almost.

The verification of the Stable Release Update for apport has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package apport - 2.20.11-0ubuntu27.3

---------------
apport (2.20.11-0ubuntu27.3) focal; urgency=medium

  * apport_python_hook.py: if python apt modules are not built for the python
    version then do capture the crash. (LP: #1774843)
  * apport/report.py: If the user is not a part of any system groups then
    set UserGroups to 'N/A'. (LP: #1427600)

 -- Brian Murray <email address hidden> Mon, 01 Jun 2020 09:44:37 -0700

$ lsb_release -cs
bionic

$ apport-unpack /var/tmp/_usr_bin_<binary>.0.crash /tmp/<dir>
Traceback (most recent call last):
  File "/usr/bin/apport-unpack", line 74, in <module>
    pr.extract_keys(f, bin_keys, dir)
  File "/usr/lib/python3/dist-packages/problem_report.py", line 270, in extract_keys
    [item for item, element in b64_block.items() if element is False])
ValueError: ['UserGroups'] has no binary content

I'm having the same stack trace from groovy, when trying to unpack

ii apport 2.20.11-0ubuntu45 all automatically generate crash reports for debugging
ii apport-symptoms 0.23 all symptom scripts for apport
ii python3-apport 2.20.11-0ubuntu45 all Python 3 library for Apport crash report handling

From what release is the crash file which you are trying to unpack?