[UBUNTU 21.10] qemu: target/s390x: Fix translation exception on illegal instruction

@IBM Please could you provide a more step-by-step test case?
That would allow us to verify the package once the fix got picked up.

Thanks for the report, it LGMT and makes sense - although uretprobe inside of s390x emulation is super rare and therefore not urgent IMHO.
I'd make this part of the qemu 6.0 (it is applied upstream after that) that I'll prep for Ubuntu 21.10 and once that is completed consider SRUs.

If you are not ok with this plan of action please speak up.

And to help the eventually happening SRU of this - an easy to use testcase would be very helpful.

------- Comment From <email address hidden> 2021-06-01 09:37 EDT-------
Here is a small test that doesn't require installing extra tools or building code:

echo "r:bash_readline /bin/bash:0x$(nm -D /bin/bash | awk '/T readline$/ {print $1}')" >/sys/kernel/debug/tracing/uprobe_events
echo 1 >/sys/kernel/debug/tracing/events/uprobes/bash_readline/enable
cat /sys/kernel/debug/tracing/trace

With the unfixed qemu you will lose your shell (provided it's bash) after step 2.

This bug was fixed in the package qemu - 1:6.0+dfsg-1~ubuntu3

---------------
qemu (1:6.0+dfsg-1~ubuntu3) impish; urgency=medium

  * d/p/u/lp-1935617-target-ppc-Fix-load-endianness-for-lxvwsx-lxvdsx.patch:
    fix TCG emulation for ppc64 (LP: #1935617)

qemu (1:6.0+dfsg-1~ubuntu2) impish; urgency=medium

  * d/control: remove fuse2 trial-build (LP 1934510)

qemu (1:6.0+dfsg-1~ubuntu1) impish; urgency=medium

  * Merge with Debian experimental, Among many other things this fixes LP Bugs:
    (LP: #1907952) broken arrow keys in -display gtk on aarch64
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type
      (LP: 1304107 1621042 1776189 1761372 1761372 1776189)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types containing release versioned machine attributes
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
        for host-phys-bits=true
      - Add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - Let qemu-utils recommend sharutils
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/control-in: Disable capstone disassembler library support (universe)
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - d/control*, d/rules: disable xen by default, but provide universe
      package qemu-system-x86-xen as alternative
      [includes compat links changes of 5.0-5ubuntu4]
    - Fix upgrade module handling (LP 1905377)
      --enable-module-upgrades for qemu-xen which doesn't exist in Debian
  * Dropped Changes [in 6.0]:
    - d/p/ubuntu/lp-1907789-build-no-pie-is-no-functional-liker-flag.patch: fix
      ld usage of -no-pie (LP 1907789)
    - d/p/u/lp-1916230-hw-s390x-fix-build-for-virtio-9p-ccw.patch: fix
      virtio-9p-ccw being missing (LP 1916230)
    - d/p/u/lp-1916705-disas-Fix-build-with-glib2.0-2.67.3.patch: Fix FTFBS due
      to glib2.0 >=2.67.3 (LP 1916705)
    - d/p/u/lp-1921754*: add EPYC-Rome-v2 as v1 missed IBRS and thereby fails
      on some HW/Guest combinations e.g. Windows 10 on Threadripper chips
  ...

Ubuntu 20.10 (Groovy Gorilla) reached its End of Life on July 22 2021, hence I'm updating now the groovy entry of this ticket to Won't Fix.

Hi again,
I know it is prio-low for everyone, but still thanks for your patience!
Plenty of other things now finally are out of the way, so I tried to test the current broken'ness of this before starting a SRU.

1. Create a s390x KVM guest, for example via, but not limited to:
$ uvt-kvm create --password=ubuntu testguest release=focal arch=s390x label=daily
# Note: do so with the various qemu versions that are to be tested.

In my example I had
Focal: 1:4.2-3ubuntu6.17
Hirsute: 1:5.2+dfsg-9ubuntu3.1

2. In that guest prep the load so kindly outlined in comment #3 (slightly adapted to match)
Note you have to run those as root
# prereq
$ apt install binutils
# define probe
$ echo "r:bash_readline /usr/bin/bash:0x$(nm -D /usr/bin/bash | awk '/T readline$/ {print $1}')" > /sys/kernel/debug/tracing/uprobe_events

# check if the probe is active and enable it
$ cat /sys/kernel/debug/tracing/uprobe_events
r:uprobes/bash_readline /usr/bin/bash:0x00000000000dd250
$ echo 1 >/sys/kernel/debug/tracing/events/uprobes/bash_readline/enable

# Start bash to trigger some traces
$ /usr/bin/bash -c 'echo $BASH_VERSION'
5.0.17(1)-release

# Check if traces happened
$ cat /sys/kernel/debug/tracing/trace
root@testguest2:~# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 2/2 #P:1
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
            bash-1653 [000] d... 540.252440: bash_readline: (0x2aa377b590e <- 0x2aa3785d250)
            bash-1653 [000] d... 555.642396: bash_readline: (0x2aa377b590e <- 0x2aa3785d250)

So far this just works, I see the uretprobes happening in the trace.
But neither my test bash nor the one that I'm in logged into the guest crashed/stopped in any way.
This is true for both tested qemu versions.

Is this only affecting emulated s390x?
Is there something else I miss to recreate this?

------- Comment From <email address hidden> 2021-10-05 09:25 EDT-------
Hi,

this issue affects only TCG (i.e. non-KVM) setups. Other repro steps look correct, so could you please try this without KVM (preferably on x86_64, just to be sure)?

Best regards,
Ilya

Ok, emulation it shall be then ...
Finalized test steps:

1.
On x86 get an emulated s390x guest the way you prefer
Example with uvtool + modifications:
$ sudo apt install qemu-system-s390x
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=s390x label=daily release=focal
$ uvt-kvm create --password=ubuntu qemu-s390x-tcg release=focal arch=s390x label=daily
$ virsh destroy qemu-s390x-tcg
That image will obviously fail to run in x86, so adapt the Guest XML to use tcg
Use something like:
<domain type='qemu'>
  <name>qemu-s390x-tcg</name>
  <uuid>cfac1691-2da7-49bb-bf09-c7f0cdefcf85</uuid>
  <metadata>
    <uvt:ssh_known_hosts xmlns:uvt="https://launchpad.net/uvtool/libvirt/1">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYv7mW5g+YoZH1w7v0BNTjfRA/IWM4xSKyH6zwly4W0Zl6pq1XZzO/HyoRDDVb9cXjzggyrMAKTY9749uGwg1X92OEEyJ5nOcuuhNcZJ2bgmWfvo+EqzYS8WR271Iuz3w+BQzpW6XQj7G/tnQPlnmkboFyz0UWdPhlVaBuS6LQbnRHrPWe/gc20NgOaRQf9Jq5GmXO4YywYiXJpQgqu4vrEwr1d/LQcd4oZMnErvLsCZ0pUlQK3WYHSk3F2kWaBigJsJHS6zhkGUdAFlIEqSbjuNuhuyFMdRUyAvHnY7whXOHKUsChnSinqescwfSjk2dHEOqxeIL1AyYxAzRtRd6U4F14BhoG5gphKFrle5oIf6tVidjReEHNCWfAaJPMSmRBhoaSRA4YKq/7T58dKp6FvqlmceT21lJOzRWtSp7C1xCxhomWNJFUBpupmL5WdPDfdvmqiUQaKrRBymP+yOp18TSEhqZrbvNUMsIOl8lkllnRlyp2CiQXA5RvgshKIA0= root@localhost
ssh-dss AAAAB3NzaC1kc3MAAACBALdhwADEpIp/K23jFoXlLeSi6bbGyXs+wIiQgITaSrOBVnEA63kyckWXCn/0A8zXEWF11Y4uNikZJL+49vJu412IFfVgzV8tHYnq7cVAT/WJMencfWkYFNnyqtn8w7jk3DoSXvhcjs/saiswWzKbxpMazPBGNuNbM32rLmaKY4LZAAAAFQCfHHCB16tMrLleWCr9+1AItKDTQwAAAIEAoyUeXONb8/Tb+qMRN6/qfHAGZAwxEhd1LM8DKLhFsUxqFCpyhZr7QXic/RDt4Op3jMaLPht/Un3+l5nW/h4Vs0vSkC9Q8l0t/92DV67hT+lKRJXdfE4OGgUMC32jPhQ574JdOru+4wSJEXR4yCymP8BAkEr8l+LE8aa8cqO5tuMAAACAU8CQAsXsLKHm01ES/HqWxwk8skxHsaJuCwxdIZMbFVzWO8TcjftJ21ymg3t97PPeTiojTZrVgpIUemfP1spqlawf+VeXUat6Q0Uh6DF7KvjRb6DNkfGlxzIvO9LBsJmvc3VohSQ9psf3JNfoQh/NeGj6F3Yz8pi3NpOKpN+DRec= root@localhost
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGsr5+bBmrFA4Qfv94Dj+Qwm8GjR1WsWORBfZP8QPO0HmL9uOEvtPn12W6zchY+svA0u4136ZFp/I2hZ6LXOCf0= root@localhost
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMstvO+G5089bnuUbur+pOz5RaDpx1IgtxOi3a510bSO root@localhost
</uvt:ssh_known_hosts>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='s390x' machine='s390-ccw-virtio-focal'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-s390x</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/uvtool/libvirt/images/qemu-s390x-tcg.qcow'/>
      <target dev='vda' bus='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/uvtool/libvirt/images/qemu-s390x-tcg-ds.qcow'/>
      <target dev='vdb' bus='virtio'/>
      <address ...

I have prepared MPs
F: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/410032
H: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/410033
and a PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4682

The hint that it is emulation helped and I now have working test steps and added an SRU Template for the bug. But it also made me wonder about the importance of having this at all.

I mean ... really ... "uretprobes"?
While nice those are not the most common use case for Ubuntu users IMHO.
And then s390x TCG emulation, that is rare as well and so far in the past was not even always really supported/wanted by IBM.
Combining the two I'm tempted to say we'd waste the time of the SRU team, waster internet BW to push a new qemu everywhere and waste testing time.

Then there also is plenty of fuzz applying it back to Focal (Hirsute is fine).
Applying patch ubuntu/lp-1929926-target-s390x-Fix-translation-exception-on-illegal-in.patch
patching file target/s390x/translate.c
Hunk #1 succeeded at 6317 with fuzz 2 (offset -95 lines).
Hunk #2 succeeded at 6334 (offset -95 lines).
Hunk #3 succeeded at 6362 (offset -95 lines).
Hunk #4 succeeded at 6371 with fuzz 1 (offset -95 lines).
Hunk #5 succeeded at 6385 with fuzz 2 (offset -103 lines).
Hunk #6 succeeded at 6445 (offset -104 lines).
Nothing evil - I reviewed and think it is fine, but still it adds to my point of "maybe not worth for what it fixes".

I wonder (no offense) if this has a real use case that IBM considers to be important for Ubuntu users or if this is just a "we have found it so let us fix it" case. If it is the latter then I'm tempted to say Won't Fix here. Or at least have it wait for another qemu SRU to come by that we can add it on top of.

Therefore my question to IBM, are there any use-cases or other strong reasons to push for this fix? If you do care about it I'm happy to help to carry this to a conclusion, but I'm asking for:

1. Please provide a use-case or reasoning why this actually deserves to be SRU-fixed and the few that want/need it can't just upgrade to e.g. 21.10? This isn't about me, we also need this to convince the SRU team

2. Please Test the linked PPA for Hirsute and Focal and report if both fulfill your needs

3. Please have a look at the Focal MP and confirm that you think the backport fuzz did not mess up anything

P.S. on potentially adding it on top of some other SRU:
- No other strong SRU in the queue right now (but there will be I'm sure).
- I'm also asking the security Team is something is inbound ...
- I'd still want 1-3 above be answered even if we piggy-back it on something else

------- Comment From <email address hidden> 2021-10-13 18:35 EDT-------
This used to be important for libbpf CI (which is meant to be used as a CI for the in-kernel BPF code too), however, in the meantime we've decided to provide the hardware and switch from TCG to KVM, so this fix is not that necessary anymore.

Thank you for the clarifications!

Then I'd say we can piggyback it onto another upload !IF! upfront you could spent the cycles to confirm that the test builds really give you what you'd want/need in Focal and Hirsute.

Until then - I'll mark this Won't Fix

I understand you want to stage this fix in Hirsute via block-proposed as by looking at Hirsute alone the benefit/cost of the SRU is not high enough, still doing this will mean that there will be a (small) regression for users upgrading from Focal to Hirsute.

OTOH if we look at Focal and Hirsute *together*, given that we have a Focal SRU to piggyback, then I think the overall benefit/cost of releasing this fix to both the releases is acceptable, even without staging the Hirsute SRU.

I'd lean towards not staging, but this is up to you and the SRU team.

Thanks, I agree and I guess if it really is a problem (I think not) then the SRU team will speak up on review.

Uploaded to F/H-unapproved

Hello bugproxy, or anyone else affected,

Accepted qemu into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:5.2+dfsg-9ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello bugproxy, or anyone else affected,

Accepted qemu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:4.2-3ubuntu6.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:5.2+dfsg-9ubuntu3.3) for hirsute have finished running.
The following regressions have been reported in tests triggered by the package:

lava/2020.12-1ubuntu2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/hirsute/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Thanks for accepting this @SRU Team
I added block-proposed-hirsute as outlined in the description.

Up to testing ...

FYI - the autopkgtest on lava was flaky and is now resolved.

Hello,
the slow verification of this blocks the more important/urgent fix of bug 1749393 that we'd really would want to get released before Christmas. I know that FHeimes has coordinated this to be tested within IBM - any chance to get this completed in the next few days?

------- Comment From <email address hidden> 2021-12-17 08:24 EDT-------
Hi!

Sorry about the delay. Both 20.04 and 21.04 packages work fine. Here is the sequence I used:

host# docker run -it ubuntu:20.04 bash
docker# apt-get update
docker# apt-get install binutils linux-image-5.11 qemu-system-s390x wget

docker# cd /tmp
docker# lz4 -d </boot/initrd.img-5.11.0-43-generic | cpio -idv
docker# objdump -d bin/sh | grep read@plt
0000000000007080 <read@plt>:

docker# wget http://launchpadlibrarian.net/571593767/qemu-system-s390x_4.2-3ubuntu6.19_s390x.deb
docker# dpkg -i --force-all qemu-system-s390x_4.2-3ubuntu6.19_s390x.deb

docker# qemu-system-s390x -kernel /boot/vmlinuz-5.11.0-43-generic -initrd /boot/initrd.img-5.11.0-43-generic -m 1024 -nographic
vm# mount -t debugfs foo /sys/kernel/debug
vm# echo "r:read_plt /bin/sh:0x7080" >/sys/kernel/debug/tracing/uprobe_events
vm# echo 1 >/sys/kernel/debug/tracing/events/uprobes/read_plt/enable

Fails without the fix, works with it. Same on 21.04.

Thank you, Ilya. I've adjusted the tags accordingly ...

There was one bad old tag left, fixed that.
Once the SRU Team is back for action this should be released.

This bug was fixed in the package qemu - 1:5.2+dfsg-9ubuntu3.3

---------------
qemu (1:5.2+dfsg-9ubuntu3.3) hirsute; urgency=medium

  * d/p/u/lp-1929926-target-s390x-Fix-translation-exception-on-illegal-in.patch:
    fix uretprobe in s390x TCG (LP: #1929926)

 -- Christian Ehrhardt <email address hidden> Tue, 12 Oct 2021 09:04:44 +0200

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:4.2-3ubuntu6.19

---------------
qemu (1:4.2-3ubuntu6.19) focal; urgency=medium

  * d/p/u/lp-1749393-linux-user-Reserve-space-for-brk.patch: fix static
    use cases needing a lot of brk space (LP: #1749393)
  * d/p/u/lp-1929926-target-s390x-Fix-translation-exception-on-illegal-in.patch:
    fix uretprobe in s390x TCG (LP: #1929926)

 -- Christian Ehrhardt <email address hidden> Mon, 26 Apr 2021 11:11:19 +0200

------- Comment From <email address hidden> 2022-02-25 08:08 EDT-------
Verification done & fix released. Hence closing the bug.
Status change: ->CLOSED