kernel: improve spectre mitigation

Bug #1790457 reported by bugproxy
288
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Unassigned
linux (Ubuntu)
Fix Released
Undecided
Skipper Bug Screeners
Bionic
Fix Released
Undecided
Unassigned
qemu (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * eToken Facility will help to mitigate spectre.
   With it in place use of expolines can be ommitted.

   Kernel https://github.com/torvalds/linux/commit/aeaf7002a76c8da60c0f503badcbddc07650678c

   KVM to pass it to guests:
https://patchwork.kernel.org/patch/10532197/

 * Backport the changes to Qemu/Kernel so that the impact of the spectre
   fixes can be minimized.

[Test Case]

 * First of all you need HW with the facility available.
   For HW without nothing should change at all, well maybe a message that
   it wasn't detected when the new kernel boots.

 * When running on HW with the Facility and a fixed kernel then the
   facility should be reported as being available.

 * With a fixed Kernel AND Qemu this facility should be passed to the
   guest so that it can benefit from the improvements as well.

 * Due to a lack of such HW IBM volunteered to do the verification on
   this bug.

[Regression Potential]

 * Detection and passing of a Facility is nothing new, s390x has plenty of
   them and this is in some sense "just one more" so regressions should be
   minimal. The one thing we thought about was how an enabled Kernel/qemu
   would behave on systems that do not have the facility, but in all tests
   that was correctly detected and continues to use expoline.

[Other Info]

 * n/a

---

Description will follow

CVE References

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-171040 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
information type: Public → Private Security
bugproxy (bugproxy)
tags: added: targetmilestone-inin1810
removed: targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Extra spectre patch for Ubuntu 16.04

------- Comment (attachment only) From <email address hidden> 2018-09-03 07:34 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Extra spectre patch for Ubuntu 18.04

------- Comment (attachment only) From <email address hidden> 2018-09-03 07:34 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-09-03 08:09 EDT-------
Following upstream commits

aeaf7002a76c8da60c0f503badcbddc07650678c (kernel 4.19 RC1)
5eda25b10297684c1f46a14199ec00210f3c346e (kernel 4.19 RC1)
c3b9e3e1ea1c1d1524b56b6734711db2a6fc2163 (kernel 4.17 RC1)
a3da7b4a3be51f37f434f14e11e60491f098b6ea (kernel 4.17 RC1)

have to be applied to

1) Ubuntu 18.10 - kernel 4.18
2) Ubuntu 18.04.1 - kernel 4.15
3) Ubuntu 16.04.5 - kernel 4.15
3) Ubuntu 16.04.1 - kernel 4.4

In addition 2 patches are applied

One for 16.04 and one for 18.04.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
importance: Undecided → High
Revision history for this message
Stefan Bader (smb) wrote :

The extra patch for 16.04 and 18.04 seems to be a follow-up of commit de5cb6eb514ebe241e3edeb290cb41deb380b81d upstream ("s390: use expoline thunks in the BPF JIT"). First question: will that extra change go upstream, too? Second question: There seems to be a related follow-up: 26f8438 "s390: fix br_r1_trampoline for machines without exrl" which is not, yet, included in 18.04 or 16.04 (fwiw neither seems to be in upstream stable 4.4.y). Should that get picked up along with the other changes? And finally (to confirm) this is "CVE-2018-3639 (s390x)" follow-ups?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-04 06:28 EDT-------
The extra patch is a follow-up for de5cb6eb51 ("s390: use expoline thunks in
the BPF JIT"), but the extra patch can not go upstream as the code it patches
is not present anymore. See git commit e1cf4befa2 ("bpf, s390x: remove
ld_abs/ld_ind").

The patch 26f8438 ("s390: fix br_r1_trampoline for machines without exrl")
has just been picked up by Greg for stable-4.4, see [PATCH 4.4 71/80] in
his latest patch series. This patch is only required when building for
machines prio to z10.

All of these patches are related to the original spectre-v2 CVE-2017-5715,
CVE-2018-3639 is speculative store bypass aka spectre variant 4, no ?

Revision history for this message
Stefan Bader (smb) wrote :

Hi Martin,

thanks for the details on the additional patch. For the follow-up, I guess if it goes into 4.4.y it will end up in our 16.04 kernel, even though we probably would not strictly need it there either. And 18.04, too. But in both cases it could be included with the cve update or not. About the CVE number, *sigh* yes, you are right. I could have sworn I was searching for Spectre.v2 and came up with the other number but now I also end up with CVE-2017-5715.

Revision history for this message
Khaled El Mously (kmously) wrote :

The patches specified above apply easily to the 4.15 (bionic) kernel (with some minor changes), but
it is taking longer to prepare the patches for xenial (4.4) since at least 2 of the above patches need significant changing and/or additional patches in order to be backported to the 4.4 kernel.

Revision history for this message
Stefan Bader (smb) wrote :

Probably also should be mentioned that the patch needing adjustment was "KVM: s390: add etoken support for guests" as it tests on the GISA address in the SIE block which does not yet exist in 4.15. We would drop that if section of the patch.

Revision history for this message
Khaled El Mously (kmously) wrote :

Actually the patch ("KVM: s390: add etoken support for guests") only needed adjustment because I incorrectly resolved a (minor) conflict when applying it and ended up keeping an unnecessary if() section. After resolving the conflict correctly, no adjustment was needed.

Frank Heimes (fheimes)
Changed in linux (Ubuntu):
status: New → In Progress
Changed in ubuntu-z-systems:
status: New → In Progress
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hello Martin,

The requested upstream fixes don't apply cleanly on top of 16.04 4.4 kernel, and backporting them is not straightforward as we first thought. So far I have identified the following commits as pre-reqs to c3b9e3e1ea1c1d1524b56b6734711db2a6fc2163 alone:

c18719638cc1 KVM: s390: generate facility mask from readable list
d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
35b3fde6203b KVM: s390: wire up bpb feature

I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.

Would it be possible to provide us the backported patches or a tree with the commits on top of our 16.04 4.4 kernel?

Please advice.

Thank you.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-10 07:03 EDT-------
> c18719638cc1 KVM: s390: generate facility mask from readable list
> d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
> bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
> 35b3fde6203b KVM: s390: wire up bpb feature
>
> I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.

With the kernel version 4.4 this indeed takes some extra patches. In addition to
that we would need a matching qemu version as well which is probably overkill.

I have talked to Christian Borntr?ger about the KVM aspect and he would be fine
if we do not backport the KVM related patches. The end result is that the KVM
guests would use the expoline mitigation by default, even on machines with
the etoken facility. You can still use the spectre_v2=off kernel parameter to
disable expolines.

That reduces the list of git commits to only two:
aeaf7002a76c8da6 "s390: detect etoken facility"
5eda25b10297684c "s390/lib: use expoline for all bcr instructions"
plus the extra patch.

Revision history for this message
bugproxy (bugproxy) wrote : Backport of "s390: detect etoken facility"

------- Comment (attachment only) From <email address hidden> 2018-09-10 07:05 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Backport of "s390/lib: use expoline for all bcr instructions"

------- Comment (attachment only) From <email address hidden> 2018-09-10 07:06 EDT-------

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Fixes for 18.04 (Bionic) and 16.04 (Xenial) are now committed to the git repos and will be available on kernel builds for the SRU cycle that's starting this week.

Revision history for this message
Frank Heimes (fheimes) wrote :

Many thanks kernel team for picking this up on short notice!
Changing status to Fix Committed.

Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-09-11 13:18 EDT-------
(In reply to comment #19)
> > c18719638cc1 KVM: s390: generate facility mask from readable list
> > d6450dc35e21 KVM: s390: Populate mask of non-hypervisor managed facility bits
> > bb0258cde7a6 s390/facilities: add helper tool to generate facility lists
> > 35b3fde6203b KVM: s390: wire up bpb feature
> >
> > I'm afraid we would take some time to backport all the requested fixes and their pre-reqs and get it right.
>
> With the kernel version 4.4 this indeed takes some extra patches. In
> addition to
> that we would need a matching qemu version as well which is probably
> overkill.
>
> I have talked to Christian Borntr?ger about the KVM aspect and he would be
> fine
> if we do not backport the KVM related patches. The end result is that the KVM
> guests would use the expoline mitigation by default, even on machines with
> the etoken facility. You can still use the spectre_v2=off kernel parameter to
> disable expolines.
>

To write down the obvious.
That is of course only true for the 4.4 kernel. We can drop the KVM etoken and cpumodel patches from 4.4 because it does not yet provide the cpu model. We certainly want to have the KVM patches in the 4.15 kernel (bionic and future HWE)

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-11 13:24 EDT-------
For etoken support for KVM guests we also need qemu support to add this to the cpumodel.

The feature is provided by the following upstream commit
https://git.qemu.org/?p=qemu.git;a=commit;h=27e84d4ebd25b981ab27cb590fe06d1b0fcd06d2
commit: 27e84d4ebd25b981ab27cb590fe06d1b0fcd06d2
title: s390x/kvm: add etoken facility

we also need the following commit to make this compile
https://git.qemu.org/?p=qemu.git;a=commit;h=d36f7de82995a42b749c29c5b60ba31483995a42
commit: d36f7de82995a42b749c29c5b60ba31483995a42
title: linux-headers: update

To simplify things, this part of the headers update is sufficient:
https://git.qemu.org/?p=qemu.git;a=blobdiff;f=linux-headers/asm-s390/kvm.h;h=1ab9901911bf55e3aba55488bb20a184d85bbce5;hp=11def143015d572f208bbdf4aa4ecfa094c700a1;hb=d36f7de82995a42b749c29c5b60ba31483995a42;hpb=c61177881cbda50704207dd9fb4811659bbf913e

Please tell me if you want a backport patch for both upstream commits or if you prefer to do the backport yourself. (Do you want a separate launchpad for qemu or can we handle that here?)

The backport would be necessary for the qemu in bionic and cosmic and all relevant cloud archive versions.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Christian, thanks for the broken down header change!
We can stick to this bug and just add a qemu task to it.

I'll take a look at doing this in cosmic soon (before things close/freeze more) and if working fine there we can take a look at making this available in Bionic's qemu as well.
If I hit anything that makes me feel unsure on the backport I'll reach out to you.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I think with changes being upstream there could be a description added :-)
Is there a TL;DR in your bugzilla that we could move to the bug description on LP?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

It is a fix in the means of it would qualify even for an SRU under the "HW exploitation" as well as the "Security" banner. Therefore I think we are ok to push for this right now unless cosmic is fully frozen for the release week or so.

But I'd need to ask you (=IBM) to give the feature some testing as it needs HW not currently available to me.

Could you check the PPA [1] in regard to the change requested here please?

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3413

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-12 05:44 EDT-------
i can verify the qemu changes in the ppa.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

As discussed you might need the proposed kernel of cosmic.
If you use 4.18.0-8.9 of [1] that should have the etoken changes as you need it.

[1]: https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/proposed

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-12 07:44 EDT-------
tested sucessfully with the ppas.

With an old host kernel, qemu reject the etoken facility (and the guest falls back to expoline).
With 4.18.0-8-generic qemu allows etoken and passes that along to the guest.

looks good.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the check, especially that you did it with both kernels to be sure on the behavior.
Uploading to cosmic then ...

Changed in qemu (Ubuntu):
status: New → Triaged
Changed in qemu (Ubuntu Bionic):
status: New → Confirmed
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Ok, cosmic is almost released.
I'm preparing Bionic - could you pre-verify that the backport worked as expected on the ppa at [1] please?
This contains an updated qemu for Bionic.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3414

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.12+dfsg-3ubuntu6

---------------
qemu (1:2.12+dfsg-3ubuntu6) cosmic; urgency=medium

  * improve s390x spectre mitigation with etoken facility (LP: #1790457)
    - debian/patches/ubuntu/lp-1790457-s390x-kvm-add-etoken-facility.patch
    - debian/patches/ubuntu/lp-1790457-partial-s390x-linux-headers-update.patch

 -- Christian Ehrhardt <email address hidden> Wed, 12 Sep 2018 10:06:48 +0200

Changed in qemu (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-09-13 10:07 EDT-------
that qemu looks good. I had to use an upstream kernel as the kernel patch is still in the pipe.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the extra check!

Uploaded to Bionic-unapproved for SRU Team review

Frank Heimes (fheimes)
Changed in linux (Ubuntu Bionic):
status: New → Fix Committed
information type: Private Security → Public Security
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

To be clear - this relies on special HW to be present, so I can't validate it.
IBM was so kind to verify the PPAs in advance, it would be great if you could do so again with the bits in proposed.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-09-27 10:42 EDT-------
qemu 2.11+dfsg-1ubuntu7.6 from proposed does work as expected.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for testing, setting tags accordingly.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Frank Heimes (fheimes) wrote :

Since bionic-updates containes kernel 4.15.0-36. and git says that 's390: detect etoken facility' is included:
$ git log --oneline | grep "s390: detect etoken facility"
cffc6b1 s390: detect etoken facility
$ git tag --contains cffc6b1
Ubuntu-4.15.0-35.38
Ubuntu-4.15.0-36.39
Ubuntu-4.15.0-37.40
Ubuntu-raspi2-4.15.0-1025.27
I'm marking bionic as Fix Released.

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

Since cosmic contains kernel 4.18.0.8.9 and git says that 's390: detect etoken facility' is included:
$ git log --oneline | grep "s390: detect etoken facility"
edb9bc2 s390: detect etoken facility
$ git tag --contains edb9bc2
Ubuntu-4.18.0-8.9
Ubuntu-4.18.0-9.10
Ubuntu-raspi2-4.18.0-1004.4
Ubuntu-raspi2-4.18.0-1004.5
Ubuntu-raspi2-4.18.0-1004.6
I'm marking cosmic ["linux (Ubuntu)"] as Fix Released, too.

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Frank Heimes (fheimes) wrote :

And just for the records, the kernel part already landed in xenial, too:
xenial updates kernel today: 4.4.0.137.
$ git log --oneline | grep "s390: detect etoken facility"
c32821c s390: detect etoken facility
$ git tag --contains c32821c
Ubuntu-4.4.0-136.162
Ubuntu-4.4.0-137.163
Ubuntu-4.4.0-138.164
Ubuntu-raspi2-4.4.0-1099.107
Ubuntu-snapdragon-4.4.0-1103.108

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.6

---------------
qemu (1:2.11+dfsg-1ubuntu7.6) bionic; urgency=medium

  [ Christian Ehrhardt ]
  * Add cpu model for z14 ZR1 (LP: #1780773)
  * d/p/ubuntu/lp-1789551-seccomp-set-the-seccomp-filter-to-all-threads.patch:
    ensure that the seccomp blacklist is applied to all threads (LP: #1789551)
    - CVE-2018-15746
  * improve s390x spectre mitigation with etoken facility (LP: #1790457)
    - debian/patches/ubuntu/lp-1790457-s390x-kvm-add-etoken-facility.patch
    - debian/patches/ubuntu/lp-1790457-partial-s390x-linux-headers-update.patch

  [ Phillip Susi ]
  * d/p/ubuntu/lp-1787267-fix-en_us-vnc-pipe.patch: Fix pipe, greater than and
    less than keys over vnc when using en_us kemaps (LP: #1787267).

 -- Christian Ehrhardt <email address hidden> Wed, 29 Aug 2018 11:46:37 +0200

Changed in qemu (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-10-19 04:59 EDT-------
IBM bugzilla status-> closed, Fix Released for all Distros in Service

Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.