[18.04] backport needed for spectre/meltdown mitigation machine type qemu patches (kvm)

Merged in the change from bug 1761371 as it is almost part of the same series and has the same dependencies.
Closed the other bug as dup and added the change to the list in the bug description.

I'm still evaluating this (quite some time now), but I want to share the interim TL;DR for you to consider.

All those patches are part of a huge rewrite and handling of sprs.
Yes the series has "only" (already a lot) 7 patches, but it seems due to dependencies this easily will get 10 or even 20+.

We need to look out if we can achive that in a much better backport.
I understand that for 2.12 and later this was the right way to do.

But for the fix in 2.11 we will need:
- do not add a 2.12 type
- modify the 2.11 type instead
- stop rewriting almost all of the spr cap handling
- just set the sane defaults (not much more)

Only then this is maintainable without making 2.11 almost 2.12.
I'll take a further look after a call that starts now, but I'll likely need your help to achive the above.

A bunch of the dependencies went in through the 2.11.1 stable updates:

$ git log --oneline hw/ppc/spapr_caps.c
eab4b51 target/ppc/spapr_caps: Add new tristate cap safe_indirect_branch
d7aa3d0 target/ppc/spapr_caps: Add new tristate cap safe_bounds_check
3dc1227 target/ppc/spapr_caps: Add new tristate cap safe_cache
e9a8747 target/ppc/spapr_caps: Add support for tristate spapr_capabilities
43a29f0 target/ppc/spapr_caps: Add macro to generate spapr_caps migration vmstate
6a47136 hw/ppc/spapr_caps: Rework spapr_caps to use uint8 internal representation
e4f4fa0 spapr: Handle Decimal Floating Point (DFP) as an optional capability
ff6f7e1 spapr: Handle VMX/VSX presence as an spapr capability flag
804e5ea spapr: Validate capabilities on migration
9070f40 spapr: Treat Hardware Transactional Memory (HTM) as an optional capability
78a38cd spapr: Capabilities infrastructure

There we ensured that all is in 2.11 and not the 2.12 type.
We need to ensure the new changes are doing the same here.

After all it might not be too bad, due to the others being already in the stable fixes.
Most apply clean (with some offsets thou).
The last patch with the new 2.12 type is what has some fails.
But those are fixable and are expected (since our 2.11/2.12 types differ).

I'll add a further cleanup to make this to 2.11 type as needed and then start testing as well as asking for your review.

So all that, to provide a -sxxm type that is NOT the default (not in your patches, so I didn't make it the default in mine).

I wonder, what is the benefit of this then.
If users need to change the machine type, then they can just as well set the CFPC/SBBC/IBS caps right?

Since we are not changing the deafault (pseries-bionic) is what we essentially support.
We don't even have to wrap it into the 2.11 type.

Todays machine types in Bionic:
[Upstream types]
pseries-2.10 pSeries Logical Partition (PAPR compliant)
pseries-2.11 pSeries Logical Partition (PAPR compliant)
pseries-2.12 pSeries Logical Partition (PAPR compliant)
[...]
[Ubuntu recommended types (the defaults)]
pseries pSeries Logical Partition (PAPR compliant) (alias of pseries-bionic)
pseries-bionic pSeries Logical Partition (PAPR compliant) (default)
[...]

Your patch will add
pseries-2.12-sxxm based on the 2.12 that was added in the stable 2.11.1 patches.
I mentioned this before on other places.
That might be a nice simplification for users, but since 2.12 is not yet set in stone (at least it is on -rc now) we can't (and won't) make any guarantees about cross release migrations.

pseries-bionic is what we can control, and the default and what we consider that shall work across releases.
Since you provided the extra type as non-default convenience type I'll do the same.

IFF there are changes to 2.12 in the remaining upstream 2.12 or the coming Ubuntu 18.10 development cycle we will not try (too hard) to keep e.g. a Bionic pseries-2.12-sxxm in sync with that of 18.10.

This is just the same as 1001 extra qemu options, if you derived from the common case it won't be migratable (it easily is with shutdown, start, just live has issues).

If you want any of these sxxm defaults to be the default of the default machine type ... ARGL too much default exception.
Rephrase:
If you want the changes to the CAPs that the 2.12-sxxm type starts to be applied to the default machine type "pseries-bionic" let me know.

Until Bionic release we could set:
     smc->default_caps.caps[SPAPR_CAP_CFPC] = SPAPR_CAP_WORKAROUND;
     smc->default_caps.caps[SPAPR_CAP_SBBC] = SPAPR_CAP_WORKAROUND;
     smc->default_caps.caps[SPAPR_CAP_IBS] = SPAPR_CAP_FIXED_CCD;
In the pseries-bionic type (but not after release anymore).

Same (=please let me know asap) is true if instead you want any of:
pseries-2.11-sxxm
pseries-bionic-sxxm

------- Comment From <email address hidden> 2018-04-05 12:50 EDT-------
(In reply to comment #9)
> So all that, to provide a -sxxm type that is NOT the default (not in your
> patches, so I didn't make it the default in mine).
>
> I wonder, what is the benefit of this then.
> If users need to change the machine type, then they can just as well set the
> CFPC/SBBC/IBS caps right?

I believe the issue is similar to what we initially hit with:

Bug 165081 - LP1752026- Ubuntu18.04:POWER9:DD2.2: Unable to start a KVM guest with default machine type(pseries-bionic) complaining "KVM implementation does not support Transactional Memory, try cap-htm=off" (kvm)

where due to lack of "-machine pseries-*,cap-*=on/off/*" support via libvirt we need to rely on machine type behavior to enable certain options in the interim.

>
> Since we are not changing the deafault (pseries-bionic) is what we
> essentially support.
> We don't even have to wrap it into the 2.11 type.
>
> Todays machine types in Bionic:
> [Upstream types]
> pseries-2.10 pSeries Logical Partition (PAPR compliant)
> pseries-2.11 pSeries Logical Partition (PAPR compliant)
> pseries-2.12 pSeries Logical Partition (PAPR compliant)
> [...]
> [Ubuntu recommended types (the defaults)]
> pseries pSeries Logical Partition (PAPR compliant) (alias of
> pseries-bionic)
> pseries-bionic pSeries Logical Partition (PAPR compliant) (default)
> [...]
>
> Your patch will add
> pseries-2.12-sxxm based on the 2.12 that was added in the stable 2.11.1
> patches.
> I mentioned this before on other places.
> That might be a nice simplification for users, but since 2.12 is not yet set
> in stone (at least it is on -rc now) we can't (and won't) make any
> guarantees about cross release migrations.
>
> pseries-bionic is what we can control, and the default and what we consider
> that shall work across releases.
> Since you provided the extra type as non-default convenience type I'll do
> the same.
>
> IFF there are changes to 2.12 in the remaining upstream 2.12 or the coming
> Ubuntu 18.10 development cycle we will not try (too hard) to keep e.g. a
> Bionic pseries-2.12-sxxm in sync with that of 18.10.

Doesn't seem likely to change at this point but that seems reasonable either way.

> If you want any of these sxxm defaults to be the default of the default
> machine type ... ARGL too much default exception.
> Rephrase:
> If you want the changes to the CAPs that the 2.12-sxxm type starts to be
> applied to the default machine type "pseries-bionic" let me know.
>
> Until Bionic release we could set:
> smc->default_caps.caps[SPAPR_CAP_CFPC] = SPAPR_CAP_WORKAROUND;
> smc->default_caps.caps[SPAPR_CAP_SBBC] = SPAPR_CAP_WORKAROUND;
> smc->default_caps.caps[SPAPR_CAP_IBS] = SPAPR_CAP_FIXED_CCD;
> In the pseries-bionic type (but not after release anymore).
>
> Same (=please let me know asap) is true if instead you want any of:
> pseries-2.11-sxxm
> pseries-bionic-sxxm

Adding pseries-2.11-sxxm/pseries-bionic-sxxm gives us some flexibility with different firmware/hardware and migration strategies so this would be our preference. This is also in line with how things are done with spectre-rel...

>
> > I wonder, what is the benefit of this then.
> > If users need to change the machine type, then they can just as well set
> the
> > CFPC/SBBC/IBS caps right?
>
> I believe the issue is similar to what we initially hit with:
>
> Bug 165081 - LP1752026- Ubuntu18.04:POWER9:DD2.2: Unable to start a KVM
> guest with default machine type(pseries-bionic) complaining "KVM
> implementation does not support Transactional Memory, try cap-htm=off"
> (kvm)
>
> where due to lack of "-machine pseries-*,cap-*=on/off/*" support via
> libvirt we need to rely on machine type behavior to enable certain
> options in the interim.
>

Yes exactly.

And this is why I was asking for more guidance on the preferred defaults,
as the addition of an extra type still needs the user to do something (e.g.
selecting that type).

[...]

> IFF there are changes to 2.12 in the remaining upstream 2.12 or the coming
> > Ubuntu 18.10 development cycle we will not try (too hard) to keep e.g. a
> > Bionic pseries-2.12-sxxm in sync with that of 18.10.
>
> Doesn't seem likely to change at this point but that seems reasonable
> either way.
>

Yeah, as much as upstream can't change 2.11 anymore (which is why you make
it a 2.12 type) we can't change 2.12 yet (in this release).

[...]

> Same (=please let me know asap) is true if instead you want any of:
> > pseries-2.11-sxxm
> > pseries-bionic-sxxm
>
> Adding pseries-2.11-sxxm/pseries-bionic-sxxm gives us some flexibility
> with different firmware/hardware and migration strategies so this would
> be our preference.

Ok, will do so.
We will keep the 2.12 type as-is in code, but it might change (if it
changes upstream - unlikely).
While together we will try to keep the bionic types stable.

> This is also in line with how things are done with
> spectre-related CPU capabilities in 2.12/2.11.1 for x86 (new explicit
> cpu types as opposed to changed defaults).
>

Yes, that is true on x86

> In terms of upstream patches that need to be backported I think we also
> want "ppc/spapr-caps: Disallow setting workaround for spapr-cap-ibs". I
> got a pretty clean backport on top of 2.11.1 just cherry-picking:
>

I already have this patch in my current build - I added it to the
description here when I was dupping the other bug onto this.

>
> ppc/spapr-caps: Define the pseries-2.12-sxxm machine type
> ppc/spapr-caps: Convert cap-ibs to custom spapr-cap
> ppc/spapr-caps: Disallow setting workaround for spapr-cap-ibs
> ppc/spapr-caps: Convert cap-sbbc to custom spapr-cap
> ppc/spapr-caps: Convert cap-cfpc to custom spapr-cap
> ppc/spapr-caps: Add support for custom spapr_capabilities
> target/ppc: Check mask when setting cap_ppc_safe_indirect_branch
>

That is exactly the series I already have in my ppa since yesterday ~noon

> Then an ubuntu-specific pseries-2.11-sxxm patch would basically mirror
> the "ppc/spapr-caps: Define the pseries-2.12-sxxm machine type"
> implementation and pseries-bionic-sxxm could point to that.
>

Exactly my thought will do so and respin the build in the ppa.

# qemu-system-ppc64 -M ? | egrep 'bionic|2.1[12]'
pseries-2.11 pSeries Logical Partition (PAPR compliant)
pseries-2.12 pSeries Logical Partition (PAPR compliant)
pseries-2.12-sxxm pSeries Logical Partition (PAPR compliant)
pseries pSeries Logical Partition (PAPR compliant) (alias of pseries-bionic)
pseries-bionic pSeries Logical Partition (PAPR compliant) (default)
pseries-bionic-sxxm pSeries Logical Partition (PAPR compliant)

Both types seem to work for me.
I have no FW to check in the actual features.

I'll drive a new general regression run over the weekend, but I'd be really happy if one of you could test the if new type fits your needs so that I can push it on monday (since everything else seems good).

Still ppa: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3225
Min Version with these changes: qemu - 1:2.11+dfsg-1ubuntu6~ppa4

------- Comment From <email address hidden> 2018-04-09 07:10 EDT-------
(In reply to comment #13)
> # qemu-system-ppc64 -M ? | egrep 'bionic|2.1[12]'
> pseries-2.11 pSeries Logical Partition (PAPR compliant)
> pseries-2.12 pSeries Logical Partition (PAPR compliant)
> pseries-2.12-sxxm pSeries Logical Partition (PAPR compliant)
> pseries pSeries Logical Partition (PAPR compliant) (alias of
> pseries-bionic)
> pseries-bionic pSeries Logical Partition (PAPR compliant) (default)
> pseries-bionic-sxxm pSeries Logical Partition (PAPR compliant)
>
> Both types seem to work for me.
> I have no FW to check in the actual features.
>
> I'll drive a new general regression run over the weekend, but I'd be really
> happy if one of you could test the if new type fits your needs so that I can
> push it on monday (since everything else seems good).
>
> Still ppa: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3225
> Min Version with these changes: qemu - 1:2.11+dfsg-1ubuntu6~ppa4

ii qemu-kvm 1:2.11+dfsg-1ubuntu6~ppa4

I tested on Power8 am able to boot with pseries-bionic-sxxm fine and works as expected.

kvm -M pseries-bionic-sxxm -monitor stdio -serial /dev/pts/6 -enable-kvm -nographic -vga none /home/sath/avocado-fvt-wrapper/data/avocado-vt/images/ubuntu-16.04.4-ppc64le.qcow2 -device virtio-net

Regards,
-Satheesh

Thanks for the confirmation.
Pushing this now ...

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu6

---------------
qemu (1:2.11+dfsg-1ubuntu6) bionic; urgency=medium

  * Remove LP: 1752026 changes to d/p/ubuntu/define-ubuntu-machine-types.patch.
    The Kernel fixes are preferred and already committed to the kernel.
    Therefore remove the default disabling of the HTM feature (LP: #1761175)
  * d/p/ubuntu/lp1739665-SSE-AVX-AVX512-cpu-features.patch: Enable new
    SSE/AVX/AVX512 cpu features (LP: #1739665)
  * d/p/ubuntu/lp1740219-continuous-space-commpage.patch: make Arm
    space+commpage continuous which avoids long startup times on
    qemu-user-static (LP: #1740219)
  * d/p/ubuntu/lp-1761372-*: provide pseries-bionic-2.11-sxxm type as
    convenience with all meltdown/spectre workarounds enabled by default.
    This is not the default type following upstream and x86 on that.
    (LP: #1761372).
  * d/p/ubuntu/lp-1704312-1-* provide means to manually handle filesystem-dax
    with pmem by backporting align and unarmed options (LP: #1704312).
  * d/p/ubuntu/lp-1762315-slirp-Add-domainname.patch: slirp: Add domainname
    option to slirp's DHCP server (LP: #1762315)

 -- Christian Ehrhardt <email address hidden> Wed, 04 Apr 2018 15:16:07 +0200

------- Comment From <email address hidden> 2018-04-18 07:12 EDT-------
This fix needs to be considered for other LTS streams aswell like xenial, Trusty.
Can we use the same bz to track there inclusion aswell?
Advice otherwise, Thanks in advance.

Regards,
-Satheesh

You could use this bug as well as I want to remind you that you need Ubuntu Releases as well as a set of Ubuntu Cloud Archive targets - like bug 1744882 essentially.

But THIS bug here is just about the addtion of a easy-to-use-enable-all-workarounds machine type.
This is
a) likely not happening to modify/add those machine types for all the former releases
   (at least I'd object atm)
   Post-adding such types would mean add all those types to all releases like a huge
   matrix, which for a convenience "enable all types" change is not a valid tradeoff.
   There "cap-cfpc=workaround,cap-sbbc=workaround,cap-ibs=fixed" is ok to be used instead of
   -M pseris-<release>-sxxm - both is a change of configs and one is not harder than the other.
   But in terms of extra types to maintain it absolutely is.
   So nack to "backport the -sxxm type"

b) This is not what you actually want which is the code that makes it possible to set those.
   So like "3dc1227 target/ppc/spapr_caps: Add new tristate cap safe_cache"
   Which was not part of this request.
   IMHO what you want is the security backport of the actual spectre/meltdown related features.
   What is in 2.11.1 as you find it here [1] and covered here [2]

I didn't find a security tracker or bug for it, but that would be the right bug to drive for SRUs into older releases not this ticket here.

I'll ping mdeslaur who likely worked or works on the actual security fixes backport and might know the right bug for you.

[1]: https://www.qemu.org/2018/02/14/qemu-2-11-1-and-spectre-update/
[2]: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

I got in contact with mdeslaur and he confirmed there is no bug yet for spectre/meltdown on qemu for ppc64.

I'd ask you to:
1. file a new bug for that as this one here is about the -sxxm machine types
2. please list the functional changes there that add the cap-xxx options
   But not the new machine types (IMHO)

That bug would then be handled by the security Team and eventually become part of [1] I think.

[1]: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

------- Comment From <email address hidden> 2018-04-19 05:39 EDT-------
(In reply to comment #20)
> I got in contact with mdeslaur and he confirmed there is no bug yet for
> spectre/meltdown on qemu for ppc64.
>
> I'd ask you to:
> 1. file a new bug for that as this one here is about the -sxxm machine types
> 2. please list the functional changes there that add the cap-xxx options
> But not the new machine types (IMHO)
>
Sure, Thanks, Have created Bug 166958 for the same, soon it will get mirrored to Canonical.

Regards,
-Satheesh
> That bug would then be handled by the security Team and eventually become
> part of [1] I think.
>
> [1]:
> https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

------- Comment From <email address hidden> 2018-04-19 11:53 EDT-------
> Sure, Thanks, Have created Bug 166958 for the same, soon it will get
> mirrored to Canonical.

Bug 166958 is now https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1765364

------- Comment From <email address hidden> 2018-05-04 10:18 EDT-------
Tested and found fixed.

Power9:
Host:
4.15.0-15-generic #16-Ubuntu
qemu: 1:2.11+dfsg-1ubuntu6
libvirt: 4.0.0-1ubuntu7
kvm -M ? 2>&1|grep sxxm
pseries-2.12-sxxm pSeries Logical Partition (PAPR compliant)
pseries-bionic-sxxm pSeries Logical Partition (PAPR compliant)

Guest:
4.15.0-15-generic #16-Ubuntu

with: pseries-bionic-sxxm
# make tests
=========
Machine details from dmesg:
dmesg | grep -e 'pSeries machine' -e 'OPAL detected' -e rfi-fixups -e rfi-flush
[ 0.000000] Using pSeries machine description
[ 0.000000] rfi-flush: Using ori type flush
[ 0.000000] rfi-flush: Using mttrig type flush
[ 0.000000] rfi-flush: patched 9 locations
=========
Running tests...
Testing mitigation for spectre (ii. indirect branch prediction)... PASS (20000066 branches, 10000008 branch misses)
Testing mitigation for meltdown (l1d flush - syscall)... PASS (196008075 misses, 192000000 expected) [10/10 pass]
Testing mitigation for meltdown (l1d flush - userspace)... SKIP (!power8)

with: pseries-bionic
make tests
=========
Machine details from dmesg:
dmesg | grep -e 'pSeries machine' -e 'OPAL detected' -e rfi-fixups -e rfi-flush
[ 0.000000] Using pSeries machine description
[ 0.000000] rfi-flush: Using fallback displacement flush
[ 0.000000] rfi-flush: patched 9 locations
=========
Running tests...
Testing mitigation for spectre (ii. indirect branch prediction)... PASS (20000058 branches, 10000007 branch misses)
Testing mitigation for meltdown (l1d flush - syscall)... PASS (197506169 misses, 192000000 expected) [10/10 pass]
Testing mitigation for meltdown (l1d flush - userspace)... SKIP (!power8)

------------

Power8:

# make tests
=========
Machine details from dmesg:
dmesg | grep -e 'pSeries machine' -e 'OPAL detected' -e rfi-fixups -e rfi-flush
[ 0.000000] Using pSeries machine description
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: patched 9 locations (no flush)
=========
Running tests...
Testing mitigation for spectre (ii. indirect branch prediction)... PASS (20000100 branches, 10000007 branch misses)
Testing mitigation for meltdown (l1d flush - syscall)... FAIL (95128 misses, 192000000 expected) [10/10 failures]
Testing mitigation for meltdown (l1d flush - userspace)... PASS (193524721 misses, 192000000 expected) [10/10 pass]
root@ubuntu:~/side-channel-tests/staging# dmesg|grep rfi
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: patched 9 locations (no flush)
root@ubuntu:~/side-channel-tests/staging# uname -a
Linux ubuntu 4.15.0-17-generic #18-Ubuntu SMP Mon Apr 16 21:16:36 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux