[KVM][CLX] CPUID_7_0_EDX_ARCH_CAPABILITIES is not enabled in VM.

Commit:2bdb76c015df7125783d8394d6339d181cb5bc30

git tag --contains 2bdb76c015df7125783d8394d6339d181cb5bc30
v5.1
v5.1-rc3
v5.1-rc4
v5.1-rc5
v5.1-rc6
v5.1-rc7

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1828495

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

List of commits needed to enable ARCH_CAPABILITIES:

Kernel:
commit 2bdb76c015df7125783d8394d6339d181cb5bc30
Author: Xiaoyao Li <email address hidden>
Date: Fri Mar 8 15:57:20 2019 +0800
    kvm/x86: Move MSR_IA32_ARCH_CAPABILITIES to array emulated_msrs

QEMU:

commit 014018e19b3c54dd1bf5072bc912ceffea40abe8
Author: Eduardo Habkost <email address hidden>
Date: Fri Jan 25 20:06:06 2019 -0200
   i386: Make arch_capabilities migratable

commit 485b1d256bcb0874bcde0223727c159b6837e6f8
Author: Eduardo Habkost <email address hidden>
Date: Fri Jan 25 20:06:05 2019 -0200
   i386: kvm: Disable arch_capabilities if MSR can't be set

commit b0a1980384fc265d91de7e09aa5fe531a69e6288
Author: Tao Xu <email address hidden>
Date: Thu Dec 27 10:43:03 2018 +0800
   i386: Update stepping of Cascadelake-Server

commit aec5e9c3a94cf8b7920f59bef69a6f426092c4a0
Author: Bandan Das <email address hidden>
Date: Sun Nov 25 23:17:28 2018 -0500
   kvm: Use KVM_GET_MSR_INDEX_LIST for MSR_IA32_ARCH_CAPABILITIES support

commit 07585923485952bf4cb7da563c9f91fecc85d09c
Author: Robert Hoo <email address hidden>
Date: Mon Oct 15 12:47:24 2018 +0800
   x86: Data structure changes to support MSR based features

commit f57bceb6ab5163ddd6c41ff4344ab8cf28a9c63d
Author: Robert Hoo <email address hidden>
Date: Mon Oct 15 12:47:23 2018 +0800
   kvm: Add support to KVM_GET_MSR_FEATURE_INDEX_LIST and KVM_GET_MSRS system ioctl

commit d86f963694df27f11b3681ffd225c9362de1b634
Author: Robert Hoo <email address hidden>
Date: Mon Oct 15 12:47:25 2018 +0800
   x86: define a new MSR based feature word – FEATURE_WORDS_ARCH_CAPABILITIES

commit c7a88b52f62b30c04158eeb07f73e3f72221b6a8
Author: Tao Xu <email address hidden>
Date: Wed Sep 19 11:11:22 2018 +0800
   i386: Add new model of Cascadelake-Server

commit 3fc7c73139d2d38ae80c3b0bc963b1ac1555924c
Author: Robert Hoo <email address hidden>
Date: Thu Jul 5 17:09:55 2018 +0800
   i386: Add CPUID bit and feature words for IA32_ARCH_CAPABILITIES MSR

commit 8a11c62da9146dd89aee98947e6bd831e65a970d
Author: Robert Hoo <email address hidden>
Date: Thu Jul 5 17:09:58 2018 +0800
   i386: Add new CPU model Icelake-{Server,Client}

commit 8c80c99fcceabd0708a5a83f08577e778c9419f5
Author: Robert Hoo <email address hidden>
Date: Thu Jul 5 17:09:54 2018 +0800
   i386: Add new MSR indices for IA32_PRED_CMD and IA32_ARCH_CAPABILITIES

The top three qemu patches are in qemu 4.0 which we plan as minimum for Ubuntu 19.10.
Tagging up to be part of the Qemu work in Ubuntu 19.10.

The rest of the qemu patches is already in qemu 3.1 which is in Ubuntu 19.04

This patch is also needed for kernels before 4.18

  cd28325249 KVM: VMX: support MSR_IA32_ARCH_CAPABILITIES as a feature MSR ( 25-6-18)

This patchset populates the structure in arch/x86/kvm/x86.c:
   msr_based_features{}

With out this patch, qemu asserts at

  kvm_get_supported_feature_msrs(KVMState *s):
         assert(msr_list.nmsrs > 0);

WORKING

This is a comment summarising a bit the statement of work in this bug:

Bellow are the commits (and the merge requests they came from) that I
could identify by the previous comments. Judging by the selected
commits, the intent is to allow MSR supportability to be queried by
guest through CPUID MSR query emulation <-> kvm ioctls interface.
Every feature to be reported has to have its MSR declared for the CPU
type to be used AND the kvm ioctl backend + kernel support (to query
the actual hardware).

(1)

Request is to allow the following features to be reported by QEMU/KVM:

CPUID.(EAX=7H,ECX=0):EDX[26] (Enable/Disable IBRS/IBPB feature flag):

    Enumerates support for indirect branch restricted speculation
    (IBRS) and the indirect branch predictor barrier (IBPB).
    Processors that set this bit support the IA32_SPEC_CTRL MSR and
    the IA32_PRED_CMD MSR. They allow software to set
    IA32_SPEC_CTRL[0] (IBRS) and IA32_PRED_CMD[0] (IBPB).

and

CPUID.(EAX=7H,ECX=0):EDX[29] (IA32_ARCH_CAPABILITIES feature flag)

   Enumerates support for the IA32_ARCH_CAPABILITIES MSR, allowing
   MSR index 10AH to be read:

   - (bit 0) RDCL_NO: not susceptible to rogue data cache
   - (bit 1) IBRS_ALL: processor supports IBRS
   - (bit 2) RSBA: processor supports RSB alternate (retpol off)
   - (bit 3) SKIP_L1DFL_VMENTRY: vm entry don't flush L1D on VM entry
   - (bit 4) SSB_NO: processor not susceptible to spec store bypass

(2)

There is *no current request* to allow following features to be
reported by EAX_7H_ECX_0_EDX QEMU/KVM right now:

* CPUID.(EAX=7H,ECX=0):EDX[27] STIBP support flag.
* CPUID.(EAX=7H,ECX=0):EDX[28] L1D_FLUSH support flag.
* CPUID.(EAX=7H,ECX=0):EDX[31] SSBD support flag.

OBS: I haven't checked patch dependencies yet, not sure if more
patches are needed yet, just realized that SSBD support flag wasn't
being asked to be backported (nor present in 2.11 version, Bionic
version which we are targetting this to). That explains the small
"statement of work" above.

For now the request was fully understood: I'll work tomorrow in a
backport attempt to check if big pieces in between 2.11 and something
around the v3.0.0-152-g8c80c99fcc .. v4.0.0-rc0-2-g014018e19b range
are missing that would require a major refactoring that would not be
possible to be done.

(3)

Possible points of pain:

- arch_capabilities unmigratable flag inside cpu data structure
  (while CPUID was being developed) turned later on into migratable
  later.

- data structure changes to support MSR based features.

QEMU:

######## MERGE REQUEST

21ee4787e53367590f284915bf4c30c684e65bdf
174a78a8a5c0cf421236fe14efc5559717f050df
bb4928c7cafe50ab2137a0034e350ef1bfa044d9
014018e19b3c54dd1bf5072bc912ceffea40abe8 +
485b1d256bcb0874bcde0223727c159b6837e6f8 +

commit 014018e19b3c54dd1bf5072bc912ceffea40abe8 - v4.0.0-rc0-2-g014018e19b
Author: Eduardo Habkost <email address hidden>
Date: Fri Jan 25 20:06:06 2019

    i386: Make arch_capabilities migratable

    Now that kvm_arch_get_supported_cpuid() will only return
    arch_capabilities if QEMU is able to initialize the MSR properly,
    we know that the feature is safely migratable.

    Signed-off-by: Eduardo Habkost...

Sorry, last comment was posted 3 timex due to bad wrapping and mix of chars because of that. Fixed now, I've hidden 2 broken comments. Thanks

I just realized that from my previous comment I said:

(2)

There is *no current request* to allow following features to be
reported by EAX_7H_ECX_0_EDX QEMU/KVM right now:

* CPUID.(EAX=7H,ECX=0):EDX[27] STIBP support flag.
* CPUID.(EAX=7H,ECX=0):EDX[28] L1D_FLUSH support flag.
* CPUID.(EAX=7H,ECX=0):EDX[31] SSBD support flag.

But the other bug:

https://bugs.launchpad.net/intel/+bug/1830821

Is about enabling SSBD support:

CPUID.(EAX=7H,ECX=0):EDX[29] (IA32_ARCH_CAPABILITIES feature flag)
...
- (bit 4) SSB_NO: processor not susceptible to spec store bypass

and to enable guest to opt out from SSB (speculative store bypass)

### For the kernel team: This QEMU patchset basically adds 2 new CPU types (IceLake and CascadeLake) support to QEMU (i386/target/{pc,kvm}) AND creates those new CPU mitigations-query features in their structure, allowing QEMU to inform guest which mitigations should be in place for the vCPUs through its own MSRs.

QEMU's

commit f57bceb6ab5163ddd6c41ff4344ab8cf28a9c63d
Author: Robert Hoo <email address hidden>
Date: Mon Oct 15 04:47:23 2018

    kvm: Add support to KVM_GET_MSR_FEATURE_INDEX_LIST and KVM_GET_MSRS system ioctl

    Add kvm_get_supported_feature_msrs() to get supported MSR feature index list.
    Add kvm_arch_get_supported_msr_feature() to get each MSR features value.

    Signed-off-by: Robert Hoo <email address hidden>
    Message-Id: <email address hidden>
    Reviewed-by: Eduardo Habkost <email address hidden>
    Signed-off-by: Eduardo Habkost <email address hidden>

Summarizes the kernel support we need for this "RFE" on Bionic (and higher) QEMUs.

Basically we need support for our ioctl() calls being added here:

kvm_ioctl(s, KVM_GET_MSRS, &msr_data);

kvm_ioctl(s, KVM_GET_MSR_FEATURE_INDEX_LIST, &msr_list);

to be backported to Bionic (and HWE in Bionic) kernels.

With this support, QEMU will be able to query IA32_ARCH_CAPABILITIES MSRs.

More information to this feature can be found:

http://kib.kiev.ua/x86docs/SDMs/336996-002.pdf

Chapter 5 (5.1 - Enumeration by CPUID).

Previous chapters describe briefly the vulnerabilities and how mitigations work HW wise.

Rafael - Is this completed for Bionic releases?
Thank you

Hi pragyansri,
If you track the progress on the merge proposal that is linked this looks good atm and will soon be in a PPA [2] with proposed changes for pre-evaluation with Bionic/Cosmic/Disco/Eoan.

From there the next steps are:
- test the new feature from the PPA on cascade lake machines
- general regression tests
- Once we know it works (Tests on newer kernels) will need to also involve the kernel Team to also backport the kernel patches
- if the above look good as well it will become an SRU [1]
- The SRU processing will have it reviewed by the SRU Team, then land in proposed to be final verified and then be released to be available

Sorry if that seems slow to you, but the changes are complex and we don't want to rush things - if you read the SRU [1] policy you'll see that the steady tradeoff of stability-vs-fixes is important to be done with the focus of avoiding regressions.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates
[2]: https://help.launchpad.net/Packaging/PPA?action=show&redirect=PPA

Alright, so we are still testing the version we're working on but here are some test results...

The merge request can be found here:
https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/qemu/+git/qemu/+merge/368804
with code reviews and comments in between the backport attempts.

PPA containing a test version can be found here:
https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1828495

And the HOST and GUEST kernels were: Linux XXX 4.18.0-23-generic #24~18.04.1-Ubuntu SMP Thu Jun 13 17:08:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

since the regular bionic kernel has to have support backported.

I have tested for the MITIGATIONS in:

1) HOST (Bionic)

2) OLD QEMU GUEST (Bionic)

3) NEW QEMU (w/ backport) GUEST (Bionic)

And will show the results:

1) HOST (Bionic)

#### CASCADE LAKE HOST INFORMATION:

Spectre and Meltdown mitigation detection tool v0.42-1-g91d0699

Checking for vulnerabilities on current system Kernel is Linux
4.18.0-23-generic #24~18.04.1-Ubuntu SMP Thu Jun 13 17:08:52 UTC 2019 x86_64
CPU is Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz

--------

Hardware check

* Hardware support (CPU microcode) for mitigation techniques

  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available: YES
    * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates STIBP capability: YES (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability: YES (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available: YES
    * CPU indicates L1D flush capability: YES (L1D flush feature bit)
  * Microarchitecture Data Sampling
    * VERW instruction is available: YES (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: YES
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES
  * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): YES
  * CPU supports Software Guard Extensions (SGX): NO
  * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x6 ucode 0x4000021 cpuid 0x50656)
  * CPU microcode is the latest known available version: NO (latest version is 0x4000024 dated 2019/04/07 according to builtin MCExtractor DB v112 - 2019/05/22)

--------

* CPU vulnerability to the speculative execution attack variants

  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): NO
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): NO
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): NO
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitect...

2) OLD QEMU GUEST (Bionic)

#### REGULAR QEMU GUEST IN OLD QEMU

Spectre and Meltdown mitigation detection tool v0.42-1-g91d0699

Checking for vulnerabilities on current system Kernel is Linux
4.18.0-23-generic #24~18.04.1-Ubuntu SMP Thu Jun 13 17:08:52 UTC 2019 x86_64
CPU is Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz

--------

Hardware check

* Hardware support (CPU microcode) for mitigation techniques

  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available: YES
    * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates STIBP capability: NO
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability: YES (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available: NO
    * CPU indicates L1D flush capability: NO
  * Microarchitecture Data Sampling
    * VERW instruction is available: YES (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO
  * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): NO
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): NO
  * CPU supports Software Guard Extensions (SGX): NO
  * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x6 ucode 0x1 cpuid 0x50656)
  * CPU microcode is the latest known available version: NO (latest version is 0x4000024 dated 2019/04/07 according to builtin MCExtractor DB v112 - 2019/05/22)

--------

* CPU vulnerability to the speculative execution attack variants

  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): YES
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): YES
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): YES
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): YES
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): YE...

3) NEW QEMU (w/ backport) GUEST: Bionic

#### REGULAR QEMU GUEST IN NEW QEMU

Spectre and Meltdown mitigation detection tool v0.42-1-g91d0699

Checking for vulnerabilities on current system Kernel is Linux
4.18.0-23-generic #24~18.04.1-Ubuntu SMP Thu Jun 13 17:08:52 UTC 2019 x86_64
CPU is Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz

-------

Hardware check

* Hardware support (CPU microcode) for mitigation techniques

  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available: YES
    * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available: YES
    * CPU indicates STIBP capability: NO
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability: YES (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available: NO
    * CPU indicates L1D flush capability: NO
  * Microarchitecture Data Sampling
    * VERW instruction is available: YES (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: YES
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES
  * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): NO
  * CPU supports Software Guard Extensions (SGX): NO
  * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x6 ucode 0x1 cpuid 0x50656)
  * CPU microcode is the latest known available version: NO (latest version is 0x4000024 dated 2019/04/07 according to builtin MCExtractor DB v112 - 2019/05/22)

--------

* CPU vulnerability to the speculative execution attack variants

  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): YES
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): YES
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): YES
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory ...

*) SUMMARY:

In between the old and new QEMU, changed results for the mitigations checker was:

#### DIFF between old QEMU and this new one

Hardware check

* Hardware support (CPU microcode) for mitigation techniques

  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: YES
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES

  * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES

  * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO

--------

* CPU vulnerability to the speculative execution attack variants

  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO

--------

## CVE-2017-5715 aka 'Spectre Variant 2, branch target injection'

OLD QEMU:

* Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling)
* Mitigation 1
  * Kernel is compiled with IBRS support: YES
    * IBRS enabled and active: YES (for firmware code only)
  * Kernel is compiled with IBPB support: YES
    * IBPB enabled and active: YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm): NO
  * Kernel compiled with retpoline option: YES
    * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation)
  * Kernel supports RSB filling: YES

> STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability)

NEW QEMU:

* Mitigated according to the /sys interface: YES (Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling)
* Mitigation 1
  * Kernel is compiled with IBRS support: YES
    * IBRS enabled and active: YES
  * Kernel is compiled with IBPB support: YES
    * IBPB enabled and active: YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm): NO
  * Kernel compiled with retpoline option: YES
  * Kernel supports RSB filling: YES

> STATUS: NOT VULNERABLE (IBRS + IBPB are mitigating the vulnerability)

--

## CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load'

OLD QEMU:

* Mitigated according to the /sys interface: YES (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI): YES
  * PTI enabled and active: YES
  * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU: NO

> STATUS: NOT VULNERABLE (Mitigation: PTI)

NEW QEMU:

* Mitigated according to the /sys interface: YES (Not affected)
* Kernel supports Page Table Isolation (PTI): YES
  * PTI enabled and active: NO
  * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU: NO

> STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable)

--

OLD QEMU:

## CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault'

* Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT ...

TL;DR:

HOST MITIGATION FEATURES REPORT:
https://bugs.launchpad.net/intel/+bug/1828495/comments/15

OLD QEMU GUEST MIT FEATURES REPORT:
https://bugs.launchpad.net/intel/+bug/1828495/comments/16

NEW QEMU GUEST MIT FEATURES REPORT:
https://bugs.launchpad.net/intel/+bug/1828495/comments/17

MIT FEATURES REPORT DELTA FROM OLD TO NEW:
https://bugs.launchpad.net/intel/+bug/1828495/comments/18

Meaning we basically have enabled INSIDE the GUEST:

* Hardware support (CPU microcode) for mitigation techniques

  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability: YES
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES
  * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO

and

* CPU vulnerability to the speculative execution attack variants

  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO

@rafaeldtinoco: I found some forther more ugly detail for the stepping change from 5 to 6.
Qemu 3.1 as in Disco had Cascade lake with the stepping 5.
So for Disco the SRU will change the definition.
Which is different to Bionic/Cosmic where we can say "our Cascade definition will just always have been 6" since it didn't exist before the SRU

You'll spot that when you start to create the branch for disco
Note: cosmic has none of the types

Due to that for those users who have set up guests to use "Cascadelake-Server" the SRU on Disco (or the respective cloud archive) will make them need to restart the guest before they can migrate it to a system with the updates applied.

For bionic/cosmic users things will be fine as they have nothing->stepping6
And Eoan will be released with stepping 6

I think that is a drawback, but a tradeoff that we will have to make for the overall gain for much more users. But we need to declare that in the regression considerations of the bug when pushing the SRU.

@rafaeldtinoco: when you have branches and PPAs for Disco as well, could you spawn a Cascadelake-Server guest on a node without the upgrade and then migrate to a note with the upgrade?
I expect an issue, but we should know how exactly it looks.
Once we do we will add that to the SRU template in the description.

This also affects "Cascadelake-Server" defined guests migrating from patched Bionic to unpatched Disco - that will fail. But requiring updates being applied is fine, only the enforced guest restart (which doesn't apply to the just mentioned use-case) is the thing that is really bad.

Overall maybe even worth a NEWS entry for the Disco upload to increase the chance of admins being aware. Please add one on the Disco branch for this.

Alright. I had in mind that (for bionic/cosmic) but we definitely have to mention what you said (for started guests). About the migration, will cause the issue and report back after cosmic/disco backport.

About needed CPU features. I could get the following... these are the needed cpuflags to have a good mitigation status (and opt out from some heavy workarounds/mitigations inside guest):

arch_capabilities=on
ssbd=on
md-clear=on,
wbnoinvd=off
bpb=off
virt-ssbd=off
rdctl-no=yes
ibrs-all=yes
rsba=yes
skip-l1dfl-vmentry=yes
ssb-no=yes

As we've discussed, I backported EOAN's libvirt 5.4.0-0ubuntu2 to Bionic and checked support...

This is the difference when starting CPU as passthrough and CPU as CascadeLake:

$ diff -y one two | grep -E "(>|<)"
arch_capabilities <
arch_perfmon <
ept <
flexpriority <
ibrs_enhanced <
md_clear <
                                                              > pti
ss <
tpr_shadow <
tsc_adjust <
umip <
vmx <
vnmi <
vpid <
xsaves <
xtopology <

As you can see we have to enable the CPU features in order to fully inform guest about HW mitigations and such. Going a bit deeper:

INTEL:

arch-capabilities
invpcid

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Cascadelake-Server</model>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='invpcid'/>
  </cpu>

AMD ONLY:

wbnoinvd
ibpb
virt-ssbd

saw support in libvirt XML files.

COULD NOT FIND specific support for the following CPU features:

ssbd
md-clear
bpb
ibrs-all
rdctl-no
rsba
skip-l1dfl-vmentry

I guess that we will have to backport this support in libvirt, in order to allow QEMU to pick specific CPU mitigation flags.

----

This is the difference from executing QEMU with passthrough versus specifying the Cascadelake Server WITHOUT picking up specific flags (as they are unsupported):

----

HOST has MD_CLEAR, CPU type not:

    * VERW instruction is available: YES (MD_CLEAR feature bit)

HOST has arch_capabilities, CPU type not:

    * CPU indicates ARCH_CAPABILITIES MSR availability: YES
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES

 * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES

 * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES

HOST reports not being vulnerable to:

  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, r...

> ssbd
> md-clear
> bpb
> ibrs-all
> rdctl-no
> rsba
> skip-l1dfl-vmentry
>
> I guess that we will have to backport this support in libvirt, in order
> to allow QEMU to pick specific CPU mitigation flags.

Those are not all missing at least. I have seen ssbd and md-clear for
sure in Bionic e.g. for the latter coming with
ubuntu/bionic-4.0:debian/patches/md-clear.patch

In the context of this bug we will take a look at arch_capabilities
and if some of the others are low hanging fruits.
Quite often - but not always - for libvirt it is just a CPU bit
definition, but as we know e.g. arch_cap is more complex.

@rafael - One more question about Stepping 5/6.
I have formerly read your explanation that stepping 6 would be
required to be able to enable arch_capabilities.
And we planned to add all SRUs with stepping 6 right away and keep the
Delta to consider all versions to be stepping 6.

But then I have seen [1] and in particular [2].
Overall this adds more complexity but also more ability to CPU model
definitions.
But in between the lines in [2] I read that arch_capabilities gets
enabled by default and stepping goes back to 5.

That seems to imply that enabling it on stepping 5 isn't an issue.

If it does work doing the SRU that way it will ease maintenance in the future.
Especially since these versioned types will make maintaining Delta in
that area even more complex.
And it would avoid the upgrade issue on Disco where the CascadeLake
would change.

Therefore I wondered if we could make the backport use stepping 5
which would allow to not carry the delta forever in >=Eoan.
Could you make a PPA build as it is right now, but with stepping 5 and
check if using arch_capabilities still works for you?

[1]: https://<email address hidden>/msg626545.html
[2]: https://<email address hidden>/msg626550.html

https://<email address hidden>/msg626552.html

On the first patch of the set, Eduardo says:

"""
The last patch in the series demonstrates how the new feature can
be used to update a CPU model: it adds a Cascadelake-Server-4.1.1
CPU model, including "arch-capabilities=on" and "stepping=5".
Unfortunately we can't enable arch-capabilities in the -4.1
version of Cascadelake-Server because it would break our existing
runnability guarantees.
"""

He likely discovered partial support for the mitigations on CPUs
with stepping=5 and wanted to implement arch_capabilities in order
to report those back to guests.

From:

https://en.wikichip.org/wiki/intel/microarchitectures/cascade_lake

we have:

"""
Hardware mitigations for:

CVE-2017-5715 - Spectre (Variant 2)
https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)

CVE-2017-5754 - Meltdown (Variant 3)
https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)

CVE-2018-3640 - RSRE - Rogue System Register Read (Variant 3a)
https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)#V3a

CVE-2018-{3620,3646} - Foreshadow (L1 Terminal Fault) VMs,VMM,OS,SMM
https://en.wikipedia.org/wiki/Foreshadow_(security_vulnerability)

*MDS* (Microarchitectural Data Sampling) attacks:
(MDS; MFBDS, RIDL, MSBDS, Fallout, MLPDS, MDSUM)
https://en.wikipedia.org/wiki/Microarchitectural_Data_Sampling

RIDL types:

1. (CVE-2018-12127) - MLPDS - Microarchitectural Load Port Data Sampling
2. (CVE-2018-12130) - MFBDS - Microarchitectural Fill Buffer Data Sampling
3. (CVE-2019-11091) - MDSUM - Microarchitectural Data Sampling Uncacheable Memory

https://mdsattacks.com/files/ridl.pdf

ZombieLoad type:

2. (CVE-2018-12130) - MFBDS - Microarchitectural Fill Buffer Data Sampling

Fallout type:

4. (CVE-2018-12126) - MSBDS - Microarchitectural StoreBuffer Data Sampling

https://mdsattacks.com/files/fallout.pdf

*** NOTE ***

- Steppings 6 and 7 are fully mitigated

- Stepping 5 is NOT PROTECTED against:

  a. MSBDS - (Faulout) Microarchitectural StoreBuffer Data Sampling
  b. MLPDS - (RIDL) Microarchitectural Load Port Data Sampling
  c. MDSUM - (RIDL) Microarchitectural Data Sampling Uncacheable Memory
"""

Commit 6/6 shows:

"""
The new feature will introduce a new host software requirement,
breaking our CPU model runnability promises. This means we can't
enable the new CPU model version by default in QEMU 4.1, because
management software isn't ready yet to resolve CPU model aliases.
This is why the feature is being enabled in a
Cascadelake-Server-4.1.1 CPU model instead of
Cascadelake-Server-4.1.
"""

Based on status on that discussion, unfinished and untested work, and
the fact that we're backporting things, and after our hangout, it is
clear the need to stay with stepping=5 in order to better maintain
qemu in Bionic/Cosmic/Disco, but, like we discussed, informing end
user about how to enable proper CPU flags to advertise HW mitigations
to the GUEST using specific cpu flags (QEMU and libvirt support):

+arch_capabilities (together with, or not):
    +rdctl-no
    +ibrs-all
    +rsba
    +skip-l1dfl-vmentry
    +ssb-no
+ssbd
+md-clear

just like you did in:

https://wiki.ubuntu.com/Security...

we have verified that on 18.04.2 with Rafael David Tinoco (rafaeldtinoco)'s PPA qemy packages. And get that guest doesn't work as expected. After detailed discussion with Intel Upstream, there still needs some patches. But they are under review. Keep watch the process of process.

https://<email address hidden>/msg627282.html

Hello quanxian,

Could you clarify exactly what is the expected behaviour and what wasn't achieved ?
We'd like to understand better the requirements, rather than having patches pointed out only. Mainly because there are some rules we usually follow for already released versions and some of them might be discussable and some might not.

Based on the thread you pointed out:

----
* Patch "i386: Infrastructure for versioned CPU models" was
  rewritten and split in two:
  * i386: Register versioned CPU models
  * i386: Make unversioned CPU models be aliases
* -IBRS, -noTSX, -IBPB CPU models are now aliases
* Enable rdctl-no, ibrs-all, skip-l1dfl-vmentry in
  Cascadelake-Server-v2
* New patch added:
  * i386: Get model-id from CPU object on "-cpu help"
----

Are you looking for the CPU versioning feature ? (Rather than only having stepping increased/decreased ?). Are you looking for Cascadelake-server-v2 CPU type ? Or on having, by default, the CPU flags rdctl-no, ibrs-all, skip-l1dfl-vmentry enabled ?

I'm asking you this because, for backports, we are looking forward to enabling those features ad-hoc, and not by default (as they seem to be in the PPA). Specially because we have to guarantee compatibility in between QEMU diff versions migrations. This has been done already before, for some other requested features.

Thank you very much in advance for your feedback.

Expected results is:
HOST and Guest should enable IA32_ARCH_CAPABILITIES MSR.
MDS_NO is bit 5 of ARCH_CAPABILITIES. Expose this bit to guest.

##cpuid -r
0x00000007 0x00: eax=0x00000000 ebx=0xd19f4fb9 ecx=0x00000818 edx=0x84000000
edx's 29 bit should be 1.

#rdmsr 0x10a -f 5:5
return value should be 1

Actual result is
host works well as expected.
guest not.

Below are details.

Host
# qemu-system-x86_64 -accel kvm -drive if=virtio,id=hd,file=ubuntu-18.04.2-server-amd64.qcow,format=qcow2 -m 4096 -smp 4 -cpu Cascadelake-Server,+arch-capabilities -serial stdio -redir tcp:2223::22

root@PLY02:~# cpuid -r |more
CPU 0:
   0x00000000 0x00: eax=0x00000016 ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
……
   0x00000007 0x00: eax=0x00000000 ebx=0xd39ffffb ecx=0x00000818 edx=0xbc000400

edx is 0xbc000400, 29bit is 1

root@PLY02:~# rdmsr 0x10AH
2b

## 5 bit is 1

Guest

root@test:~# cpuid -r |more
CPU 0:
   0x00000000 0x00: eax=0x0000000d ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69
   ……
   0x00000007 0x00: eax=0x00000000 ebx=0xd19f4fb9 ecx=0x00000818 edx=0x84000000

## edx's 29 bit is 0

root@PLY02:~# rdmsr 0x10AH
0
## 5 bit is 0

Reason:
If you want to see the features that enumerated by MSR_IA32_ARCH_CAPABILITIES in guest with Cascadelake-Server cpu model, just using “-cpu Cascadelake-Server,+arch-capabilities” is not enough.
“-cpu Cascadelake-Server,+arch-capabilities” only let guest see MSR_IA32_ARCH_CAPABILITIES, but it doesn’t contain any feature enumerated by this msr, so the result of rdmsr 0x10a is 0.

If you want to see feature MDS_NO (bit 5) in guest, you should use “-cpu Cascadelake-Server,+arch-capabilities,+mds_no”.

Further, we get 0x2b (bit 0,1,3,5) when rdmsr 0x10a in host, which means host supports “rdctl-no”, “ibrs-all”, “skip-l1dfl-vmentry”, "mds-no".
If we want guest has the same ability as host, not only should we add arch_capabilities explicitly, but also add the features list above explicitly. Otherwise we cannot see these features in guest.

In a word, it’s all due to current Cascadelake-Server cpu model. It lacks all above.
After new version of Cascadelake-Server added in qemu, we can get rid of all these manually adding features annoyance.

Eduardo has sent out the qemu patch for versioned cpu model and patch 09 of which contains new version of Cascadelake Server cpu model. It depends on when they are merged.
https://<email address hidden>/msg627282.html

The series was merged in 3a1acf5d47295d22ffdae0982a2fd808b802a7da as a prep to qemu 4.1.
But the changes are rather invasive and after a review I think for the SRU we will not add them.

For example the changes around:
  "model runnability guarantees won't apply to unversioned CPU models anymore"
seems non backportable to me.
I agree it is the right path going forward to keep the proliferation of CPU models under control and provide something like a "moving head" weakening the runability guarantees that can be regained by fetching the exact versions via alias-of.

But none of the old systems management stacks will be able to do so.
Yes there is [2] but I'm still scared in an SRU context to add that when it is a valid (more effort but working) option to add model+flags for the time being.

And in addition the code for making the models versionable will be another great set of backports with potential backport flaws added.

Hence at least for now my decision would be:
- make the features available at least to the last LTS (Bionic) which will reach
  LTS-1 for many (to admit not all) users via Ubuntu cloud archive
- do not backport the versioned CPU model changes (too much risk / incompatibilities)
  - pro: no changes to the higher virt stack needed
  - pro: less regression risk for the SRUs
  - pro: the new security features will be available
  - con: security features must be enabled individually via feature flags

@Rafael that should match what you had prepared in your MPs already.
I'll create a PPA with your any my changes for general regression tests.

@Quaxlan - for the versioned CPUs an all that belongs to that context I'd ask you to file a new bug. I expect that will be in qemu 4.1 (just entered rc0) and a later (yet to be defined) libvirt version. Once all of that has properly landed we can consider if we will pull it back into Eoan (as long as it isn't released and under the SRU policy) OR if we wait for these new features to properly arrive in the next Ubuntu release being 20.04. Once you happen to know which libvirt version will have the appropriate changes to properly tolerate and exploit what was added to qemu via [1] (version and commits if you have both) let us know there.

[1]: https://<email address hidden>/msg627282.html
[2]: https://<email address hidden>/msg628326.html

And qemu 4.0 is in Eoan, so this release is "fix released" (for the scope of adding the features, not the just discussed versioned CPUs)

Tags pushed and uploaded to B/D unapproved for the SRU Team to do a final review and accept.

I also marked Cosmic as Won't Fix as it will be out of support before this SRU completes.

Further I added libvirt Tasks where the next step is for rafaeldtinoco to find if/what we'd need to make the added qemu feature more accessible/usable by higher levels (Bionic/Disco).

Some updates to libvirt:
- we already have ssbd/md-clear through security updates
- rdctl-no, ibrs-all, skip-l1dfl-vmentry, mds-no are part of
  c8ec678f cpu_map: Introduce IA32_ARCH_CAPABILITIES MSR features
- arch_capabilities itself comes in 511df17a

There are also updates to the cascade/icelake cpu types. They related to the qemu commits, but not necessary for the feature support itself. Yet having those named types recognized would be nice to have along the qemu feature backports (If they cause trouble we will skip them).
 2878278c cpu_map: Add Cascadelake-Server CPU model
 5cae1f47 cpu_map: Use and install Icelake model definitions
 993d85ae cpu_map: Add Icelake CPU models

We have not backported any of the features associated with:
 98130811 cpu_map: Add features for Icelake CPUs
Further there also is stibp in:
  eb1b551d cpu: Add support for "stibp" x86_64 feature
But that isn't the preferred way to mititgate anyway and hence isn't backported in qemu for now, see https://lwn.net/Articles/773118/ for some details.
OTOH it would not really hurt to "detect" those properly, that does not mean they would be used on a qemu not supporting them.
And having them reduces some context noise, as above we will try to backport but if those cause trouble we might skip them.

All the pacthes above have some context/series they need on top.
There also were some file renaming actions, so this needs quite some effort to have the backports stay sane.
We have to check that and create a full list of dependent changes.

Some of the changes realize that qemu can't always present what it can support in older versions (unavailable-features probing). So host-model woudl skip them, but the core code for that is in qemu since 2016. I have not found a unavailable-features-MSR patch for qemu - but if there is one this might be needed.

The list above extended by some context patches that we might need creates this overall list that might be a good start:
(further indent being context to main changes)
   8eb4a89f qemu: Forbid MSR features with old QEMU
   2674d00e qemu: Drop MSR features from host-model with old QEMU
 c8ec678f cpu_map: Introduce IA32_ARCH_CAPABILITIES MSR features
   bcfed7f1 cpu_x86: Introduce virCPUx86FeatureFilter*MSR
   b8e086a5 cpu_x86: Turn virCPUx86DataIteratorInit into a function
   4e6f58b8 conf: Introduce virCPUDefCheckFeatures
 4a0f604d cpu_map: Distribute x86_Cascadelake-Server.xml
 2878278c cpu_map: Add Cascadelake-Server CPU model
 511df17a cpu_map: Add support for arch-capabilities feature
   eb1b551d cpu: Add support for "stibp" x86_64 feature
 5cae1f47 cpu_map: Use and install Icelake model definitions
 993d85ae cpu_map: Add Icelake CPU models
   98130811 cpu_map: Add features for Icelake CPUs

Hello quanxian, or anyone else affected,

Accepted qemu into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:3.1+dfsg-2ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello quanxian, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted qemu (1:2.11+dfsg-1ubuntu7.16) for bionic have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#bionic

All autopkgtests for the newly accepted qemu (1:3.1+dfsg-2ubuntu3.3) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

I still need to verify this for the SRUs. There are no - up to now - regressions being showed in excuses for Disco and/or Bionic. Will provide verification soon.

## USING HOST CPU ONLY:

sudo /usr/bin/qemu-system-x86_64 -name guest="guest" -machine accel=kvm -cpu host -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 7e55c71a-558f-412c-8445-db0e95fc549f -display none -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -kernel /var/lib/libvirt/images/guest/vmlinuz -initrd /var/lib/libvirt/images/guest/initrd.img -append "root=/dev/vda noresume console=tty0 console=ttyS0,38400n8 apparmor=0 net.ifnames=0 crashkernel=256M" -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/guest/disk01.ext4.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on -serial stdio

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke avx512_vnni md_clear arch_capabilities

bugs : spectre_v1 spectre_v2 spec_store_bypass mds

## ARCH_CAPABILITIES + MITIGATION FLAGS ARGUMENTS GIVEN

sudo /usr/bin/qemu-system-x86_64 -name guest="guest" -machine accel=kvm -cpu host,arch_capabilities=on,ssbd=on,md-clear=on,rdctl-no=yes,ibrs-all=yes,skip-l1dfl-vmentry=yes -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 7e55c71a-558f-412c-8445-db0e95fc549f -display none -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -kernel /var/lib/libvirt/images/guest/vmlinuz -initrd /var/lib/libvirt/images/guest/initrd.img -append "root=/dev/vda noresume console=tty0 console=ttyS0,38400n8 apparmor=0 net.ifnames=0 crashkernel=256M" -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/guest/disk01.ext4.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on -serial stdio

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fa...

I still have to check if default CascadeLake CPU with no flags will enable the features by default (should not) and verify Disco SRU. Unfortunately I have shutdown the remote machine (virsh console's fault :o) and will come back to this as soon as it is up again.

Hello Rafael,

Testing results to share, looks like the exposure of eIBRS into the Guest is complete. Don't see Bit 5 implemented yet for MDS bit for Arch Capability.

See below for details and let me know if you need any specific based on these configuration.

Regards, Ai B.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Tested on Host:- Ubuntu 18.04.1 Kernel 4.15.0-55-generic

#virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1

#lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
Stepping: 6
CPU MHz: 800.144
CPU max MHz: 2100.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

#rdmsr 0x10a
2b

qemu:
  Installed: (none)
  Candidate: 1:2.11+dfsg-1ubuntu7.16~ppa1
  Version table:
     1:2.11+dfsg-1ubuntu7.16~ppa1 500
        500 http://ppa.launchpad.net/rafaeldtinoco/lp1828495/ubuntu bionic/main amd64 Packages
     1:2.11+dfsg-1ubuntu7.15 500
        500 http://cn.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     1:2.11+dfsg-1ubuntu7.14 500
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     1:2.11+dfsg-1ubuntu7 500
        500 http://cn.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

#/usr/bin/qemu-system-x86_64 -name guest=vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-vm1/master-key.aes -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 32768 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid ee83a263-89e0-47ca-81a3-9fa41c73b645 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-vm1/moni...

Hello Ai Lim,

Thanks for your feedback.. indeed we have not backported the following patch:

commit 20140a82c67467f53814ca197403d5e1b561a5e5
Author: Paolo Bonzini <email address hidden>
Date: Thu May 16 15:53:20 2019

    target/i386: add MDS-NO feature

    Microarchitectural Data Sampling is a hardware vulnerability which allows
    unprivileged speculative access to data which is available in various CPU
    internal buffers.

    Some Intel processors use the ARCH_CAP_MDS_NO bit in the
    IA32_ARCH_CAPABILITIES
    MSR to report that they are not vulnerable, make it available to guests.

    Signed-off-by: Paolo Bonzini <email address hidden>
    Message-Id: <email address hidden>
    Signed-off-by: Eduardo Habkost <email address hidden>

The documentation I had:

336996-Speculative-Execution-Side-Channel-Mitigations.pdf, from Intel, showed bits 0-4 only, last feature I had documented for ARCH_CAPABILITIES was SSB_NO. Turns out there is MDS-NO feature, in bit 5, to be backported (Disco & Bionic). Do you know if there is a newer document from Intel showing specs for MDS-NO + ARCH_CAPABILITIES ?

Nevertheless, I'll provide you the backports in a PPA, for testing, first thing in my morning.

Sorry for missing this one.

Best Regards

Rafael

Hello Ai Lim,

I have just uploaded a new version (~ppa2) to the same PPA:

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1828495

for Bionic. Could you test this version and let me know if it presents MDS_NO MSR flag ?

I'll test on my side (as soon as the package is compiled in the PPA) as well.

Thanks a lot for the feedback. I'll provide the same change for Ubuntu Disco.

@Ai Lim, it would be great to get your testing on the PPA asap.

@Rafael - we already have the fixup. I have provided feedback on the MP, I think we can quickly sponsor something new. That means we don't need to go the full verification-failed, reject, new upload path - instead we will ask to accept the new version with the minor fixup that was needed.

Alright, working on it.

Okay,

So said in the merge requests, Bionic needed a new version with:

e6891e7... by Rafael David Tinoco <email address hidden> 3 minutes ago
changelog

c27fa94... by Rafael David Tinoco <email address hidden> on 2019-08-02
    - 0017-target-i386-add-MDS-NO-feature.patch (LP: #1828495):
      target/i386: add MDS-NO feature
      (upstream: 20140a82c674, desc: v4.0.0-721-g20140a82c6)

And the changelog stayed:

qemu (1:2.11+dfsg-1ubuntu7.17) bionic; urgency=medium

  * {Ice,Cascade}Lake IA32_ARCH_CAPABILITIES support (LP: 1828495)
    Needed patch is in d/p/u/lp1828495-:
    - 0017-target-i386-add-MDS-NO-feature.patch:
      target/i386: add MDS-NO feature

 -- Rafael David Tinoco <email address hidden> Mon, 05 Aug 2019 19:12:08 +0000

qemu (1:2.11+dfsg-1ubuntu7.16) bionic; urgency=medium

  * {Ice,Cascade}Lake CPUs + IA32_ARCH_CAPABILITIES support (LP: #1828495)
    Needed patches are in d/p/u/lp1828495-:
    - 0001-guidance-cpu-models.patch:
      docs: add guidance on configuring CPU models for x86
      + d/qemu-system-common.install: include man/man7/qemu-cpu-models.7
    - 0002-msr-new-msr-indices.patch:
      i386: Add new MSR indices for IA32_PRED_CMD and IA32_ARCH_CAPABILITIES
    - 0003-cpuid-feature-ia32-arch-capabilities.patch:
      i386: Add CPUID bit and feature words for IA32_ARCH_CAPABILITIES MSR
    - 0004-cpuid-bit-for-wbnoinvd.patch:
      i386: Add CPUID bit for WBNOINVD
    - 0005-new-cpu-model-for-icelake.patch:
      i386: Add new CPU model Icelake-{Server,Client}
    - 0006-update-headers-to-4.16-rc5.patch:
      update Linux headers to 4.16-rc5
    - 0007-kvm-get-msr-feature-index_list.patch:
      kvm: Add support to KVM_GET_MSR_FEATURE_INDEX_LIST and
    - 0008-x86-msr-related-data-structure-changes.patch:
      x86: Data structure changes to support MSR based features
    - 0009-feature-wordS-arch-capabilities.patch:
      x86: define a new MSR based feature word -- FEATURE_WORDS_ARCH
    - 0010-use-kvm-get-msr-index-list.patch:
      kvm: Use KVM_GET_MSR_INDEX_LIST for MSR_IA32_ARCH_CAPABILITIES support
    - 0011-disable-arch-cap-when-no-msr.patch:
      i386: kvm: Disable arch_capabilities if MSR can't be set
    - 0012-arch-capabilities-migratable.patch:
      i386: Make arch_capabilities migratable
    - 0013-cascadelake-server.patch:
      i386: Add new model of Cascadelake-Server
    - 0014-remove-cpuid-pconfig.patch:
      i386: remove the new CPUID 'PCONFIG' from Icelake-Server CPU model
    - 0015-remove-cpuid-intel_pt.patch:
      i386: remove the 'INTEL_PT' CPUID bit from named CPU models
    - 0016-no-ospke-on-some.patch:
      i386: Disable OSPKE on CPU model definitions

 -- Rafael David Tinoco <email address hidden> Wed, 19 Jun 2019 19:48:48 +0000

We need the SRU for the last 2 commits (already pushed in the MR).

=================

For Disco, we're good because of security maintenance you had already done previously:

qemu (1:3.1+dfsg-2ubuntu3.1) disco-security; urgency=medium

  * SECURITY UPDATE: Add support for exposing md-clear functionality
    to guests
    - d/p/ubuntu/enable-md-clear.patch
    - d/p/ubuntu/enable-md-no.patch
    ...

And enable-md-no.patch made the tr...

# Disco verification:

ubuntu@disco:~$ sudo /usr/bin/qemu-system-x86_64 -name guest="guest" -machine accel=kvm -cpu host,+arch-capabilities,+ssbd,+md-clear,+rdctl-no,+ibrs-all,+skip-l1dfl-vmentry,+mds-no -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 7e55c71a-558f-412c-8445-db0e95fc549f -display none -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -kernel /var/lib/libvirt/images/guest/vmlinuz -initrd /var/lib/libvirt/images/guest/initrd.img -append "root=/dev/vda noresume console=tty0 console=ttyS0,38400n8 apparmor=0 net.ifnames=0 crashkernel=256M" -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/guest/disk01.ext4.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on -serial stdio

-> changing "host" to CascadeLake-Server also works the same way.

Provided me:

inaddy@guest:~$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
stepping : 6
microcode : 0x1
cpu MHz : 2095.076
cache size : 16384 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke avx512_vnni md_clear arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass
bogomips : 4190.15
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

AND reading the MSR directly:

inaddy@guest:~$ sudo rdmsr 0x10a
2b

We have bits: 0 1 3 and 5 like it should be.

-------------------------

Running same QEMU cmd line, enabling +arch-capabilities in CascadeLake-Server but without specifying any other CPU flags:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase...

I'll wait for the next SRU for bionic to be in -proposed to provide Bionic full verification just like I did to Disco. It shall not take too long and the version currently in -proposed will be superseded by the one we have in the merge request right now.

Thanks Rafael, I reviewed the MP, thanks for the fixup.

I sponsored it into bionic-unapproved.
It would be great if the SRU Team could evaluate and accept this minor update to the Former upload.

Hello quanxian, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

From the MR:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/qemu/+git/qemu/+merge/368804/+index?

I'm testing now the Bionic version in -proposed: 7.17

Containing the missing MDS-NO arch-capabilities flag.

Running the following command:

$ sudo /usr/bin/qemu-system-x86_64 -name guest="guest" -machine accel=kvm -cpu host,+arch-capabilities,+ssbd,+md-clear,+rdctl-no,+ibrs-all,+skip-l1dfl-vmentry,+mds-no -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 7e55c71a-558f-412c-8445-db0e95fc549f -display none -no-user-config -nodefaults -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -kernel /var/lib/libvirt/images/guest/vmlinuz -initrd /var/lib/libvirt/images/guest/initrd.img -append "root=/dev/vda noresume console=tty0 console=ttyS0,38400n8 apparmor=0 net.ifnames=0 crashkernel=256M" -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/guest/disk01.ext4.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on -serial stdio

This is the cpu_flags:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpxavx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke avx512_vnni md_clear arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass

And reading MSR directly:

$ sudo rdmsr 0x10a
2b

We also have bits in place: 0 1 3 5, meaning it is good for all announced mitigations.

Note: Running the same command with Cascadelake-Server as cpu has the same results.

------------------------------

Running Cascadelake-Server CPU without setting any CPU flags:

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat pku ospke avx512_vnni arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds

$ sudo rdmsr 0x10a
0

Expected as we are not enabling the mitigations by default.

------------------------------

With all the tests I'm settin...

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:3.1+dfsg-2ubuntu3.3

---------------
qemu (1:3.1+dfsg-2ubuntu3.3) disco; urgency=medium

  [ Christian Ehrhardt ]
  * d/p/ubuntu/lp-1830243-s390-bios-Skip-bootmap-signature-entries.patch:
    tolerate guests with secure boot loaders (LP: #1830243)

  [ Rafael David Tinoco ]
  * {Ice,Cascade}Lake CPUs IA32_ARCH_CAPABILITIES support (LP: #1828495)
    Needed patches are in d/p/u/lp1828495-:
    - 0011-disable-arch-cap-when-no-msr.patch (LP: #1828495):
      i386: kvm: Disable arch_capabilities if MSR can't be set
    - 0012-arch-capabilities-migratable.patch (LP: #1828495):
      i386: Make arch_capabilities migratable
    - 0014-remove-cpuid-pconfig.patch
      i386: remove the new CPUID 'PCONFIG' from Icelake-Server CPU model
    - 0015-remove-cpuid-intel_pt.patch
      i386: remove the 'INTEL_PT' CPUID bit from named CPU models
    - 0016-no-ospke-on-some.patch (LP: #1828495):
      i386: Disable OSPKE on CPU model definitions

 -- Christian Ehrhardt <email address hidden> Thu, 04 Jul 2019 14:47:56 +0200

Hello Rafael,

Testing results to share, Bit 5 Arch Capability is verified implemented.
See below for details, please feel free to let me know if you need more information.
Thanks.

Regards, Ai B.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Tested on Host:- Ubuntu 18.04.1 Kernel 4.15.0-55-generic

#virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.11.1

#lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
Stepping: 6
CPU MHz: 800.144
CPU max MHz: 2100.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,40-59
NUMA node1 CPU(s): 20-39,60-79
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

#rdmsr 0x10a
2b

qemu:
  Installed: (none)
  Candidate: 1:2.11+dfsg-1ubuntu7.17~ppa1
  Version table:
     1:2.11+dfsg-1ubuntu7.17~ppa1 500
        500 http://ppa.launchpad.net/rafaeldtinoco/lp1828495/ubuntu bionic/main amd64 Packages
     1:2.11+dfsg-1ubuntu7.15 500
        500 http://cn.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     1:2.11+dfsg-1ubuntu7.14 500
        500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     1:2.11+dfsg-1ubuntu7 500
        500 http://cn.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Guest OS CentOS 7.6 kernel 3.10.0-957.12.2.el7.x86_64
#lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) ...

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.17

---------------
qemu (1:2.11+dfsg-1ubuntu7.17) bionic; urgency=medium

  * {Ice,Cascade}Lake IA32_ARCH_CAPABILITIES support (LP: 1828495)
    Needed patch is in d/p/u/lp1828495-:
    - 0017-target-i386-add-MDS-NO-feature.patch:
      target/i386: add MDS-NO feature

qemu (1:2.11+dfsg-1ubuntu7.16) bionic; urgency=medium

  [ Christian Ehrhardt ]
  * d/p/ubuntu/lp-1830243-s390-bios-Skip-bootmap-signature-entries.patch:
    tolerate guests with secure boot loaders (LP: #1830243)

  [ Rafael David Tinoco ]
  * {Ice,Cascade}Lake CPUs + IA32_ARCH_CAPABILITIES support (LP: #1828495)
    Needed patches are in d/p/u/lp1828495-:
    - 0001-guidance-cpu-models.patch:
      docs: add guidance on configuring CPU models for x86
      + d/qemu-system-common.install: include man/man7/qemu-cpu-models.7
    - 0002-msr-new-msr-indices.patch:
      i386: Add new MSR indices for IA32_PRED_CMD and IA32_ARCH_CAPABILITIES
    - 0003-cpuid-feature-ia32-arch-capabilities.patch:
      i386: Add CPUID bit and feature words for IA32_ARCH_CAPABILITIES MSR
    - 0004-cpuid-bit-for-wbnoinvd.patch:
      i386: Add CPUID bit for WBNOINVD
    - 0005-new-cpu-model-for-icelake.patch:
      i386: Add new CPU model Icelake-{Server,Client}
    - 0006-update-headers-to-4.16-rc5.patch:
      update Linux headers to 4.16-rc5
    - 0007-kvm-get-msr-feature-index_list.patch:
      kvm: Add support to KVM_GET_MSR_FEATURE_INDEX_LIST and
    - 0008-x86-msr-related-data-structure-changes.patch:
      x86: Data structure changes to support MSR based features
    - 0009-feature-wordS-arch-capabilities.patch:
      x86: define a new MSR based feature word -- FEATURE_WORDS_ARCH
    - 0010-use-kvm-get-msr-index-list.patch:
      kvm: Use KVM_GET_MSR_INDEX_LIST for MSR_IA32_ARCH_CAPABILITIES support
    - 0011-disable-arch-cap-when-no-msr.patch:
      i386: kvm: Disable arch_capabilities if MSR can't be set
    - 0012-arch-capabilities-migratable.patch:
      i386: Make arch_capabilities migratable
    - 0013-cascadelake-server.patch:
      i386: Add new model of Cascadelake-Server
    - 0014-remove-cpuid-pconfig.patch:
      i386: remove the new CPUID 'PCONFIG' from Icelake-Server CPU model
    - 0015-remove-cpuid-intel_pt.patch:
      i386: remove the 'INTEL_PT' CPUID bit from named CPU models
    - 0016-no-ospke-on-some.patch:
      i386: Disable OSPKE on CPU model definitions

 -- Rafael David Tinoco <email address hidden> Mon, 05 Aug 2019 19:12:08 +0000

Recent libvirt versions (5.5) added more for arch_capabilities.
Lets get that into Eoan before considering the SRUs back to Bionic afterwards.

commit ver subject
2674d00e 5.5 qemu: Drop MSR features from host-model with old QEMU
8eb4a89f 5.5 qemu: Forbid MSR features with old QEMU
c8ec678f 5.5 cpu_map: Introduce IA32_ARCH_CAPABILITIES MSR features
56b254dc 5.5 cpu_x86: Read CPU features from IA32_ARCH_CAPABILITIES MSR
bcfed7f1 5.5 cpu_x86: Introduce virCPUx86FeatureFilter*MSR
b8e086a5 5.5 cpu_x86: Turn virCPUx86DataIteratorInit into a function
4e6f58b8 5.5 conf: Introduce virCPUDefCheckFeatures

511df17a 5.0 cpu_map: Add support for arch-capabilities feature

I started a test PPA [1] and an MP [2] to get this into Eoan.
@Rafael could you test this on the same system you used last time and review the MP?

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1828495-archcap-eoan
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/371504

Definitely! Will provide feedback soon!

Another bunch of related changes might be important.
Not sure how much of that will go into SRUs - I hope not all of it.
Already in Eoan we should try to use a reduced set or we could go directly to 5.5 which has other known issues.

63acb7bf qemu_process: Prefer generic qemuMonitorGetGuestCPU
cc6d6b3c qemu: Introduce generic qemuMonitorGetGuestCPU
430023e5 qemu: Add type filter to qemuMonitorJSONParsePropsList
df73078c cpu: Introduce virCPUDataAddFeature
055f8f6b qemu: Make qemuMonitorGetGuestCPU usable on x86 only
a3f2c802 qemu: Don't use full CPU model expansion
ec232c5d qemu: Translate feature names from query-cpu-model-expansion
5030a745 qemu_command: Use canonical names of CPU features
6f6401fb qemu: Probe host CPU after capabilities
XX 0d254bce qemu: Probe for "unavailable-features" CPU property
XX 2a4c2321 qemu: Probe for max-x86_64-cpu type
61ee757e qemu: Add APIs for translating CPU features
24aa210d qemuxml2argvtest: Add test for CPU features translation
0b763774 qemu: Filter CPU features in active XML
XX c145b660 cpu_conf: Introduce virCPUDefFilterFeatures
955fd6e7 qemu_process: Drop cleanup label from qemuProcessUpdateGuestCPU
b1286526 qemu: Drop qemuFeatureNoEffect

I hope I might get away with only those I marked with XX, but still this keeps growing to an uncompfortable amount

Tests confirmed that the recent libvirt did in fact work fine.
FYI - this was analyzed in bug 1841066 which will also be the bug to track this little qemu addition which has to go alongside the libvirt portion of this once considering SRUs.

I was by accident changing the kernel task, fixed now.

@Christian,

I'm flagging the libvirt SRUs as Opinion. So far I have done quite a lot cherry-picks and backports identifying all the structures that have changed and I'm not currently seeing how this can be done as a SRU (even for Disco). There are quite a few data model changes to how libvirt was acquiring CPU features (and, now, missing features) that simply wont fit into SRU rules.

The base need for the features would be:

commit c145b660b8225f73db16660461077ef931730939
Author: Jiri Denemark <email address hidden>
Date: Fri Jun 7 09:07:10 2019

    cpu_conf: Introduce virCPUDefFilterFeatures

    This new internal API can be used for in place filtering of CPU features
    in virCPUDef.

    Signed-off-by: Jiri Denemark <email address hidden>
    Reviewed-by: Ján Tomko <email address hidden>

----

commit 2a4c23210674b453f91569f0f4b9fd5ebe8d7906
Author: Jiri Denemark <email address hidden>
Date: Mon Jun 10 11:46:10 2019

    qemu: Probe for max-x86_64-cpu type

    We will use it to check whether QEMU supports a specific CPU property.

    Signed-off-by: Jiri Denemark <email address hidden>
    Reviewed-by: Ján Tomko <email address hidden>

----

commit 0d254bce4ec6fd62c0277d24e28bf018a4c6cb37
Author: Jiri Denemark <email address hidden>
Date: Mon Jun 10 11:49:22 2019

    qemu: Probe for "unavailable-features" CPU property

    It is similar to "filtered-features" property, which reports CPUID bits
    corresponding to disabled features, but more general. The
    "unavailable-features" property supports both CPUID and MSR features by
    listing their names.

    Signed-off-by: Jiri Denemark <email address hidden>
    Reviewed-by: Ján Tomko <email address hidden>

----

commit 4e6f58b8d55d44fa9f80736b2745b44710f6e25a
Author: Jiri Denemark <email address hidden>
Date: Wed Jun 19 14:01:30 2019

    conf: Introduce virCPUDefCheckFeatures

    This API can be used to check whether a CPU definition contains features
    matching a given filter.

    Signed-off-by: Jiri Denemark <email address hidden>
    Reviewed-by: Ján Tomko <email address hidden>

----
commit b8e086a570b14b1f83fc07e25df6da758abe7706
Author: Jiri Denemark <email address hidden>
Date: Wed Jun 19 16:58:01 2019

    cpu_x86: Turn virCPUx86DataIteratorInit into a function

    Until now, this was a macro usable for direct initialization when a
    variable is defined. Turning the macro into a function makes it more
    general.

    Signed-off-by: Jiri Denemark <email address hidden>
    Reviewed-by: Ján Tomko <email address hidden>

< compiles good >

Then we would need the backport of:

cpu_x86: Introduce virCPUx86DataItem container struct

The following patches introduce CPU features read from MSR in addition
to those queried via CPUID instruction. Let's introduce a container
struct which will be able to describe either feature type.

Signed-off-by: Jiri Denemark <email address hidden>
Reviewed-by: Ján Tomko <email address hidden>

AND all the ones bellow:

fcf4846a6b cpu_x86: Add support for storing MSR features in CPU map
370177e2f6 cpu_x86: Store virCPUx86DataItem content in union
10b80165db cpu_x86: Make x86cpuidMatch more general
2eea67a98e cpu_x86: Make x86cpuidMatchMasked more genera...

Hi Rafael,
thanks for giving it try. We knew it might be complex when we first saw the changes.
And I consider it wise to - at some point - step back and realize this won't be SRUable.
I'll summarize what we know about the libvirt portion of this:

## SRUability ##
The changes are rather complex, and will most likely not be SRUable.
Even if prepared for an upload they are massive and with a huge risk of regressing other use cases which we'd generally want to avoid.
I haven't done/tried the porting myself, but your doc reads fine and it grows and grows to become borderline to considering a major version bump which isn't an option here.

## How strict is the need for these ##
1. I know - after all I was a performance engineer for a decade - that speed is important. And those extra mitigation flags are all about improved mitigations for speed. But TBH, that also makes them not strictly required for the function.
2. After all the CPU features we are talking about here are still rare. You only get them at the very latest CPUs. So the chances that an existing server Farm needs those changes desperately are low. This will mostly be for consideration of new setups, and they can/should use the new code of the new release.

## Availability ##
1. The majority of this code is upstream in 5.5 and we backported it to 5.4 for Eoan - so there is a Ubuntu release that can use this code already.
2. LTS users in Bionic (way more than Eoan I'd think) can get also access to it via the Ubuntu Cloud Archive [1]. And that is not only true for the now released 5.4, but when we release Ubuntu 20.04 there will be a new UCA along it containing all the final upstream fixes (not only our backports) - and that will be supported for an even longer time.

## Verdict ##
I second your call on these patches after reading your summary, lets call the related libvirt backports Won't Fix for now.

## TODOs ##
@Rafael - do you still have a TODO on the kernel side as there are tasks open and assigned?
@Rafael - while I generally dislike raw qemu-cmdline in XML documenting an example as you suggested might be useful, will you come up with a draft for it?
@Rafael/Christian - lets also talk with the Team about it to get everyone on the same page.

[1]: https://wiki.ubuntu.com/OpenStack/CloudArchive

+1 on your summary, looks very good.

I will review kernel changes (if they are present or not) and change status accordingly.

+1 on the team update.

o/

For the Kernel Support:

---------------

commit 28c1c9fabf48d6ad596273a11c46e0d0da3e14cd
Author: KarimAllah Ahmed <email address hidden>
Date: Thu Feb 1 19:59:44 2018

    KVM/VMX: Emulate MSR_IA32_ARCH_CAPABILITIES

Disco: OK - https://bugs.launchpad.net/bugs/1823060
Bionic: OK - https://bugs.launchpad.net/bugs/1838116

-

commit 801e459a6f3a63af9d447e6249088c76ae16efc4
Author: Tom Lendacky <email address hidden>
Date: Wed Feb 21 16:39:51 2018

    KVM: x86: Add a framework for supporting MSR-based features

Disco: OK
Bionic: OK - since Ubuntu-4.15.0-32.34

-

commit 772439717dbf703b39990be58d8d4e3e4ad0598a
Author: Konrad Rzeszutek Wilk <email address hidden>
Date: Wed Apr 25 23:04:22 2018

    x86/bugs/intel: Set proper CPU features and setup RDS

Disco: OK
Bionic: OK - since Ubuntu-4.15.0-22.24

-

commit 9f65fb29374ee37856dbad847b4e121aab72b510
Author: Konrad Rzeszutek Wilk <email address hidden>
Date: Wed May 9 16:41:38 2018

    x86/bugs: Rename _RDS to _SSBD

Disco: OK
Bionic: OK - since Ubuntu-4.15.0-22.24

-

commit 1eaafe91a0df4157521b6417b3dd8430bf5f52f0
Author: Jim Mattson <email address hidden>
Date: Wed May 9 18:29:35 2018

    kvm: x86: IA32_ARCH_CAPABILITIES is always supported

Disco: OK
Bionic: OK - http://bugs.launchpad.net/bugs/1786352

-

commit cd28325249a1ca0d771557ce823e0308ad629f98
Author: Paolo Bonzini <email address hidden>
Date: Mon Jun 25 09:04:37 2018

    KVM: VMX: support MSR_IA32_ARCH_CAPABILITIES as a feature MSR

Disco: OK
Bionic: OK - since Ubuntu-4.15.0-32.34

-

commit 0cf9135b773bf32fba9dd8e6699c1b331ee4b749
Author: Sean Christopherson <email address hidden>
Date: Thu Mar 7 20:43:02 2019

    KVM: x86: Emulate MSR_IA32_ARCH_CAPABILITIES on AMD hosts

Disco: OK - https://bugs.launchpad.net/bugs/1823060
Bionic: OK - https://bugs.launchpad.net/bugs/1838116

-

commit 2bdb76c015df7125783d8394d6339d181cb5bc30
Author: Xiaoyao Li <email address hidden>
Date: Fri Mar 8 04:57:20 2019

    kvm/x86: Move MSR_IA32_ARCH_CAPABILITIES to array emulated_msrs

Disco: OK - https://bugs.launchpad.net/bugs/1830934
Bionic: Missing - Possibly no need

---------------

I believe we're all set. I'll have to test in CascadeLake machine to make though.