Unable to use Skylake capability on Cascadelake server

Upstream fixes https://listman.redhat.com/archives/libvir-list/2020-March/msg01197.html

Hi,
what a coincidence - I was just about backporting these changes for a special build.
Thereby I know that one more or less needs the full series that is seen in that link.

Many are reworks of the allocation which is rather safe as Focal already had started the glib allocation - so everything is in place. Never the less it is a 39 patch series that needs to be thoroughly checked for potential regressions.

I'll let you know once there is a PPA to test.

The code is in v6.3.0 and later thereby Groovy and later are fixed.
For Pre-Focal these changes make not much sense as the HW support for both chips isn't in Bionic anyway.

I will be happy to check the assembly as soon as it is ready, because this is the only option for us now to expand the cluster

I have a first stab at this ready in a PPA [1] and would appreciate it very much if you could test that in regard to your reported case. If it works in general for you I'd be very interested if you could also try a migration between a normal and an updated host as that can sometimes also expose issues when updating CPU definitions.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4525/+packages

I tried

dpkg -l |grep libvirt
ii libvirt-clients 6.0.0-0ubuntu8.9~focalppa3 amd64 Programs for the libvirt library
ii libvirt-daemon 6.0.0-0ubuntu8.9~focalppa3 amd64 Virtualization daemon
ii libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9~focalppa3 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9~focalppa3 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 6.0.0-0ubuntu8.9~focalppa3 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9~focalppa3 amd64 Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 6.0.0-0ubuntu8.9~focalppa3 amd64 library for interfacing with different virtualization systems

But still has error:

virsh cpu-compare /tmp/old_host.xml
CPU described in /tmp/old_host.xml is incompatible with host CPU

I rechecked the flags and it seems that the new CPU have't support for TSX

virsh domcapabilities |grep "Skylake"
      <model usable='yes'>Skylake-Server-noTSX-IBRS</model>
      <model usable='no'>Skylake-Server-IBRS</model>
      <model usable='no'>Skylake-Server</model>

I added `tsx=on tsx_async_abort=off` for kernel cmdline and it worked

virsh cpu-compare /tmp/old_host.xml
Host CPU is a superset of CPU described in /tmp/old_host.xml

Also successfully migrated a virtual machine to a new host

That tsx issue is orthogonal to the sky-/cascade-lake detection.
It is due to all the spectre/meltdown/l1tf/... fixes, one of them had to more or less turn TSX off (by default) and is discussed in 1853200. There isn't a neat works-for-all solution yet - and admins have to decide to either adapt guest definitions (and other software using TSX) OR to opt into using TSX as you've done with the kernel cmdline.

But to be sure, was this - for you - only the tsx mismatch all along. Which means does this work fine for you with just the kernel cmdline but without the build that I provided?
Or do you need both to get it going?

FYI as links
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1853200
https://ubuntu.com/blog/ubuntu-updates-to-mitigate-latest-intel-hardware-vulnerabilities
https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/tsx_async_abort.html

I'm sorry, the problem was really only in TSX

But I see another problem, I can't migrate server back for old-host with error:
Instance launched has CPU info: {"arch": "x86_64", "model": "Cascadelake-Server", "vendor": "Intel", "topology": {"cells": 2, "sockets": 1, "cores": 12, "threads": 2}, "features": ["ss", "invpcid", "vme", "sse4.2", "acpi", "cx8", "ssse3", "adx", "clflush", "f16c", "pge", "avx512dq", "bmi1", "tsc", "invtsc", "dtes64", "xtpr", "pschange-mc-no", "movbe", "lm", "nx", "xsaves", "fpu", "spec-ctrl", "xsave", "avx2", "stibp", "sse2", "msr", "rdrand", "avx512vl", "ht", "avx512f", "ds", "pcid", "smap", "pni", "arat", "x2apic", "vmx", "tm", "tsc-deadline", "fxsr", "cx16", "ssbd", "pat", "avx512cd", "ds_cpl", "abm", "lahf_lm", "mca", "ibrs-all", "sep", "xsavec", "pse36", "sse4.1", "pse", "erms", "skip-l1dfl-vmentry", "smep", "fsgsbase", "clflushopt", "tm2", "xsaveopt", "monitor", "apic", "mce", "dca", "pae", "avx", "avx512vnni", "rdtscp", "smx", "tsc_adjust", "mtrr", "arch-capabilities", "mmx", "mpx", "intel-pt", "3dnowprefetch", "avx512bw", "fma", "rdctl-no", "pbe", "hle", "syscall", "pdcm", "rdseed", "pdpe1gb", "popcnt", "bmi2", "cmov", "aes", "rtm", "est", "sse", "pclmuldq", "xgetbv1", "md-clear", "clwb", "pku", "de"]}
CPU doesn't have compatibility. Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
Exception during message handling: nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility. Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 8402, in check_can_live_migrate_destination
     self._compare_cpu(None, source_cpu_info, instance)
   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 8772, in _compare_cpu
     raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
 nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.

virsh dumpxml instance
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='1'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='avx512vnni'/>
    <feature policy='disable' name='mpx'/>
    <numa>
      <cell id='0' cpus='0-1' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>

Old host:

virsh domcapabilities |grep "Cascadelak...

I can see that both servers (without patch) identify themselves the same, although they are not. The only difference is in the instruction set avx512vnni

Old host

virsh domcapabilities kvm
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='avx512vnni'/>
    </mode>

New host

virsh domcapabilities kvm
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='pschange-mc-no'/>
    </mode>

I solved the problems without a patch, but the patch would make life much easier by correctly defining the CPU on old servers

Yeah I agree that the patch would still be helpful,
and thanks for your prompt responses all the time!

I'd keep bug 1853200 separate for any further discussion on TSX, and this bug here open to evaluate backporting the ability to process stepping and thereby better differentiate these CPU types.

Even though you resolved your issue otherwise now, would you further down the road (this will take while for all the verifications) still be available to help testing these builds as they take the steps into being released for Focal/Groovy?

For Focal, yes, I can help with testing, though not so fast :)

And thank you for handling this bug very quickly.

FYI First regression tests LGTM, but the feature freeze on Hirsute is to hard atm to push this right now.

Therefore this will likely be an early change in 21.10 and then SRUs for Focal/Groovy/Hirsute.

I was now able to get hold of a cascade lake chip (there are many types) that was formerly already correctly recognized and working. In this case I can confirm that the change didn't have any negative impact.

It was before&after:
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='hle'/>
      <feature policy='disable' name='rtm'/>
    </mode>

And also the usability of that chip didn't change and stayed
      <model usable='yes'>Cascadelake-Server-noTSX</model>

So far so good, waiting for the upload to be accepted into Hirsute as zero day SRU - but stalled by the release.

@Rascal - might I ask you which cpu type exactly you were using on your old&new hosts?

In my case it was an (lscpu) "Xeon(R) Gold 6252" with microcode (dmesg) "0x4003003, date = 2020-06-18".
It is not required, but could be useful later on or for other affected users to know where the different chip types settle in here :-)

The lscpu output was added in the first two comments to this bug. These were 4114 and 4214 silver processors.

> The lscpu output was added in the first two comments to this bug. These
> were 4114 and 4214 silver processors.

I didn't remember or read back all the former content :-/
Thanks for providing the answer before I asked :-)

For completeness here the `lscpu` and `cpuid -1` of my test system.

A few associated SRUs completed - now is the time.
This was uploaded to focal-unapproved for consideration by the SRU team.

Hello Rascal, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Testing before with libvirt 6.0.0-0ubuntu8.8 on Xeon 6130 (Skylake):

    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='ibrs'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='avx512vnni'/>
    </mode>

With 6.0.0-0ubuntu8.9

    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Server-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='clflushopt'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='ibrs'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='pschange-mc-no'/>
    </mode>

On Xeon 5218 (CascadeLake)

Before/After (no change):

    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='ibrs'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <fea...

Thank Markus you for chiming in as well here with your impressive set of chips available to you.

I've also not seen the detection change on cascade lakes I've got.

Regression tests are still running, I'll mark verified once completed.

The regression tests are good as well with the only hiccups being the known postcopy-after-precopy denial and a known bug in hirsute guest kernels on s390x. Nothing new introduced by the changes as far as I can see.

prep (x86_64) : Pass 20 F/S/N 0/0/0 - RC 0 (10 min 38961 lin)
migrate (x86_64) : Pass 720 F/S/N 36/0/0 - RC 36 (444 min 776248 lin)
cross (x86_64) : Pass 46 F/S/N 0/0/1 - RC 0 (53 min 72466 lin)
misc (x86_64) : Pass 219 F/S/N 0/0/0 - RC 0 (87 min 120975 lin)

prep (s390x) : Pass 20 F/S/N 0/0/0 - RC 0 (10 min 26102 lin)
migrate (s390x) : Pass 718 F/S/N 22/16/0 - RC 22 (501 min 582924 lin)
cross (s390x) : Pass 46 F/S/N 0/0/1 - RC 0 (42 min 66256 lin)
misc (s390x) : Pass 199 F/S/N 2/0/0 - RC 2 (74 min 95916 lin)

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.9

---------------
libvirt (6.0.0-0ubuntu8.9) focal; urgency=medium

  * d/p/u/lp-1921754*: add EPYC-Rome-v2 as v1 missed IBRS and thereby fails
    on some HW/Guest combinations e.g. Windows 10 on Threadripper
    (LP: #1921754)
  * d/p/u/lp-1921880*: add EPYC-Milan features and named cpu type support
    (LP: #1921880)
  * d/p/u/lp-1922907: add ability to parse cpu stepping and thereby correctly
    differentiate skylake and cascadelake chips (LP: #1922907)

 -- Christian Ehrhardt <email address hidden> Wed, 07 Apr 2021 13:33:46 +0200

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.