KVM internal error. Suberror: 1 emulation failure

Which kernel are you using on the physical host?

I am seeing this (almost the same Code line) on 4.20.5 kernel. 4.19.11 works fine.

In my case the code is Code=cb 66 ba 52 d0 0f 00 e9 cb fe bc 00 80 0a 00 e8 0e 4f ff ff <0f> aa fa fc 66 ba 6b d0 0f 00 e9 b4 fe f3 90 f0 0f ba 2d 30 51 0f 00 00 72 f3 8b 25 2c 51
qemu 2.5 on an i5-2500k

Yes. I was using a 4.20.x kernel. But i guess the bug can be closed because the vm does not exist anymore and the new one does not show this behaviour.

Hmm, I've also just hit this in a nest:

KVM internal error. Suberror: 1
emulation failure
EAX=00000001 EBX=000f7874 ECX=00000002 EDX=00000001
ESI=7ffbdca4 EDI=000069f2 EBP=000069b2 ESP=000a8000
EIP=000fd099 EFL=00010046 [---Z-P-] CPL=0 II=0 A20=1 SMM=1 HLT=0
ES =0010 00000000 ffffffff 00c09300
CS =0000 00000000 00000fff 00809b00
SS =0010 00000000 ffffffff 00c09300
DS =0010 00000000 ffffffff 00c09300
FS =0010 00000000 ffffffff 00c09300
GS =0010 00000000 ffffffff 00c09300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 10387cfe 0000fe6c
IDT= 0010387c 00003810
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffffcffc DR7=000000000e1e0400
EFER=0000000000000000
Code=cb 66 ba 8f d0 0f 00 e9 ce fe bc 00 80 0a 00 e8 24 4d ff ff <0f> aa fa fc 66 ba a8 d0 0f 00 e9 b7 fe f3 90 f0 0f ba 2d 7c 4f 0f 00 00 72 f3 8b 25 78 4f
^Cqemu-system-x86_64: terminating on signal 2

given thw EIP I suspect the address is actually in seabios.

Host: Fedora 29, 4.20.5-200.fc29.x86_64, qemu 3.0.0 on i7-8650U
L1: Fedora 29, 5.0.0-0.rc4.git2.2.fc30.x86_64 (and also 4.20), qemu head
L2: Doesn't even seem to need a useful guest

Note the error here is on stderr of L1's qemu; there's nothing in dmesg on host or L1 and nothing in the libvirt log.

This is related to SMM usage in SeaBIOS. The QEMU register dump states SMM=1, plus "<0f> aa" from the dumped code stands for the RSM instruction (0F AA -- RSM—Resume from System Management Mode, see it in the Intel SDM.)

In RHEL7 downstream, we disabled SMM usage in SeaBIOS.
- https://bugzilla.redhat.com/show_bug.cgi?id=1378006
- https://bugzilla.redhat.com/show_bug.cgi?id=1464654#c21

It's conceivable that the upstream host kernel suffered a regression 4.19 and 4.20; in particular when it comes to nesting. For example, Ladi fixed <https://bugzilla.redhat.com/show_bug.cgi?id=1488203> in <https://www.spinics.net/lists/kvm/msg156709.html>:

0234bf885236 KVM: x86: introduce ISA specific SMM entry/exit callbacks
72d7b374b14d KVM: x86: introduce ISA specific smi_allowed callback
21f2d5511838 KVM: nVMX: set IDTR and GDTR limits when loading L1 host state
72e9cbdb4338 KVM: nVMX: fix SMI injection in guest mode
c26340651b75 KVM: nSVM: refactor nested_svm_vmrun
05cade71cf3b KVM: nSVM: fix SMI injection in guest mode

These were part of v4.15. But, based on <https://bugzilla.redhat.com/show_bug.cgi?id=1661979>, more recent kernels may have regressed those fixes.

(Bunch of non-public BZ references above; sorry about that, I can't open them up.)

I have bisected the kernel and ended with first bad commit:

commit 14c07ad89f4d728a468caaea6a769c018c2b8dd6
Author: Vitaly Kuznetsov <email address hidden>
Date: Mon Oct 8 21:28:08 2018 +0200

    x86/kvm/mmu: introduce guest_mmu

    When EPT is used for nested guest we need to re-init MMU as shadow
    EPT MMU (nested_ept_init_mmu_context() does that). When we return back
    from L2 to L1 kvm_mmu_reset_context() in nested_vmx_load_cr3() resets
    MMU back to normal TDP mode. Add a special 'guest_mmu' so we can use
    separate root caches; the improved hit rate is not very important for
    single vCPU performance, but it avoids contention on the mmu_lock for
    many vCPUs.

It seems to me this is some timing issue. If a L2 vm fails to start I can get it running with a few hard reboots.

Also changing the L2 config helps, -net none, -nodefaults and -vga cirrus let the vm start most of the time.

Thanks for the bisect! I've cc'd in Vitaly.

Ack, thanks for the bisect! It seems something was overlooked when we did host/guest mmu split. I'll try to investigate.

Thomas, Albert, David,

I'm having hard times trying to reproduce the issue in my environment; could you please provide your qemu command lines for both L0 and L1? It would also be great if you could try to come up with some 'minimal' configuration (my guess is that in L1 having just "qemu-system-x86_64 -machine q35,smm=on,accel=kvm -cpu host -vnc :0" would do).

Thanks!

Right, an L1 line of :
./x86_64-softmmu/qemu-system-x86_64 -M q35,accel=kvm -cpu host

does trigger it for me.

My L0 is:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=fedora29,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/home/dgilbert/.config/libvirt/qemu/lib/domain-1-fedora29/master-key.aes -machine pc-q35-3.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,clflushopt=on,ssbd=on,xsaves=on,pdpe1gb=on -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid d070c898-4323-46f6-b8c2-566061a2f88d -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=27,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -drive file=/home/vmimages/fedora29.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.4,addr=0x0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-sata0-0-0,media=cdrom,readonly=on -device ide-cd,bus=ide.0,drive=drive-sata0-0-0,id=sata0-0-0 -netdev user,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c3:dc:36,bus=pci.1,addr=0x0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=29,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

using a f29 ...

Thank you David, I see the issue now.

I sent a patch which is supposed to fix the issue:
https://marc.info/?l=kvm&m=155085391830663&w=2

it would be great if someone could give it a spin!

>>> On 2/22/2019 at 9:50 AM, Vitaly Kuznetsov <email address hidden> wrote:
> I sent a patch which is supposed to fix the issue:
> https://marc.info/?l=kvm&m=155085391830663&w=2
>
> it would be great if someone could give it a spin!
>

I've been trying to get down to the bottom of this.
It looks like you've got it.
I can confirm that this fixes the issue for me. Thanks!

Tested-by: Bruce Rogers <email address hidden>

> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1813165
>
> Title:
> KVM internal error. Suberror: 1 emulation failure
>
> Status in QEMU:
> New
>
> Bug description:
> Hello Devs.
>
> Having problems getting VM to run with qemu 3.1.0. I should mention
> it's a nested configuration.
>
> 2019-01-24 13:46:08.648+0000: starting up libvirt version: 4.10.0, qemu
> version: 3.1.0, kernel: 4.14.94, hostname: one....
> LC_ALL=C
> PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/b
> in:/usr/local/sbin:/opt/bin HOME=/root USER=root QEMU_AUDIO_DRV=none
> /usr/bin/kvm -name guest=one-266,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-one-266/m
> aster-key.aes -machine pc-i440fx-2.9,accel=kvm,usb=off,dump-guest-core=off
> -cpu
> Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,ssbd=on,x
> saves=on,pdpe1gb=on -m 1024 -realtime mlock=off -smp
> 2,sockets=2,cores=1,threads=1 -uuid b219b45d-a2f0-4128-a948-8673a7abf968
> -no-user-config -nodefaults -chardev
> socket,id=charmonitor,fd=21,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot
> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/var/lib/one//datastores/0/266/disk.0,format=qcow2,if=none,id=drive-virt
> io-disk0,cache=none -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio
> -disk0,bootindex=1,write-cache=on -drive
> file=/var/lib/one//datastores/0/266/disk.1,format=raw,if=none,id=drive-ide0-0
> -0,readonly=on -device
> ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev
> tap,fd=23,id=hostnet0 -device
> rtl8139,netdev=hostnet0,id=net0,mac=02:00:00:76:69:85,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
> -vnc 0.0.0.0:266 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg
> timestamp=on
> char device redirected to /dev/pts/1 (label charserial0)
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000001 EBX=000f7c2c ECX=00000001 EDX=00000001
> ESI=00006a26 EDI=3ffbdc48 EBP=000069e6 ESP=000a8000
> EIP=000fd057 EFL=00010016 [----AP-] CPL=0 II=0 A20=1 SMM=1 HLT=0
> ES =0010 00000000 ffffffff 00c09300
> CS =0000 00000000 00000fff 00809b00
> SS =0010 00000000 ffffffff 00c09300
> DS =0010 00000000 ffffffff 00c09300
> FS =0010 00000000 ffffffff 00c09300
> GS =0010 00000000 ffffffff 00c09300
> LDT=0000 0...

The patch works fine for me too, many thanks!

Hey folks, I am having the same problem.

I recognize there was a patch a year ago (almost exactly one year ago) and I wonder it was included in later updates.

That said, Ubuntu has upgraded. Recently I upgrade my machine. Here are some key stats.

DistroRelease: Ubuntu 20.04
Uname: Linux 5.4.0-14-generic x86_64
UpgradeStatus: Upgraded to focal on 2020-02-12 (22 days ago)
ProcVersionSignature: Ubuntu 5.4.0-14.17-generic 5.4.18
RelatedPackageVersions:
 kvmtool N/A
 gir1.2-libvirt-glib-1.0 2.0.0-2
 gir1.2-libvirt-sandbox-1.0 N/A
 libnss-libvirt N/A
 libvirt-bin N/A
 libvirt-clients 6.0.0-0ubuntu4
 libvirt-daemon 6.0.0-0ubuntu4
 libvirt-daemon-driver-lxc N/A
 libvirt-daemon-driver-qemu 6.0.0-0ubuntu4
 libvirt-daemon-driver-storage-gluster N/A
 libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu4
 libvirt-daemon-driver-storage-zfs N/A
 libvirt-daemon-driver-vbox N/A
 libvirt-daemon-driver-xen N/A
 libvirt-daemon-system 6.0.0-0ubuntu4
 libvirt-daemon-system-systemd 6.0.0-0ubuntu4
 libvirt-daemon-system-sysv N/A
 libvirt-dbus N/A
 libvirt-dev 6.0.0-0ubuntu4
 libvirt-doc N/A
 libvirt-glib-1.0-0 2.0.0-2
 libvirt-glib-1.0-dev N/A
 libvirt-ocaml N/A
 libvirt-ocaml-dev N/A
 libvirt-sandbox-1.0-5 N/A
 libvirt-sandbox-1.0-dev N/A
 libvirt-sanlock N/A
 libvirt-wireshark N/A
 libvirt0 6.0.0-0ubuntu4
 libvirtodbc0 N/A
 libvirtualpg-dev N/A
 libvirtualpg0 N/A
 libvirtuoso5.5-cil N/A
 nbdkit-plugin-libvirt N/A
 nova-compute-libvirt N/A
 php-libvirt-php N/A
 python3-libvirt 6.0.0-0ubuntu3
 ruby-fog-libvirt 0.6.0-1
 ruby-libvirt 0.7.1-1
 uvtool-libvirt N/A
 vagrant-libvirt 0.0.45-2
 virt-manager 1:2.2.1-3ubuntu1
 qemu 1:4.2-3ubuntu1
 qemu-block-extra 1:4.2-3ubuntu1
 qemu-efi N/A
 qemu-efi-aarch64 N/A
 qemu-efi-arm N/A
 qemu-guest-agent N/A
 qemu-kvm 1:4.2-3ubuntu1
 qemu-slof N/A
 qemu-system N/A
 qemu-system-arm N/A
 qemu-system-common 1:4.2-3ubuntu1
 qemu-system-data 1:4.2-3ubuntu1
 qemu-system-gui 1:4.2-3ubuntu1
 qemu-system-mips N/A
 qemu-system-misc N/A
 qemu-system-ppc N/A
 qemu-system-s390x N/A
 qemu-sy...

Here is the error:

2020-03-05T02:01:00.287202Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
KVM internal error. Suberror: 1
emulation failure

Here is the CLI error I get:

Error unpausing domain: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1311, in resume
    self._backend.resume()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 2174, in resume
    if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
libvirt.libvirtError: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

If you're seeing "KVM internal error. Suberror: 1" it can be multiple things, not necessarily the same bug. Could you please confirm that:
- You are running a nested configuration
- The issue is observed with a UEFI booted guest

BTW, kernel 5.4 you have has the patch fixing the original bug.

Ok Vitaly I will take a look and report back.

Nested Configuration check
tstrike39@islandhealthcenter-media:~$ sudo cat /sys/module/kvm_intel/parameters/nested
[sudo] password for tstrike39:
Y

Not UEFI enabled

'nested' parameter for kvm_intel module controls whether you're able to run nested configurations and it is enabled by default, it doesn't say anything about whether your configuration is nested or not.

Could you please describe your environment? In case it is nested, it will look like:
L0(host) Linux 5.4.0 .... with KVM
L1(guest) Linux xxxx with KVM
L2(nested guest) Linux xxxx

Note that L2 in this scenario is running "inside" L1 and that's why it is nested.

Also, it is important to know if L1 or L2 in this configuration are UEFI booted, it is not very important how you boot the host (L0).

L0 DistroRelease: Ubuntu 20.04 on Kernel Linux 5.4.0-14-generic x86_64
L1 3 guests Windows 10, Centos 8
No L2s

No guests are enabled for UEFI Boot

With Win10 you need to make sure it is not running Hyper-V under the hood (e.g. when you enable Hyper-V role Windows will put itself in a VM -- and thus you will get a nested environment).

To be 100% sure do the following:
# rmmod kvm_intel
# modprobe kvm_intel nested=0

And see if the issue reproduces. In case it does, this is definitely something different, the original bug only affects nested environments.

Just performed what you asked Vitaly.

Same result.

Thanks for checking,

this is a different issue then, please open a new bug. Also, if I understood you correctly, the problem appeared after an upgrade? It would make sense to try to bisect between qemu and kernel versions (personally, I'd start with kernel because it's easier to rollback and has higher chances of being the root cause) to see when the issue appeared. Knowing which exact version brought the regression will likely help Ubuntu packagers in fixing the issue.

Ok. I will open another bug report. Give me about an hour or so I can regather my logs (plus get my workout in) for a repost.

I am not a libvirt expert per se (although a LONG time user). Should I run an apport report to post up on the new bug so that the packages can get a birds eye view on my system?

Sorry but I'm not at all familiar with bug reporting process in Ubuntu. The "KVM internal error. Suberror: 1" issue is definitely not libvirt related (may be induced by the VM configuration created in libvirt but that's it).

I created a new bug request to track the Ubuntu 20.04 package (bug #1867545)

According to comment #12, the bug underlying this LP ticket was fixed in Linux kernel commit ad7dc69aeb23 ("x86/kvm/mmu: fix switch between root and guest MMUs", 2019-02-22), released in v5.0.