Ubuntu
libvirt package

virt-manager misses xen guests shutting down

Bug #929626 reported by Stefan Bader on 2012-02-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	libvirt	Fix Released	Undecided	redhat-bugs #746007
	libvirt (Ubuntu)	Fix Released	Medium	Stefan Bader

Bug Description

Release: Precise, Oneiric
Architecture: amd64

Test:
- Connect to a xen host via ssh
- Start an existing guest via virt-manager interface
- Shut down the guest via virt-manager

Result:
- The guest is still shown as running
- Running virsh list --all shows the guest is not running

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: virt-manager 0.9.0-3ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-14.23-generic 3.2.3
Uname: Linux 3.2.0-14-generic x86_64
ApportVersion: 1.91-0ubuntu1
Architecture: amd64
Date: Thu Feb 9 17:10:12 2012
PackageArchitecture: all
ProcEnviron:
LANGUAGE=en_US:
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: virt-manager
UpgradeStatus: No upgrade log present (probably fresh install)

Tags:

Related branches

lp:ubuntu/precise/libvirt

Revision history for this message

In Red Hat Bugzilla #746007, Tim (tim-redhat-bugs) wrote on 2011-10-13:

Created attachment 528059
screenshot of xm, virsh and virt-manager showing status

Description of problem:
When I shutdown a xen guest, vmm still shows it as running even though virsh and xm don't (see attached screenshot).

This means that I can't start the domain from virt-manager again because the only options are shutdown related (pause, force-off etc.)

Version-Release number of selected component (if applicable):
virt-manager-0.9.0-6.fc16

How reproducible:
Every time

Steps to Reproduce:
1. start Xen domain
2. stop xen domain

Actual results:
xen domain still shows up as running in virt-manager after it has been stopped and the domain cannot be "powered on" from virt-manager

Expected results:
Xen domain shows up as stopped in virt-manager, able to "power on" domain from virt-manager

Revision history for this message

In Red Hat Bugzilla #746007, Cole (cole-redhat-bugs) wrote on 2012-01-29:

This is a libvirt issue, present in f16 and current upstream. The issue is that invoking shutdown() on a xen guest leaves it reporting domain state 0 (unknown) for the vm object that was shut down. However, if you re-fetch the domain object, it reports the correct state of 5 (shutoff). So something is funky in xen or the xen driver.

This doesn't seem to affect the 'destroy'/'force poweroff' command

Reproducer:

$ cat test.py
import libvirt
import time

conn = "xen:///"
vmname = "xenlivecd"

conn = libvirt.open(conn)
vm = conn.lookupByName(vmname)

if vm.info()[0] == libvirt.VIR_DOMAIN_SHUTOFF:
    print "Starting %s" % vmname
    vm.create()
    time.sleep(5)

print "Shutting off %s" % vmname
vm.shutdown()
time.sleep(5)

print "Domain state is %s" % vm.info()[0]
print "Refreshed domain state is %s" % conn.lookupByName(vmname).info()[0]
print "Old domain is still %s" % vm.info()[0]

$ sudo python test.py
Starting xenlivecd
Shutting off xenlivecd
Domain state is 0
Refreshed domain state is 5
Old domain is still 0

Reassigning to libvirt but this could be a xen issue.

Revision history for this message

In Red Hat Bugzilla #746007, Cole (cole-redhat-bugs) wrote on 2012-01-29:

*** Bug 733433 has been marked as a duplicate of this bug. ***

Revision history for this message

Stefan Bader (smb) wrote on 2012-02-09:

Dependencies.txt Edit (6.3 KiB, text/plain; charset="utf-8")

Revision history for this message

Stefan Bader (smb) wrote on 2012-02-09:

Screenshot at 2012-02-09 17:08:01.png Edit (98.5 KiB, image/png)

Revision history for this message

Stefan Bader (smb) wrote on 2012-03-02:

As off today, this is still showing in Precise.

Marc Deslauriers (mdeslaur) on 2012-03-22

affects:	virt-manager (Ubuntu) → libvirt (Ubuntu)
affects:	virt-manager → libvirt
Changed in libvirt (Ubuntu):
status:	New → Confirmed

Revision history for this message

Stefan Bader (smb) wrote on 2012-03-23:

Ok, so the problem is the special way inactive domains are handled in xend. And one expectation which is wrong. I don't have any xen versions older than 3, so I cannot really check backwards. But from other places in the current code there was a change about then to have no state and domid presented for inactive domains. This is handled by assigning new connections a domain->id of -1.
As soon as an instance is started, this gets replaced by the id which is then assigned. At this point I am not sure whether and where this should get reset to -1 as soon as the instance goes down.

But fact is that when the instance is shut down, and an established connection calls GetDomainInfo, the unified driver calls out to its various sub-drivers. The hypervisor driver won't find the domain anymore, so the next in sequence is the xend driver. That checks for different states looking at the domain/state node (which is not present anymore after shutdown) and then checks for domain->id to be -1 in order to handle the inactive instance. But this only works as long as the instance has not been started before.

From my hosts it seems that there is an additional node (domain/status) which is always present and contains 0 for inactive instances (2 for running). So I replaced the check by looking for that.

Changed in libvirt (Ubuntu):
assignee:	nobody → Stefan Bader (stefan-bader-canonical)
importance:	Undecided → Medium

Revision history for this message

Stefan Bader (smb) wrote on 2012-03-23:

xen-xend-shutdown-detection.patch Edit (1.9 KiB, text/plain)

Using the attached patch, I was able to get virt-manager detect a previously running Xen instance as being shut down. I still had my log filling with a lot of internal error messages which I reported as bug 963006)

Stefan Bader (smb) on 2012-03-23

Changed in libvirt (Ubuntu):
status:	Confirmed → In Progress

Ubuntu Foundations Team Bug Bot (crichton) on 2012-03-23

tags:

added: patch

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-03-30:

This bug was fixed in the package libvirt - 0.9.8-2ubuntu15

---------------
libvirt (0.9.8-2ubuntu15) precise; urgency=low

  [ Stefan Bader ]
  * Use domain/status to check for inactive domains in the xend sub-
    driver. (LP: #929626)
  * Prevent the hypervisor sub-driver from logging an internal error
    just because it cannot find a certain domain when looking for
    the number of vcpus. (LP: #963006)
-- Stefan Bader <email address hidden> Fri, 23 Mar 2012 11:38:24 +0100

Changed in libvirt (Ubuntu):
status:	In Progress → Fix Released

Revision history for this message

In Red Hat Bugzilla #746007, Cole (cole-redhat-bugs) wrote on 2012-04-10:

#10

Possible patch upstream:

https://www.redhat.com/archives/libvir-list/2012-April/msg00023.html

Revision history for this message

In Red Hat Bugzilla #746007, Cole (cole-redhat-bugs) wrote on 2012-06-07:

#11

*** Bug 746503 has been marked as a duplicate of this bug. ***

Revision history for this message

In Red Hat Bugzilla #746007, Fedora (fedora-redhat-bugs) wrote on 2012-06-24:

#12

libvirt-0.9.6.1-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/libvirt-0.9.6.1-1.fc16

Revision history for this message

In Red Hat Bugzilla #746007, Fedora (fedora-redhat-bugs) wrote on 2012-06-26:

#13

Package libvirt-0.9.6.1-1.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-0.9.6.1-1.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-9913/libvirt-0.9.6.1-1.fc16
then log in and leave karma (feedback).