OpenStack Compute (nova)

live migration with disabled vnc/spice not possible

Bug #1595962 reported by Markus Zoeller (markus_z)
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Medium
Markus Zoeller (markus_z)

Bug Description

Description
===========
If the graphical consoles VNC and SPICE are disabled, the live-migration of an instance will result in an ERROR state.

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* disable VNC in the "nova.conf"

  [VNC]
  enabled=False

* enabled the serial console in "nova.conf"

  [serial_console]
  enabled=True

* launch an instance
* live migrate this instance to another compute node

Expected result
===============
The instance will be on the destination host.

Actual result
=============
The instance is still on the source host and has the status "ERROR" and task "MIGRATING".

Environment
===========
1. Master (Newton cycle) - 2016-06-20
   974b158 Merge "Tear down of os-disk-config part 2"

2. libvirt 1.3.3 + qemu 2.6.0

2. IBM Storewize V7K
   cinder.volume.drivers.ibm.storwize_svc.storwize_svc_fc.StorwizeSVCFCDriver

3. Neutron with OVS

Logs & Configs
==============
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server [req-76217a6e-d28c-4487-b36d-6660fc0b4b21 admin demo] Exception during message handling
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception.py", line 104, in wrapped
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server payload)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 221, in __exit__
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 197, in force_reraise
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception.py", line 83, in wrapped
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 233, in decorated_function
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 211, in decorated_function
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 221, in __exit__
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server self.force_reraise()
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 197, in force_reraise
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 199, in decorated_function
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 5033, in check_can_live_migrate_source
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server block_device_info)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5442, in check_can_live_migrate_source
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server self._verify_serial_console_is_disabled()
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5826, in _verify_serial_console_is_disabled
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server raise exception.MigrationError(reason=msg)
2016-06-24 11:03:18.784 75155 ERROR oslo_messaging.rpc.server MigrationError: Migration error: Your libvirt version does not support the VIR_DOMAIN_XML_MIGRATABLE flag or your destination node does not support retrieving listen addresses. In order for live migration to work properly you must either disable serial console or upgrade your libvirt version.

See original description
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

I suspect change [1]. "listen_addrs" will always be None if VNC and SPICE are disabled [2].

References:
[1] https://github.com/openstack/nova/commit/917afd2831be8107177f50f277e400e03f896f74#diff-f4019782d93a196a0d026479e6aa61b1L6001
[2] https://github.com/openstack/nova/blob/11b15e8321f73d8b4401c3e9ce386282c683f293/nova/virt/libvirt/driver.py#L5333-L5333

tags: added: libvirt live-migration
Markus Zoeller (markus_z) (mzoeller)
description: updated
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

This regression could occur as we don't have a functional test or tempest integration test for the "serial console" feature.

tags: added: needs-functional-test
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/335132

Changed in nova:
assignee: nobody → Markus Zoeller (markus_z) (mzoeller)
status: New → In Progress
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

The functional test for this: https://review.openstack.org/#/c/338416/

Markus Zoeller (markus_z) (mzoeller)
tags: added: newton-rc-potential
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

That is clearly a regression that was introduced early in Newton that we need to fix before the general release.

Changed in nova:
importance: Undecided → Medium
Revision history for this message
Matt Riedemann (mriedem) wrote :

I believe this is supposed to be fixed by this revert: https://review.openstack.org/#/c/368732/

tags: removed: newton-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Markus Zoeller (markus_z) (<email address hidden>) on branch: master
Review: https://review.openstack.org/335132
Reason: This is resolved by the revert in https://review.openstack.org/#/c/368732/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/338416
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=845adc652862a72c4c46f516ed02c4e05af358d9
Submitter: Jenkins
Branch: master

commit 845adc652862a72c4c46f516ed02c4e05af358d9
Author: Markus Zoeller <email address hidden>
Date: Thu Jun 30 11:12:02 2016 +0200

    Add functional regression test for bug 1595962

    If the graphical consoles VNC and SPICE are disabled, the live-migration
    of an instance will result in an ERROR state. VNC and SPICE are usually
    disabled on IBM z systems platforms where graphical consoles are not
    available. The serial console is then enabled and VNC + SPICE are
    disabled. The logged error message will be:

        "Migration error: Your libvirt version does not support the
        VIR_DOMAIN_XML_MIGRATABLE flag or your destination node does
        not support retrieving listen addresses. In order for live
        migration to work properly you must either disable serial
        console or upgrade your libvirt version."

    This functional test executes the live migration flow with enabled serial
    console and disabled VNC + SPICE and reproduces bug 1595962.

    Related-Bug: 1595962

    Change-Id: Ib1b22ce77d8a36e55944acc1fbca042906aa41ef

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.