libvirtError exceptions during volume attach leave volume connected to host

https://review.opendev.org/#/c/655712/

Reviewed: https://review.opendev.org/655712
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=091a910576d9b580678f1881fffa425ab4632f48
Submitter: Zuul
Branch: master

commit 091a910576d9b580678f1881fffa425ab4632f48
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/657109

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/657110

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/657111

Reviewed: https://review.opendev.org/657109
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=048d5b790f3da2756a0a1bf2bc015812cb24d53a
Submitter: Zuul
Branch: stable/stein

commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)

Reviewed: https://review.opendev.org/657110
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=786083c91fa69ba9ff90bafe80d3b83b9f9bbc69
Submitter: Zuul
Branch: stable/rocky

commit 786083c91fa69ba9ff90bafe80d3b83b9f9bbc69
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)
    (cherry picked from commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a)

Reviewed: https://review.opendev.org/657111
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=99a63a6633f623cc4e6c9b361965a9c0935113cf
Submitter: Zuul
Branch: stable/queens

commit 99a63a6633f623cc4e6c9b361965a9c0935113cf
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 15:34:41 2019 +0100

    libvirt: Always disconnect volumes after libvirtError exceptions

    Building on Ib440f4f2e484312af5f393722363846f6c95b760 we should always
    attempt to disconnect volumes from the host when exceptions are thrown
    while attempting to attach a volume to a domain. This was previously
    done for generic exceptions but not for libvirtError exceptions.

    Closes-Bug: #1826523
    Change-Id: If21230869826c992e7d0398434b9a4b255940213
    (cherry picked from commit 091a910576d9b580678f1881fffa425ab4632f48)
    (cherry picked from commit 048d5b790f3da2756a0a1bf2bc015812cb24d53a)
    (cherry picked from commit 786083c91fa69ba9ff90bafe80d3b83b9f9bbc69)

Hello Lee, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Lee, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Lee, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package nova - 2:19.0.0-0ubuntu5

---------------
nova (2:19.0.0-0ubuntu5) eoan; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:41:27 +0200

Hello Lee, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Lee, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.9-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Lee, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have verified that 2:19.0.0-0ubuntu2.3 fixes the issue on disco,
raising the following exception:

 libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)

No attachment has been found at the device after this.

I have verified the following versions:

* bionic: 17.0.9-0ubuntu3
* cosmic: 18.1.0-0ubuntu3
* disco: 19.0.0-0ubuntu2.3

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

 libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)

No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property.

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3

---------------
nova (2:17.0.9-0ubuntu3) bionic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:17.0.9-0ubuntu2) bionic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).
  * d/p/skip-double-word-hacking-test.patch: Cherry-picked from upstream
    to work-around test_hacking failures with new versions of python
    (LP: #1782786).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 11:06:11 +0200

This bug was fixed in the package nova - 2:18.1.0-0ubuntu3

---------------
nova (2:18.1.0-0ubuntu3) cosmic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:18.1.0-0ubuntu2) cosmic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:58:45 +0200

This issue was fixed in the openstack/nova 19.0.1 release.

OK, verified S/Q/R

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

** libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)
** No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property. (https://pastebin.canonical.com/p/kkmRcfrsjT/)

This issue was fixed in the openstack/nova 18.2.1 release.

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3~cloud0
---------------

 nova (2:19.0.0-0ubuntu2.3~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3~cloud0
---------------

 nova (2:17.0.9-0ubuntu3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:17.0.9-0ubuntu3) bionic; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3

---------------
nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:19.0.0-0ubuntu2.2) disco; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

nova (2:19.0.0-0ubuntu2.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/p/eventlet-monkey-patching-should-be-as-early-as-possible.patch:
    Cherry-picked from upstream stable/stein review to fix py3+wsgi+ssl crash
    (LP: #1808951).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:54:46 +0200

This issue was fixed in the openstack/nova 17.0.11 release.

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.