Merge ~andersson123/utah:make_sure_vm_dead_before_undefine into utah:master

[Tim Andersson]

add check for vm shutoff before undefine

`https://libvirt.gitlab.io/libvirt-appdev-guide-python/libvirt_application_development_guide_using_python-Guest_Domains-Information-State.html`

add check for vm shutoff before undefine

`https://libvirt.gitlab.io/libvirt-appdev-guide-python/libvirt_application_development_guide_using_python-Guest_Domains-Information-State.html`

e205ab1... by Tim Andersson on 2023-06-13

add check for vm shutoff before undefine

`https://libvirt.gitlab.io/libvirt-appdev-guide-python/libvirt_application_development_guide_using_python-Guest_Domains-Information-State.html`

[Paride Legovini (paride)]

I am not convinced. UTAH leaks VMs in running state:

 Id Name State
----------------------------------------------------
 75 utah-129602-mantic-amd64 running
 82 utah-129604-mantic-amd64 running
 86 utah-129606-mantic-amd64 running
 130 utah-129626-mantic-amd64 running
 139 utah-129630-jammy-amd64 running
 143 utah-129632-jammy-amd64 running
 151 utah-129633-jammy-amd64 running
 195 utah-129654-jammy-amd64 running
 232 utah-129672-mantic-amd64 running
 252 utah-129679-mantic-amd64 running
 261 utah-129683-jammy-amd64 running
 269 utah-129686-jammy-amd64 running
 311 utah-129707-mantic-amd64 running
 323 utah-129712-jammy-amd64 running
 327 utah-129713-jammy-amd64 running
 330 utah-129714-jammy-amd64 running
 374 utah-129735-jammy-amd64 running

which means that destroy() is never reached, I believe. Let's fix this traceback, which maybe it getting in the way of the cleanup:

Traceback (most recent call last):
  File "/usr/bin/run_utah_tests.py", line 129, in <module>
    sys.stderr.write('UTAH timeout: {}\n'.format(exception.message))
AttributeError: 'UTAHTimeout' object has no attribute 'message'

[Paride Legovini (paride)]

This should help figuring out what's actually going on:

https://code.launchpad.net/~paride/utah/+git/utah-1/+merge/444849

review: Needs Information

Unmerged commits

e205ab1... by Tim Andersson on 2023-06-13

add check for vm shutoff before undefine

`https://libvirt.gitlab.io/libvirt-appdev-guide-python/libvirt_application_development_guide_using_python-Guest_Domains-Information-State.html`

diff --git a/utah/provisioning/vm.py b/utah/provisioning/vm.py
index ad0f35b..9c82866 100644
--- a/utah/provisioning/vm.py
+++ b/utah/provisioning/vm.py
@@ -468,7 +468,10 @@ class CustomVM(CustomInstallMixin, SSHMixin, LibvirtVM):
             try:
                 vm.undefineFlags(libvirt.VIR_DOMAIN_UNDEFINE_NVRAM)
                 vm.destroy()
-                time.sleep(5)  # Workaround for LP: #1822096
+                ############################## changes
+                # https://libvirt.gitlab.io/libvirt-appdev-guide-python/libvirt_application_development_guide_using_python-Guest_Domains-Information-State.html
+                while vm.state()[0] != libvirt.VIR_DOMAIN_SHUTOFF:
+                    time.sleep(5)
                 vm.undefine()
             except (libvirt.libvirtError, UnboundLocalError):
                 pass