autopkgtest fails Bionic@amd64 (service) and Hirsute@arm64 (racy)

Hmm,
we had issues in the past with this one related to various kernel images and other little details.
I'll take a look.

Ok, we indeed have ~45% rate of fails in the last 20 runs :-/

All we got from the log is that "dependency job" issue mentioned in the report.

Indeed it does pass for me locally as well.

But before had 0% issues, so it seems something has changed.
I checked the history and there was no upload of libvirt anywhere near the switch to the bad state.

Various kernel upload and systemd/237-3ubuntu10.42 is close, but I see no clear connection between those and the fails yet.

    smoke-lxc (F 15% f 0% S 0% B 0% => P 85%/) ..FF.FFF.FF..F...F..........................................

The test switches to set -x for the actual test steps, so we only have very few commands that run before. In fact it only is
1. export LIBVIRT_DEFAULT_URI='lxc:///'
2. systemctl restart libvirtd

The latter is what fails with our: "dependency job for libvirtd.service failed"

That restart is for
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848317
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867379
But we don't see the effect of these bugs, instead we need to track down what dependency issue happens on autopkgtest-infra when running this test.

Furthermore worth to note that the issue only affects x64.
I see it on amd64 + i386 but not on arm/s390x/ppc64el.

The code didn't change much to Focal, but there the test did not degrade.

I added some debugging for the state of the service(s) before and after that call.
Hopefully I'm able to trigger the issue on the autopkgtest infrastructure that way to get some insight what is going on.

PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4311

We know it won't reproduce in local VM, but haunts tests on infra.
I've added some debug (no functional change) to see what dependent job might fail on that restart, but even running on LP-infra the fail won't trigger ... :-/

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201013_115528_1f7a3@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201013_140034_003f0@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201014_060703_651fd@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201014_083426_e5163@/log.gz
...

Another three runs without issues :-/
So we have a heisenbug :-/
I must say that I clearly showed this isn't due to a change in libvirt NOR does it reveal any debug data - so I'm puzzled at what it might be.

I'm ok to mark the test flaky if the situation remains that bad in Bionic (after all it is "only" smoke-lxc which isn't really supported anyway).

But an SRU upload just for that feels wrong, I'd think we will do so whenever we do an Bionic SRU anyway. An Block-proposed SRU upload won't be enough as we are not looking at build, but autopkgtest time issues. Those would only be picking up -proposed if triggered that way and if we do that we can as well retry (as we do already).

Until then please speak up here if the situation continues (tests keep failing) or get worse (rate increases further).

I've encountered this failure, too:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/libv/libvirt/20201109_214647_148b5@/log.gz

FYI: Still no other libvirt change in the queue for bionic that would justify an SRU upload.
Current test ratio on the flaky test ~60% passes on smoke-lxc and all other subtests pass 100%.

On Hirsute this now also became (from a yet unknown change in a different component) flaky.
Essentially having timing issues between "start" and "lxc-enter-namespace".
Since we like the test, but not actually care too much about libvirt-lxc (we favor LXD by orders of magnitude for that use case) I think what I'll implement is:

1. make the test skippable
2. skip if the restart of libvirtd failed (we still want to test the consequence of guests being alive if it worked, but if the restart itself failed we should skip)
3. on check_domain since we seem to have races between start and being ready add a retry mechanism

I'll test that in hirsute where the current arm64 issues happen and if working well there that can be what we backport along the next SRUs.

Recent Hirsute stats:
  arm64
    smoke-lxc (F 22% f 0% S 0% B 15% => P 62%/) ..FF.F.......BF..F...BF.FB.F...BBB.F....

All seem to be like:
+ check_domain
+ grep+ -qs sl[[:space:]]\+running
virsh list
+ virsh lxc-enter-namespace --noseclabel sl /bin/ls /bin/ls
error: Requested operation is not valid: Init pid is not yet available

I've created two PPAs to iterate on amd64+arm64 tests for this:
- https://launchpad.net/~paelzer/+archive/ubuntu/lp-1899180-unmodified
- https://launchpad.net/~paelzer/+archive/ubuntu/lp-1899180-new

I've triggered just the arm64 test 4 times on each to see if we can reproduce the bug on one but not the other. While we eventually will need more I'm not triggering more to ensure the behavior is ok as-is. Maybe over the weekend we then can run more as needed.

Note to my self, trigger and result check via
$ lp-test-ppa ppa:paelzer/lp-1899180-unmodified --release hirsute -a amd64 --showpass
$ lp-test-ppa ppa:paelzer/lp-1899180-new --release hirsute -a amd64 --showpass

So far:
- unmodified 2/5 fails
- new 0/5 fails
=> I'll schedule a bunch more for the weekend to get a better feeling if this really works out.

Ok the results over the weekend look promising, tested on arm64:
- unmodified 6/31 fails
- new 0/31 fails

But when checking the good cases in detail what I found was that in all
cases the domain was considered running, but then continuously (= not a race) failed to enter namespace. I've had 9 cases where this happened, so it wasn't a one off.

libvirt-lxc still isn't the most important case, so reworking the test still is right, but this seems to have a semi-real-issue on armhf (not reproducible anywhere else).
I'd like to avoid to mask errors on other architectures by e.g. re-starting the domain if it failed - therefore I'd not want to remove the test or unconditionally accept it as flaky, but maybe it is ok to do so on armhf only.

With that fixed up I'll do some retries on hirsute@arm64 (to see if it detects ans skip/pass correctly now) and the same backported to bionic@amd64 (for the service restart issue).

Ok, the new issue can be seen on arm64 as well, see
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute-paelzer-lp-1899180-new/hirsute/arm64/libv/libvirt/20201207_083430_0fb55@/log.gz

I need to extend the skipping if for it.

Also the try-restart needs to cover more on bionic (added as well now).
New builds uploaded to the PPA ...

If affected by the bug we see in Bionic things would now look like:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-paelzer-lp-1899180-new/bionic/amd64/libv/libvirt/20201208_112813_c5366@/log.gz

autopkgtest [11:24:35]: test smoke-lxc: [-----------------------
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
Flaky restart failed while checking libvirt-lxc - skipping.
autopkgtest [11:24:56]: test smoke-lxc: -----------------------]
autopkgtest [11:24:56]: test smoke-lxc: - - - - - - - - - - results - - - - - - - - - -
smoke-lxc SKIP exit status 77 and marked as skippable

In most cases it directly works, and in a few it recovered on the retry.
That would be fine, but I'm waiting on a few more test results to be sure and also how Hirsute now looks like.

Good on Hirsute as well

good:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute-paelzer-lp-1899180-new/hirsute/arm64/libv/libvirt/20201208_113807_f0d9e@/log.gz

bad:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute-paelzer-lp-1899180-new/hirsute/arm64/libv/libvirt/20201208_114418_bd955@/log.gz
...
+ virsh lxc-enter-namespace --noseclabel sl /bin/ls /bin/ls
error: Requested operation is not valid: Init pid is not yet available

+ rc=1
+ return 1
+ sleep 2s
+ echo Known to be unreliable on test infrastructure - skipping
+ exit 77
+ cleanup
+ [ -z ]
+ virsh destroy sl
Known to be unreliable on test infrastructure - skipping
Domain sl destroyed

+ virsh undefine sl
Domain sl has been undefined

+ CLEANED_UP=1
autopkgtest [11:33:34]: test smoke-lxc: -----------------------]
autopkgtest [11:33:34]: test smoke-lxc: - - - - - - - - - - results - - - - - - - - - -

As mentioned before, the change on it's own isn't SRUable.
But we can push it to Hirsute and to all active releases as block-proposed.
There it can
a) be used in custom autopkgtest triggers if needed
b) more likely is picked up on the next upload

I'm preparing merge proposals for that ...

I've reviewed and approved all the SRU MPs (as well as the Hirsute MP). FWIW, full SRU information for just autopkgtest changes aren't usually required; I'd have accepted with just "fixing autopkgtest; block-proposed-* please" :-)

Going fine in Hirsute - even though full proposed migration is behind a wall of tests.
Uploaded for B/F/G SRUs.

block-proposed is for the development release (Hirsute) only. I'll leave you to remove that if you want to unblock Hirsute.

block-proposed-{bionic,focal,groovy} are needed to mark your uploads as "staged" for these releases. Please leave these in place. See the documentation at https://wiki.ubuntu.com/StableReleaseUpdates#Staging_an_upload

Hello Balint, or anyone else affected,

Accepted libvirt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.6.0-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Balint, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Balint, or anyone else affected,

Accepted libvirt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/4.0.0-1ubuntu8.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted libvirt (4.0.0-1ubuntu8.18) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

cockpit/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#libvirt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The failure of "cockpit/unknown (armhf)" was an infrastructure issue unrelated to this upload and resolved on a retry.
The modified libvirt autopkgtests worked as expected.

E.g. on Bionic it would have hit the issue of this bug.
See
https://autopkgtest.ubuntu.com/packages/l/libvirt/bionic/amd64
There this run
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/libv/libvirt/20201210_000643_37485@/log.gz

Would have been bad if not skipping the problematic test

A dependency job for libvirtd.service failed. See 'journalctl -xe' for details.
Flaky restart failed while checking libvirt-lxc - skipping.

This allows me to set verification-done on all these releases.
But as outlined before - it will be in block-proposed until something else comes by that suffice for an SRU.

If affected by this issue and blocked, please use a test trigger against the new version that is in -proposed now.

This bug was fixed in the package libvirt - 6.9.0-1ubuntu4

---------------
libvirt (6.9.0-1ubuntu4) hirsute; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Christian Ehrhardt <email address hidden> Fri, 04 Dec 2020 08:12:02 +0100

Hello Balint, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Balint, or anyone else affected,

Accepted libvirt into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.6.0-1ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

These tests worked that way for quite some time (held in -proposed via block-proposed).
But they were used only when custom triggered against proposed.

Now the new code ran as part of this upload as the other SRU "picked it u" and it is fine:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/libv/libvirt/20210217_072030_10e74@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/libv/libvirt/20210210_183646_142d4@/log.gz

The groovy run would have been one that formerly was broken on the flaky case.

Setting verified

All autopkgtests for the newly accepted libvirt (6.6.0-1ubuntu3.3) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

libvirt-dbus/1.4.0-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#libvirt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libvirt - 6.6.0-1ubuntu3.3

---------------
libvirt (6.6.0-1ubuntu3.3) groovy; urgency=medium

  * d/p/u/lp-1913266-qemu-Add-virtio-related-options-to-vsock.patch: allow
    vsock to work in secure execution environments. (LP: #1913266)

libvirt (6.6.0-1ubuntu3.2) groovy; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Christian Ehrhardt <email address hidden> Tue, 09 Feb 2021 16:14:26 +0100

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.7

---------------
libvirt (6.0.0-0ubuntu8.7) focal; urgency=medium

  * d/p/u/lp-1913266-qemu-Add-virtio-related-options-to-vsock.patch: allow
    vsock to work in secure execution environments. (LP: #1913266)

libvirt (6.0.0-0ubuntu8.6) focal; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Christian Ehrhardt <email address hidden> Tue, 09 Feb 2021 16:09:39 +0100

Hello Balint, or anyone else affected,

Accepted libvirt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/4.0.0-1ubuntu8.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

The SRU for bug 1915811 has picked this one up which wasn't worth to be released on its own. I rechecked the tests [1] and they are all good again/still (=>verified). Since the only reason for block-proposed was to not release it "on its own" we can drop that tag to let it release together with that other SRU once ready.

[1]: https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#libvirt

This bug was fixed in the package libvirt - 4.0.0-1ubuntu8.19

---------------
libvirt (4.0.0-1ubuntu8.19) bionic; urgency=medium

  * d/p/lp1915811-fix-numa-topology.patch: Fix NUMA topology population for
    machines with a high number of CPUs (LP: #1915811)

libvirt (4.0.0-1ubuntu8.18) bionic; urgency=medium

  * Improve flaky smoke-lxc test (LP: #1899180)
    - d/t/control, d/t/smoke-lxc: retry service restart and skip test if
      failing; This was flaky on some release/architectures
    - d/t/smoke-lxc: retry check_domain being flaky on arm64

 -- Victor Manuel Tapia King <email address hidden> Thu, 18 Feb 2021 17:40:38 +0100