Merge ~p-pisati/britney/+git/hints-ubuntu:devel-skiptest-meta into ~ubuntu-release/britney/+git/hints-ubuntu:devel

[Paolo Pisati]

hint two flaky tests (dracut/amd64) and (initramfs-tools/s390x) while everything else looks sound

[Benjamin Drung (bdrung)]

Have you proof that these tests are flaky and not just starting to fail with linux-meta/6.8.0-11.11+1?

[Paolo Pisati (p-pisati)]

Yes, all the previous upload were fine and 11.11+1 is a no-change respin to fix some dependencies in -meta and -lrm - the kernel binary (e.g. 11.11) didn't change at all.

[Timo Aaltonen (tjaalton)]

review: Approve

[Brian Murray (brian-murray)]

> Yes, all the previous upload were fine and 11.11+1 is a no-change respin to
> fix some dependencies in -meta and -lrm - the kernel binary (e.g. 11.11)
> didn't change at all.

Generally speaking for these types of merge proposals we prefer to be able to inspect the logs ourselves. Is there something you can point us to?

[Paolo Pisati (p-pisati)]

initramfs-tools s390x:

good:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/s390x/i/initramfs-tools/20240218_052553_120c1@/log.gz

bad:
https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-noble/noble/amd64/d/dracut/20240222_113253_2a01e@/log.gz

dracut amd64:

good:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/d/dracut/20240218_143715_3deda@/log.gz

bad:
https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-noble/noble/amd64/d/dracut/20240222_113253_2a01e@/log.gz

[Graham Inggs (ginggs)]

Both dracut/amd64 and initramfs-tools/s390x passed eventually, so no need for a hint for these.

However, all the linux-meta* package seem to regress glibc/2.39-0ubuntu1 that has just migrated.

I have just retried the glibc tests against linux-meta/6.8.0-11.11+1 with additional triggeres linux-signed/6.8.0-11.11 and linux/6.8.0-11.11.

[Steve Langasek (vorlon)]

These appear to be real regressions in the glibc tests with the new kernel. The following tests have moved from 'UNSUPPORTED' to 'FAIL':
debug/tst-fortify-syslog
elf/tst-dlopen-self-container
elf/tst-dlopen-tlsmodid-container
elf/tst-glibc-hwcaps-2-cache
elf/tst-glibc-hwcaps-cache
elf/tst-glibc-hwcaps-prepend-cache

And the tests show an error such as:

 3052s error: test-container.c:1136: could not create a private mount namespace

We should not simply let the glibc autopkgtests regress here as that is a bad regression in test coverage. It needs to get sorted out - there at least needs to be agreement that it's a bug on the glibc side and that the glibc tests will need to be fixed, before we ignore the failure.

review: Needs Information

[Paolo Pisati (p-pisati)]

We think it's LP: #2046844, and JJ is looking into it.

In the mean time, i updated the hint to cover all kernels blocked by failing glibc autopkgtests.

[Paolo Pisati (p-pisati)]

Citing schopin (see #43 LP: #2046844):

"We had a mitigation for this in glibc but the latest change from simply denying the unshare() call to allowing it but then denying anything requiring capabilities *presumably* broke the glibc test suite again. I'm only basing this from looking at the test logs, as I'm temporarily unable to run autopkgtests locally and am lacking the time to fix it.

2 classes of errors:

2770s FAIL: stdlib/tst-system
2770s original exit status 1
2770s error: test-container.c:1136: could not create a private mount namespace

That one is clearly userns-related, as it's due to a failing mount() call right after unshare()

2770s FAIL: sunrpc/tst-svc_register
2770s original exit status 1
2770s error: xwrite.c:32: write of 12 bytes failed after 0: Operation not permitted
2770s error: 1 test failures

I can't tell for sure what this one is about since this is your basic write() call and I don't have a stack trace at hand, but the EPERM would suggest that it's related.

I think a first fix would be to amend the test script to disable the userns restriction entirely for the duration of the tests (using 'needs-sudo'), while I'll still need to patch the test suite eventually to handle this new failure mode gracefully and simply ignore the tests, akin to https://sourceware.org/pipermail/libc-alpha/2024-February/154754.html"

[Simon Chopin (schopin)]

From the glibc side, I agree that it's something that needs to be fixed in glibc rather than in the kernel and I'll be working on it in the coming weeks. Better to hint it for now.

[Steve Langasek (vorlon)]

review: Approve

diff --git a/ubuntu-release b/ubuntu-release
index 8e605b4..5cf80cd 100644
--- a/ubuntu-release
+++ b/ubuntu-release
@@ -36,3 +36,12 @@ force-badtest nvidia-graphics-drivers-390/all/armhf
 
 # buggy test on s390x, will be skipped in next upload (LP: #2052794)
 force-badtest glib2.0/2.79.1-1/s390x
+
+# hint AA regression affecting glibc autopkgtests and stalling all kernels
+# in -proposed, see LP: #2046844
+force-skiptest linux-meta/6.8.0-11.11+1
+force-skiptest linux-meta-aws/6.8.0-1001.1
+force-skiptest linux-meta-azure/6.8.0-1001.1
+force-skiptest linux-meta-gcp/6.8.0-1002.2
+force-skiptest linux-meta-lowlatency/6.8.0-7.7.1
+force-skiptest linux-meta-oracle/6.8.0-1001.1