Merge ~ubuntu-release/autopkgtest/+git/development:hide-warnings into ~ubuntu-release/autopkgtest/+git/development:master

[Brian Murray]

We were seeing issues on arm64 test instances where tests would fail immediately after running the following commands:

112s autopkgtest [18:56:44]: ERROR: "sh -ec mkdir -p /etc/apt/preferences.d; PKGS=""; PKGS="$PKGS $(apt-cache showsrc binutils | awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1 ":*"; next } { show=0 }' |sort -u | tr '\n' ' ')"; printf "Package: $PKGS\nPin: release a=noble-proposed\nPin-Priority: 995\n" > /etc/apt/preferences.d/autopkgtest-noble-proposed; printf "
112s Package: *\nPin: release a=noble-updates\nPin-Priority: 990\n" >> /etc/apt/preferences.d/autopkgtest-noble-proposed; " failed with stderr "W: Target Packages (main/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:1 and /etc/apt/sources.list.d/ubuntu.sources:1

The problem was that $PKGS contained warning messages and then some how this tried as a command:

binutils-aarch64-linux-gnu-dbg:*: command not found

There seems to be a problem with our images in that we have /etc/apt/sources.list and /etc/apt/sources.list.d/ubuntu.sources which needs investigating. However, we should get arm64 (and other arches?) tests running again while we investigate the root cause of the issue.

[Brian Murray (brian-murray)]

I said other arches because of the following amd64 log file:

https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/i/initramfs-tools/20231208_161152_a97ae@/log.gz

[Simon Quigley (tsimonq2)]

You're probably looking for https://discourse.ubuntu.com/t/spec-apt-deb822-sources-by-default/29333/11

This will be an issue until the appropriate tooling is fixed. Until then, looks like we have both.

[Steve Langasek (vorlon)]

As I understand it the code currently fails because there is output on stderr.

But by redirecting stderr to stdout (2>&1), there will now be no output on stderr. So if we intend to detect and fail on other error output, this prevents that.

[Steve Langasek (vorlon)]

Here is a horrible shell pipeline that filters out only 'W: ' lines from stderr, and then puts the rest of stdout and stderr back on the file descriptors where it's supposed to be.

(apt-cache showsrc e2fsprogs 3>&1 1>&2 2>&3 3>&- | grep -v ^W: ) 3>&1 1>&2 2>&3 3>&-

review: Needs Fixing

[Steve Langasek (vorlon)]

I suggest filing a bug against apt about not having any way to suppress warning messages on stderr.

[Paride Legovini (paride)]

That `apt-cache showsrc | awk` logic is gone in newer versions of autopkgtest:

https://salsa.debian.org/ci-team/autopkgtest/-/commit/39290ef8c66d4d2362720383643fd32cf88bf2a7

I'm working at merging that latest Debian version of autopkgtest, however there are plenty of conflicts with our cross architecture testing patchset, so this is not a trivial merge.

01bff8e... by Brian Murray on 2024-01-10

Use Steve's horrible shell pipeline

[Brian Murray (brian-murray)]

Both versions return the same results:

ubuntu@creation-test-amd64-1:~$ (apt-cache showsrc e2fsprogs 3>&1 1>&2 2>&3 3>&- | grep -v ^W: ) 3>&1 1>&2 2>&3 3>&- | awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1 ":*"; next } { show=0 }' | sort -u | tr '\\n' ' '
comerr-dev:*
e2fsck-static:*
e2fsprogs-l10 :*
e2fsprogs-udeb:*
e2fsprogs:*
fuse2fs:*
libcom-err2:*
libext2fs-dev:*
libext2fs2:*
libss2:*
logsave:*
ss-dev:*
ubuntu@creation-test-amd64-1:~$ apt-cache showsrc e2fsprogs 2>&1 | grep -v ^W: | awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1 ":*"; next } { show=0 }' | sort -u | tr '\\n' ' '
comerr-dev:*
e2fsck-static:*
e2fsprogs-l10 :*
e2fsprogs-udeb:*
e2fsprogs:*
fuse2fs:*
libcom-err2:*
libext2fs-dev:*
libext2fs2:*
libss2:*
logsave:*
ss-dev:*

[Steve Langasek (vorlon)]

Just to confirm, yes, the output should be identical between the two commands when there are no other sources of stderr messages

[Tim Andersson (andersson123)]

If the most recent commit gets squashed, this LGTM. I know Paride mentioned when merging our branch with upstream, this code is completely different already, but I think this is a valuable patch that we could carry for the time being.

review: Approve

[Steve Langasek (vorlon)]

review: Approve

[Brian Murray (brian-murray)]

review: Needs Fixing

9f38deb... by Brian Murray on 2024-01-16

Don't hard code a package name here!

Unmerged commits

9f38deb... by Brian Murray on 2024-01-16

Don't hard code a package name here!

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

01bff8e... by Brian Murray on 2024-01-10

Use Steve's horrible shell pipeline

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

ed423be... by Brian Murray on 2023-12-08

adt_testbed.py: supress warnings from apt

test:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

diff --git a/lib/adt_testbed.py b/lib/adt_testbed.py
index cfb0c64..a322847 100644
--- a/lib/adt_testbed.py
+++ b/lib/adt_testbed.py
@@ -1380,7 +1380,7 @@ Description: satisfy autopkgtest test dependencies
 
         # translate src:name entries into binaries of that source
         if srcpkgs:
-            script += 'PKGS="$PKGS $(apt-cache showsrc %s | ' \
+            script += 'PKGS="$PKGS $((apt-cache showsrc %s 3>&1 1>&2 2>&3 3>&- | grep -v ^W: ) 3>&1 1>&2 2>&3 3>&- | ' \
                 '''awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1 ":*"; next } { show=0 }' |''' \
                 '''sort -u | tr '\\n' ' ')"; ''' % \
                 ' '.join(srcpkgs)