Merge ~jawn-smith/autopkgtest-cloud:long-software-properties into autopkgtest-cloud:master

Proposed by William Wilson
Status: Merged
Merged at revision: 7ebd7aadbaa0e5dbef41b2fa2daaeb4e8660943e
Proposed branch: ~jawn-smith/autopkgtest-cloud:long-software-properties
Merge into: autopkgtest-cloud:master
Diff against target: 13 lines (+1/-1)
1 file modified
worker-config-production/worker-lxd.conf (+1/-1)
Reviewer Review Type Date Requested Status
Iain Lane Needs Information
Brian Murray Approve
Review via email: mp+397452@code.launchpad.net

Commit message

Adding software-properties to long_tests to resolve armhf timeouts

Description of the change

Autopkgtests for software-properties are frequently timing out on armhf. The error is below:

autopkgtest [23:26:11]: ERROR: timed out on command "su -s /bin/bash root -c set -e; export USER=`id -nu`; . /etc/profile >/dev/null 2>&1 || true; . ~/.profile >/dev/null 2>&1 || true; buildtree="/tmp/autopkgtest.2dp2gm/build.2eo/src"; mkdir -p -m 1777 -- "/tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-artifacts"; export AUTOPKGTEST_ARTIFACTS="/tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-artifacts"; export ADT_ARTIFACTS="$AUTOPKGTEST_ARTIFACTS"; mkdir -p -m 755 "/tmp/autopkgtest.2dp2gm/autopkgtest_tmp"; export AUTOPKGTEST_TMP="/tmp/autopkgtest.2dp2gm/autopkgtest_tmp"; export ADTTMP="$AUTOPKGTEST_TMP"; export DEBIAN_FRONTEND=noninteractive; export LANG=C.UTF-8; export DEB_BUILD_OPTIONS=parallel=4; unset LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION LC_ALL;rm -f /tmp/autopkgtest_script_pid; set -C; echo $$ > /tmp/autopkgtest_script_pid; set +C; trap "rm -f /tmp/autopkgtest_script_pid" EXIT INT QUIT PIPE; cd "$buildtree"; export AUTOPKGTEST_NORMAL_USER=ubuntu; export ADT_NORMAL_USER=ubuntu; export 'ADT_TEST_TRIGGERS=ubuntu-drivers-common/1:0.8.6.4'; chmod +x /tmp/autopkgtest.2dp2gm/build.2eo/src/debian/tests/add-apt-repository-ppa; touch /tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-stdout /tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-stderr; /tmp/autopkgtest.2dp2gm/build.2eo/src/debian/tests/add-apt-repository-ppa 2> >(tee -a /tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-stderr >&2) > >(tee -a /tmp/autopkgtest.2dp2gm/add-apt-repository-ppa-stdout);" (kind: test)

This should be resolved by adding software-packages to the long_tests, therefore doubling the test-timeout to 20,000 seconds

To post a comment you must log in.
Revision history for this message
Brian Murray (brian-murray) wrote :

This looks good to me, but I'd rather have somebody who can actually update the server merge the code.

review: Approve
Revision history for this message
Iain Lane (laney) wrote :

I think this should be OK, but can you confirm that making the test long lets it complete, and it's not hung?

I'm seeing quite some variation in the history with test durations. That could be due to server load causing the tests to run slower sometimes, but I'd ideally like to know that it's not actually that there's a race condition which means we can hang forever - in which case making the tests 'long' would just hold up nodes for more time.

  https://autopkgtest.ubuntu.com/packages/software-properties/hirsute/armhf

If you can't reproduce the timeout then the alternative strategy would be to merge this and someone agrees to watch the results for a bit to confirm they are consistently green now.

review: Needs Information
Revision history for this message
Julian Andres Klode (juliank) wrote :

I've rolled this out, we're fairly confident this fixes the issue and it's load and not a race that blocks forever

Revision history for this message
William Wilson (jawn-smith) wrote :

> I think this should be OK, but can you confirm that making the test long lets
> it complete, and it's not hung?
>
> I'm seeing quite some variation in the history with test durations. That could
> be due to server load causing the tests to run slower sometimes, but I'd
> ideally like to know that it's not actually that there's a race condition
> which means we can hang forever - in which case making the tests 'long' would
> just hold up nodes for more time.
>
> https://autopkgtest.ubuntu.com/packages/software-properties/hirsute/armhf
>
> If you can't reproduce the timeout then the alternative strategy would be to
> merge this and someone agrees to watch the results for a bit to confirm they
> are consistently green now.

I've taken the following steps to demonstrate that adding this to long_tests will resolve the issue:

1) Run the test in a local armhf lxd instance 5 times with --timeout-test=999999999
    * All tests succeeded.
2) Run the test in a local armhf lxd instance with --timeout-test=10
    * Observed the same exact error as in the failed armhf autopkgtests.
3) Reviewed the source code to look for any possible race conditions
    * Found nothing. The test that is timing out simply iterates
      through all possible combinations of flags to 'add-apt-respository'.
    * This is a total of 638 combinations, which takes quite a while
    * None of these commands happen in parallel or asynchronously,
      so a race condition seems unlikely

In addition to these steps I'd be happy to watch the results for a bit to confirm they are consistently green now

Revision history for this message
Iain Lane (laney) wrote :

Did you see my last paragraph?

Revision history for this message
Iain Lane (laney) wrote :

On Thu, Feb 04, 2021 at 04:44:43PM -0000, William Wilson wrote:
> > I think this should be OK, but can you confirm that making the test long lets
> In addition to these steps I'd be happy to watch the results for a bit
> to confirm they are consistently green now

Thanks William. Sounds good to me. I just guard capacity, especially
armhf capacity, a bit closely as it is relatively limited & slow. So I
don't want us to accidentally make this worse.

Cheers!

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/worker-config-production/worker-lxd.conf b/worker-config-production/worker-lxd.conf
index 457bb82..bc52375 100644
--- a/worker-config-production/worker-lxd.conf
+++ b/worker-config-production/worker-lxd.conf
@@ -17,7 +17,7 @@ architectures = $LXD_ARCH
17setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh17setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
18setup_command2 = sed -E -i "s/(us.)?ports.ubuntu.com/ftpmaster.internal/; s/ubuntu-ports/ubuntu/" /etc/apt/sources.list `ls /etc/apt/sources.list.d/*.list 2>/dev/null || true`; ln -s /dev/null /etc/systemd/system/bluetooth.service; printf "http_proxy=http://squid.internal:3128\nhttps_proxy=http://squid.internal:3128\nno_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,launchpad.net,10.24.0.0/24\n" >> /etc/environment18setup_command2 = sed -E -i "s/(us.)?ports.ubuntu.com/ftpmaster.internal/; s/ubuntu-ports/ubuntu/" /etc/apt/sources.list `ls /etc/apt/sources.list.d/*.list 2>/dev/null || true`; ln -s /dev/null /etc/systemd/system/bluetooth.service; printf "http_proxy=http://squid.internal:3128\nhttps_proxy=http://squid.internal:3128\nno_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,launchpad.net,10.24.0.0/24\n" >> /etc/environment
19big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts heat-dashboard samtools satpy python-cffi yt skimage scikit-learn silx statsmodels puma dask19big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts heat-dashboard samtools satpy python-cffi yt skimage scikit-learn silx statsmodels puma dask
20long_tests = glibc gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 mercurial salt epson-inkjet-printer-escpr haveged pandas simde linux-hwe-5.0 linux-hwe-5.4 openjdk-820long_tests = glibc gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 mercurial salt epson-inkjet-printer-escpr haveged pandas simde linux-hwe-5.0 linux-hwe-5.4 openjdk-8 software-properties
21blacklist = xenial/armhf/libnih artful/armhf/libnih bionic/armhf/libnih bionic/armhf/r-cran-vioplot bionic/armhf/openjdk-lts xenial/armhf/virtualbox-ext-pack bionic/armhf/virtualbox-ext-pack focal/armhf/openjdk-13 groovy/armhf/openjdk-13 focal/armhf/openjdk-14 groovy/armhf/openjdk-14 hirsute/armhf/openjdk-13 hirsute/armhf/openjdk-14 hirsute/armhf/openjdk-lts hirsute/armhf/rust-onig21blacklist = xenial/armhf/libnih artful/armhf/libnih bionic/armhf/libnih bionic/armhf/r-cran-vioplot bionic/armhf/openjdk-lts xenial/armhf/virtualbox-ext-pack bionic/armhf/virtualbox-ext-pack focal/armhf/openjdk-13 groovy/armhf/openjdk-13 focal/armhf/openjdk-14 groovy/armhf/openjdk-14 hirsute/armhf/openjdk-13 hirsute/armhf/openjdk-14 hirsute/armhf/openjdk-lts hirsute/armhf/rust-onig
2222
23[virt]23[virt]

Subscribers

People subscribed via source and target branches