Merge ~ginggs/autopkgtest-cloud:big_packages-pyfai into autopkgtest-cloud:master

[Graham Inggs]

Add pyfai to big_packages on x86 and arm64

[Adam Conrad (adconrad)]

It looks like you're doing this for OOM reasons, not timeout reasons. As such, I don't understand why you're only bumping it on some arches, rather than all. Yes, it currently doesn't run on ppc64el and s390x for other reasons, but if those reasons cleared up, it would OOM there all the same, surely?

[Adam Conrad (adconrad)]

(ditto for armhf, where the reason it passes is because our current infra doesn't constrain RAM usage on armhf containers, but if we moved to VMs, they'd be identical to the arm64 ones)

[Graham Inggs (ginggs)]

I conservatively only added it on affected architectures. I figured it is easy to add it later, where required.

(I looked at previous additions to big_packages, and many don't seem to be added for all architectures.)

[Adam Conrad (adconrad)]

The reason the others aren't affected isn't because they use less RAM, though, it's cause either (a) the infra is weird (armhf) or (b) the test isn't being run at all.

Many big_packages are for timeouts, which is more sensitive to CPU speed, and thus much more arch-specific. Our VMs, however, are all sized the same, so OOM issues should be similar across the board (with possible exceptions for 64-vs-32-bit, or bugs that we'd want to know about).

I'd contend that only changing it on the arches where it would pass today is a ticking confusion bomb for arches where it could pass tomorrow, if only we'd remembered to set this everywhere.

[Graham Inggs (ginggs)]

Shall I go ahead and add pyfai to worker-*-ppc64el.conf and worker-*-s390x.conf?

And worker-lxd.conf for armhf too?

[Brian Murray (brian-murray)]

big_packages are now handled in a separate git branch so I'm rejecting this.

Unmerged commits

50a07bf... by Graham Inggs on 2019-10-01

Add pyfai to big_packages on x86 and arm64

diff --git a/worker-config-production/worker-bos01-arm64.conf b/worker-config-production/worker-bos01-arm64.conf
index 491936e..134598e 100644
--- a/worker-config-production/worker-bos01-arm64.conf
+++ b/worker-config-production/worker-bos01-arm64.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic disco eoan
 architectures = arm64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad pyfai
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 salt
 blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bionic/arm64/r-cran-vioplot disco/arm64/stress-ng disco/arm64/upstart disco/arm64/libnih disco/arm64/r-cran-vioplot bionic/arm64/openjdk-lts disco/arm64/openjdk-lts xenial/arm64/virtualbox-ext-pack bionic/arm64/virtualbox-ext-pack disco/arm64/virtualbox-ext-pack
 
diff --git a/worker-config-production/worker-bos02-arm64.conf b/worker-config-production/worker-bos02-arm64.conf
index 491936e..134598e 100644
--- a/worker-config-production/worker-bos02-arm64.conf
+++ b/worker-config-production/worker-bos02-arm64.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic disco eoan
 architectures = arm64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad pyfai
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 salt
 blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bionic/arm64/r-cran-vioplot disco/arm64/stress-ng disco/arm64/upstart disco/arm64/libnih disco/arm64/r-cran-vioplot bionic/arm64/openjdk-lts disco/arm64/openjdk-lts xenial/arm64/virtualbox-ext-pack bionic/arm64/virtualbox-ext-pack disco/arm64/virtualbox-ext-pack
 
diff --git a/worker-config-production/worker.conf b/worker-config-production/worker.conf
index 347683c..4681534 100644
--- a/worker-config-production/worker.conf
+++ b/worker-config-production/worker.conf
@@ -16,7 +16,7 @@ releases = precise trusty xenial bionic disco eoan
 architectures = i386 amd64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts nodejs cl-ironclad python-scipy cross-toolchain-base-mipsen mkl-dnn
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts nodejs cl-ironclad python-scipy cross-toolchain-base-mipsen mkl-dnn pyfai
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft cross-toolchain-base
 blacklist = bionic/i386/upstart bionic/amd64/upstart bionic/i386/libnih bionic/amd64/libnih bionic/i386/r-cran-vioplot bionic/amd64/r-cran-vioplot bionic/amd64/stress-ng bionic/i386/stress-ng disco/amd64/openjdk-lts bionic/i386/openjdk-lts disco/i386/openjdk-lts xenial/amd64/virtualbox-ext-pack bionic/amd64/virtualbox-ext-pack disco/amd64/virtualbox-ext-pack xenial/i386/virtualbox-ext-pack bionic/i386/virtualbox-ext-pack disco/i386/virtualbox-ext-pack