Merge ~xnox/autopkgtest-cloud:master into autopkgtest-cloud:master

[Dimitri John Ledkov]

Add all kernel flavours to big_packages.

LP: https://bugs.launchpad.net/auto-package-testing/+bug/1831446

Add all kernel flavours to big_packages.

LP: https://bugs.launchpad.net/auto-package-testing/+bug/1831446

[Iain Lane (laney)]

Thanks for this. Seems like a real problem, but it feels like updating this static list will very easily get forgotten.

What about making the entries into globs?

review: Needs Fixing

[Dimitri John Ledkov (xnox)]

@laney

A simple .startswith('linux-*') would result in roughly half the packages being false positives. If we don't care about those ending up as big_packages (and i think most of them don't have autopkgtests) we could merge just a check for 'linux-*'.

The below is the list of unaffected source packages, that start with 'linux-*'

Package: linux-atm
Package: linux-base
Package: linux-ftpd
Package: linux-ftpd-ssl
Package: linux-igd
Package: linux-meta
Package: linux-meta-aws
Package: linux-meta-aws-edge
Package: linux-meta-azure
Package: linux-meta-azure-edge
Package: linux-meta-gcp
Package: linux-meta-gcp-edge
Package: linux-meta-gke-4.15
Package: linux-meta-hwe
Package: linux-meta-hwe-edge
Package: linux-meta-kvm
Package: linux-meta-oem
Package: linux-meta-oem-osp1
Package: linux-meta-oracle
Package: linux-meta-raspi2
Package: linux-meta-snapdragon
Package: linux-minidisc
Package: linux-show-player
Package: linux-signed
Package: linux-signed-azure
Package: linux-signed-azure-edge
Package: linux-signed-gcp
Package: linux-signed-gcp-edge
Package: linux-signed-gke-4.15
Package: linux-signed-hwe
Package: linux-signed-hwe-edge
Package: linux-signed-oem
Package: linux-signed-oem-osp1
Package: linux-signed-oracle
Package: linux-user-chroot
Package: linux-wlan-ng

[Iain Lane (laney)]

That seems a better option to me. If it started to be a problem we could think again then.

I wouldn't merge "just" a check for "linux-*" (literally). It should be something like (untested):

big_pkg = [ fnmatch.fnmatchcase(pkgname, bigpkg) for bigpkg in glob.glob(cfg.get('autopkgtest', 'big_packages').strip().split() ] is not None

or something more efficient maybe, to stop at the first match.

[Dimitri John Ledkov (xnox)]

hmmm. let me test all that.

[Iain Lane (laney)]

thanks

...dunno what that 'glob.glob' randomly in the middle there is about...

[Dimitri John Ledkov (xnox)]

i was staring at the glob.glob until my brain timed out.

"i was like, do i miss the bit of where packages are unpacked on disk or stamp files created..."

[Dimitri John Ledkov (xnox)]

Ok, added pattern matching support, and then added 'linux-*' as a big package pattern.

[Iain Lane (laney)]

thanks!

review: Approve

diff --git a/worker-config-production/worker-bos01-arm64.conf b/worker-config-production/worker-bos01-arm64.conf
index cecd4bf..6f449d6 100644
--- a/worker-config-production/worker-bos01-arm64.conf
+++ b/worker-config-production/worker-bos01-arm64.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = arm64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 salt
 blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bionic/arm64/r-cran-vioplot cosmic/arm64/stress-ng cosmic/arm64/upstart cosmic/arm64/libnih cosmic/arm64/r-cran-vioplot disco/arm64/stress-ng disco/arm64/upstart disco/arm64/libnih disco/arm64/r-cran-vioplot bionic/arm64/openjdk-lts cosmic/arm64/openjdk-lts disco/arm64/openjdk-lts
 
diff --git a/worker-config-production/worker-bos01-ppc64el.conf b/worker-config-production/worker-bos01-ppc64el.conf
index 546d25c..60bb893 100644
--- a/worker-config-production/worker-bos01-ppc64el.conf
+++ b/worker-config-production/worker-bos01-ppc64el.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = ppc64el
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base
 blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libnih bionic/ppc64el/r-cran-vioplot cosmic/ppc64el/stress-ng cosmic/ppc64el/upstart cosmic/ppc64el/libnih cosmic/ppc64el/r-cran-vioplot disco/ppc64el/stress-ng disco/ppc64el/upstart disco/ppc64el/libnih disco/ppc64el/r-cran-vioplot bionic/ppc64el/openjdk-lts cosmic/ppc64el/openjdk-lts disco/ppc64el/openjdk-lts
 
diff --git a/worker-config-production/worker-bos01-s390x.conf b/worker-config-production/worker-bos01-s390x.conf
index 37137a9..2060b42 100644
--- a/worker-config-production/worker-bos01-s390x.conf
+++ b/worker-config-production/worker-bos01-s390x.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = s390x
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base
 blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bionic/s390x/r-cran-vioplot cosmic/s390x/stress-ng cosmic/s390x/upstart cosmic/s390x/libnih cosmic/s390x/r-cran-vioplot disco/s390x/stress-ng disco/s390x/upstart disco/s390x/libnih disco/s390x/r-cran-vioplot bionic/s390x/openjdk-lts cosmic/s390x/openjdk-lts disco/s390x/openjdk-lts
 
diff --git a/worker-config-production/worker-bos01.conf b/worker-config-production/worker-bos01.conf
index 546d25c..60bb893 100644
--- a/worker-config-production/worker-bos01.conf
+++ b/worker-config-production/worker-bos01.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = ppc64el
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base
 blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libnih bionic/ppc64el/r-cran-vioplot cosmic/ppc64el/stress-ng cosmic/ppc64el/upstart cosmic/ppc64el/libnih cosmic/ppc64el/r-cran-vioplot disco/ppc64el/stress-ng disco/ppc64el/upstart disco/ppc64el/libnih disco/ppc64el/r-cran-vioplot bionic/ppc64el/openjdk-lts cosmic/ppc64el/openjdk-lts disco/ppc64el/openjdk-lts
 
diff --git a/worker-config-production/worker-bos02-arm64.conf b/worker-config-production/worker-bos02-arm64.conf
index cecd4bf..6f449d6 100644
--- a/worker-config-production/worker-bos02-arm64.conf
+++ b/worker-config-production/worker-bos02-arm64.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = arm64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts lintian cl-ironclad
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 salt
 blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bionic/arm64/r-cran-vioplot cosmic/arm64/stress-ng cosmic/arm64/upstart cosmic/arm64/libnih cosmic/arm64/r-cran-vioplot disco/arm64/stress-ng disco/arm64/upstart disco/arm64/libnih disco/arm64/r-cran-vioplot bionic/arm64/openjdk-lts cosmic/arm64/openjdk-lts disco/arm64/openjdk-lts
 
diff --git a/worker-config-production/worker-bos02-ppc64el.conf b/worker-config-production/worker-bos02-ppc64el.conf
index 546d25c..60bb893 100644
--- a/worker-config-production/worker-bos02-ppc64el.conf
+++ b/worker-config-production/worker-bos02-ppc64el.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = ppc64el
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base
 blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libnih bionic/ppc64el/r-cran-vioplot cosmic/ppc64el/stress-ng cosmic/ppc64el/upstart cosmic/ppc64el/libnih cosmic/ppc64el/r-cran-vioplot disco/ppc64el/stress-ng disco/ppc64el/upstart disco/ppc64el/libnih disco/ppc64el/r-cran-vioplot bionic/ppc64el/openjdk-lts cosmic/ppc64el/openjdk-lts disco/ppc64el/openjdk-lts
 
diff --git a/worker-config-production/worker-bos02-s390x.conf b/worker-config-production/worker-bos02-s390x.conf
index 37137a9..2060b42 100644
--- a/worker-config-production/worker-bos02-s390x.conf
+++ b/worker-config-production/worker-bos02-s390x.conf
@@ -16,7 +16,7 @@ releases = trusty xenial bionic cosmic disco eoan
 architectures = s390x
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base
 blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bionic/s390x/r-cran-vioplot cosmic/s390x/stress-ng cosmic/s390x/upstart cosmic/s390x/libnih cosmic/s390x/r-cran-vioplot disco/s390x/stress-ng disco/s390x/upstart disco/s390x/libnih disco/s390x/r-cran-vioplot bionic/s390x/openjdk-lts cosmic/s390x/openjdk-lts disco/s390x/openjdk-lts
 
diff --git a/worker-config-production/worker-canonistack.conf b/worker-config-production/worker-canonistack.conf
index 21ed604..a91e685 100644
--- a/worker-config-production/worker-canonistack.conf
+++ b/worker-config-production/worker-canonistack.conf
@@ -17,7 +17,7 @@ releases = trusty xenial
 architectures = i386 amd64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope
 
 [virt]
diff --git a/worker-config-production/worker-lxd.conf b/worker-config-production/worker-lxd.conf
index f681599..3e00aba 100644
--- a/worker-config-production/worker-lxd.conf
+++ b/worker-config-production/worker-lxd.conf
@@ -16,7 +16,7 @@ releases = precise trusty xenial bionic cosmic disco eoan
 architectures = $LXD_ARCH
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = sed -i "s/ports.ubuntu.com/ftpmaster.internal/; s/ubuntu-ports/ubuntu/" /etc/apt/sources.list `ls /etc/apt/sources.list.d/*.list 2>/dev/null || true`; ln -s /dev/null /etc/systemd/system/bluetooth.service; printf "http_proxy=http://squid.internal:3128\nhttps_proxy=http://squid.internal:3128\nno_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24\n" >> /etc/environment
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts heat-dashboard
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python-ruffus openjdk-8 openjdk-lts heat-dashboard
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft snapd cross-toolchain-base automake-1.16 mercurial salt
 blacklist = xenial/armhf/libnih artful/armhf/libnih bionic/armhf/libnih bionic/armhf/r-cran-vioplot cosmic/armhf/r-cran-vioplot disco/armhf/r-cran-vioplot bionic/armhf/openjdk-lts cosmic/armhf/openjdk-lts disco/armhf-openjdk-lts
 
diff --git a/worker-config-production/worker.conf b/worker-config-production/worker.conf
index 6c528ba..03de5c2 100644
--- a/worker-config-production/worker.conf
+++ b/worker-config-production/worker.conf
@@ -16,7 +16,7 @@ releases = precise trusty xenial bionic cosmic disco eoan
 architectures = i386 amd64
 setup_command = $AUTOPKGTEST_CLOUD_DIR/worker-config-production/setup-canonical.sh
 setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts nodejs cl-ironclad
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia python-cryptography libtext-bidi-perl python-hypothesis nova rocs camitk kineticstools r-cran-igraph botch mathicgb ganeti lttng-modules python2.7 python3.6 python3.7 python3.8 python-ruffus openjdk-8 openjdk-lts nodejs cl-ironclad
 long_tests = gutenprint gmp-ecm open-iscsi diffoscope snapcraft cross-toolchain-base
 blacklist = bionic/i386/upstart bionic/amd64/upstart bionic/i386/libnih bionic/amd64/libnih bionic/i386/r-cran-vioplot bionic/amd64/r-cran-vioplot bionic/amd64/stress-ng bionic/i386/stress-ng cosmic/i386/upstart cosmic/amd64/upstart cosmic/i386/libnih cosmic/amd64/libnih cosmic/i386/r-cran-vioplot cosmic/amd64/r-cran-vioplot cosmic/amd64/stress-ng cosmic/i386/stress-ng bionic/amd64/openjdk-lts cosmic/amd64/openjdk-lts disco/amd64/openjdk-lts bionic/i386/openjdk-lts cosmic/i386/openjdk-lts disco/i386/openjdk-lts
 
diff --git a/worker/worker b/worker/worker
index f431a6c..8de7057 100755
--- a/worker/worker
+++ b/worker/worker
@@ -20,6 +20,7 @@ import urllib.request
 import re
 import hashlib
 import random
+import fnmatch
 
 import amqplib.client_0_8 as amqp
 import distro_info
@@ -74,8 +75,6 @@ FAIL_PKG_STRINGS = {'systemd*': ['timed out waiting for testbed to reboot',
 
 
 def getglob(d, glob, default=None):
-    import fnmatch
-
     for k in d:
         if fnmatch.fnmatch(glob, k):
             return d[k]
@@ -345,7 +344,7 @@ def request(msg):
             testpkg_version.write('%s blacklisted' % pkgname)
 
     container = 'autopkgtest-' + release
-    big_pkg = pkgname in cfg.get('autopkgtest', 'big_packages').strip().split()
+    big_pkg = any((fnmatch.fnmatchcase(pkgname, pkg) for pkg in cfg.get('autopkgtest', 'big_packages').strip().split()))
 
     autopkgtest_checkout = cfg.get('autopkgtest', 'checkout_dir').strip()
     if autopkgtest_checkout:
diff --git a/worker/worker.conf b/worker/worker.conf
index 5b12065..3ae8b03 100644
--- a/worker/worker.conf
+++ b/worker/worker.conf
@@ -17,7 +17,7 @@ releases = trusty xenial
 architectures = i386 amd64
 setup_command =
 # setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux tdb firefox akonadi julia libtext-bidi-perl rocs camitk kineticstools r-cran-igraph botch mathicgb openjdk-8 openjdk-lts
+big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl rocs camitk kineticstools r-cran-igraph botch mathicgb openjdk-8 openjdk-lts
 long_tests = gutenprint gmp-ecm open-iscsi
 blacklist =