Merge autopkgtest-cloud:i386-as-amd64-cross into autopkgtest-cloud:master

[Steve Langasek]

[Iain Lane (laney)]

Thanks for this, some specific comments inline. Nothing too major.

One general question: I'm concerned about the hardcoding of "== focal" here. Can you elaborate on whether there is a plan to drop i386 completely after focal which will make this be okay?

I think it is probably better in any event to avoid hardcoding that policy / plan in autopkgtest-cloud, and move to checking for ">= focal" instead, so that we don't have to think about revisiting that if the plan is to drop i386 and yet it doesn't come off for some reason - or doesn't happen immediately in the GG cycle. The requester (the Ubuntu archive and/or proposed-migration) would then have the policy about which tests it wants to see results for. Using something like the following class which lets you compare releases:

https://gist.github.com/iainlane/eaee9f2b1e681d49fb6ed51343dee215

review: Needs Fixing

[Iain Lane (laney)]

Steve, can I nudge you to address these review comments please? It's clear now that we should make this apply to all releases >= focal. At the start of the hirsute cycle, because this is currently cowboyed on the production infrastructure and requires manual editing, I made a mistake and all i386 tests ran as amd64 for a few days. Now this is fixed but we have regressions reported that need to be manually hinted away.

We should not be in this position again - both because there's no cowboy and because it doesn't require adjustment at each opening.

Here's a "release_ge" script I wrote for something else, which you can use like "if release_ge ${RELEASE} focal; then ... fi" in create-nova-image-new-release, to inject the --add-architecture only when needed.

https://gist.github.com/iainlane/a6df7f4c8e928fd7d1cccc6bef643211

[Balint Reczey (rbalint)]

I've added handling of later releases in:
https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/395503

diff --git a/worker-config-production/setup-canonical.sh b/worker-config-production/setup-canonical.sh
index af76f24..f61f1c0 100644
--- a/worker-config-production/setup-canonical.sh
+++ b/worker-config-production/setup-canonical.sh
@@ -39,3 +39,10 @@ if [ -d /run/systemd/system ] && systemd-detect-virt --quiet --vm; then
 ExecStart=/usr/bin/perl -E 'open \$\$f, "/bin/bash" or die; open \$\$rnd, ">/dev/random" or die; for (\$\$i = 0; \$\$i < 10; ++\$\$i) {read \$\$f, \$\$d, 64; ioctl \$\$rnd, 0x40085203, pack("ii", 64*8, 64) . \$\$d}'
 EOF
 fi
+
+# Make our amd64 runners capable of testing i386
+if grep -q focal /etc/os-release && [ "$(dpkg --print-architecture)" = amd64 ]
+then
+    dpkg --add-architecture i386
+    apt-get update
+fi
diff --git a/worker-config-production/worker-bos01-arm64.conf b/worker-config-production/worker-bos01-arm64.conf
index 5a84577..38c3975 100644
--- a/worker-config-production/worker-bos01-arm64.conf
+++ b/worker-config-production/worker-bos01-arm64.conf
@@ -24,4 +24,4 @@ blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bion
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos01-ppc64el.conf b/worker-config-production/worker-bos01-ppc64el.conf
index e8a6d38..cc80e9b 100644
--- a/worker-config-production/worker-bos01-ppc64el.conf
+++ b/worker-config-production/worker-bos01-ppc64el.conf
@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos01-s390x.conf b/worker-config-production/worker-bos01-s390x.conf
index 9fc3d7e..43a1e77 100644
--- a/worker-config-production/worker-bos01-s390x.conf
+++ b/worker-config-production/worker-bos01-s390x.conf
@@ -24,4 +24,4 @@ blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bion
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos01.conf b/worker-config-production/worker-bos01.conf
index f7a8f57..d034ec0 100644
--- a/worker-config-production/worker-bos01.conf
+++ b/worker-config-production/worker-bos01.conf
@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos02-arm64.conf b/worker-config-production/worker-bos02-arm64.conf
index 5a84577..38c3975 100644
--- a/worker-config-production/worker-bos02-arm64.conf
+++ b/worker-config-production/worker-bos02-arm64.conf
@@ -24,4 +24,4 @@ blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bion
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos02-ppc64el.conf b/worker-config-production/worker-bos02-ppc64el.conf
index 1f74340..fbffa47 100644
--- a/worker-config-production/worker-bos02-ppc64el.conf
+++ b/worker-config-production/worker-bos02-ppc64el.conf
@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-bos02-s390x.conf b/worker-config-production/worker-bos02-s390x.conf
index 9fc3d7e..43a1e77 100644
--- a/worker-config-production/worker-bos02-s390x.conf
+++ b/worker-config-production/worker-bos02-s390x.conf
@@ -24,4 +24,4 @@ blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bion
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker-config-production/worker-canonistack.conf b/worker-config-production/worker-canonistack.conf
index a91e685..32a6c19 100644
--- a/worker-config-production/worker-canonistack.conf
+++ b/worker-config-production/worker-canonistack.conf
@@ -24,4 +24,4 @@ long_tests = gutenprint gmp-ecm open-iscsi diffoscope
 # flavors
 package_size_default = m1.small
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net'
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname #KEYNAME# -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net'
diff --git a/worker-config-production/worker.conf b/worker-config-production/worker.conf
index d16e926..f184909 100644
--- a/worker-config-production/worker.conf
+++ b/worker-config-production/worker.conf
@@ -24,4 +24,4 @@ blacklist = bionic/i386/upstart bionic/amd64/upstart bionic/i386/libnih bionic/a
 # flavors
 package_size_default = autopkgtest
 package_size_big = m1.large
-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$HOSTARCH-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
diff --git a/worker/worker b/worker/worker
index 6a769a6..aaec435 100755
--- a/worker/worker
+++ b/worker/worker
@@ -176,11 +176,16 @@ def series_to_version(series):
 
 
 def subst(s, autopkgtest_checkout, big_package, release, architecture, pkgname):
+    if release == 'focal' and architecture == 'i386':
+        host_arch = 'amd64'
+    else:
+        host_arch = architecture
     subst = {
         'CHECKOUTDIR': autopkgtest_checkout,
         'AUTOPKGTEST_CLOUD_DIR': root_path,
         'RELEASE': release,
         'ARCHITECTURE': architecture,
+        'HOSTARCH': host_arch,
         'PACKAGENAME': pkgname,
         'PACKAGESIZE': cfg.get('virt',
                                big_package and 'package_size_big' or 'package_size_default'),
@@ -359,6 +364,9 @@ def request(msg):
         argv = ['autopkgtest']
     argv += ['--output-dir', out_dir, '--timeout-copy=6000']
 
+    if release in ('focal',) and architecture == 'i386':
+        argv += ['-a', 'i386']
+
     c = cfg.get('autopkgtest', 'extra_args')
     if c:
         argv += c.strip().split()
@@ -479,9 +487,14 @@ def request(msg):
     if 'testname' in params:
         argv.append('--testname=%s' % params['testname'])
 
+    if release in ('focal',) and architecture == 'i386':
+        img_arch = 'amd64'
+    else:
+        img_arch = architecture
+
     argv.append('--')
     argv += subst(cfg.get('virt', 'args'), autopkgtest_checkout, big_pkg,
-                  release, architecture, pkgname).split()
+                  release, img_architecture, pkgname).split()
 
     # run autopkgtest; retry up to three times on tmpfail issues
     if not blacklisted: