Merge ~juliank/autopkgtest-cloud:multiple-cloud-workers into autopkgtest-cloud:master

Proposed by Julian Andres Klode
Status: Work in progress
Proposed branch: ~juliank/autopkgtest-cloud:multiple-cloud-workers
Merge into: autopkgtest-cloud:master
Diff against target: 179 lines (+31/-9)
13 files modified
deployment/charms/xenial/autopkgtest-cloud-worker/hooks/leader-elected (+9/-0)
tools/build-adt-image-all-clouds (+5/-0)
tools/cleanup-instances (+5/-0)
worker-config-production/worker-bos01-arm64.conf (+1/-1)
worker-config-production/worker-bos01-ppc64el.conf (+1/-1)
worker-config-production/worker-bos01-s390x.conf (+1/-1)
worker-config-production/worker-bos01.conf (+1/-1)
worker-config-production/worker-bos02-arm64.conf (+1/-1)
worker-config-production/worker-bos02-ppc64el.conf (+1/-1)
worker-config-production/worker-bos02-s390x.conf (+1/-1)
worker-config-production/worker-canonistack.conf (+1/-1)
worker-config-production/worker.conf (+1/-1)
worker/worker (+3/-0)
Reviewer Review Type Date Requested Status
Ubuntu Release Team Pending
Review via email: mp+366571@code.launchpad.net

Description of the change

WIP adding support for multiple workers. Will fail to cleanup old instances created before this change.

To post a comment you must log in.
09a43f0... by Julian Andres Klode

cloud worker: Only enable image building on the leader

This simply touches a file on the leader and the script then does
nothing if the file does not exist.

Unmerged commits

09a43f0... by Julian Andres Klode

cloud worker: Only enable image building on the leader

This simply touches a file on the leader and the script then does
nothing if the file does not exist.

86771fc... by Julian Andres Klode

cleanup-instances: Only cleanup instances with matching hostnames (if any)

If the instance name has the correct number of components, and the
one supposed to be the hostname is not our hostname, ignore it.

29bffda... by Julian Andres Klode

worker-config-production: Add $HOSTNAME to worker names

This allows us to know which cloud worker started the task,
in case there are multiple ones.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/leader-elected b/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/leader-elected
2new file mode 100755
3index 0000000..4e6cb5c
4--- /dev/null
5+++ b/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/leader-elected
6@@ -0,0 +1,9 @@
7+#!/bin/sh
8+set -e
9+
10+LEADER=$(is-leader)
11+if [ "${LEADER}" = "True" ]; then
12+ touch /tmp/autopkgtest-cloud-worker-is-leader
13+else
14+ rm /tmp/autopkgtest-cloud-worker-is-leader || true
15+fi
16diff --git a/tools/build-adt-image-all-clouds b/tools/build-adt-image-all-clouds
17index a0de5bd..57b2882 100755
18--- a/tools/build-adt-image-all-clouds
19+++ b/tools/build-adt-image-all-clouds
20@@ -3,6 +3,11 @@
21 # architectures.
22 set -eu
23
24+if ! test -e /tmp/autopkgtest-cloud-worker-is-leader; then
25+ echo "Not the leader, exiting"
26+ exit
27+fi
28+
29 SETUP_SCRIPT="$1"
30 RELEASE=${RELEASE:-$(distro-info --devel)}
31 case "${2:-}" in
32diff --git a/tools/cleanup-instances b/tools/cleanup-instances
33index 578f606..2b00c8e 100755
34--- a/tools/cleanup-instances
35+++ b/tools/cleanup-instances
36@@ -3,6 +3,7 @@
37 import os
38 import time
39 import logging
40+import socket
41 import subprocess
42
43 import novaclient.client
44@@ -31,6 +32,7 @@ else:
45 tenant_name=os.environ['OS_TENANT_NAME'])
46
47 sess = session.Session(auth=auth)
48+our_hostname = socket.gethostname()
49
50
51 nova = novaclient.client.Client('2', session=sess,
52@@ -52,6 +54,9 @@ for instance in nova.servers.list():
53
54 if not instance.name.startswith('adt-'):
55 continue
56+ if not instance.name.endswith("-" + our_hostname):
57+ logging.debug('instance %s is managed by different worker, ignoring', instance.name)
58+ continue
59
60 # check age
61 if age > MAX_AGE:
62diff --git a/worker-config-production/worker-bos01-arm64.conf b/worker-config-production/worker-bos01-arm64.conf
63index cecd4bf..3553b89 100644
64--- a/worker-config-production/worker-bos01-arm64.conf
65+++ b/worker-config-production/worker-bos01-arm64.conf
66@@ -24,4 +24,4 @@ blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bion
67 # flavors
68 package_size_default = autopkgtest
69 package_size_big = m1.large
70-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
71+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
72diff --git a/worker-config-production/worker-bos01-ppc64el.conf b/worker-config-production/worker-bos01-ppc64el.conf
73index 546d25c..6a5a457 100644
74--- a/worker-config-production/worker-bos01-ppc64el.conf
75+++ b/worker-config-production/worker-bos01-ppc64el.conf
76@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
77 # flavors
78 package_size_default = autopkgtest
79 package_size_big = m1.large
80-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
81+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
82diff --git a/worker-config-production/worker-bos01-s390x.conf b/worker-config-production/worker-bos01-s390x.conf
83index 37137a9..44da227 100644
84--- a/worker-config-production/worker-bos01-s390x.conf
85+++ b/worker-config-production/worker-bos01-s390x.conf
86@@ -24,4 +24,4 @@ blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bion
87 # flavors
88 package_size_default = autopkgtest
89 package_size_big = m1.large
90-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
91+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
92diff --git a/worker-config-production/worker-bos01.conf b/worker-config-production/worker-bos01.conf
93index 546d25c..6a5a457 100644
94--- a/worker-config-production/worker-bos01.conf
95+++ b/worker-config-production/worker-bos01.conf
96@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
97 # flavors
98 package_size_default = autopkgtest
99 package_size_big = m1.large
100-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
101+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
102diff --git a/worker-config-production/worker-bos02-arm64.conf b/worker-config-production/worker-bos02-arm64.conf
103index cecd4bf..3553b89 100644
104--- a/worker-config-production/worker-bos02-arm64.conf
105+++ b/worker-config-production/worker-bos02-arm64.conf
106@@ -24,4 +24,4 @@ blacklist = bionic/arm64/stress-ng bionic/arm64/upstart bionic/arm64/libnih bion
107 # flavors
108 package_size_default = autopkgtest
109 package_size_big = m1.large
110-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
111+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
112diff --git a/worker-config-production/worker-bos02-ppc64el.conf b/worker-config-production/worker-bos02-ppc64el.conf
113index 546d25c..6a5a457 100644
114--- a/worker-config-production/worker-bos02-ppc64el.conf
115+++ b/worker-config-production/worker-bos02-ppc64el.conf
116@@ -24,4 +24,4 @@ blacklist = bionic/ppc64el/stress-ng bionic/ppc64el/upstart bionic/ppc64el/libni
117 # flavors
118 package_size_default = autopkgtest
119 package_size_big = m1.large
120-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
121+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
122diff --git a/worker-config-production/worker-bos02-s390x.conf b/worker-config-production/worker-bos02-s390x.conf
123index 37137a9..44da227 100644
124--- a/worker-config-production/worker-bos02-s390x.conf
125+++ b/worker-config-production/worker-bos02-s390x.conf
126@@ -24,4 +24,4 @@ blacklist = bionic/s390x/stress-ng bionic/s390x/upstart bionic/s390x/libnih bion
127 # flavors
128 package_size_default = autopkgtest
129 package_size_big = m1.large
130-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
131+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --nova-reboot --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server-.* --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,changelogs.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
132diff --git a/worker-config-production/worker-canonistack.conf b/worker-config-production/worker-canonistack.conf
133index 21ed604..76304cb 100644
134--- a/worker-config-production/worker-canonistack.conf
135+++ b/worker-config-production/worker-canonistack.conf
136@@ -24,4 +24,4 @@ long_tests = gutenprint gmp-ecm open-iscsi diffoscope
137 # flavors
138 package_size_default = m1.small
139 package_size_big = m1.large
140-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net'
141+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,ppa.launchpad.net'
142diff --git a/worker-config-production/worker.conf b/worker-config-production/worker.conf
143index 6c528ba..82f73fc 100644
144--- a/worker-config-production/worker.conf
145+++ b/worker-config-production/worker.conf
146@@ -24,4 +24,4 @@ blacklist = bionic/i386/upstart bionic/amd64/upstart bionic/i386/libnih bionic/a
147 # flavors
148 package_size_default = autopkgtest
149 package_size_big = m1.large
150-args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
151+args = ssh -s $CHECKOUTDIR/ssh-setup/nova -- --flavor $PACKAGESIZE --security-groups $SECGROUP --name adt-$RELEASE-$ARCHITECTURE-$PACKAGENAME-$TIMESTAMP-$HOSTNAME --image adt/ubuntu-$RELEASE-$ARCHITECTURE-server --keyname #KEYNAME# --net-id=net_ues_proposed_migration -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,ppa.launchpad.net' --mirror=http://ftpmaster.internal/ubuntu
152diff --git a/worker/worker b/worker/worker
153index f431a6c..f7b4cd8 100755
154--- a/worker/worker
155+++ b/worker/worker
156@@ -20,6 +20,7 @@ import urllib.request
157 import re
158 import hashlib
159 import random
160+import socket
161
162 import amqplib.client_0_8 as amqp
163 import distro_info
164@@ -40,6 +41,7 @@ running_test = False
165 status_exchange_name = 'teststatus.fanout'
166 amqp_con = None
167 systemd_logging_handler = systemd.journal.JournalHandler()
168+hostname = socket.gethostname()
169
170 # In the case of a tmpfail, look for these strings in the log and if they're
171 # found, consider this a real failure instead. This is useful if the test
172@@ -180,6 +182,7 @@ def subst(s, autopkgtest_checkout, big_package, release, architecture, pkgname):
173 'PACKAGESIZE': cfg.get('virt',
174 big_package and 'package_size_big' or 'package_size_default'),
175 'TIMESTAMP': time.strftime('%Y%m%d-%H%M%S'),
176+ 'HOSTNAME': hostname,
177 }
178 for i in args.variable:
179 k, v = i.split('=', 1)

Subscribers

People subscribed via source and target branches