Merge autopkgtest-cloud:wip/mojo-juju-2 into autopkgtest-cloud:master

[Iain Lane]

https://trello.com/c/oB0tznGF/31-use-juju-2

I've moved from the old Juju and custom deployment script to new Juju and deployment via Mojo (https://mojo.canonical.com).

This ***still needs work*** to be ready for the production environment (see the above Trello card) - so do not merge(!), but it is ready for some review now.

[Iain Lane (laney)]

Don't worry about the conflicts IMHO; the worker-configs are irrelevant in the new deployment anyhow.

[Julian Andres Klode (juliank)]

This branch is working for me locally, so it seems good to go.

review: Approve

c761ac9... by Iain Lane on 2021-03-11

autopkgtest-cloud worker, charm: Collect big/long/no run packages from git

Instead of in the charm's own config. This lets any release team member
merge changes to these list, without needing to go to an
autopkgtest-cloud admin for deployment.

800824f... by Iain Lane on 2021-03-24

copy-security-group: Omit copying the project_id and tenant_id attributes

These will be implicitly created. On newer Openstack versions including
these causes an error:

  neutronclient.common.exceptions.BadRequest: Specifying 'project_id' or
  'tenant_id' other than the authenticated project in request requires
  admin privileges

956412d... by Iain Lane on 2021-04-08

download-results: Make sure to ack the complete notifications all the time

Otherwise when we ignore them they'll be left hanging around in the
queue.

3e30d78... by Iain Lane on 2021-04-12

download-all-results: Fetch results from Swift API

This lets us fetch all results, not just the newest ones. That's useful
in case `download-results` breaks.

a5cdac9... by Iain Lane on 2021-04-12

service-bundle: Add a first armhf worker

Just one for now - the current user is still up and we mainly just need
to make sure that it's working.

3f5c036... by Iain Lane on 2021-04-12

.gitignore: Ignore sphinx build artifacts

22663d7... by Iain Lane on 2021-04-12

make-lxd-secgroup: Update router IP, add haproxy

2b9414c... by Iain Lane on 2021-04-12

autopkgtest-lxd-remote@: Be wanted by autopkgtest.target

It needs to get installed somewhere.

68823f3... by Iain Lane on 2021-04-12

update-github-jobs: Exit if PENDING_DIR doesn't exit

0250661... by Iain Lane on 2021-04-12

service-bundle: Fix swift creds

eac3f7b... by Iain Lane on 2021-04-21

docs/: Add docs for readthedocs

666dd40... by Iain Lane on 2021-04-21

.gitignore: Ignore built docs

5aff327... by Iain Lane on 2021-04-22

Trim the README now we have readthedocs docs

eaa78aa... by Iain Lane on 2021-04-23

service-bundle: expose rabbitmq

It needs to be externally accessible, e.g. by snakefruit.

3ec671a... by Iain Lane on 2021-04-26

docs: Add instructions for how to update

until we get webhooks

9f16fe1... by Iain Lane on 2021-04-26

service-bundle: Add impish

07e8a1f... by Iain Lane on 2021-04-26

webcontrol: Add link to RTD docs

7ce7401... by Iain Lane on 2021-04-26

docs: Document running download-all-results

It needs to be run after seed-new-release to get the results into
autopkgtest.db.

7ed840c... by Iain Lane on 2021-04-26

docs: add a TODO note

3d07862... by Iain Lane on 2021-04-26

docs: Specify the mirror when building new lxd images

a731689... by Iain Lane on 2021-04-26

docs: Fix whitespace errors, thanks git diff

Unmerged commits

a731689... by Iain Lane on 2021-04-26

docs: Fix whitespace errors, thanks git diff

3d07862... by Iain Lane on 2021-04-26

docs: Specify the mirror when building new lxd images

7ed840c... by Iain Lane on 2021-04-26

docs: add a TODO note

7ce7401... by Iain Lane on 2021-04-26

docs: Document running download-all-results

It needs to be run after seed-new-release to get the results into
autopkgtest.db.

07e8a1f... by Iain Lane on 2021-04-26

webcontrol: Add link to RTD docs

9f16fe1... by Iain Lane on 2021-04-26

service-bundle: Add impish

3ec671a... by Iain Lane on 2021-04-26

docs: Add instructions for how to update

until we get webhooks

eaa78aa... by Iain Lane on 2021-04-23

service-bundle: expose rabbitmq

It needs to be externally accessible, e.g. by snakefruit.

5aff327... by Iain Lane on 2021-04-22

Trim the README now we have readthedocs docs

666dd40... by Iain Lane on 2021-04-21

.gitignore: Ignore built docs

diff --git a/.gitignore b/.gitignore
index 74e308d..accb26a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1 @@
-deployment/charms/xenial/rabbitmq-server
+docs/_build
diff --git a/COPYRIGHT b/COPYRIGHT
index a07e413..234b0d9 100644
--- a/COPYRIGHT
+++ b/COPYRIGHT
@@ -1,5 +1,6 @@
-autopkgtest-cloud is (C) 2015-2016 Canonical Ltd.
-Author: Martin Pitt <[e-mail]>
+autopkgtest-cloud is (C) 2015-2019 Canonical Ltd.
+Author: Martin Pitt <[e-mail]>,
+        Iain Lane <[e-mail]>
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..75a195d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,11 @@
+autopkgtest-cloud
+=================
+
+Introduction
+------------
+
+This is Ubuntu's environment for running autopkgtests, and collecting /
+delivering their results.
+
+See docs/ or [our readthedocs page](https://autopkgtest-cloud.readthedocs.io)
+for the complete documentation for developers and admins of autopkgtest-cloud.
diff --git a/autopkgtest-cloud b/autopkgtest-cloud
new file mode 120000
index 0000000..a64523e
--- /dev/null
+++ b/autopkgtest-cloud
@@ -0,0 +1 @@
+charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud
\ No newline at end of file
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/README.md b/charms/focal/autopkgtest-cloud-worker/README.md
similarity index 87%
rename from deployment/charms/xenial/autopkgtest-cloud-worker/README.md
rename to charms/focal/autopkgtest-cloud-worker/README.md
index a66b49e..44da3e0 100644
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/README.md
+++ b/charms/focal/autopkgtest-cloud-worker/README.md
@@ -6,4 +6,4 @@ This charm sets up an autopkgtest cloud worker. This receives test requests from
 Contact Information
 -------------------
 
-Author: Martin Pitt <[e-mail]>
+Author: Iain Lane <[e-mail]>
diff --git a/charms/focal/autopkgtest-cloud-worker/actions.yaml b/charms/focal/autopkgtest-cloud-worker/actions.yaml
new file mode 100644
index 0000000..98243ff
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/actions.yaml
@@ -0,0 +1,5 @@
+reload-units:
+    description: Reload all systemd units. Use this after you upgrade the charm to change the worker code, for example.
+
+update-sources:
+    description: Update (git pull) autopkgtest/autodep8
diff --git a/charms/focal/autopkgtest-cloud-worker/actions/reload-units b/charms/focal/autopkgtest-cloud-worker/actions/reload-units
new file mode 100755
index 0000000..52ab519
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/actions/reload-units
@@ -0,0 +1,21 @@
+#!/usr/local/sbin/charm-env python3
+
+from systemd import get_units, reload_unit
+
+
+if __name__ == "__main__":
+    (cloud, lxd, _) = get_units()
+    cloud_object_paths = [
+        cloud[region][arch][n]
+        for region in cloud
+        for arch in cloud[region]
+        for n in cloud[region][arch]
+    ]
+    lxd_object_paths = [
+        lxd[arch][ip][n]
+        for arch in lxd
+        for ip in lxd[arch]
+        for n in lxd[arch][ip]
+    ]
+    for op in cloud_object_paths + lxd_object_paths:
+        reload_unit(op)
diff --git a/charms/focal/autopkgtest-cloud-worker/actions/update-sources b/charms/focal/autopkgtest-cloud-worker/actions/update-sources
new file mode 100755
index 0000000..7bd2f9d
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/actions/update-sources
@@ -0,0 +1,19 @@
+#!/usr/local/sbin/charm-env python3
+
+import sys
+
+from reactive.autopkgtest_cloud_worker import (
+    AUTOPKGTEST_LOCATION,
+    AUTODEP8_LOCATION,
+)
+
+import pygit2
+from utils import pull, install_autodep8, UnixUser
+
+
+if __name__ == "__main__":
+    for location in [AUTOPKGTEST_LOCATION, AUTODEP8_LOCATION]:
+        repo = pygit2.Repository(location)
+        with UnixUser("ubuntu"):
+            pull(repo)
+    install_autodep8(AUTODEP8_LOCATION)
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/armhf-lxd-cluster-template.userdata.in b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/armhf-lxd-cluster-template.userdata.in
new file mode 100644
index 0000000..aca8491
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/armhf-lxd-cluster-template.userdata.in
@@ -0,0 +1,174 @@
+#cloud-config
+
+ # cloud-init user-data file for setting up an autopkgtest slave for running
+ # armhf LXD containers. This should be booted on an arm64 nova m1.large instance.
+
+manage_etc_hosts: true
+apt_update: true
+apt_upgrade: true
+
+packages:
+ - rng-tools
+ - distro-info
+ - libdpkg-perl
+ - python3-netifaces
+ - python3-jinja2
+
+snap:
+ commands:
+   00: ['refresh', '--channel=latest/stable', 'lxd']
+
+# we don't need much space for the OS, keep most of the disk free for a btrfs partition for lxd
+growpart:
+  mode: off
+
+write_files:
+ - path: /etc/environment
+   content: |
+    PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
+    http_proxy=http://squid.internal:3128
+    https_proxy=http://squid.internal:3128
+    no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,10.0.0.0/8
+ - path: /etc/sysctl.d/99-autopkgtest-inotify.conf
+   content: |
+    # Increase the user inotify instance limit to allow for about
+    # 100 containers to run before the limit is hit again
+    fs.inotify.max_user_instances = 1024
+
+# LP: #1572026, RT#90771
+ - path: /etc/rc.local
+   permissions: '0755'
+   content: |
+     #!/bin/sh -e
+     iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
+
+{%- if role == 'bootstrap' or role == 'leader' %}
+ - path: /var/tmp/setup/ubuntu-crontab
+   content: |
+     PATH=/snap/bin:/usr/bin:/bin
+     # development release
+     37 2 * * * MIRROR=http://ftpmaster.internal/ubuntu/ ~/autopkgtest/tools/autopkgtest-build-lxd images:ubuntu/$(distro-info --devel)/armhf
+     # stable releases
+     1 0 * * 0 for r in $(distro-info --supported | grep -v $(distro-info --devel)); do MIRROR=http://ftpmaster.internal/ubuntu/ ~/autopkgtest/tools/autopkgtest-build-lxd images:ubuntu/$r/armhf; done
+{%- endif %}
+
+ - path: /var/tmp/setup/lxd-init
+   content: |
+     config:
+       core.https_address: '{{ '{{' }} myip {{ '}}' }}:8443'
+       core.trust_password: autopkgtest
+     networks:
+     - config:
+         bridge.mtu: "1458"
+         ipv4.address: 10.0.4.1/24
+         ipv4.nat: "true"
+         ipv6.address: none
+         raw.dnsmasq: dhcp-option=6,91.189.91.130
+       description: ""
+       name: lxdbr0
+       type: bridge
+     storage_pools:
+     - config:
+         source: /srv/lxd
+       description: ""
+       name: default
+       driver: dir
+     profiles:
+     - config:
+         raw.lxc: |-
+           lxc.apparmor.profile=unconfined
+           lxc.seccomp.profile=
+         security.nesting: "true"
+       description: Default LXD profile
+       devices:
+         eth0:
+           name: eth0
+           nictype: bridged
+           parent: lxdbr0
+           type: nic
+         root:
+           path: /
+           pool: default
+           type: disk
+       name: default
+     cluster:
+       enabled: true
+       images_minimal_replica: -1
+       server_name: {{uuid}}
+{%- if role == 'node' %}
+       server_address: {{ '{{' }} myip {{ '}}' }}:8443
+       cluster_address: {{server}}:8443
+       cluster_certificate: "{{cert}}
+     "
+       cluster_password: autopkgtest
+       member_config:
+       - entity: storage-pool
+         name: default
+         key: source
+         value: ""
+{%- endif %}
+
+# we can't do these things during runcmd as this is run *during* boot
+ - path: /var/tmp/setup/post-boot-setup.sh
+   content: |
+     # wait until we are fully booted
+     while true; do O=$(systemctl is-system-running || true); [ "$O" = running -o "$O" = degraded ] && break; sleep 1; done
+     systemctl daemon-reload
+     systemctl start rng-tools
+     systemctl stop snap.lxd.daemon.unix.socket snap.lxd.daemon.service || true
+     systemctl start snap.lxd.daemon.unix.socket snap.lxd.daemon.service
+     python3 -c "from jinja2 import Template; import netifaces as ni; ni.ifaddresses('eth0'); ip = ni.ifaddresses('eth0')[ni.AF_INET][0]['addr']; f = open('/var/tmp/setup/lxd-init'); print(Template(f.read()).render(myip=ip))" | lxd init --preseed
+{%- if role == 'bootstrap' or role == 'leader' %}
+     for r in $(distro-info --supported); do /home/ubuntu/autopkgtest/tools/autopkgtest-build-lxd images:ubuntu/$r/armhf; done
+{%- endif %}
+     echo "Finished building - rebooting so changed kernel command line comes into effect..."
+     reboot
+
+# use gdisk to fix GPT partition table to span the whole disk
+# http://askubuntu.com/questions/446991/gparted-claims-whole-hard-drive-is-unallocated-and-gives-warning-about-gpt-table
+# then create new ~ 80 GB partition and mount it as /srv
+# use compat_uts_machine=armv7l so that 'linux32 uname -m' shows armv7l, like the buildds
+runcmd:
+ - printf 'x\ne\nw\ny\n' | gdisk /dev/vda
+ - printf "d\n1\nn\n\n\n+8000M\n\n\nw\ny\n" | gdisk /dev/vda
+ - partprobe
+ - resize2fs /dev/vda1
+ - lsblk -dno PARTUUID /dev/vda1 | awk '{ print "GRUB_FORCE_PARTUUID=" $0 }' | tee /etc/default/grub.d/40-force-partuuid.cfg
+ - update-grub
+ - echo 'mkpart primary btrfs 8495MB 100%\n' | parted /dev/vda
+ - for timeout in `seq 10`; do [ -e /dev/vda2 ] && break; sleep 1; done
+
+ - mkfs.btrfs -L autopkgtest /dev/vda2
+ - systemctl stop snap.lxd.daemon.service snap.lxd.daemon.unix.socket
+ - echo 'LABEL=autopkgtest   /srv   btrfs defaults 0 0' >> /etc/fstab
+ - echo '/srv/var-snap-lxd-common /var/snap/lxd/common none defaults,bind 0 0' >> /etc/fstab
+ - mount /srv
+ - mkdir -m 0755 /srv/var-snap-lxd-common
+ - mount /var/snap/lxd
+ - mkdir -m 1777 -p /srv/tmp/
+ - echo '/srv/tmp  /tmp   none rw,bind 0 0' >> /etc/fstab
+ - mount /tmp
+ - mkdir -p /srv/lxd
+ - fallocate -l 512M /swapfile
+ - chmod 600 /swapfile
+ - mkswap /swapfile
+ - echo '/swapfile none swap defaults 0 0' >> /etc/fstab
+ - swapon /swapfile
+ - systemctl start snap.lxd.daemon.service snap.lxd.daemon.unix.socket
+
+ - /etc/rc.local
+
+ - sed -i '/GRUB_CMDLINE_LINUX_DEFAULT=/ { s/quiet splash//; s/"$/ compat_uts_machine=armv7l"/ }' /etc/default/grub
+ - update-grub
+
+ - apt-get install -y linux-image-generic
+ - apt-get purge --auto-remove -y linux-headers-generic rsyslog mlocate
+ - dpkg -P `dpkg -l linux-headers* | grep ^ii | awk '{print $2}'`
+ - adduser ubuntu lxd
+
+{%- if role == 'bootstrap' or role == 'leader' %}
+ - apt-get install -y --no-install-recommends git
+ - su - -c '[ -e autopkgtest ] || git clone https://git.launchpad.net/~ubuntu-release/autopkgtest/+git/development autopkgtest' ubuntu
+ - su - -c 'cat /var/tmp/setup/ubuntu-crontab | crontab -' ubuntu
+{%- endif %}
+ - sh -xe /var/tmp/setup/post-boot-setup.sh &
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/build-adt-image b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/build-adt-image
new file mode 100755
index 0000000..8d29c52
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/build-adt-image
@@ -0,0 +1,73 @@
+#!/bin/bash
+# Build adt cloud images with create-nova-image-new-release for the given
+# cloud, release and arch
+
+set -eu
+
+IFS="[- ]" read -r RELEASE REGION ARCH bootstrap <<< "$@"
+
+if [ -z "${RELEASE}" ] || [ -z "${REGION}" ] || [ -z "${ARCH}" ]; then
+        echo "Usage: $0 RELEASE REGION ARCH" >&2
+        exit 1
+fi
+
+[ -z "${MIRROR:-}" ] && . ~/mirror.rc
+export MIRROR
+
+[ -z "${NET_NAME:-}" ] && . ~/net-name.rc
+export NET_NAME
+
+if [ -z "${USE_CLOUD_CONFIG_FROM_ENV:-}" ]; then
+        if [ -e ~/cloudrcs/${REGION}-${ARCH}.rc ]; then
+                . ~/cloudrcs/${REGION}-${ARCH}.rc
+        else
+                . ~/cloudrcs/${REGION}.rc
+        fi
+fi
+
+IMAGE_NAME="adt/ubuntu-${RELEASE}-${ARCH}-server-$(date '+%Y%m%d').img"
+
+if openstack image show "${IMAGE_NAME}" >/dev/null 2>/dev/null; then
+        echo "$0: Image '${IMAGE_NAME}' already exists, not re-creating." >&2
+        exit 0
+fi
+
+IMGS="$(openstack image list -f value)"
+
+PREVIMG=$(echo "$IMGS" | grep "adt/ubuntu-${RELEASE}-${ARCH}" | tail -n1 | awk '{print $2}')
+
+if [ -n "${ENSURE_ONLY:-}" ] && [ -n "${PREVIMG}" ]; then
+        echo "$0: ENSURE_ONLY passed and we have a previous image (${PREVIMG}), doing nothing." >&2
+        exit 0
+fi
+
+RELEASE_VERSION="$(distro-info --series "${RELEASE}" --release | awk '{print $1}')"
+IMG=$(echo "$IMGS" | grep -E "${RELEASE}-(?:daily|${RELEASE_VERSION})-${ARCH}" | tail -n1 | awk '{print $2}')
+if [ -z "$IMG" ]; then
+        # Fall back to previous adt image and upgrade that
+        IMG="${PREVIMG}"
+fi
+
+# If we have neither, fall back to the latest stable release
+BASE_RELEASE=${BASE_RELEASE:-$(distro-info --stable)}
+if [ -z "$IMG" ] && [ "${RELEASE}" = "$(distro-info --devel)" ] && [ -n "${BASE_RELEASE:-}" ]; then
+        echo "$0: No image exists for ${RELEASE}; trying to bootstrap from ${BASE_RELEASE}"
+        IMG=$(echo "$IMGS" | grep "${BASE_RELEASE}-daily-${ARCH}" | tail -n1 | awk '{print $2}')
+        [ -n "$IMG" ] || IMG=$(echo "$IMGS" | grep "adt/ubuntu-${BASE_RELEASE}-${ARCH}" | tail -n1 | awk '{print $2}')
+fi
+
+if [ -z "$IMG" ]; then
+        echo "$0: No matching images found" >&2
+        exit 1
+fi
+
+echo "$REGION-$ARCH: using image $IMG"
+KEYNAME=${KEYNAME:-testbed-$(hostname)}
+$(dirname $0)/create-nova-image-new-release $RELEASE $ARCH $IMG "${KEYNAME}" "$IMAGE_NAME"
+# clean old images
+openstack image list --private -f value | grep --color=none -v "$IMAGE_NAME" | while read id img state; do
+        if $(echo ${img} | grep -qs "adt/ubuntu-${RELEASE}-${ARCH}") && [ ${state} = active ]; then
+                echo "Cleaning up old image $img ($id)"
+                openstack image delete $id
+        fi
+done
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances
new file mode 100755
index 0000000..aad8328
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-instances
@@ -0,0 +1,164 @@
+#!/usr/bin/python3
+# clean up broken/orphaned instances
+import logging
+import os
+import re
+import subprocess
+import time
+from urllib.error import HTTPError
+
+import novaclient.client
+from influxdb import InfluxDBClient
+from keystoneauth1 import session
+from keystoneauth1.identity import v2, v3
+
+try:
+    INFLUXDB_CONTEXT = os.environ["INFLUXDB_CONTEXT"]
+    INFLUXDB_DATABASE = os.environ["INFLUXDB_DATABASE"]
+    INFLUXDB_HOSTNAME = os.environ["INFLUXDB_HOSTNAME"]
+    INFLUXDB_PASSWORD = os.environ["INFLUXDB_PASSWORD"]
+    INFLUXDB_PORT = os.environ["INFLUXDB_PORT"]
+    INFLUXDB_USERNAME = os.environ["INFLUXDB_USERNAME"]
+
+    influx_client = InfluxDBClient(
+        INFLUXDB_HOSTNAME,
+        INFLUXDB_PORT,
+        INFLUXDB_USERNAME,
+        INFLUXDB_PASSWORD,
+        INFLUXDB_DATABASE,
+    )
+except KeyError:
+    influx_client = None
+
+try:
+    region = os.environ["REGION"]
+except KeyError:  # probably being run manually; exec-in-region sets this
+    region = None
+    influx_client = None
+
+# number of seconds at which an instance counts as "implausibly old" long_tests
+# has a copy timeout, test timeout and a maximum build time (Restrictions:
+# build-needed) of 40,000 + 40,000s + 40,000s ~ 33 h
+MAX_AGE = 36 * 3600
+
+measurements = []
+
+if os.environ.get("DEBUG"):
+    logging.basicConfig(level=logging.DEBUG)
+
+if os.environ.get("OS_IDENTITY_API_VERSION") == "3":
+    auth = v3.Password(
+        auth_url=os.environ["OS_AUTH_URL"],
+        username=os.environ["OS_USERNAME"],
+        password=os.environ["OS_PASSWORD"],
+        project_name=os.environ["OS_PROJECT_NAME"],
+        user_domain_name=os.environ["OS_USER_DOMAIN_NAME"],
+        project_domain_name=os.environ["OS_PROJECT_DOMAIN_NAME"],
+    )
+else:
+    auth = v2.Password(
+        auth_url=os.environ["OS_AUTH_URL"],
+        username=os.environ["OS_USERNAME"],
+        password=os.environ["OS_PASSWORD"],
+        tenant_name=os.environ["OS_TENANT_NAME"],
+    )
+
+sess = session.Session(auth=auth)
+
+
+nova = novaclient.client.Client(
+    "2", session=sess, region_name=os.environ["OS_REGION_NAME"]
+)
+
+for instance in nova.servers.list():
+    age = time.mktime(time.gmtime()) - time.mktime(
+        time.strptime(instance.created, "%Y-%m-%dT%H:%M:%SZ")
+    )
+    logging.debug(
+        "%s: status %s, age %is, networks %s"
+        % (instance.name, instance.status, age, instance.networks)
+    )
+
+    # check state
+    if instance.status == "ERROR":
+        try:
+            message = str(instance.fault)
+        except AttributeError:
+            message = "fault message not available"
+        msg = "instance {} ({}) is in error state (message: {})".format(
+            instance.name, instance.id, message
+        )
+        logging.warning("{}, deleting".format(msg))
+        measurements.append(
+            {
+                "measurement": "autopkgtest_delete_event",
+                "fields": {"deleted": True},
+                "tags": {
+                    "deletion_reason": "instance_in_error",
+                    "instance": INFLUXDB_CONTEXT,
+                    "message": msg,
+                    "region": region,
+                },
+            }
+        )
+        instance.delete()
+        continue
+
+    if not instance.name.startswith("adt-"):
+        continue
+
+    # check age
+    if age > MAX_AGE:
+        message = "instance {} ({}) is {:.1f} hours old, deleting".format(
+            instance.name, instance.id, (float(age) / 3600)
+        )
+        logging.warning("{}, deleting".format(message))
+        measurements.append(
+            {
+                "measurement": "autopkgtest_delete_event",
+                "fields": {"deleted": True},
+                "tags": {
+                    "deletion_reason": "instance_too_old",
+                    "instance": INFLUXDB_CONTEXT,
+                    "message": message,
+                    "region": region,
+                },
+            }
+        )
+        instance.delete()
+
+    # check matching adt-run process for instance name
+    try:
+        if (
+            subprocess.call(
+                [
+                    "pgrep",
+                    "-f",
+                    "python.*autopkgtest.* --name %s"
+                    % re.escape(instance.name),
+                ],
+                stdout=subprocess.PIPE,
+            )
+            != 0
+        ):
+            message = "instance {} ({}) has no associated autopkgtest".format(
+                instance.name, instance.id
+            )
+            measurements.append(
+                {
+                    "measurement": "autopkgtest_delete_event",
+                    "fields": {"deleted": True},
+                    "tags": {
+                        "deletion_reason": "instance_no_autopkgtest",
+                        "instance": INFLUXDB_CONTEXT,
+                        "message": message,
+                        "region": region,
+                    },
+                }
+            )
+            instance.delete()
+    except IndexError:
+        logging.warning("instance %s has invalid name" % instance.name)
+
+    if measurements and influx_client:
+        influx_client.write_points(measurements)
diff --git a/tools/cleanup-lxd b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-lxd
similarity index 100%
rename from tools/cleanup-lxd
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cleanup-lxd
diff --git a/tools/cloud-worker-maintenance b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cloud-worker-maintenance
similarity index 55%
rename from tools/cloud-worker-maintenance
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cloud-worker-maintenance
index 06d01d4..b1471a9 100755
--- a/tools/cloud-worker-maintenance
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/cloud-worker-maintenance
@@ -3,41 +3,39 @@
 # Run this from cron a few times a day.
 set -eu
 
-CLOUD_RCS="$(find -maxdepth 1 -name '*.rc' ! -type l)"
-TOOLS_DIR="$(dirname $0)"
+TOOLS_DIR="$(dirname "${0}")"
 
 # clean up orphaned/broken instances
-for cloudrc in $CLOUD_RCS; do
-    . $cloudrc
-    # this might fail because a cloud is temporarily down
-    $TOOLS_DIR/cleanup-instances || true
-done
+if [ -d "${HOME}/cloudrcs/" ]; then
+    for cloudrc in "${HOME}"/cloudrcs/*.rc; do
+        if [ -e "${cloudrc}" ]; then
+            region="$(basename "${cloudrc%*.rc}")"
+            # this might fail because a cloud is temporarily down
+            ( "${TOOLS_DIR}/exec-in-region" "${region}" "$TOOLS_DIR/cleanup-instances" || true )
+        fi
+    done
+fi
 
 # restart died workers (they commonly die because of hitting the quota with
 # lots of orphaned instances)
-if [ -d /run/systemd/system ]; then
-    # check for failed workers and show their log tails
-    for unit in $(systemctl --no-legend --all --state=inactive list-units 'autopkgtest@*.service' | awk '{print $1}'); do
-        if [ "$(systemctl show --property=ExecMainStatus $unit)" = "ExecMainStatus=99" ] ||
-           [ "$(systemctl show --property=Result $unit)" != "Result=success" ]; then
-            # ignore "quota exceeded" failures, not much that we can do about them
-            log="$(journalctl --output=cat --no-pager --lines=300 -u $unit)"
-            if ! echo "$log" | grep -q 'novaclient.exceptions.Forbidden: Quota exceeded'; then
-                cat <<EOF
+# check for failed workers and show their log tails
+for unit in $(systemctl --no-legend --all --state=inactive list-units 'autopkgtest@*.service' | awk '{print $1}'); do
+if [ "$(systemctl show --property=ExecMainStatus "${unit}")" = "ExecMainStatus=99" ] ||
+   [ "$(systemctl show --property=Result "${unit}")" != "Result=success" ]; then
+    # ignore "quota exceeded" failures, not much that we can do about them
+    log="$(journalctl --output=cat --no-pager --lines=300 -u "${unit}")"
+    if ! echo "${log}" | grep -q 'novaclient.exceptions.Forbidden: Quota exceeded'; then
+        cat <<EOF
 ==== worker $unit failed ====
 $log
 
 EOF
-            fi
-        fi
-        sudo systemctl reset-failed "$unit"
-    done
-
-    sudo systemctl stop autopkgtest.target
-    sudo systemctl start autopkgtest.target
-else
-    sudo initctl --quiet start autopkgtest-worker-launch
+    fi
 fi
+sudo systemctl reset-failed "${unit}"
+done
+
+sudo systemctl restart autopkgtest.target
 
 # cleanup lxd cruft
-$TOOLS_DIR/cleanup-lxd
+"${TOOLS_DIR}/cleanup-lxd"
diff --git a/tools/copy-security-group b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/copy-security-group
similarity index 95%
rename from tools/copy-security-group
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/copy-security-group
index 5a956a1..6a6a072 100755
--- a/tools/copy-security-group
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/copy-security-group
@@ -18,7 +18,8 @@ from neutronclient.v2_0 import client
 
 # Members in a security group rule that cannot be copied.
 RULE_MEMBERS_IGNORE = ["id", "tags", "updated_at",
-                       "created_at", "revision_number"]
+                       "created_at", "revision_number",
+                       "project_id", "tenant_id", ]
 
 
 def main():
@@ -30,6 +31,11 @@ def main():
                         help='only delete group')
     args = parser.parse_args()
 
+    # we get called from ExecStartPre of lxd units too (where
+    # copy-security-group isn't required), just bail out if that's the case
+    if 'lxd' in args.name:
+        return
+
     if os.environ.get('OS_IDENTITY_API_VERSION') == '3':
         auth = v3.Password(auth_url=os.environ['OS_AUTH_URL'],
                            username=os.environ['OS_USERNAME'],
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-armhf-cluster-member b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-armhf-cluster-member
new file mode 100755
index 0000000..75e4c74
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-armhf-cluster-member
@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+
+# There are three modes.
+# - bootstrap: The first node of a new cluster. Doesn't connect to anything
+#              else. Sets up cron jobs to regularly rebuild images.
+# - node: Any subsequent nodes. Connect to the given target IP (can be any
+#         other node). No cron job created.
+# - leader: A new bootstrap-like node, but for an existing cluster. Cron jobs
+#           are set up as for bootstrap nodes. Use this when the bootstrap node goes
+#           down.
+
+import os
+import subprocess
+import sys
+import uuid
+
+from jinja2 import Template
+
+IN_FILE = os.path.join(
+    os.path.dirname(os.path.realpath(__file__)),
+    "armhf-lxd-cluster-template.userdata.in",
+)
+
+
+def get_ssl_cert(ip):
+    out = subprocess.check_output(
+        [
+            "ssh",
+            "-o",
+            "UserKnownHostsFile=/dev/null",
+            "-o",
+            "StrictHostKeyChecking=no",
+            "ubuntu@{}".format(ip),
+            "sed ':a;N;$!ba;s/\\n/\\n\\n/g' /var/snap/lxd/common/lxd/cluster.crt",
+        ]
+    )
+
+    return "     ".join(out.decode("utf-8").rstrip("\n").splitlines(True))
+
+
+def usage():
+    print(
+        "usage: {} [bootstrap|leader|node] [IP of cluster to join]".format(
+            os.path.basename(sys.argv[0])
+        ),
+        file=sys.stderr,
+    )
+
+
+try:
+    role = sys.argv[1]
+    if role in ("leader", "node"):
+        to_join = sys.argv[2]
+except IndexError:
+    usage()
+    sys.exit(1)
+
+if role not in ["bootstrap", "leader", "node"]:
+    usage()
+    sys.exit(1)
+
+with open(IN_FILE, "r") as f:
+    template = Template(f.read())
+    if role == "bootstrap":
+        print(template.render(role=role, uuid=uuid.uuid1()))
+    else:
+        cert = get_ssl_cert(to_join)
+        print(
+            template.render(
+                cert=cert,
+                role=role,
+                server=to_join,
+                uuid=uuid.uuid1(),
+            )
+        )
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-nova-image-new-release b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-nova-image-new-release
new file mode 100755
index 0000000..0451144
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-nova-image-new-release
@@ -0,0 +1,180 @@
+#!/bin/bash
+# create an autopkgtest nova image for a new release, based on a generic image
+# Author: Martin Pitt <[e-mail]>
+set -eu
+RELEASE="${1:-}"
+ARCH="${2:-}"
+BASEIMG="${3:-}"
+KEYNAME="${4:-}"
+IMAGE_NAME="${5:-}"
+SETUP_TESTBED="${SETUP_TESTBED:-/home/ubuntu/autopkgtest/setup-commands/setup-testbed}"
+
+function release_ge() {
+    distro-info --all | mawk "
+        /^$1\$/ { l = NR };
+        /^$2\$/ { r = NR };
+        END {
+            if (!l) {
+                print \"Unknown release $1\" > \"/dev/stderr\";
+                exit 1;
+            }
+            if (!r) {
+                print \"Unknown release $2\" > \"/dev/stderr\";
+                exit 1;
+            }
+            if (l >= r) {
+                exit 0
+            } else {
+                exit 1
+            }
+        }"
+}
+
+if [ -z "$IMAGE_NAME" ]; then
+    echo "Usage: $0 <release> <base image name> <key name> <output image name>" >&2
+    exit 1
+fi
+
+if [ "${ARCH}" = amd64 ] && release_ge "${RELEASE}" focal; then
+    ARCHES="[amd64, i386]"
+    RUNCMD="$(printf "runcmd:\n   - [ dpkg, --add-architecture, i386]\n")"
+else
+    ARCHES="default"
+    RUNCMD=
+fi
+
+# unbreak my server option :-(
+userdata=`mktemp`
+trap "rm $userdata" EXIT TERM INT QUIT PIPE
+cat <<EOF > $userdata
+#cloud-config
+
+manage_etc_hosts: true
+package_update: true
+
+apt:
+  primary:
+    - arches: ${ARCHES}
+      uri: ${MIRROR}
+packages:
+  - linux-generic
+${RUNCMD}
+EOF
+
+# create new instance
+INSTNAME="${BASEIMG}-adt-prepare"
+eval "$(openstack network show -f shell ${NET_NAME})"
+
+NET_ID=${id}
+
+retries=20
+while true; do
+    eval "$(openstack server create -f shell --flavor m1.small --image $BASEIMG --user-data $userdata --key-name $KEYNAME --wait $INSTNAME --nic net-id=${NET_ID})"
+    if openstack server show "${id}" >/dev/null 2>/dev/null; then
+        break
+    fi
+    retries=$((retries - 1))
+    if [ $retries -lt 0 ]; then
+        echo "failed 20 times, giving up: openstack server create -f shell --flavor m1.small --image $BASEIMG --user-data $userdata --key-name $KEYNAME --wait $INSTNAME --nic net-id=${NET_ID}" >&2
+        exit 1
+    fi
+    echo "openstack server create failed, waiting ten seconds before retrying"
+    # sometimes this makes a zombie instance
+    openstack server delete "${IMAGE_NAME}" >/dev/null 2>/dev/null || true
+    sleep 10
+done
+
+SRVID="${id}"
+
+trap "openstack server delete ${SRVID}" EXIT TERM INT QUIT PIPE
+
+# determine IP address
+eval "$(openstack server show -f shell ${SRVID})"
+ipaddr=$(echo ${addresses} | awk 'BEGIN { FS="=" } { print $2 }')
+
+SSH_CMD="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@$ipaddr"
+echo "Waiting for ssh (may cause some error messages)..."
+timeout 300 sh -c "while ! $SSH_CMD true; do sleep 5; done"
+
+echo "Waiting until cloud-init is done..."
+timeout 25m $SSH_CMD 'while [ ! -e /var/lib/cloud/instance/boot-finished ]; do sleep 1; done'
+
+echo "Running setup script..."
+cat "${SETUP_TESTBED}" | $SSH_CMD "sudo env MIRROR='${MIRROR:-}' RELEASE='$RELEASE' sh -"
+
+echo "Running Canonical setup script..."
+CANONICAL_SCRIPT=$(dirname $(dirname $(readlink -f $0)))/worker-config-production/setup-canonical.sh
+cat "$CANONICAL_SCRIPT" | $SSH_CMD "sudo env MIRROR='${MIRROR:-}' RELEASE='$RELEASE' sh -"
+
+arch=$($SSH_CMD dpkg --print-architecture)
+
+echo "Check that the upgraded image boots..."
+while true; do
+    if openstack server reboot --wait "${SRVID}"; then
+            break
+    fi
+
+    retries=$((retries - 1))
+    if [ $retries -lt 0 ]; then
+        echo "failed 20 times, giving up: openstack server reboot --wait ${SRVID}" >&2
+        exit 1
+    fi
+    echo "openstack server reboot failed, waiting 10 seconds before retrying" >&2
+    sleep 10
+done
+
+if ! timeout 300 sh -c "while ! $SSH_CMD true; do sleep 5; done"; then
+    echo "ERROR: upgraded instance does not reboot; console-log:" >&2
+    openstack console log show "${SRVID}" >&2 || true
+    exit 1
+fi
+
+echo "Cleaning systemd journal..."
+$SSH_CMD sudo journalctl --rotate --vacuum-time=12h || true
+
+echo "Powering off to get a clean file system..."
+$SSH_CMD sudo poweroff || true
+eval "$(openstack server show -f shell ${SRVID})"
+while [ ${os_ext_sts_vm_state} != "stopped" ]; do
+        sleep 1
+        eval "$(openstack server show -f shell ${SRVID})"
+done
+
+echo "Creating image $IMAGE_NAME ..."
+# this tends to fail occasionally, retry
+retries=20
+while true; do
+    if openstack server image create --wait --name "${IMAGE_NAME}" "${SRVID}"; then
+        break
+    fi
+    inner_retries=20
+    while [ $inner_retries -gt 0 ]; do
+        # server image create often loses its connection but it's actually
+        # working - if the image is uploading, wait a bit for it to finish
+        eval $(openstack image show -f shell --prefix=image_ "${IMAGE_NAME}")
+        eval $(openstack server show -f shell --prefix=server_ "${SRVID}")
+        if [ "${server_os_ext_sts_task_state}" = "image_uploading" ] ||
+           [ "${image_status}" = "saving" ]; then
+            echo "image ${IMAGE_NAME} is uploading, waiting..." >&2
+            sleep 20
+            inner_retries=$((inner_retries - 1))
+        elif [ "${image_status}" = "active" ]; then
+            exit 0
+        elif [ "${image_status}" = "queued" ]; then # borked, try the outer retry loop
+            break
+        else
+            echo "image creation failed, aborting (status: ${image_status})." >&2
+            openstack image delete "${IMAGE_NAME}" >/dev/null 2>/dev/null || true
+            exit 1
+        fi
+    done
+    retries=$((retries - 1))
+    if [ $retries -lt 0 ]; then
+        echo "failed 20 times, giving up: openstack image create --wait --name ${IMAGE_NAME} ${SRVID}" >&2
+        exit 1
+    fi
+    echo "openstack server image create failed, waiting 10 seconds before retrying" >&2
+    # sometimes this makes a zombie image
+    openstack image delete "${IMAGE_NAME}" >/dev/null 2>/dev/null || true
+    sleep 10
+done
diff --git a/tools/create-nova-image-with-proposed-package b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-nova-image-with-proposed-package
similarity index 100%
rename from tools/create-nova-image-with-proposed-package
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/create-nova-image-with-proposed-package
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/ensure-keypair b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/ensure-keypair
new file mode 100755
index 0000000..be664d6
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/ensure-keypair
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+set -eu
+
+IFS="[- ]" read -r RELEASE REGION ARCH bootstrap <<< "$@"
+
+if [ -e ~/cloudrcs/${REGION}-${ARCH}.rc ]; then
+        . ~/cloudrcs/${REGION}-${ARCH}.rc
+else
+        . ~/cloudrcs/${REGION}.rc
+fi
+
+if ! [ -e "${HOME}/.ssh/id_rsa" ]; then
+    echo "No SSH key exists, generating one..."
+    ssh-keygen -N '' -f ~/.ssh/id_rsa
+fi
+
+KEYNAME="testbed-$(hostname)"
+
+if ! openstack keypair show "${KEYNAME}" >/dev/null 2>/dev/null; then
+    echo "Uploading SSH key ~/.ssh/id_rsa.pub to ${REGION} with key name: ${KEYNAME}"
+    openstack keypair create --public-key ~/.ssh/id_rsa.pub "${KEYNAME}"
+fi
diff --git a/tools/exec-in-region b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/exec-in-region
similarity index 76%
rename from tools/exec-in-region
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/exec-in-region
index aac5bea..0261108 100755
--- a/tools/exec-in-region
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/exec-in-region
@@ -9,14 +9,23 @@ if [ $# -lt 2 ]; then
 fi
 
 REGION="$1"
-REGION="${REGION%-*}"
+if ! [ -e "${HOME}/cloudrcs/${REGION}.rc" ]; then
+	REGION="${REGION%-*}"
+fi
+
+if ! [ -e "${HOME}/cloudrcs/${REGION}.rc" ]; then
+	echo "${REGION}.rc not found, exiting." >&2
+	exit 0
+fi
 
 shift 1
 
+export REGION
+
 if [ "${REGION#lxd-}" != "$REGION" ]; then
 	LXD_ARCH=${REGION#*-}; LXD_ARCH=${LXD_ARCH%%-*}
 else
-	. ./${REGION}.rc
+	. ${HOME}/cloudrcs/${REGION}.rc
 fi
 
 exec "$@"
diff --git a/tools/filter-amqp b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/filter-amqp
similarity index 100%
rename from tools/filter-amqp
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/filter-amqp
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/metrics b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/metrics
new file mode 100755
index 0000000..d2e14a8
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/metrics
@@ -0,0 +1,142 @@
+#!/usr/bin/python3
+
+from gi.repository import GLib, Gio
+from influxdb import InfluxDBClient
+
+import json
+import os
+import subprocess
+
+SYSTEM_BUS = Gio.bus_get_sync(Gio.BusType.SYSTEM)
+
+INFLUXDB_CONTEXT = os.environ["INFLUXDB_CONTEXT"]
+INFLUXDB_DATABASE = os.environ["INFLUXDB_DATABASE"]
+INFLUXDB_HOSTNAME = os.environ["INFLUXDB_HOSTNAME"]
+INFLUXDB_PASSWORD = os.environ["INFLUXDB_PASSWORD"]
+INFLUXDB_PORT = os.environ["INFLUXDB_PORT"]
+INFLUXDB_USERNAME = os.environ["INFLUXDB_USERNAME"]
+
+
+def make_submission(counts, measurement):
+    out = []
+    for arch in counts:
+        (active, error) = counts[arch]
+        m = {
+            "measurement": measurement,
+            "fields": {"count": active},
+            "tags": {
+                "arch": arch,
+                "status": "active",
+                "instance": INFLUXDB_CONTEXT,
+            },
+        }
+        out.append(m)
+        m = {
+            "measurement": measurement,
+            "fields": {"count": error},
+            "tags": {
+                "arch": arch,
+                "status": "error",
+                "instance": INFLUXDB_CONTEXT,
+            },
+        }
+        out.append(m)
+    return out
+
+
+def get_units():
+    counts = {}
+
+    (units,) = SYSTEM_BUS.call_sync(
+        "org.freedesktop.systemd1",
+        "/org/freedesktop/systemd1",
+        "org.freedesktop.systemd1.Manager",
+        "ListUnits",
+        None,  # parameters
+        GLib.VariantType("(a(ssssssouso))"),  # reply type
+        Gio.DBusCallFlags.NONE,
+        -1,  # timeout
+        None,
+    )  # cancellable
+
+    for unit in units:
+        (name, _, _, _, sub, _, _, _, _, _) = unit
+        if not name.startswith("autopkgtest@"):
+            continue
+
+        name_cloud = name.strip("autopkgtest@").rstrip(".service")
+
+        if name_cloud.startswith("lxd-"):
+            # lxd-armhf-1.2.3.4-1
+            name_cloud = name_cloud[4:]
+            try:
+                (arch, _, _) = name_cloud.split("-", -1)
+                (active, error) = counts.setdefault(arch, (0, 0))
+                if sub == "running":
+                    active += 1
+                else:
+                    error += 1
+                counts[arch] = (active, error)
+            except ValueError:
+                # XXX: for some reason we get autopkgtest@lxd-armhf-a.b.c.d
+                # (without the number), not sure why
+                pass
+            continue
+
+        try:
+            (region, arch, n) = name_cloud.split("-", -1)
+        except ValueError:
+            # [e-mail]
+            (region, n) = name_cloud.split("-", -1)
+            arch = "amd64"
+        (active, error) = counts.setdefault(arch, (0, 0))
+
+        if sub == "running":
+            active += 1
+        else:
+            error += 1
+        counts[arch] = (active, error)
+
+    return make_submission(counts, "autopkgtest_unit_status")
+
+
+def get_remotes():
+    counts = {}
+    out = subprocess.check_output(
+        ["lxc", "remote", "list", "--format=json"], universal_newlines=True
+    )
+
+    for r in json.loads(out):
+        if not r.startswith("lxd"):
+            continue
+
+        (_, arch, ip) = r.split("-", 3)
+        (active, error) = counts.setdefault(arch, (0, 0))
+
+        try:
+            cl = subprocess.check_output(
+                ["lxc", "cluster", "list", f"{r}:", "--format=json"],
+                universal_newlines=True,
+            )
+        except subprocess.CalledProcessError:  # all backends down, eek
+            continue
+
+        for node in json.loads(cl):
+            if node["status"] == "Online":
+                active += 1
+            else:
+                error += 1
+            counts[arch] = (active, error)
+
+    return make_submission(counts, "autopkgtest_cluster_status")
+
+
+influx_client = InfluxDBClient(
+    INFLUXDB_HOSTNAME,
+    INFLUXDB_PORT,
+    INFLUXDB_USERNAME,
+    INFLUXDB_PASSWORD,
+    INFLUXDB_DATABASE,
+)
+
+influx_client.write_points(get_units() + get_remotes())
diff --git a/tools/retry-github-test b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/retry-github-test
similarity index 100%
rename from tools/retry-github-test
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/retry-github-test
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/run-autopkgtest b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/run-autopkgtest
new file mode 100755
index 0000000..5c2bd67
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/run-autopkgtest
@@ -0,0 +1,198 @@
+#!/usr/bin/python3
+# Request runs of autopkgtests for packages
+# Imported from lp:ubuntu-archive-scripts, lightly modified to not rely on a
+# britney config file, to be used for administration or testing.
+
+from datetime import datetime
+import os
+import sys
+import argparse
+import json
+import urllib.parse
+
+import amqplib.client_0_8 as amqp
+
+my_dir = os.path.dirname(os.path.realpath(sys.argv[0]))
+
+
+def parse_args():
+    """Parse command line arguments"""
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "-s", "--series", required=True, help="Distro series name (required)."
+    )
+    parser.add_argument(
+        "-a",
+        "--architecture",
+        action="append",
+        default=[],
+        help="Only run test(s) on given architecture name(s). "
+        "Can be specified multiple times (default: all).",
+    )
+    parser.add_argument(
+        "--trigger",
+        action="append",
+        default=[],
+        metavar="SOURCE/VERSION",
+        help="Add triggering package to request. "
+        "Can be specified multiple times.",
+    )
+    parser.add_argument(
+        "--ppa",
+        metavar="LPUSER/PPANAME",
+        action="append",
+        default=[],
+        help="Enable PPA for requested test(s). "
+        "Can be specified multiple times.",
+    )
+    parser.add_argument(
+        "--env",
+        metavar="KEY=VALUE",
+        action="append",
+        default=[],
+        help="List of VAR=value strings. "
+        "This can be used to influence a test's behaviour "
+        "from a test request. "
+        "Can be specified multiple times.",
+    )
+    parser.add_argument(
+        "--test-git",
+        metavar="URL [branchname]",
+        help="A single URL or URL branchname. "
+        "The test will be git cloned from that URL and ran "
+        "from the checkout. This will not build binary "
+        "packages from the branch and run tests against "
+        "those, the test dependencies will be taken from the "
+        "archive, or PPA if given. In this case the "
+        "srcpkgname will only be used for the result path in "
+        "swift and be irrelevant for the actual test.",
+    )
+    parser.add_argument(
+        "--build-git",
+        metavar="URL [branchname]",
+        help="A single URL or URL branchname. "
+        "Like --test-git`, but will first build binary "
+        "packages from the branch and run tests against those.",
+    )
+    parser.add_argument(
+        "--test-bzr",
+        help="A single URL. "
+        "The test will be checked out with bzr from that URL. "
+        "Otherwise this has the same behaviour as test-git.",
+    )
+    parser.add_argument(
+        "--all-proposed",
+        action="store_true",
+        help="Disable apt pinning and use all of -proposed",
+    )
+    parser.add_argument(
+        "--bulk",
+        action="store_true",
+        help="Mark this as a bulk (low priority) test where possible",
+    )
+    parser.add_argument(
+        "package", nargs="+", help="Source package name(s) whose tests to run."
+    )
+    args = parser.parse_args()
+
+    if not args.trigger and not args.ppa:
+        parser.error("One of --trigger or --ppa must be given")
+
+    if not args.architecture:
+        parser.error("--architecture must be given")
+
+    # verify syntax of triggers
+    for t in args.trigger:
+        try:
+            (src, ver) = t.split("/")
+        except ValueError:
+            parser.error(
+                'Invalid trigger format "%s", must be "sourcepkg/version"' % t
+            )
+
+    # verify syntax of PPAs
+    for t in args.ppa:
+        try:
+            (user, name) = t.split("/")
+        except ValueError:
+            parser.error(
+                'Invalid ppa format "%s", must be "lpuser/ppaname"' % t
+            )
+
+    return args
+
+
+if __name__ == "__main__":
+    args = parse_args()
+
+    context = ""
+    params = {}
+    if args.bulk:
+        context = "huge-"
+    if args.trigger:
+        params["triggers"] = args.trigger
+    if args.ppa:
+        params["ppas"] = args.ppa
+        context = "ppa-"
+    if args.env:
+        params["env"] = args.env
+    if args.test_git:
+        params["test-git"] = args.test_git
+        context = "upstream-"
+    elif args.build_git:
+        params["build-git"] = args.build_git
+        context = "upstream-"
+    if args.test_bzr:
+        params["test-bzr"] = args.test_bzr
+        context = "upstream-"
+    if args.all_proposed:
+        params["all-proposed"] = True
+    try:
+        params["requester"] = os.environ["SUDO_USER"]
+    except KeyError:
+        pass
+    params["submit-time"] = datetime.strftime(
+        datetime.utcnow(), "%Y-%m-%d %H:%M:%S%z"
+    )
+    params = "\n" + json.dumps(params)
+
+    try:
+        creds = urllib.parse.urlsplit(
+            "amqp://{user}:{password}@{host}".format(
+                user=os.environ["RABBIT_USER"],
+                password=os.environ["RABBIT_PASSWORD"],
+                host=os.environ["RABBIT_HOST"],
+            ),
+            allow_fragments=False,
+        )
+    except KeyError:
+        with open(os.path.expanduser("~/rabbitmq.cred"), "r") as f:
+            env_dict = dict(
+                tuple(line.replace("\n", "").replace('"', "").split("="))
+                for line in f.readlines()
+                if not line.startswith("#")
+            )
+        creds = urllib.parse.urlsplit(
+            "amqp://{user}:{password}@{host}".format(
+                user=env_dict["RABBIT_USER"],
+                password=env_dict["RABBIT_PASSWORD"],
+                host=env_dict["RABBIT_HOST"],
+            ),
+            allow_fragments=False,
+        )
+    assert creds.scheme == "amqp"
+
+    with amqp.Connection(
+        creds.hostname, userid=creds.username, password=creds.password
+    ) as amqp_con:
+        with amqp_con.channel() as ch:
+            for arch in args.architecture:
+                queue = "debci-%s%s-%s" % (context, args.series, arch)
+                for pkg in args.package:
+                    ch.basic_publish(
+                        amqp.Message(
+                            pkg + params, delivery_mode=2
+                        ),  # persistent
+                        routing_key=queue,
+                    )
diff --git a/tools/seed-new-release b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/seed-new-release
similarity index 100%
rename from tools/seed-new-release
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/seed-new-release
diff --git a/worker-config-production/setup-canonical.sh b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
similarity index 92%
rename from worker-config-production/setup-canonical.sh
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
index 5bcbc23..af76f24 100644
--- a/worker-config-production/setup-canonical.sh
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
@@ -39,10 +39,3 @@ if [ -d /run/systemd/system ] && systemd-detect-virt --quiet --vm; then
 ExecStart=/usr/bin/perl -E 'open \$\$f, "/bin/bash" or die; open \$\$rnd, ">/dev/random" or die; for (\$\$i = 0; \$\$i < 10; ++\$\$i) {read \$\$f, \$\$d, 64; ioctl \$\$rnd, 0x40085203, pack("ii", 64*8, 64) . \$\$d}'
 EOF
 fi
-
-# Make our amd64 runners capable of testing i386 starting with 20.04
-if [ $(lsb_release -r -s | sed 's/\..*//') -gt 19 ]  && [ "$(dpkg --print-architecture)" = amd64 ]
-then
-    dpkg --add-architecture i386
-    apt-get update
-fi
diff --git a/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
similarity index 72%
rename from worker/worker
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index d1916d2..685f07d 100755
--- a/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -2,7 +2,7 @@
 # autopkgtest cloud worker
 # Author: Martin Pitt <[e-mail]>
 #
-# Requirements: python3-amqplib python3-swiftclient
+# Requirements: python3-amqplib python3-swiftclient python3-influxdb
 # Requirements for running autopkgtest from git: python3-debian libdpkg-perl
 
 import os
@@ -27,10 +27,29 @@ import distro_info
 import swiftclient
 import systemd.journal
 
+from influxdb import InfluxDBClient
 from urllib.error import HTTPError
 
 ALL_RELEASES = distro_info.UbuntuDistroInfo().get_all(result='object')
 
+try:
+    INFLUXDB_CONTEXT = os.environ["INFLUXDB_CONTEXT"]
+    INFLUXDB_DATABASE = os.environ["INFLUXDB_DATABASE"]
+    INFLUXDB_HOSTNAME = os.environ["INFLUXDB_HOSTNAME"]
+    INFLUXDB_PASSWORD = os.environ["INFLUXDB_PASSWORD"]
+    INFLUXDB_PORT = os.environ["INFLUXDB_PORT"]
+    INFLUXDB_USERNAME = os.environ["INFLUXDB_USERNAME"]
+
+    influx_client = InfluxDBClient(
+        INFLUXDB_HOSTNAME,
+        INFLUXDB_PORT,
+        INFLUXDB_USERNAME,
+        INFLUXDB_PASSWORD,
+        INFLUXDB_DATABASE,
+    )
+except KeyError:
+    influx_client = None
+
 my_path = os.path.dirname(__file__)
 root_path = os.path.dirname(os.path.abspath(my_path))
 args = None
@@ -39,9 +58,15 @@ swift_creds = {}
 exit_requested = None
 running_test = False
 status_exchange_name = 'teststatus.fanout'
+complete_exchange_name = 'testcomplete.fanout'
 amqp_con = None
 systemd_logging_handler = systemd.journal.JournalHandler()
 
+# Read in from config files
+big_packages = set()
+long_tests = set()
+never_run = set()
+
 FAIL_CODES = (4, 6, 12, 14, 20)
 
 # In the case of a tmpfail, look for these strings in the log and if they're
@@ -104,6 +129,32 @@ FAIL_PKG_STRINGS = {'systemd*': ['timed out waiting for testbed to reboot',
                     'llvm-toolchain-*': ['clang: error: unable to execute command: Segmentation fault (core dumped)']}
 
 
+def submit_metric(architecture, code, pkgname, current_region, retry, release):
+    if influx_client is None:
+        return
+
+    region = (
+        current_region
+        if current_region.startswith("lxd")
+        else current_region.split("-")[0]
+    )
+
+    point = {
+        "measurement": "autopkgtest_exit_event",
+        "fields": {"exited": True},
+        "tags": {
+            "arch": architecture,
+            "exit_status": code,
+            "instance": INFLUXDB_CONTEXT,
+            "package": pkgname,
+            "region": region,
+            "retry": retry,
+            "series": release,
+        },
+    }
+    influx_client.write_points([point])
+
+
 def getglob(d, glob, default=None):
     for k in d:
         if fnmatch.fnmatch(glob, k):
@@ -147,6 +198,44 @@ def parse_args():
     return parser.parse_args()
 
 
+def read_per_package_configs(cfg):
+    def read_per_package_file(filename):
+        out = set()
+        with open(filename, 'r') as f:
+            entries = {
+                line.strip()
+                for line in f.readlines()
+                if not line.startswith("#")
+            }
+
+            for entry in entries:
+                try:
+                    (package, arch, release) = entry.split("/", 3)
+                except ValueError:
+                    release = "all"
+                    try:
+                        (package, arch) = entry.split("/", 2)
+                    except ValueError:
+                        arch = "all"
+                        package = entry
+                finally:
+                    out.add(f"{package}/{arch}/{release}")
+            return out
+    global big_packages, long_tests, never_run
+
+    dir = cfg.get('autopkgtest', 'per_package_config_dir').strip()
+
+    big_packages = read_per_package_file(os.path.join(dir, "big_packages"))
+    long_tests = read_per_package_file(os.path.join(dir, "long_tests"))
+    never_run = read_per_package_file(os.path.join(dir, "never_run"))
+
+
+def request_matches_per_package(package, arch, release, s):
+    return (any(fnmatch.fnmatchcase(f"{package}/{arch}/{release}", entry) for entry in s) or
+            any(fnmatch.fnmatchcase(f"{package}/{arch}/all", entry) for entry in s) or
+            any(fnmatch.fnmatchcase(f"{package}/all/all", entry) for entry in s))
+
+
 def process_output_dir(dir, pkgname, code):
     '''Post-process output directory'''
 
@@ -162,6 +251,21 @@ def process_output_dir(dir, pkgname, code):
             testpkg_version.write('%s unknown' % pkgname)
         files.add('testpkg-version')
 
+    with open(os.path.join(dir, 'testpkg-version'), 'r') as tpv:
+        testpkg_version = tpv.read().split()[1]
+
+    try:
+        with open(os.path.join(dir, 'duration'), 'r') as dur:
+            duration = dur.read()
+    except FileNotFoundError:
+        duration = None
+
+    try:
+        with open(os.path.join(dir, 'requester'), 'r') as req:
+            requester = req.read()
+    except FileNotFoundError:
+        requester = None
+
     # these are small and we need only these for gating and indexing
     resultfiles = ['exitcode']
     # these might not be present in infrastructure failure cases
@@ -191,6 +295,8 @@ def process_output_dir(dir, pkgname, code):
             else:
                 os.unlink(path)
 
+    return (testpkg_version, duration, requester)
+
 
 def series_to_version(series):
     versions = [x.version for x in ALL_RELEASES if x.series == series]
@@ -201,24 +307,24 @@ def series_to_version(series):
 def i386_cross_series(series):
     # the first version where i386 is a partiel architecture and only cross
     # testing is done is 20.04
-    ver = series_to_version(series)
-    if ver:
-        return ver > '19.10'
+    return series_to_version(series) > '19.10'
 
-    return False
 
+def host_arch(release, architecture):
+    if architecture != 'i386':
+        return architecture
 
-def subst(s, autopkgtest_checkout, big_package, release, architecture, pkgname):
-    if i386_cross_series(release) and architecture == 'i386':
-        host_arch = 'amd64'
-    else:
-        host_arch = architecture
+    if not i386_cross_series(release):
+        return architecture
+
+    return 'amd64'
+
+
+def subst(s, big_package, release, architecture, hostarch, pkgname):
     subst = {
-        'CHECKOUTDIR': autopkgtest_checkout,
-        'AUTOPKGTEST_CLOUD_DIR': root_path,
         'RELEASE': release,
         'ARCHITECTURE': architecture,
-        'HOSTARCH': host_arch,
+        'HOSTARCH': hostarch,
         'PACKAGENAME': pkgname,
         'PACKAGESIZE': cfg.get('virt',
                                big_package and 'package_size_big' or 'package_size_default'),
@@ -322,7 +428,11 @@ def request(msg):
         if extra.startswith("ADT_"):
             del systemd_logging_handler._extra[extra]
 
-    blacklisted = False
+    # Re-read in case the big/long/no run lists changed, would be better to
+    # this only when needed via inotify.
+    read_per_package_configs(cfg)
+
+    dont_run = False
 
     # FIXME: make this more elegant
     fields = msg.delivery_info['routing_key'].split('-')
@@ -368,23 +478,14 @@ def request(msg):
                  pkgname, release, architecture, params)
 
     current_region = os.environ.get('REGION')
-    if current_region:
-        if ('%s/%s/%s/%s' % (release, architecture, pkgname, current_region) in
-                cfg.get('autopkgtest', 'blacklist').split()) or \
-            ('all/%s/%s/%s' % (architecture, pkgname, current_region) in
-                cfg.get('autopkgtest', 'blacklist').split()):
-            logging.warning('Blacklisted on this region (%s), ignoring and sleeping for 5 minutes' % current_region)
-            msg.channel.basic_reject(msg.delivery_tag, requeue=True)
-            time.sleep(300)
-            return
 
     # build autopkgtest command line
     work_dir = tempfile.mkdtemp(prefix='autopkgtest-work.')
     out_dir = os.path.join(work_dir, 'out')
 
-    if '%s/%s/%s' % (release, architecture, pkgname) in cfg.get('autopkgtest', 'blacklist').split():
-        logging.warning('Blacklisted, ignoring')
-        blacklisted = True
+    if request_matches_per_package(pkgname, architecture, release, never_run):
+        logging.warning('Marked to never run, ignoring')
+        dont_run = True
 
         # these will be written later on
         code = 99
@@ -394,21 +495,28 @@ def request(msg):
 
         # now let's fake up a log file
         with open(os.path.join(out_dir, 'log'), 'w') as log:
-            log.write('This package is blacklisted. To get the entry removed, contact a member of the release team.')
+            log.write('This package is marked to never run. To get the entry removed, contact a member of the release team.')
 
+        triggers = None
         # a json file containing the env
         if 'triggers' in params:
+            triggers = ' '.join(params['triggers'])
             with open(os.path.join(out_dir, 'testinfo.json'), 'w') as testinfo:
                 d = {'custom_environment':
-                     ['ADT_TEST_TRIGGERS=%s' % ' '.join(params['triggers'])]}
+                     ['ADT_TEST_TRIGGERS=%s' % triggers]}
                 json.dump(d, testinfo, indent=True)
 
         # and the testpackage version (pkgname blacklisted)
+        # XXX: replace "blacklisted" here, but needs changes in
+        # proposed-migration and hints
         with open(os.path.join(out_dir, 'testpkg-version'), 'w') as testpkg_version:
             testpkg_version.write('%s blacklisted' % pkgname)
 
     container = 'autopkgtest-' + release
-    big_pkg = any((fnmatch.fnmatchcase(pkgname, pkg) for pkg in cfg.get('autopkgtest', 'big_packages').strip().split()))
+    big_pkg = request_matches_per_package(pkgname,
+                                          architecture,
+                                          release,
+                                          big_packages)
 
     autopkgtest_checkout = cfg.get('autopkgtest', 'checkout_dir').strip()
     if autopkgtest_checkout:
@@ -426,11 +534,11 @@ def request(msg):
 
     c = cfg.get('autopkgtest', 'setup_command').strip()
     if c:
-        c = subst(c, autopkgtest_checkout, big_pkg, release, architecture, pkgname)
+        c = subst(c, big_pkg, release, architecture, host_arch(release, architecture), pkgname)
         argv += ['--setup-commands', c]
     c = cfg.get('autopkgtest', 'setup_command2').strip()
     if c:
-        c = subst(c, autopkgtest_checkout, big_pkg, release, architecture, pkgname)
+        c = subst(c, big_pkg, release, architecture, host_arch(release, architecture), pkgname)
         argv += ['--setup-commands', c]
 
     if 'ppas' in params and params['ppas']:
@@ -506,7 +614,7 @@ def request(msg):
     argv += testargs
     if args.debug:
         argv.append('--debug')
-    if pkgname in cfg.get('autopkgtest', 'long_tests').strip().split():
+    if request_matches_per_package(pkgname, architecture, release, long_tests):
         argv.append('--timeout-copy=40000')
         argv.append('--timeout-test=40000')
         argv.append('--timeout-build=40000')
@@ -521,8 +629,10 @@ def request(msg):
     for e in params.get('env', []):
         argv.append('--env=%s' % e)
 
+    triggers = None
     if 'triggers' in params:
-        argv.append('--env=ADT_TEST_TRIGGERS=%s' % ' '.join(params['triggers']))
+        triggers = ' '.join(params['triggers'])
+        argv.append('--env=ADT_TEST_TRIGGERS=%s' % triggers)
 
         # want to run against a non-default kernel?
         for t in params['triggers']:
@@ -576,17 +686,14 @@ def request(msg):
     if 'testname' in params:
         argv.append('--testname=%s' % params['testname'])
 
-    if i386_cross_series(release) and architecture == 'i386':
-        img_arch = 'amd64'
-    else:
-        img_arch = architecture
-
     argv.append('--')
-    argv += subst(cfg.get('virt', 'args'), autopkgtest_checkout, big_pkg,
-                  release, img_arch, pkgname).split()
+    argv += subst(cfg.get('virt', 'args'), big_pkg,
+                  release, architecture,
+                  host_arch(release, architecture),
+                  pkgname).split()
 
     # run autopkgtest; retry up to three times on tmpfail issues
-    if not blacklisted:
+    if not dont_run:
         global running_test
         running_test = True
         start_time = time.time()
@@ -606,6 +713,7 @@ def request(msg):
                     logging.warning('%sLog follows:', "Retrying in 5 minutes. " if retry < 2 else "")
                     logging.error(contents)
                     if retry < 2:
+                        submit_metric(architecture, code, pkgname, current_region, True, release)
                         cleanup_and_sleep(out_dir)
                 else:
                     break
@@ -637,6 +745,7 @@ def request(msg):
                 logging.warning('Testbed failure%s. Log follows:', ", retrying in 5 minutes" if retry < 2 else "")
                 logging.error(contents)
                 if retry < 2:
+                    submit_metric(architecture, code, pkgname, current_region, True, release)
                     cleanup_and_sleep(out_dir)
             else:  # code == 0, no retry needed
                 break
@@ -645,6 +754,7 @@ def request(msg):
                 logging.warning('Three fails in a row - considering this a failure rather than tmpfail')
                 code = 4
             else:
+                submit_metric(architecture, code, pkgname, current_region, False, release)
                 logging.error('Three tmpfails in a row, aborting worker. Log follows:')
                 logging.error(log_contents(out_dir))
                 sys.exit(99)
@@ -652,6 +762,7 @@ def request(msg):
         duration = int(time.time() - retry_start_time)
 
     logging.info('autopkgtest exited with code %i', code)
+    submit_metric(architecture, code, pkgname, current_region, False, release)
     if code == 1:
         logging.error('autopkgtest exited with unexpected error code 1')
         sys.exit(1)
@@ -664,7 +775,7 @@ def request(msg):
         with open(os.path.join(out_dir, 'requester'), 'w') as f:
             f.write('%s\n' % params['requester'])
 
-    process_output_dir(out_dir, pkgname, code)
+    (testpkg_version, duration, requester) = process_output_dir(out_dir, pkgname, code)
 
     # If two tests for the same package with different triggers finish at the
     # same second, we get collisions with just the timestamp; disambiguate with
@@ -740,6 +851,23 @@ def request(msg):
     swift_con.close()
     shutil.rmtree(work_dir)
 
+    global amqp_con
+    complete_amqp = amqp_con.channel()
+    complete_amqp.access_request('/complete', active=True, read=False, write=True)
+    complete_amqp.exchange_declare(complete_exchange_name, 'fanout', durable=True, auto_delete=False)
+    complete_msg = json.dumps ({'architecture': architecture,
+                                'container': container,
+                                'duration': duration,
+                                'exitcode': code,
+                                'package': pkgname,
+                                'testpkg_version': testpkg_version,
+                                'release': release,
+                                'requester': requester,
+                                'swift_dir': swift_dir,
+                                'triggers': triggers})
+    complete_amqp.basic_publish(amqp.Message(complete_msg, delivery_mode=2),
+                                complete_exchange_name, '')
+
     logging.info('Acknowledging request %s' % body)
     msg.channel.basic_ack(msg.delivery_tag)
     running_test = False
@@ -754,13 +882,11 @@ def amqp_connect(cfg, callback):
     Return queue object.
     '''
     global amqp_con
-    logging.info('Connecting to AMQP server %s', cfg.get('amqp', 'host'))
-    if cfg.has_option('amqp', 'user') and cfg.get('amqp', 'user'):
-        amqp_con = amqp.Connection(cfg.get('amqp', 'host'),
-                                   userid=cfg.get('amqp', 'user'),
-                                   password=cfg.get('amqp', 'password'))
-    else:
-        amqp_con = amqp.Connection(cfg.get('amqp', 'host'))
+    logging.info('Connecting to AMQP server %s', os.environ['RABBIT_HOST'])
+    amqp_con = amqp.Connection(os.environ['RABBIT_HOST'],
+                               userid=os.environ['RABBIT_USER'],
+                               password=os.environ['RABBIT_PASSWORD'],
+                               confirm_publish=True)
     queue = amqp_con.channel()
     # avoids greedy grabbing of the entire queue while being too busy
     queue.basic_qos(0, 1, True)
@@ -813,15 +939,9 @@ def main():
     # load configuration
     cfg = configparser.ConfigParser(
         {'setup_command': '', 'setup_command2': '',
-         'long_tests': '', 'big_packages': '',
-         'checkout_dir': '', 'blacklist': '',
+         'checkout_dir': '',
          'package_size_default': '', 'package_size_big': '',
-         'extra_args': '',
-         'region_name': os.environ.get('OS_REGION_NAME', '<undefined swift region>'),
-         'username': os.environ.get('OS_USERNAME', '<undefined swift user>'),
-         'auth_url': os.environ.get('OS_AUTH_URL', '<undefined swift auth URL>'),
-         'password': os.environ.get('OS_PASSWORD', '<undefined swift password>'),
-         'tenant': os.environ.get('OS_TENANT_NAME', '<undefined swift tenant name>')},
+         'extra_args': ''},
         allow_no_value=True)
     cfg.read(args.config)
 
@@ -829,19 +949,32 @@ def main():
                         format='%(levelname)s: %(message)s',
                         handlers=[systemd_logging_handler])
 
-    # build swift credentials
-    os_options = {}
-    os_options['region_name'] = cfg.get('swift', 'region_name')
-    if '/v2.0' in cfg.get('swift', 'auth_url'):
-        auth_version = '2.0'
-    else:
-        auth_version = '1'
-    swift_creds = {'authurl': cfg.get('swift', 'auth_url'),
-                   'user': cfg.get('swift', 'username'),
-                   'key': cfg.get('swift', 'password'),
-                   'tenant_name': cfg.get('swift', 'tenant'),
-                   'os_options': os_options,
-                   'auth_version': auth_version}
+    auth_version = os.environ['SWIFT_AUTH_VERSION']
+
+    if auth_version == '2':
+        swift_creds = {
+            'authurl': os.environ['SWIFT_AUTH_URL'],
+            'user': os.environ['SWIFT_USERNAME'],
+            'key': os.environ['SWIFT_PASSWORD'],
+            'tenant_name': os.environ['SWIFT_TENANT'],
+            'os_options': {
+                'region_name': os.environ['SWIFT_REGION']
+            },
+            'auth_version': os.environ['SWIFT_AUTH_VERSION']
+        }
+    else:  # 3
+        swift_creds = {
+            'authurl': os.environ['SWIFT_AUTH_URL'],
+            'user': os.environ['SWIFT_USERNAME'],
+            'key': os.environ['SWIFT_PASSWORD'],
+            'os_options': {
+                'region_name': os.environ['SWIFT_REGION'],
+                'project_domain_name': os.environ['SWIFT_PROJECT_DOMAIN_NAME'],
+                'project_name': os.environ['SWIFT_PROJECT_NAME'],
+                'user_domain_name': os.environ['SWIFT_USER_DOMAIN_NAME']
+                },
+            'auth_version': auth_version
+        }
 
     # ensure that we can connect to swift
     swiftclient.Connection(**swift_creds).close()
diff --git a/worker/worker.conf b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker.conf
similarity index 90%
rename from worker/worker.conf
rename to charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker.conf
index 3ae8b03..bee4851 100644
--- a/worker/worker.conf
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker.conf
@@ -17,9 +17,7 @@ releases = trusty xenial
 architectures = i386 amd64
 setup_command =
 # setup_command2 = $CHECKOUTDIR/setup-commands/setup-testbed
-big_packages = binutils chromium-browser glibc libreoffice linux linux-* tdb firefox akonadi julia libtext-bidi-perl rocs camitk kineticstools r-cran-igraph botch mathicgb openjdk-8 openjdk-lts
-long_tests = gutenprint gmp-ecm open-iscsi
-blacklist =
+per_package_config_dir = ~/autopkgtest-package-configs/
 
 [virt]
 # normal packages get $PACKAGESIZE set to this value
diff --git a/charms/focal/autopkgtest-cloud-worker/config.yaml b/charms/focal/autopkgtest-cloud-worker/config.yaml
new file mode 100644
index 0000000..dc4d4fa
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/config.yaml
@@ -0,0 +1,104 @@
+options:
+  nova-rcs:
+    type: string
+    description: "base64 encoded tarball of nova OpenStack credential .rc files"
+    default: ~
+  n-workers:
+    type: string
+    description: "yaml dict region -> arch -> n_workers corresponding to the number of worker instances to run"
+    default: ~
+  lxd-remotes:
+    type: string
+    description: "yaml dict arch -> ip -> n_workers corresponding to the number of LXD workers to run"
+    default: ~
+  mail-notify:
+    type: string
+    description: "email addresses (space separated) to notify on worker failure"
+    default: ~
+  swift-auth-url:
+    type: string
+    description: "swift auth URL"
+    default: ~
+  swift-username:
+    type: string
+    description: "username for swift"
+    default: ~
+  swift-password:
+    type: string
+    description: "password for swift"
+    default: ~
+  swift-region:
+    type: string
+    description: "the swift region to use"
+    default: ~
+  swift-project-domain-name:
+    type: string
+    description: "the swift project domain (v3 only)"
+    default: ~
+  swift-project-name:
+    type: string
+    description: "the swift project name (v3 only)"
+    default: ~
+  swift-user-domain-name:
+    type: string
+    description: "the swift user domain name (v3 only)"
+    default: ~
+  swift-auth-version:
+    type: int
+    description: "the keystone API version to use for swift (2 or 3)"
+    default: 2
+  swift-tenant:
+    type: string
+    description: "the swift tenant name (v2 only)"
+    default: ~
+  releases:
+    type: string
+    description: "the releases to test for"
+    default: ~
+  mirror:
+    type: string
+    description: "the archive mirror to use for tests"
+    default: "http://archive.ubuntu.com/ubuntu/"
+  net-name:
+    type: string
+    description: "the neutron network name to use for the SSH/nova runner (must have port 22 inbound open)"
+  worker-setup-command:
+    type: string
+    description: "autopkgtest's --setup-command argument"
+    default: ~
+  worker-setup-command2:
+    type: string
+    description: "second autopkgtest --setup-command argument"
+    default: ~
+  worker-default-flavor:
+    type: string
+    description: "the default nova flavor to run tests under"
+    default: "m1.small"
+  worker-big-flavor:
+    type: string
+    description: "the flavor to run big-packages tests under"
+    default: "m1.large"
+  worker-args:
+    type: string
+    description: "autopkgtest virt args"
+    default: "null"
+  influxdb-hostname:
+    default: ~
+    description: The hostname of the remote influxdb to submit metrics to
+    type: string
+  influxdb-port:
+    default: 8086
+    description: The port of the remote influxdb to submit metrics to
+    type: int
+  influxdb-username:
+    default: ~
+    description: The influxdb username
+  influxdb-password:
+    default: ~
+    description: The influxdb password
+  influxdb-database:
+    default: ~
+    description: The influxdb database to use
+  influxdb-context:
+    default: "production"
+    description: Submit all metrics with this as the "context" tag, to differentiate staging vs. production submissions
diff --git a/charms/focal/autopkgtest-cloud-worker/layer.yaml b/charms/focal/autopkgtest-cloud-worker/layer.yaml
new file mode 100644
index 0000000..e0a99e0
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/layer.yaml
@@ -0,0 +1,33 @@
+includes:
+  - 'layer:apt'
+  - 'layer:basic'
+  - 'layer:status'
+  - 'interface:rabbitmq'
+  - 'interface:juju-info'
+options:
+  basic:
+    packages:
+      - gir1.2-glib-2.0
+      - python3-gi
+      - python3-pygit2
+    include_system_packages: true
+  apt:
+    packages:
+      - bsd-mailx
+      - dctrl-tools
+      - distro-info
+      - git
+      - libdpkg-perl
+      - lxd-client
+      - make
+      - python3-amqplib
+      - python3-debian
+      - python3-distro-info
+      - python3-glanceclient
+      - python3-influxdb
+      - python3-keystoneauth1
+      - python3-neutronclient
+      - python3-novaclient
+      - python3-openstackclient
+      - python3-swiftclient
+      - ssmtp
diff --git a/charms/focal/autopkgtest-cloud-worker/lib/systemd.py b/charms/focal/autopkgtest-cloud-worker/lib/systemd.py
new file mode 100644
index 0000000..8c9856e
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/lib/systemd.py
@@ -0,0 +1,402 @@
+import os
+import shutil
+from textwrap import dedent
+
+from collections import defaultdict
+from gi.repository import GLib, Gio
+from lib.utils import UbuntuRelease
+
+
+SYSTEM_BUS = Gio.bus_get_sync(Gio.BusType.SYSTEM)
+
+
+def get_unit_names(region, arch, ns):
+    if arch == "amd64":
+        unit_names = ["autopkgtest@{}-{}.service".format(region, n) for n in ns]
+    else:
+        unit_names = [
+            "autopkgtest@{}-{}-{}.service".format(region, arch, n) for n in ns
+        ]
+
+    return unit_names
+
+
+def reload():
+    SYSTEM_BUS.call_sync(
+        "org.freedesktop.systemd1",
+        "/org/freedesktop/systemd1",
+        "org.freedesktop.systemd1.Manager",
+        "Reload",
+        None,  # parameters
+        None,  # reply type
+        Gio.DBusCallFlags.NONE,
+        -1,  # timeout
+        None,
+    )  # cancellable
+
+
+def enabledisable(unit_names, enabledisable, enabledisablearg, startstop):
+    print(
+        "calling {enabledisable} then {startstop} on {unit_names}".format(
+            **locals()
+        )
+    )
+    SYSTEM_BUS.call_sync(
+        "org.freedesktop.systemd1",
+        "/org/freedesktop/systemd1",
+        "org.freedesktop.systemd1.Manager",
+        enabledisable,
+        enabledisablearg,
+        GLib.VariantType(
+            "*"
+        ),  # reply type, using '*' because we don't actually care
+        Gio.DBusCallFlags.NONE,
+        -1,  # timeout
+        None,
+    )  # cancellable
+
+    for unit in unit_names:
+        (_,) = SYSTEM_BUS.call_sync(
+            "org.freedesktop.systemd1",
+            "/org/freedesktop/systemd1",
+            "org.freedesktop.systemd1.Manager",
+            startstop,
+            GLib.Variant("(ss)", (unit, "replace")),
+            GLib.VariantType("(o)"),  # reply type
+            Gio.DBusCallFlags.NONE,
+            -1,  # timeout
+            None,
+        )  # cancellable
+
+
+def enable(unit_names):
+    enabledisable(
+        unit_names,
+        "EnableUnitFiles",
+        GLib.Variant("(asbb)", (unit_names, False, True)),
+        "StartUnit",
+    )
+    reload()
+
+
+def disable(unit_names):
+    enabledisable(
+        unit_names,
+        "DisableUnitFiles",
+        GLib.Variant("(asb)", (unit_names, False)),
+        "StopUnit",
+    )
+    for unit in unit_names:
+        SYSTEM_BUS.call_sync(
+            "org.freedesktop.systemd1",
+            "/org/freedesktop/systemd1",
+            "org.freedesktop.systemd1.Manager",
+            "ResetFailedUnit",
+            GLib.Variant("(s)", (unit,)),
+            None,  # reply type
+            Gio.DBusCallFlags.NONE,
+            -1.0,  # timeout
+            None,
+        )  # cancellable
+        try:
+            dropindir = os.path.join(
+                os.path.sep, "etc", "systemd", "system", "{}.d".format(unit)
+            )
+            shutil.rmtree(dropindir)
+        except FileNotFoundError:
+            pass
+
+        reload()
+
+
+def get_units():
+    (units,) = SYSTEM_BUS.call_sync(
+        "org.freedesktop.systemd1",
+        "/org/freedesktop/systemd1",
+        "org.freedesktop.systemd1.Manager",
+        "ListUnits",
+        None,  # parameters
+        GLib.VariantType("(a(ssssssouso))"),  # reply type
+        Gio.DBusCallFlags.NONE,
+        -1,  # timeout
+        None,
+    )  # cancellable
+
+    build_adt_image_object_paths = defaultdict(lambda: defaultdict(dict))
+    unit_object_paths = defaultdict(lambda: defaultdict(dict))
+    lxd_object_paths = defaultdict(lambda: defaultdict(dict))
+
+    for unit in units:
+        (name, _, _, active, _, _, object_path, _, _, _) = unit
+        if name.startswith("build-adt-image@") and name.endswith(".timer"):
+            name_release_region_arch = name[16:][:-6]
+            (release, region, arch) = name_release_region_arch.split("-", -1)
+            build_adt_image_object_paths[region][arch][release] = object_path
+            continue
+
+        if not name.startswith("autopkgtest@"):
+            continue
+
+        name_cloud = name[12:][:-8]
+
+        if name_cloud.startswith("lxd-"):
+            # lxd-armhf-1.2.3.4-1
+            name_cloud = name_cloud[4:]
+            try:
+                (arch, ip, n) = name_cloud.split("-", -1)
+                lxd_object_paths[arch][ip][n] = object_path
+            except ValueError:
+                # XXX: for some reason we get autopkgtest@lxd-armhf-a.b.c.d
+                # (without the number), not sure why
+                pass
+            continue
+
+        try:
+            (region, arch, n) = name_cloud.split("-", -1)
+        except ValueError:
+            # [e-mail]
+            (region, n) = name_cloud.split("-", -1)
+            arch = "amd64"
+        unit_object_paths[region][arch][n] = object_path
+
+    return (unit_object_paths, lxd_object_paths, build_adt_image_object_paths)
+
+
+def update_cloud_dropins(region, arch, n, releases):
+    # re-create, to support fiddling on upgrade
+    def get_arches(release):
+        if arch == "amd64" and UbuntuRelease(release) < UbuntuRelease("focal"):
+            return ["amd64", "i386"]
+        else:
+            return [arch]
+
+    ensure_adt_units = " ".join(
+        [
+            "ensure-adt-image@{}-{}-{}.service".format(release, region, arch)
+            for release in releases
+            for arch in get_arches(release)
+        ]
+    )
+
+    for unit in get_unit_names(region, arch, range(1, n + 1)):
+        dropindir = os.path.join(
+            os.path.sep, "etc", "systemd", "system", "{}.d".format(unit)
+        )
+        try:
+            os.makedirs(dropindir)
+        except FileExistsError:
+            shutil.rmtree(dropindir)
+            os.makedirs(dropindir)
+
+        with open(os.path.join(dropindir, "ensure-adt-image.conf"), "w") as f:
+            f.write(
+                dedent(
+                    """\
+                    [Unit]
+                    Requires={ensureadt}
+                    After={ensureadt}
+                    """
+                ).format(ensureadt=ensure_adt_units)
+            )
+    reload()
+
+
+def update_lxd_dropins(arch, ip, n):
+    # re-create, to support fiddling on upgrade
+    units = [
+        "autopkgtest@lxd-{}-{}-{}.service".format(arch, ip, m)
+        for m in range(1, n + 1)
+    ]
+
+    for unit in units:
+        dropindir = os.path.join(
+            os.path.sep, "etc", "systemd", "system", "{}.d".format(unit)
+        )
+        try:
+            os.makedirs(dropindir)
+        except FileExistsError:
+            pass
+
+        with open(
+            os.path.join(dropindir, "autopkgtest-lxd-remote.conf"), "w"
+        ) as f:
+            remote_unit = "autopkgtest-lxd-remote@lxd-{}-{}.service".format(
+                arch, ip
+            )
+            f.write(
+                dedent(
+                    """\
+                [Unit]
+                Requires={remote_unit}
+                After={remote_unit}
+                """
+                ).format(remote_unit=remote_unit)
+            )
+    reload()
+
+
+def enable_timer(region, arch, releases):
+    unit_names = [
+        "build-adt-image@{}-{}-{}.timer".format(release, region, arch)
+        for release in releases
+    ]
+    enabledisable(
+        unit_names,
+        "EnableUnitFiles",
+        GLib.Variant("(asbb)", (unit_names, False, True)),
+        "StartUnit",
+    )
+    reload()
+
+
+def disable_timer(region, arch, releases):
+    unit_names = [
+        "build-adt-image@{}-{}-{}.timer".format(release, region, arch)
+        for release in releases
+    ]
+    enabledisable(
+        unit_names,
+        "DisableUnitFiles",
+        GLib.Variant("(asb)", (unit_names, False)),
+        "StopUnit",
+    )
+    reload()
+
+
+def set_up_systemd_units(target_cloud_config, target_lxd_config, releases):
+    """There are two relevant factors: the units that are currently enabled, and the
+    units that the configuration file is asking for. Work out the
+    difference between the two, and enable / disable units as required.
+    This is split per region-arch. amd64 is a special case. The unit names
+    do not have the arch in them."""
+
+    (
+        cloud_unit_object_paths,
+        lxd_unit_object_paths,
+        build_adt_image_object_paths,
+    ) = get_units()
+    all_cloud_regions = set(
+        list(cloud_unit_object_paths.keys()) + list(target_cloud_config.keys())
+    )
+
+    focal = UbuntuRelease("focal")
+    i386_releases = [
+        release for release in releases if UbuntuRelease(release) < focal
+    ]
+
+    for region in sorted(all_cloud_regions):
+        all_arches_for_this_region = set(
+            list(cloud_unit_object_paths[region].keys())
+            + list(target_cloud_config.get(region, {}).keys())
+        )
+        for arch in sorted(all_arches_for_this_region):
+            n_units = len(cloud_unit_object_paths[region][arch])
+            target_n_units = target_cloud_config.get(region, {}).get(arch, 0)
+            print(
+                "Got {target_n_units} units for {region}/{arch}".format(
+                    **locals()
+                )
+            )
+            if target_n_units > 0:
+                update_cloud_dropins(region, arch, target_n_units, releases)
+                enable_timer(region, arch, releases)
+                if arch == "amd64":
+                    enable_timer(region, "i386", i386_releases)
+            if releases == []:
+                target_n_units = 0
+            if target_n_units == 0:
+                disable_timer(region, arch, releases)
+                if arch == "amd64":
+                    disable_timer(region, "i386", i386_releases)
+            if n_units < target_n_units:
+                # need to enable some units
+                delta = target_n_units - n_units
+                unit_names = get_unit_names(
+                    region, arch, range(n_units + 1, n_units + delta + 1)
+                )
+                enable(unit_names)
+            elif n_units > target_n_units:
+                # need to disable some units
+                delta = n_units - target_n_units
+                unit_names = get_unit_names(
+                    region, arch, range(target_n_units + 1, n_units + 1)
+                )
+                disable(unit_names)
+
+            currently_enabled_releases = set(
+                build_adt_image_object_paths[region][arch].keys()
+            )
+            releases_to_disable = currently_enabled_releases - set(releases)
+
+            if releases_to_disable:
+                print(
+                    "Disabling build-adt-image timers for {region}/{arch}/{releases_to_disable}".format(
+                        **locals()
+                    )
+                )
+                disable_timer(region, arch, releases_to_disable)
+
+    # now do lxd. the target config is a dict arch -> IP -> nworkers
+    all_lxd_arches = set(
+        list(lxd_unit_object_paths.keys()) + list(target_lxd_config.keys())
+    )
+    for arch in sorted(all_lxd_arches):
+        all_ips_for_this_arch = set(
+            list(lxd_unit_object_paths[arch].keys())
+            + list(target_lxd_config.get(arch, {}).keys())
+        )
+        for ip in sorted(all_ips_for_this_arch):
+            n_units = len(lxd_unit_object_paths[arch][ip])
+            target_n_units = target_lxd_config.get(arch, {}).get(ip, 0)
+            if target_n_units > 0:
+                update_lxd_dropins(arch, ip, target_n_units)
+            if n_units == target_n_units:
+                continue
+            elif n_units < target_n_units:
+                # need to enable some units
+                delta = target_n_units - n_units
+                unit_names = [
+                    "autopkgtest@lxd-{}-{}-{}.service".format(arch, ip, n)
+                    for n in range(n_units + 1, n_units + delta + 1)
+                ]
+                enable(unit_names)
+            else:
+                # need to disable some units
+                delta = n_units - target_n_units
+                unit_names = [
+                    "autopkgtest@lxd-{}-{}-{}.service".format(arch, ip, n)
+                    for n in range(target_n_units + 1, n_units + 1)
+                ]
+                disable(unit_names)
+
+
+def reload_unit(object_path):
+    SYSTEM_BUS.call_sync(
+        "org.freedesktop.systemd1",
+        object_path,
+        "org.freedesktop.systemd1.Unit",
+        "Reload",
+        GLib.Variant("(s)", ("replace",)),
+        GLib.VariantType("(o)"),  # reply type
+        Gio.DBusCallFlags.NONE,
+        -1.0,  # timeout
+        None,
+    )  # cancellable
+
+
+def reload_units():
+    (cloud_unit_object_paths, lxd_unit_object_paths, _) = get_units()
+
+    for region in cloud_unit_object_paths:
+        for arch in cloud_unit_object_paths[region]:
+            for n in cloud_unit_object_paths[region][arch]:
+                object_path = cloud_unit_object_paths[region][arch][n]
+                print("Reloading {}".format(object_path))
+                reload_unit(object_path)
+
+    for arch in lxd_unit_object_paths:
+        for ip in lxd_unit_object_paths[arch]:
+            for n in lxd_unit_object_paths[arch][ip]:
+                object_path = lxd_unit_object_paths[arch][ip][n]
+                print("Reloading {}".format(object_path))
+                reload_unit(object_path)
diff --git a/charms/focal/autopkgtest-cloud-worker/lib/utils.py b/charms/focal/autopkgtest-cloud-worker/lib/utils.py
new file mode 100644
index 0000000..22d52ee
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/lib/utils.py
@@ -0,0 +1,79 @@
+from charmhelpers.core.hookenv import log
+from distro_info import UbuntuDistroInfo
+from functools import total_ordering
+
+import os
+import pwd
+import subprocess
+
+
+class UnixUser(object):
+    def __init__(self, username):
+        self.username = username
+        pwnam = pwd.getpwnam(username)
+        self.uid = pwnam.pw_uid
+        self.gid = pwnam.pw_gid
+
+    def __enter__(self):
+        log(
+            "Raising UID to {uid} and GID to {uid}".format(
+                uid=self.uid, gid=self.gid
+            ),
+            "INFO",
+        )
+        os.setresgid(self.gid, self.gid, os.getgid())
+        os.setresuid(self.uid, self.uid, os.getuid())
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        _, _, suid = os.getresuid()
+        _, _, sgid = os.getresgid()
+        log(
+            "Restoring UID to {suid} and GID to {sgid}".format(
+                suid=suid, sgid=sgid
+            ),
+            "INFO",
+        )
+        os.setresuid(suid, suid, suid)
+        os.setresgid(sgid, sgid, sgid)
+
+
+def install_autodep8(location):
+    os.chdir(location)
+    with UnixUser("ubuntu"):
+        subprocess.check_call(["make"])
+    subprocess.check_call(["make", "install"])
+
+
+def pull(repository):
+    """  This will do a sort of git fetch origin && git reset --hard origin/master """
+    origin = [
+        remote for remote in repository.remotes if remote.name == "origin"
+    ][0]
+    origin.fetch()
+    remote_master_id = repository.lookup_reference(
+        "refs/remotes/origin/master"
+    ).target
+    repository.checkout_tree(repository.get(remote_master_id))
+    master_ref = repository.lookup_reference("refs/heads/master")
+    master_ref.set_target(remote_master_id)
+    repository.head.set_target(remote_master_id)
+
+
+@total_ordering
+class UbuntuRelease(object):
+    all_releases = UbuntuDistroInfo().all
+
+    def __init__(self, release):
+        if release not in UbuntuRelease.all_releases:
+            raise ValueError("Unknown release '{}'".format(release))
+        self.release = release
+
+    def __eq__(self, other):
+        return UbuntuRelease.all_releases.index(
+            self.release
+        ) == UbuntuRelease.all_releases.index(other)
+
+    def __le__(self, other):
+        return UbuntuRelease.all_releases.index(
+            self.release
+        ) <= UbuntuRelease.all_releases.index(other)
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/metadata.yaml b/charms/focal/autopkgtest-cloud-worker/metadata.yaml
similarity index 50%
rename from deployment/charms/xenial/autopkgtest-cloud-worker/metadata.yaml
rename to charms/focal/autopkgtest-cloud-worker/metadata.yaml
index b96a32b..ba9667d 100644
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/metadata.yaml
+++ b/charms/focal/autopkgtest-cloud-worker/metadata.yaml
@@ -1,12 +1,24 @@
 name: autopkgtest-cloud-worker
 summary: autopkgtest cloud worker
-maintainer: Martin Pitt <[e-mail]>
+maintainer: Iain Lane <[e-mail]>
+series:
+  - focal
 description: |
   An autopkgtest worker receives test requests from AMQP, runs autopkgtest with
-  the ssh/nova runner, and puts the test results into Swift.
-categories:
- - ops
+  the ssh/nova or lxd runner, and puts the test results into Swift.
+tags:
+  - ops
 requires:
   amqp:
     interface: rabbitmq
+  haproxy-lxd-armhf:
+    interface: juju-info
 subordinate: false
+storage:
+  tmp:
+    type: filesystem
+    description: working files for running tests
+    shared: false
+    read-only: false
+    minimum-size: 200G
+    location: /tmp
diff --git a/charms/focal/autopkgtest-cloud-worker/reactive/autopkgtest_cloud_worker.py b/charms/focal/autopkgtest-cloud-worker/reactive/autopkgtest_cloud_worker.py
new file mode 100644
index 0000000..d459b31
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/reactive/autopkgtest_cloud_worker.py
@@ -0,0 +1,597 @@
+from charms.layer import status
+from charms.reactive import (
+    when,
+    when_all,
+    when_any,
+    when_not,
+    when_not_all,
+    clear_flag,
+    set_flag,
+    hook,
+    not_unless,
+)
+from charms.reactive.relations import endpoint_from_flag
+from charmhelpers.core.hookenv import (
+    charm_dir,
+    config,
+    log,
+    storage_get,
+    storage_list,
+)
+from utils import install_autodep8, UnixUser
+
+from textwrap import dedent
+
+import glob
+import os
+import pygit2
+import socket
+import subprocess
+import yaml
+
+
+AUTOPKGTEST_LOCATION = os.path.expanduser("~ubuntu/autopkgtest")
+AUTOPKGTEST_CLOUD_LOCATION = os.path.expanduser("~ubuntu/autopkgtest-cloud")
+AUTODEP8_LOCATION = os.path.expanduser("~ubuntu/autodep8")
+AUTOPKGTEST_PER_PACKAGE_LOCATION = os.path.expanduser(
+    "~ubuntu/autopkgtest-package-configs/"
+)
+
+AUTOPKGTEST_CLONE_LOCATION = (
+    "https://git.launchpad.net/~ubuntu-release/autopkgtest/+git/development"
+)
+AUTODEP8_CLONE_LOCATION = (
+    "https://git.launchpad.net/~ubuntu-release/+git/autodep8"
+)
+
+AUTOPKGTEST_PER_PACKAGE_CLONE_LOCATION = "https://git.launchpad.net/~ubuntu-release/autopkgtest-cloud/+git/autopkgtest-package-configs"
+
+RABBITMQ_CRED_PATH = os.path.expanduser("~ubuntu/rabbitmq.cred")
+
+
+@when_not("autopkgtest.autopkgtest_cloud_symlinked")
+def symlink_autopkgtest_cloud():
+    with UnixUser("ubuntu"):
+        try:
+            autopkgtest_cloud = os.path.join(charm_dir(), "autopkgtest-cloud")
+            os.symlink(autopkgtest_cloud, AUTOPKGTEST_CLOUD_LOCATION)
+        except FileExistsError:
+            pass
+    set_flag("autopkgtest.autopkgtest_cloud_symlinked")
+
+
+@when("apt.installed.git")
+@when_not("autopkgtest.autodep8_cloned")
+def clone_autodep8():
+    status.maintenance("Cloning autodep8")
+    log("Cloning autodep8", "INFO")
+    with UnixUser("ubuntu"):
+        try:
+            pygit2.clone_repository(AUTODEP8_CLONE_LOCATION, AUTODEP8_LOCATION)
+            log("autodep8 cloned", "INFO")
+            status.maintenance("autodep8 cloned")
+            set_flag("autopkgtest.autodep8_cloned")
+        except ValueError:
+            log("autodep8 already cloned", "INFO")
+
+
+@when("apt.installed.git")
+@when_not("autopkgtest.autopkgtest_package_configs_cloned")
+def clone_autopkgtest_package_configs():
+    status.maintenance("Cloning autopkgtest-package-configs")
+    log("Cloning autopkgtest-package-configs", "INFO")
+    with UnixUser("ubuntu"):
+        try:
+            pygit2.clone_repository(
+                AUTOPKGTEST_PER_PACKAGE_CLONE_LOCATION,
+                AUTOPKGTEST_PER_PACKAGE_LOCATION,
+            )
+            log("autopkgtest-package-configs cloned", "INFO")
+            status.maintenance("autopkgtest-package-configs cloned")
+            set_flag("autopkgtest.autopkgtest_package_configs_cloned")
+        except ValueError:
+            log("autopkgtest-package-configs already cloned", "INFO")
+
+
+@when("autopkgtest.autodep8_cloned")
+@when_not("autopkgtest.autodep8_installed")
+def initially_install_autodep8():
+    install_autodep8(AUTODEP8_LOCATION)
+    set_flag("autopkgtest.autodep8_installed")
+
+
+@when("apt.installed.git")
+@when_not("autopkgtest.autopkgtest_cloned")
+def clone_autopkgtest():
+    status.maintenance("Cloning autopkgtest")
+    log("Cloning autopkgtest", "INFO")
+    with UnixUser("ubuntu"):
+        try:
+            pygit2.clone_repository(
+                AUTOPKGTEST_CLONE_LOCATION, AUTOPKGTEST_LOCATION
+            )
+            status.maintenance("autopkgtest cloned")
+            log("autopkgtest cloned", "INFO")
+            set_flag("autopkgtest.autopkgtest_cloned")
+        except ValueError:
+            # exists
+            log("autopkgtest already cloned", "INFO")
+
+
+@when_all(
+    "autopkgtest.autopkgtest_cloud_symlinked",
+    "autopkgtest.rabbitmq-configured",
+    "autopkgtest.influx-creds-written",
+)
+def set_up_systemd_units():
+    for unit in glob.glob(os.path.join(charm_dir(), "units", "*")):
+        base = os.path.basename(unit)
+        dest = os.path.join(os.path.sep, "etc", "systemd", "system", base)
+
+        def link_and_enable():
+            os.symlink(unit, dest)
+            if "@" not in base:
+                subprocess.check_call(["systemctl", "enable", base])
+
+        try:
+            link_and_enable()
+        except FileExistsError:
+            if not os.path.islink(dest):
+                os.unlink(dest)
+                link_and_enable()
+    set_flag("autopkgtest.systemd_units_linked_and_enabled")
+
+
+@when("autopkgtest.systemd_units_linked_and_enabled")
+@when_not("autopkgtest.target_running")
+def start():
+    subprocess.check_call(
+        ["systemctl", "enable", "--now", "autopkgtest.target"]
+    )
+    set_flag("autopkgtest.target_running")
+
+
+@hook("stop")
+@when("autopkgtest.target_running")
+def stop():
+    subprocess.check_call(
+        ["systemctl", "disable", "--now", "autopkgtest.target"]
+    )
+    clear_flag("autopkgtest.target_running")
+
+
+@when_all("autopkgtest.target-restart-needed", "autopkgtest.target_running")
+def restart_target():
+    subprocess.check_call(["systemctl", "restart", "autopkgtest.target"])
+    clear_flag("autopkgtest.target-restart-needed")
+
+
+@when_not("autopkgtest.target-restart-needed", "apt.queued_installs")
+@when("autopkgtest.target_running")
+def is_active():
+    status.active("autopkgtest workers running and processing requests")
+
+
+@when_all(
+    "autopkgtest.systemd_units_linked_and_enabled",
+    "autopkgtest.daemon-reload-needed",
+)
+@hook("upgrade-charm")
+def daemon_reload():
+    subprocess.check_call(["systemctl", "daemon-reload"])
+    clear_flag("autopkgtest.daemon-reload-needed")
+
+
+@hook("install")
+def enable_persistent_journal():
+    try:
+        journal_dir = os.path.join(os.path.sep, "var", "log", "journal")
+        os.makedirs(journal_dir)
+        subprocess.check_call(
+            ["systemd-tmpfiles", "--create", "--prefix", journal_dir]
+        )
+        subprocess.check_call(["systemctl", "restart", "systemd-journald"])
+    except FileExistsError:
+        pass
+
+
+@when("amqp.connected")
+@when_not("amqp.available")
+def setup_rabbitmq(rabbitmq):
+    rabbitmq.request_access("autopkgtest-worker", "/")
+    status.waiting("Waiting on RabbitMQ to configure vhost")
+
+
+@when("amqp.available")
+@when_not("autopkgtest.rabbitmq-configured")
+def set_up_rabbitmq(rabbitmq):
+    username = rabbitmq.username()
+    password = rabbitmq.password()
+    host = rabbitmq.private_address()
+    status.maintenance("Configuring rabbitmq")
+    log("Setting up rabbitmq connection to: {}@{}".format(username, host))
+    with open(RABBITMQ_CRED_PATH, "w") as cred_file:
+        cred_file.write(
+            dedent(
+                """\
+            RABBIT_HOST="{host}"
+            RABBIT_USER="{username}"
+            RABBIT_PASSWORD="{password}"
+            """.format(
+                    **locals()
+                )
+            )
+        )
+    status.maintenance("rabbitmq configured")
+    set_flag("autopkgtest.rabbitmq-configured")
+
+
+@when_not("amqp.available")
+def clear_rabbitmq():
+    try:
+        log("rabbitmq not available, deleting credentials file")
+        os.unlink(RABBITMQ_CRED_PATH)
+        clear_flag("autopkgtest.rabbitmq-configured")
+    except FileNotFoundError:
+        pass
+
+
+@when("config.changed.nova-rcs")
+def update_nova_rcs():
+    from tarfile import TarFile
+    from io import BytesIO
+
+    import base64
+
+    rctar = config().get("nova-rcs")
+
+    if not rctar:
+        # set to the empty value
+        return
+
+    log("Extracting cloud .rc files...", "INFO")
+
+    clear_old_rcs()
+
+    bytes_file = BytesIO(base64.b64decode(rctar))
+    tar = TarFile(fileobj=bytes_file)
+
+    log("...got {}".format(", ".join(tar.getnames())), "INFO")
+
+    tar.extractall(os.path.expanduser("~ubuntu/cloudrcs/"))
+
+
+@when("config.default.nova-rcs")
+def clear_old_rcs():
+    rcfiles = glob.glob(os.path.expanduser("~ubuntu/cloudrcs/*.rc"))
+
+    if not rcfiles:
+        return
+
+    log("Deleting old cloud .rc files...", "INFO")
+
+    for rcfile in rcfiles:
+        log("...deleting {}...".format(rcfile))
+        os.unlink(rcfile)
+
+    log("...done", "INFO")
+
+
+@when_all(
+    "autopkgtest.autopkgtest_cloud_symlinked",
+    "autopkgtest.systemd_units_linked_and_enabled",
+    "config.set.releases",
+)
+@when_any(
+    "config.set.n-workers",
+    "config.set.lxd-remotes",
+)
+@when_not("autopkgtest.systemd-units-initially-enabled")
+def enable_units_initially():
+    enable_disable_units()
+
+
+@when_all(
+    "autopkgtest.autopkgtest_cloud_symlinked",
+    "autopkgtest.systemd_units_linked_and_enabled",
+    "config.set.releases",
+)
+@when_any(
+    "config.changed.n-workers",
+    "config.changed.lxd-remotes",
+    "config.changed.releases",
+)
+def enable_disable_units():
+    from lib.systemd import set_up_systemd_units
+
+    nworkers = config().get("n-workers") or ""
+    lxdremotes = config().get("lxd-remotes") or ""
+    releases = config().get("releases") or ""
+
+    nworkers_yaml = yaml.load(nworkers, Loader=yaml.CSafeLoader)
+    lxdremotes_yaml = yaml.load(lxdremotes, Loader=yaml.CSafeLoader)
+
+    log("Setting up systemd worker units", "INFO")
+    status.maintenance("Enabling and starting worker units")
+    set_up_systemd_units(
+        nworkers_yaml or {}, lxdremotes_yaml or {}, releases.split()
+    )
+
+    set_flag("autopkgtest.reload-needed")
+    set_flag("autopkgtest.daemon-reload-needed")
+    set_flag("autopkgtest.target-restart-needed")
+    set_flag("autopkgtest.systemd-units-initially-enabled")
+
+
+@when("apt.installed.lxd-client")
+@when_any(
+    "config.set.n-workers", "config.set.lxd-remotes", "config.set.releases"
+)
+@when_not("autopkgtest.ubuntu_added_to_lxd_group")
+def add_ubuntu_user_to_lxd_group():
+    subprocess.check_call(["adduser", "ubuntu", "lxd"])
+    set_flag("autopkgtest.ubuntu_added_to_lxd_group")
+
+
+@when("config.changed.swift-auth-version")
+def auth_version_changed():
+    auth_version = config().get("swift-auth-version")
+
+    if auth_version not in [2, 3]:
+        raise ValueError("Only auth versions 2 and 3 are supported")
+
+    if auth_version == 2:
+        set_flag("autopkgtest.v2_auth")
+        clear_flag("autopkgtest.v3_auth")
+    else:
+        clear_flag("autopkgtest.v2_auth")
+        set_flag("autopkgtest.v3_auth")
+
+
+@not_unless("autopkgtest.v3_auth")
+@when_any(
+    "config.changed.swift-auth-url",
+    "config.changed.swift-username",
+    "config.changed.swift-password",
+    "config.changed.swift-region",
+    "config.changed.swift-project-domain-name",
+    "config.changed.swift-project-name",
+    "config.changed.swift-user-domain-name",
+    "config.changed.swift-auth-version",
+)
+@when_all(
+    "config.set.swift-auth-url",
+    "config.set.swift-username",
+    "config.set.swift-password",
+    "config.set.swift-region",
+    "config.set.swift-project-domain-name",
+    "config.set.swift-project-name",
+    "config.set.swift-user-domain-name",
+    "config.set.swift-auth-version",
+)
+def write_v3_config():
+    write_swift_config()
+
+
+@not_unless("autopkgtest.v2_auth")
+@when_any(
+    "config.changed.swift-auth-url",
+    "config.changed.swift-username",
+    "config.changed.swift-password",
+    "config.changed.swift-region",
+    "config.changed.swift-auth-version",
+    "config.changed.swift-tenant",
+)
+@when_all(
+    "config.set.swift-auth-url",
+    "config.set.swift-username",
+    "config.set.swift-password",
+    "config.set.swift-region",
+    "config.set.swift-auth-version",
+    "config.set.swift-tenant",
+)
+def write_v2_config():
+    write_swift_config()
+
+
+def write_swift_config():
+    with open(
+        os.path.expanduser("~ubuntu/swift-password.cred"), "w"
+    ) as swift_password_file:
+        for key in config():
+            if key.startswith("swift") and config()[key] is not None:
+                swift_password_file.write(
+                    '{}="{}"\n'.format(
+                        key.upper().replace("-", "_"),
+                        str(config()[key]).strip(),
+                    )
+                )
+
+
+@when_any(
+    "config.changed.worker-default-flavor",
+    "config.changed.worker-big-flavor",
+    "config.changed.worker-args",
+    "config.changed.worker-setup-command",
+    "config.changed.worker-setup-command2",
+    "config.changed.releases",
+    "config.changed.n-workers",
+    "config.changed.lxd-remotes",
+    "config.changed.mirror",
+)
+@when_any("config.set.nova-rcs", "config.set.lxd-remotes")
+def write_worker_config():
+    import configparser
+
+    nworkers = config().get("n-workers") or ""
+    lxdremotes = config().get("lxd-remotes") or ""
+
+    if not nworkers and not lxdremotes:
+        return
+
+    nworkers_yaml = yaml.load(nworkers, Loader=yaml.CSafeLoader) or {}
+    lxdremotes_yaml = yaml.load(lxdremotes, Loader=yaml.CSafeLoader) or {}
+
+    conf = {
+        "autopkgtest": {
+            "checkout_dir": AUTOPKGTEST_LOCATION,
+            "releases": config().get("releases"),
+            "setup_command": config().get("worker-setup-command"),
+            "setup_command2": config().get("worker-setup-command2"),
+            "per_package_config_dir": AUTOPKGTEST_PER_PACKAGE_LOCATION,
+        },
+        "virt": {
+            "package_size_default": config().get("worker-default-flavor"),
+            "package_size_big": config().get("worker-big-flavor"),
+            "args": config().get("worker-args"),
+        },
+    }
+
+    def subst(s):
+        replacements = {
+            "/CHECKOUTDIR/": AUTOPKGTEST_LOCATION,
+            "/HOSTNAME/": socket.gethostname(),
+            "/AUTOPKGTEST_CLOUD_DIR/": AUTOPKGTEST_CLOUD_LOCATION,
+            "/MIRROR/": config().get("mirror"),
+            "/NET_NAME/": config().get("net-name"),
+        }
+
+        for k in replacements:
+            if replacements[k]:
+                s = s.replace(k, replacements[k])
+
+        return s
+
+    # prune anything which wasn't set (nested dict)
+    conf = {
+        k: {l: subst(w) for l, w in v.items() if w is not None}
+        for k, v in conf.items()
+    }
+
+    def write(conf_file):
+        with open(conf_file, "w") as cf:
+            cp = configparser.ConfigParser()
+            cp.read_dict(conf)
+            cp.write(cf)
+
+    for region in nworkers_yaml:
+        for arch in nworkers_yaml[region]:
+            if arch == "amd64":
+                conf_file = os.path.join(
+                    os.path.expanduser("~ubuntu"),
+                    "worker-{}.conf".format(region),
+                )
+                conf["autopkgtest"]["architectures"] = "amd64 i386"
+                write(conf_file)
+                break
+            else:
+                conf_file = os.path.join(
+                    os.path.expanduser("~ubuntu"),
+                    "worker-{}-{}.conf".format(region, arch),
+                )
+                conf["autopkgtest"]["architectures"] = arch
+                write(conf_file)
+
+    for arch in lxdremotes_yaml:
+        conf_file = os.path.join(
+            os.path.expanduser("~ubuntu"), "worker-lxd-{}.conf".format(arch)
+        )
+        conf["autopkgtest"]["architectures"] = arch
+        write(conf_file)
+
+    set_flag("autopkgtest.daemon-reload-needed")
+    set_flag("autopkgtest.reload-needed")
+
+
+@when("config.changed.net-name")
+def write_net_name():
+    clear_flag("autopkgtest.net-name-written")
+    with open(os.path.expanduser("~ubuntu/net-name.rc"), "w") as f:
+        f.write('NET_NAME="{}"\n'.format(config().get("net-name")))
+    set_flag("autopkgtest.net-name-written")
+    set_flag("autopkgtest.reload-needed")
+
+
+@when("config.changed.mirror")
+def write_mirror():
+    with open(os.path.expanduser("~ubuntu/mirror.rc"), "w") as f:
+        f.write('MIRROR="{}"\n'.format(config().get("mirror")))
+    set_flag("autopkgtest.reload-needed")
+
+
+@when("autopkgtest.reload-needed")
+@when_not("autopkgtest.daemon-reload-needed")
+def reload_systemd_units():
+    from lib.systemd import reload_units
+
+    reload_units()
+    clear_flag("autopkgtest.reload-needed")
+
+
+@hook("tmp-storage-attached")
+def fix_tmp_permissions():
+    storageids = storage_list("tmp")
+    if not storageids:
+        status.blocked("Cannot locate attached storage")
+        return
+    storageid = storageids[0]
+
+    mount = storage_get("location", storageid)
+
+    os.chmod(mount, 0o777)
+
+
+@when_any(
+    "config.changed.influxdb-hostname",
+    "config.changed.influxdb-port",
+    "config.changed.influxdb-username",
+    "config.changed.influxdb-password",
+    "config.changed.influxdb-database",
+    "config.changed.influxdb-context",
+)
+@when_all(
+    "config.set.influxdb-hostname",
+    "config.set.influxdb-port",
+    "config.set.influxdb-username",
+    "config.set.influxdb-password",
+    "config.set.influxdb-database",
+    "config.set.influxdb-context",
+)
+def write_influx_creds():
+    influxdb_hostname = config().get("influxdb-hostname")
+    influxdb_port = config().get("influxdb-port")
+    influxdb_username = config().get("influxdb-username")
+    influxdb_password = config().get("influxdb-password")
+    influxdb_database = config().get("influxdb-database")
+    influxdb_context = config().get("influxdb-context")
+
+    with open(os.path.expanduser("~ubuntu/influx.cred"), "w") as cf:
+        cf.write(
+            dedent(
+                f"""\
+        INFLUXDB_HOSTNAME={influxdb_hostname}
+        INFLUXDB_PORT={influxdb_port}
+        INFLUXDB_USERNAME={influxdb_username}
+        INFLUXDB_PASSWORD={influxdb_password}
+        INFLUXDB_DATABASE={influxdb_database}
+        INFLUXDB_CONTEXT={influxdb_context}
+        """
+            )
+        )
+    set_flag("autopkgtest.influx-creds-written")
+
+
+@when_not_all(
+    "config.set.influxdb-hostname",
+    "config.set.influxdb-port",
+    "config.set.influxdb-username",
+    "config.set.influxdb-password",
+    "config.set.influxdb-database",
+    "config.set.influxdb-context",
+)
+def unset_influx_creds():
+    try:
+        os.unlink(os.path.expanduser("~ubuntu/influx.cred"))
+    except FileNotFoundError:
+        pass
+    clear_flag("autopkgtest.influx-creds-written")
diff --git a/charms/focal/autopkgtest-cloud-worker/tests/00-setup b/charms/focal/autopkgtest-cloud-worker/tests/00-setup
new file mode 100755
index 0000000..f0616a5
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/tests/00-setup
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+sudo add-apt-repository ppa:juju/stable -y
+sudo apt-get update
+sudo apt-get install amulet python-requests -y
diff --git a/charms/focal/autopkgtest-cloud-worker/tests/10-deploy b/charms/focal/autopkgtest-cloud-worker/tests/10-deploy
new file mode 100755
index 0000000..2cd32f6
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/tests/10-deploy
@@ -0,0 +1,35 @@
+#!/usr/bin/python3
+
+import amulet
+import requests
+import unittest
+
+
+class TestCharm(unittest.TestCase):
+    def setUp(self):
+        self.d = amulet.Deployment()
+
+        self.d.add('autopkgtest-cloud-worker')
+        self.d.expose('autopkgtest-cloud-worker')
+
+        self.d.setup(timeout=900)
+        self.d.sentry.wait()
+
+        self.unit = self.d.sentry['autopkgtest-cloud-worker'][0]
+
+    def test_service(self):
+        # test we can access over http
+        page = requests.get('http://{}'.format(self.unit.info['public-address']))
+        self.assertEqual(page.status_code, 200)
+        # Now you can use self.d.sentry[SERVICE][UNIT] to address each of the units and perform
+        # more in-depth steps. Each self.d.sentry[SERVICE][UNIT] has the following methods:
+        # - .info - An array of the information of that unit from Juju
+        # - .file(PATH) - Get the details of a file on that unit
+        # - .file_contents(PATH) - Get plain text output of PATH file from that unit
+        # - .directory(PATH) - Get details of directory
+        # - .directory_contents(PATH) - List files and folders in PATH on that unit
+        # - .relation(relation, service:rel) - Get relation data from return service
+
+        
+if __name__ == '__main__':
+    unittest.main()
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest-lxd-socat@.service b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest-lxd-remote@.service
similarity index 68%
rename from deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest-lxd-socat@.service
rename to charms/focal/autopkgtest-cloud-worker/units/autopkgtest-lxd-remote@.service
index 368468c..99cd02a 100644
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest-lxd-socat@.service
+++ b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest-lxd-remote@.service
@@ -1,22 +1,21 @@
 # instance format is lxd-<arch>-<ip>
 [Unit]
 Description=LXD remote and socat for %i
-# TODO: This probably does not work
-Before=autopkgtest@.service
 Wants=network-online.target
 After=network-online.target
+StopWhenUnneeded=yes
 
 [Service]
 User=ubuntu
+Type=oneshot
 ExecStartPre=-/usr/bin/lxc remote remove %i
-ExecStart=/bin/sh -ec 'I="%i"; exec socat UNIX-LISTEN:/tmp/%i.socket,unlink-early,mode=666,fork TCP:$${I##*-}:8443'
-ExecStartPost=/bin/sleep 2
-ExecStartPost=/usr/bin/lxc remote add %i unix:///tmp/%i.socket
-ExecStopPost=-/usr/bin/lxc remote remove %i
+ExecStart=/bin/sh -ec 'I="%i"; exec lxc remote add --protocol lxd --auth-type tls --password autopkgtest --accept-certificate $${I} $${I##*-}:8443'
+ExecStop=/usr/bin/lxc remote remove %i
 Restart=on-failure
 RestartSec=5min
 StartLimitInterval=1h
 StartLimitBurst=3
+RemainAfterExit=yes
 
 [Install]
 WantedBy=autopkgtest.target
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest.target b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest.target
similarity index 100%
rename from deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest.target
rename to charms/focal/autopkgtest-cloud-worker/units/autopkgtest.target
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest@.service b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
similarity index 73%
rename from deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest@.service
rename to charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
index 266b43f..00fc900 100644
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/autopkgtest@.service
+++ b/charms/focal/autopkgtest-cloud-worker/units/autopkgtest@.service
@@ -4,28 +4,30 @@ ConditionFileNotEmpty=/home/ubuntu/rabbitmq.cred
 ConditionFileNotEmpty=/home/ubuntu/swift-password.cred
 Wants=network-online.target
 After=network-online.target
+StopWhenUnneeded=yes
 
 [Service]
 User=ubuntu
 WorkingDirectory=~
-Environment=WORKER_CONF_DIR=autopkgtest-cloud/worker-config-production
+TimeoutStartSec=5min
 Environment=SECGROUP=autopkgtest@%i.secgroup
+EnvironmentFile=/home/ubuntu/influx.cred
 EnvironmentFile=/home/ubuntu/rabbitmq.cred
 EnvironmentFile=/home/ubuntu/swift-password.cred
 ExecStartPre=/home/ubuntu/autopkgtest-cloud/tools/exec-in-region %i /home/ubuntu/autopkgtest-cloud/tools/copy-security-group $SECGROUP
 ExecStopPost=/home/ubuntu/autopkgtest-cloud/tools/exec-in-region %i /home/ubuntu/autopkgtest-cloud/tools/copy-security-group --delete-only  $SECGROUP
-ExecStart=/bin/sh -ec 'REGION="%i"; REGION="$${REGION%-*}"; CONF=/tmp/worker-$${REGION}.conf; \
-    CONF_T="$WORKER_CONF_DIR/worker-$${REGION}.conf"; \
-    [ -e $$CONF_T ] || CONF_T="$WORKER_CONF_DIR/worker-$${REGION%%%%-*}.conf"; \
-    [ -e $$CONF_T ] || CONF_T="$WORKER_CONF_DIR/worker.conf"; \
+ExecStart=/bin/sh -ec 'REGION="%i"; REGION="$${REGION%-*}"; \
+    CONF="$HOME/worker-$${REGION}.conf"; \
+    [ -e $$CONF ] || CONF="$HOME/worker-$${REGION%%%%-*}.conf"; \
+    [ -e $$CONF ] || CONF="$HOME/worker-$${REGION%%-*}.conf"; \
+    [ -e $$CONF ] || CONF="$HOME/worker.conf"; \
     if [ "$${REGION#lxd-}" != "$$REGION" ]; then \
         LXD_ARCH=$${REGION#*-}; LXD_ARCH=$${LXD_ARCH%%-*}; \
     else \
-        . ./$${REGION}.rc; \
+        . ./cloudrcs/$${REGION}.rc; \
     fi; \
-    sed "s/#RABBITHOST#/$RABBIT_HOST/; s/#RABBITUSER#/$RABBIT_USER/; s/#RABBITPASSWORD#/$RABBIT_PASSWORD/; s/#SWIFTPASSWORD#/$SWIFT_PASSWORD/; s/#KEYNAME#/testbed-$$(hostname)/" $$CONF_T > $${CONF}.$$$$ ; \
-    mv $${CONF}.$$$$ $${CONF}; \
     exec env REGION=$${REGION} $HOME/autopkgtest-cloud/worker/worker --config $$CONF -v LXD_ARCH=$$LXD_ARCH -v LXD_REMOTE=$$REGION -v SECGROUP=$SECGROUP'
+ExecReload=/bin/kill -HUP $MAINPID
 SuccessExitStatus=0 99 SIGTERM SIGHUP
 RestartSec=5min
 Restart=on-failure
diff --git a/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.service b/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.service
new file mode 100644
index 0000000..2a9b9c9
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Build autopkgtest image (%i)
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+User=ubuntu
+WorkingDirectory=~
+ExecStartPre=/usr/bin/flock -x -w 60 /home/ubuntu/ensure-keypair.lock /home/ubuntu/autopkgtest-cloud/tools/ensure-keypair "%i"
+ExecStart=/home/ubuntu/autopkgtest-cloud/tools/build-adt-image "%i"
diff --git a/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.timer b/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.timer
new file mode 100644
index 0000000..5688e72
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/build-adt-image@.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=Timer to build autopkgtest image (%i)
+After=autopkgtest.target
+
+[Timer]
+OnBootSec=24h
+OnUnitInactiveSec=24h
+RandomizedDelaySec=6h
+
+[Install]
+WantedBy=autopkgtest.target
diff --git a/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.service b/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.service
new file mode 100644
index 0000000..452c798
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cloud worker maintenance
+
+[Service]
+Type=oneshot
+User=ubuntu
+EnvironmentFile=/home/ubuntu/influx.cred
+ExecStart=/home/ubuntu/autopkgtest-cloud/tools/cloud-worker-maintenance
diff --git a/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.timer b/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.timer
new file mode 100644
index 0000000..a1a53ae
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/cloud-worker-maintenance.timer
@@ -0,0 +1,10 @@
+[Unit]
+Description=Cloud worker maintenance (timer)
+
+[Timer]
+OnBootSec=1h
+OnUnitInactiveSec=1h
+
+[Install]
+WantedBy=autopkgtest.target
+
diff --git a/charms/focal/autopkgtest-cloud-worker/units/ensure-adt-image@.service b/charms/focal/autopkgtest-cloud-worker/units/ensure-adt-image@.service
new file mode 100644
index 0000000..1ed07c6
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/ensure-adt-image@.service
@@ -0,0 +1,12 @@
+[Unit]
+Description=Ensure there is an autopkgtest image (%i)
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+User=ubuntu
+Environment=ENSURE_ONLY=1
+WorkingDirectory=~
+ExecStartPre=/usr/bin/flock -x -w 3600 /home/ubuntu/ensure-keypair.lock /home/ubuntu/autopkgtest-cloud/tools/ensure-keypair "%i"
+ExecStart=/home/ubuntu/autopkgtest-cloud/tools/build-adt-image "%i"
diff --git a/charms/focal/autopkgtest-cloud-worker/units/metrics.service b/charms/focal/autopkgtest-cloud-worker/units/metrics.service
new file mode 100644
index 0000000..0546655
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/metrics.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Collect & submit metrics
+
+[Service]
+Type=oneshot
+User=ubuntu
+Group=ubuntu
+EnvironmentFile=/home/ubuntu/influx.cred
+ExecStart=/home/ubuntu/autopkgtest-cloud/tools/metrics
diff --git a/charms/focal/autopkgtest-cloud-worker/units/metrics.timer b/charms/focal/autopkgtest-cloud-worker/units/metrics.timer
new file mode 100644
index 0000000..4bfe712
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/metrics.timer
@@ -0,0 +1,8 @@
+[Unit]
+Description=collect metrics (timer)
+
+[Timer]
+OnCalendar=*:0/5
+
+[Install]
+WantedBy=autopkgtest.target
diff --git a/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.service b/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.service
new file mode 100644
index 0000000..ba15be6
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Pull /home/ubuntu/autopkgtest-package-configs/
+
+[Service]
+Type=oneshot
+User=ubuntu
+Group=ubuntu
+WorkingDirectory=/home/ubuntu/autopkgtest-package-configs/
+ExecStart=/usr/bin/git pull
diff --git a/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.timer b/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.timer
new file mode 100644
index 0000000..51b53ec
--- /dev/null
+++ b/charms/focal/autopkgtest-cloud-worker/units/pull-package-configs.timer
@@ -0,0 +1,9 @@
+[Unit]
+Description=Pull package configs repo (timer)
+After=autopkgtest.target
+
+[Timer]
+OnCalendar=minutely
+
+[Install]
+WantedBy=autopkgtest.target
diff --git a/deployment/charms/xenial/autopkgtest-web/README.md b/charms/focal/autopkgtest-web/README.md
similarity index 87%
rename from deployment/charms/xenial/autopkgtest-web/README.md
rename to charms/focal/autopkgtest-web/README.md
index 95d0265..8c6d189 100644
--- a/deployment/charms/xenial/autopkgtest-web/README.md
+++ b/charms/focal/autopkgtest-web/README.md
@@ -6,5 +6,5 @@ browsable through a web server.
 
 Contact Information
 -------------------
-Author: Martin Pitt <[e-mail]>
+Author: Iain Lane <[e-mail]>
 Report bugs at: http://bugs.launchpad.net/auto-package-testing
diff --git a/deployment/charms/xenial/autopkgtest-web/config.yaml b/charms/focal/autopkgtest-web/config.yaml
similarity index 83%
rename from deployment/charms/xenial/autopkgtest-web/config.yaml
rename to charms/focal/autopkgtest-web/config.yaml
index a4dd11c..9786139 100644
--- a/deployment/charms/xenial/autopkgtest-web/config.yaml
+++ b/charms/focal/autopkgtest-web/config.yaml
@@ -7,14 +7,6 @@ options:
     type: string
     default:
     description: "Public external swift storage URL"
-  ssl-cert:
-    type: string
-    default:
-    description: "Apache SSL certificate (SSL will be disabled if not present)"
-  ssl-key:
-    type: string
-    default:
-    description: "Apache SSL key (SSL will be disabled if not present)"
   hostname:
     type: string
     default: autopkgtest.local
@@ -35,7 +27,3 @@ options:
     type: string
     default:
     description: "$no_proxy environment variable (for accessing sites other than github, like login.ubuntu.com and launchpad.net)"
-  mail-notify:
-    type: string
-    description: "email addresses (space separated) to notify on worker failure"
-    default:
diff --git a/charms/focal/autopkgtest-web/layer.yaml b/charms/focal/autopkgtest-web/layer.yaml
new file mode 100644
index 0000000..e6e30c2
--- /dev/null
+++ b/charms/focal/autopkgtest-web/layer.yaml
@@ -0,0 +1,19 @@
+includes:
+  - 'layer:apt'
+  - 'layer:basic'
+  - 'layer:status'
+  - 'interface:rabbitmq'
+  - 'interface:apache-website'
+options:
+  apt:
+    packages:
+      - amqp-tools
+      - distro-info
+      - git
+      - libjs-bootstrap
+      - libjs-jquery
+      - make
+      - python3-amqplib
+      - python3-distro-info
+      - python3-flask
+      - python3-flask-openid
diff --git a/deployment/charms/xenial/autopkgtest-web/metadata.yaml b/charms/focal/autopkgtest-web/metadata.yaml
similarity index 64%
rename from deployment/charms/xenial/autopkgtest-web/metadata.yaml
rename to charms/focal/autopkgtest-web/metadata.yaml
index ff188a4..ea04545 100644
--- a/deployment/charms/xenial/autopkgtest-web/metadata.yaml
+++ b/charms/focal/autopkgtest-web/metadata.yaml
@@ -1,16 +1,18 @@
 name: autopkgtest-web
 summary: autopkgtest web frontend with results from Swift
-maintainer: Martin Pitt <[e-mail]>
+series:
+  - focal
+maintainer: Iain Lane <[e-mail]>
 description: |
   This regularly downloads new autopkgtest results from Swift and makes them
   browsable through a web server.
-categories:
+tags:
   - web_server
   - monitoring
-subordinate: false
+subordinate: true
 requires:
   amqp:
     interface: rabbitmq
-provides:
   website:
-    interface: http
+    interface: apache-website
+    scope: container
diff --git a/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py b/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py
new file mode 100644
index 0000000..b2c1b5b
--- /dev/null
+++ b/charms/focal/autopkgtest-web/reactive/autopkgtest_web.py
@@ -0,0 +1,327 @@
+from charms.layer import status
+from charms.reactive import (
+    when,
+    when_all,
+    when_any,
+    when_not,
+    set_flag,
+    clear_flag,
+    hook,
+)
+from charmhelpers.core.hookenv import charm_dir, config
+
+from textwrap import dedent
+
+import glob
+import os
+import shutil
+import subprocess
+
+AUTOPKGTEST_CLOUD_CONF = os.path.expanduser("~ubuntu/autopkgtest-cloud.conf")
+GITHUB_SECRETS_PATH = os.path.expanduser("~ubuntu/github-secrets.json")
+GITHUB_STATUS_CREDENTIALS_PATH = os.path.expanduser(
+    "~ubuntu/github-status-credentials.txt"
+)
+
+
+@when_not("autopkgtest-web.autopkgtest_web_symlinked")
+def symlink_autopkgtest_cloud():
+    autopkgtest_cloud = os.path.join(charm_dir(), "webcontrol")
+    os.symlink(autopkgtest_cloud, os.path.expanduser("~ubuntu/webcontrol"))
+    set_flag("autopkgtest-web.autopkgtest_web_symlinked")
+
+
+@when("amqp.connected")
+@when_not("amqp.available")
+def setup_rabbitmq(rabbitmq):
+    rabbitmq.request_access("webcontrol", "/")
+    status.waiting("Waiting on RabbitMQ to configure vhost")
+
+
+@when_all(
+    "amqp.available",
+    "config.set.storage-url-internal",
+    "config.set.hostname",
+)
+def write_autopkgtest_cloud_conf(rabbitmq):
+    swiftinternal = config().get("storage-url-internal")
+    hostname = config().get("hostname")
+    rabbituser = rabbitmq.username()
+    rabbitpassword = rabbitmq.password()
+    rabbithost = rabbitmq.private_address()
+    clear_flag("autopkgtest-web.config-written")
+    with open(f"{AUTOPKGTEST_CLOUD_CONF}.new", "w") as f:
+        f.write(
+            dedent(
+                """\
+            [web]
+            database=/home/ubuntu/autopkgtest.db
+            database_ro=/home/ubuntu/public/autopkgtest.db
+            SwiftURL={swiftinternal}
+            ExternalURL=https://{hostname}/results
+
+            [amqp]
+            uri=amqp://{rabbituser}:{rabbitpassword}@{rabbithost}""".format(
+                    **locals()
+                )
+            )
+        )
+    os.rename(f"{AUTOPKGTEST_CLOUD_CONF}.new", AUTOPKGTEST_CLOUD_CONF)
+    set_flag("autopkgtest-web.config-written")
+
+
+@when_all(
+    "autopkgtest-web.autopkgtest_web_symlinked",
+    "autopkgtest-web.config-written",
+)
+def set_up_systemd_units():
+    any_changed = False
+    for unit in glob.glob(os.path.join(charm_dir(), "units", "*")):
+        base = os.path.basename(unit)
+        try:
+            os.symlink(
+                unit,
+                os.path.join(os.path.sep, "etc", "systemd", "system", base),
+            )
+            any_changed = True
+            status.maintenance("Installing systemd units")
+            subprocess.check_call(["systemctl", "enable", base])
+        except FileExistsError:
+            pass
+
+    if any_changed:
+        subprocess.check_call(["systemctl", "daemon-reload"])
+        subprocess.check_call(
+            ["systemctl", "restart", "autopkgtest-web.target"]
+        )
+
+    status.active("systemd units installed and enabled")
+
+
+@when_all(
+    "config.set.hostname",
+    "config.set.storage-url-internal",
+    "website.available",
+)
+@when_not("autopkgtest-web.website-initially-configured")
+def initially_configure_website(website):
+    set_up_web_config(website)
+
+
+@when_any(
+    "config.changed.hostname",
+    "config.changed.https-proxy",
+    "config.changed.no-proxy",
+    "config.changed.storage-url-internal",
+)
+@when_all(
+    "config.set.hostname",
+    "config.set.storage-url-internal",
+    "website.available",
+    "autopkgtest-web.website-initially-configured"
+)
+def set_up_web_config(apache):
+    webcontrol_dir = os.path.join(charm_dir(), "webcontrol")
+    sn = config().get("hostname")
+    https_proxy = config().get("https-proxy")
+    no_proxy = config().get("no-proxy")
+    storage_url_internal = config().get("storage-url-internal")
+
+    conf_enabled = os.path.join(os.path.sep, "etc", "apache2", "conf-enabled")
+    environment_d = os.path.join(os.path.sep, "etc", "environment.d")
+
+    if https_proxy or no_proxy:
+        try:
+            os.makedirs(environment_d)
+        except FileExistsError:
+            pass
+
+    https_proxy_env = os.path.join(environment_d, "https_proxy.conf")
+    https_proxy_apache = os.path.join(conf_enabled, "https_proxy.conf")
+
+    try:
+        os.unlink(https_proxy_env)
+    except FileNotFoundError:
+        pass
+    try:
+        os.unlink(https_proxy_apache)
+    except FileNotFoundError:
+        pass
+
+    if https_proxy:
+        with open(https_proxy_env, "w") as f:
+            f.write("https_proxy={}".format(https_proxy))
+        with open(https_proxy_apache, "w") as f:
+            f.write("SetEnv https_proxy {}".format(https_proxy))
+
+    no_proxy_env = os.path.join(environment_d, "no_proxy.conf")
+    no_proxy_apache = os.path.join(conf_enabled, "no_proxy.conf")
+
+    try:
+        os.unlink(no_proxy_env)
+    except FileNotFoundError:
+        pass
+    try:
+        os.unlink(no_proxy_apache)
+    except FileNotFoundError:
+        pass
+
+    if no_proxy:
+        with open(no_proxy_env, "w") as f:
+            f.write("no_proxy={}".format(no_proxy))
+        with open(no_proxy_apache, "w") as f:
+            f.write("SetEnv no_proxy {}".format(no_proxy))
+
+    server_name = "ServerName {}".format(sn) if sn else ""
+    apache.send_site_config(
+        dedent(
+            """\
+        <Directory {webcontrol_dir}>
+                Require all granted
+        </Directory>
+
+        <VirtualHost *:80>
+                DocumentRoot {webcontrol_dir}
+                ErrorLog ${{APACHE_LOG_DIR}}/error.log
+                CustomLog ${{APACHE_LOG_DIR}}/access.log combined
+                AddType 'text/plain' .log
+                AddCharset UTF-8 .log
+                DirectoryIndex browse.cgi
+                RedirectMatch permanent "^/packages(.*)/$" "/packages$1"
+                Redirect permanent /running.shtml /running
+
+                # webcontrol CGI scripts
+                ScriptAlias /request.cgi {webcontrol_dir}/request.cgi/
+                ScriptAlias /login {webcontrol_dir}/request.cgi/login
+                ScriptAlias /logout {webcontrol_dir}/request.cgi/logout
+                ScriptAlias / {webcontrol_dir}/browse.cgi/
+
+                <Location "/results/">
+                    ProxyPass "{storage_url_internal}/"
+                    ProxyPassReverse "{storage_url_internal}/"
+                </Location>
+
+                {server_name}
+        </VirtualHost>
+        """.format(
+                **locals()
+            )
+        )
+    )
+    set_flag("autopkgtest-web.website-initially-configured")
+    apache.send_ports([80])  # haproxy is doing SSL termination
+    apache.send_enabled()
+
+
+@when_all("config.changed.github-secrets", "config.set.github-secrets")
+def write_github_secrets():
+    github_secrets = config().get("github-secrets")
+
+    with open(GITHUB_SECRETS_PATH, "w") as f:
+        f.write(github_secrets)
+
+    try:
+        os.symlink(
+            GITHUB_SECRETS_PATH,
+            os.path.expanduser("~www-data/github-secrets.json"),
+        )
+    except FileExistsError:
+        pass
+
+
+@when_not("config.set.github-secrets")
+def clear_github_secrets():
+    try:
+        os.unlink(GITHUB_SECRETS_PATH)
+    except FileNotFoundError:
+        pass
+
+    try:
+        os.unlink(os.path.expanduser("~www-data/github-secrets.json"))
+    except FileNotFoundError:
+        pass
+
+
+@when_all(
+    "config.changed.github-status-credentials",
+    "config.set.github-status-credentials",
+)
+def write_github_status_credentials():
+    github_status_credentials = config().get("github-status-credentials")
+
+    with open(GITHUB_STATUS_CREDENTIALS_PATH, "w") as f:
+        f.write(github_status_credentials)
+
+    try:
+        os.symlink(
+            GITHUB_STATUS_CREDENTIALS_PATH,
+            os.path.expanduser("~www-data/github-status-credentials.txt"),
+        )
+    except FileExistsError:
+        pass
+
+
+@when_not("config.set.github-status-credentials")
+def clear_github_status_credentials():
+    try:
+        os.unlink(GITHUB_STATUS_CREDENTIALS_PATH)
+    except FileNotFoundError:
+        pass
+
+    try:
+        os.unlink(
+            os.path.expanduser("~www-data/github-status-credentials.json")
+        )
+    except FileNotFoundError:
+        pass
+
+
+@when_not("autopkgtest-web.bootstrap-symlinked")
+def symlink_bootstrap():
+    try:
+        os.symlink(
+            os.path.join(
+                os.path.sep, "usr", "share", "javascript", "bootstrap"
+            ),
+            os.path.join(charm_dir(), "webcontrol", "static", "bootstrap"),
+        )
+        set_flag("autopkgtest-web.bootstrap-symlinked")
+    except FileExistsError:
+        pass
+
+
+@when_not("autopkgtest-web.runtime-dir-created")
+def make_runtime_tmpfiles():
+    with open("/etc/tmpfiles.d/autopkgtest-web-runtime.conf", "w") as r:
+        r.write("D %t/autopkgtest_webcontrol 0755 www-data www-data\n")
+    subprocess.check_call(["systemd-tmpfiles", "--create"])
+    set_flag("autopkgtest-web.runtime-dir-created")
+
+
+@when_not("autopkgtest-web.running-json-symlinked")
+def symlink_running():
+    try:
+        os.symlink(
+            os.path.join(
+                os.path.sep, "run", "amqp-status-collector", "running.json"
+            ),
+            os.path.join(charm_dir(), "webcontrol", "static", "running.json"),
+        )
+        set_flag("autopkgtest-web.running-json-symlinked")
+    except FileExistsError:
+        pass
+
+
+@when_not("autopkgtest-web.public-db-symlinked")
+def symlink_public_db():
+    try:
+        publicdir = os.path.expanduser("~ubuntu/public/")
+        os.makedirs(publicdir)
+        shutil.chown(publicdir, user="ubuntu", group="ubuntu")
+        os.symlink(
+            os.path.join(publicdir, "autopkgtest.db"),
+            os.path.join(charm_dir(), "webcontrol", "static", "autopkgtest.db"),
+        )
+        set_flag("autopkgtest-web.public-db-symlinked")
+    except FileExistsError:
+        pass
diff --git a/charms/focal/autopkgtest-web/units/amqp-status-collector.service b/charms/focal/autopkgtest-web/units/amqp-status-collector.service
new file mode 100644
index 0000000..fbbc297
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/amqp-status-collector.service
@@ -0,0 +1,10 @@
+[Unit]
+Description=Read running test status from RabbitMQ (running.json)
+
+[Service]
+RuntimeDirectory=amqp-status-collector
+ExecStart=/home/ubuntu/webcontrol/amqp-status-collector
+Restart=on-failure
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/charms/focal/autopkgtest-web/units/autopkgtest-web.target b/charms/focal/autopkgtest-web/units/autopkgtest-web.target
new file mode 100644
index 0000000..52b8a1c
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/autopkgtest-web.target
@@ -0,0 +1,5 @@
+[Unit]
+Description=autopkgtest web
+
+[Install]
+WantedBy=multi-user.target
diff --git a/charms/focal/autopkgtest-web/units/cache-amqp.service b/charms/focal/autopkgtest-web/units/cache-amqp.service
new file mode 100644
index 0000000..d7aa53d
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/cache-amqp.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cache queue entries from AMQP
+
+[Service]
+Type=oneshot
+User=ubuntu
+StateDirectory=cache-amqp
+ExecStart=/home/ubuntu/webcontrol/cache-amqp
diff --git a/charms/focal/autopkgtest-web/units/cache-amqp.timer b/charms/focal/autopkgtest-web/units/cache-amqp.timer
new file mode 100644
index 0000000..c9cf3df
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/cache-amqp.timer
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cache queue entries from AMQP (timer)
+
+[Timer]
+OnCalendar=minutely
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/charms/focal/autopkgtest-web/units/download-all-results.service b/charms/focal/autopkgtest-web/units/download-all-results.service
new file mode 100644
index 0000000..d605794
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/download-all-results.service
@@ -0,0 +1,12 @@
+[Unit]
+Description=Download all results
+Before=download-results.service
+Conflicts=download-results.service
+
+[Service]
+User=ubuntu
+Type=oneshot
+ExecStart=/home/ubuntu/webcontrol/download-all-results
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/charms/focal/autopkgtest-web/units/download-results.service b/charms/focal/autopkgtest-web/units/download-results.service
new file mode 100644
index 0000000..7ea4171
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/download-results.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Download results
+After=download-all-results.service
+
+[Service]
+User=ubuntu
+ExecStart=/home/ubuntu/webcontrol/download-results
+Restart=on-failure
+
+[Install]
+WantedBy=download-all-results.service
diff --git a/charms/focal/autopkgtest-web/units/publish-db.service b/charms/focal/autopkgtest-web/units/publish-db.service
new file mode 100644
index 0000000..ce6e5a4
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/publish-db.service
@@ -0,0 +1,6 @@
+[Unit]
+Description=publish autopkgtest.db
+
+[Service]
+Type=oneshot
+ExecStart=/home/ubuntu/webcontrol/publish-db
diff --git a/charms/focal/autopkgtest-web/units/publish-db.timer b/charms/focal/autopkgtest-web/units/publish-db.timer
new file mode 100644
index 0000000..4d00b15
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/publish-db.timer
@@ -0,0 +1,8 @@
+[Unit]
+Description=publish autopkgtest.db (timer)
+
+[Timer]
+OnCalendar=minutely
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/charms/focal/autopkgtest-web/units/update-github-jobs.service b/charms/focal/autopkgtest-web/units/update-github-jobs.service
new file mode 100644
index 0000000..5681a9f
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/update-github-jobs.service
@@ -0,0 +1,9 @@
+[Unit]
+Description=Update GitHub job status
+
+[Service]
+Type=oneshot
+User=www-data
+Group=www-data
+EnvironmentFile=/etc/environment.d/*.conf
+ExecStart=/home/ubuntu/webcontrol/update-github-jobs
diff --git a/charms/focal/autopkgtest-web/units/update-github-jobs.timer b/charms/focal/autopkgtest-web/units/update-github-jobs.timer
new file mode 100644
index 0000000..f9df3bd
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/update-github-jobs.timer
@@ -0,0 +1,8 @@
+[Unit]
+Description=Update GitHub job status (timer)
+
+[Timer]
+OnCalendar=minutely
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/webcontrol/Makefile b/charms/focal/autopkgtest-web/webcontrol/Makefile
similarity index 100%
rename from webcontrol/Makefile
rename to charms/focal/autopkgtest-web/webcontrol/Makefile
diff --git a/webcontrol/amqp-status-collector b/charms/focal/autopkgtest-web/webcontrol/amqp-status-collector
similarity index 92%
rename from webcontrol/amqp-status-collector
rename to charms/focal/autopkgtest-web/webcontrol/amqp-status-collector
index bd45a07..66a9ed4 100755
--- a/webcontrol/amqp-status-collector
+++ b/charms/focal/autopkgtest-web/webcontrol/amqp-status-collector
@@ -1,6 +1,6 @@
 #!/usr/bin/python3
 # Pick up running tests, their status and logtail from the "teststatus" fanout
-# queue, and regularly write it into /tmp/running.json
+# queue, and regularly write it into /run/running.json
 
 import os
 import json
@@ -13,11 +13,17 @@ import logging
 import amqplib.client_0_8 as amqp
 
 exchange_name = 'teststatus.fanout'
+running_name = os.path.join(os.path.sep,
+                            'run',
+                            'amqp-status-collector',
+                            'running.json')
+running_name_new = "{}.new".format(running_name)
 
 # package -> runhash -> release -> arch -> (params, duration, logtail)
 running_tests = {}
 last_update = 0
 
+
 def amqp_connect():
     '''Connect to AMQP server'''
 
@@ -42,9 +48,9 @@ def update_output(amqp_channel, force_update=False):
     if not force_update and now - last_update < 10:
         return
 
-    with open('/tmp/running.json.new', 'w', encoding='utf-8') as f:
+    with open(running_name_new, 'w', encoding='utf-8') as f:
         json.dump(running_tests, f)
-    os.rename('/tmp/running.json.new', '/tmp/running.json')
+    os.rename(running_name_new, running_name)
 
 
 def process_message(msg):
@@ -91,7 +97,7 @@ amqp_con = amqp_connect()
 status_ch = amqp_con.channel()
 status_ch.access_request('/data', active=True, read=True, write=False)
 status_ch.exchange_declare(exchange_name, 'fanout', durable=False, auto_delete=True)
-queue_name = 'listener-%s' % socket.getfqdn()
+queue_name = 'running-listener-%s' % socket.getfqdn()
 status_ch.queue_declare(queue_name, durable=False, auto_delete=True)
 status_ch.queue_bind(queue_name, exchange_name, queue_name)
 
diff --git a/webcontrol/browse.cgi b/charms/focal/autopkgtest-web/webcontrol/browse.cgi
similarity index 98%
rename from webcontrol/browse.cgi
rename to charms/focal/autopkgtest-web/webcontrol/browse.cgi
index 949d622..429876f 100755
--- a/webcontrol/browse.cgi
+++ b/charms/focal/autopkgtest-web/webcontrol/browse.cgi
@@ -31,7 +31,7 @@ def init_config():
 
     db_con = sqlite3.connect('file:%s?mode=ro' % cp['web']['database_ro'], uri=True)
     try:
-        url = cp['web']['SwiftURLExternal']
+        url = cp['web']['ExternalURL']
     except KeyError:
         url = cp['web']['SwiftURL']
     swift_container_url = os.path.join(url, 'autopkgtest-%s')
@@ -121,7 +121,7 @@ def get_queue_info():
     Return (releases, arches, context -> release -> arch -> (queue_size, [requests])).
     '''
 
-    with open('/tmp/queued.json', 'r') as json_file:
+    with open('/var/lib/cache-amqp/queued.json', 'r') as json_file:
         queue_info_j = json.load(json_file)
 
         releases = queue_info_j["releases"]
@@ -281,7 +281,7 @@ def running():
                 queue_lengths.setdefault(c, {}).setdefault(r, {})[a] = queue_length
 
     try:
-        with open('/tmp/running.json') as f:
+        with open('/run/amqp-status-collector/running.json') as f:
             # package -> runhash -> release -> arch -> (params, duration, logtail)
             running_info = json.load(f)
     except FileNotFoundError:
diff --git a/webcontrol/cache-amqp b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
similarity index 85%
rename from webcontrol/cache-amqp
rename to charms/focal/autopkgtest-web/webcontrol/cache-amqp
index bb3497f..8992943 100755
--- a/webcontrol/cache-amqp
+++ b/charms/focal/autopkgtest-web/webcontrol/cache-amqp
@@ -97,9 +97,13 @@ class AutopkgtestQueueContents:
                         exclusive=False,
                         auto_delete=False,
                     )
-                    logging.info("Queue %s has %d items", queue_name, queue_size)
+                    logging.info(
+                        "Queue %s has %d items", queue_name, queue_size
+                    )
                     requests = self.get_queue_requests(queue_name)
-                    result.setdefault(context, {}).setdefault(release, {})[arch] = {
+                    result.setdefault(context, {}).setdefault(release, {})[
+                        arch
+                    ] = {
                         "size": queue_size,
                         "requests": requests,
                     }
@@ -115,7 +119,14 @@ class AutopkgtestQueueContents:
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Fetch AMQP queues into a file")
+    try:
+        state_directory = os.environ["STATE_DIRECTORY"]
+    except KeyError:
+        state_directory = "/var/lib/cache-amqp/"
+
+    parser = argparse.ArgumentParser(
+        description="Fetch AMQP queues into a file"
+    )
     parser.add_argument(
         "--debug",
         dest="debug",
@@ -124,7 +135,11 @@ if __name__ == "__main__":
         help="Print debugging (give twice for super verbose output)",
     )
     parser.add_argument(
-        "-o", "--output", dest="output", type=str, default="/tmp/queued.json"
+        "-o",
+        "--output",
+        dest="output",
+        type=str,
+        default=os.path.join(state_directory, "queued.json"),
     )
 
     args = parser.parse_args()
@@ -133,7 +148,9 @@ if __name__ == "__main__":
     cp.read(os.path.expanduser("~ubuntu/autopkgtest-cloud.conf"))
 
     logger = logging.getLogger()
-    formatter = logging.Formatter("%(asctime)s: %(message)s", "%Y-%m-%d %H:%M:%S")
+    formatter = logging.Formatter(
+        "%(asctime)s: %(message)s", "%Y-%m-%d %H:%M:%S"
+    )
     ch = logging.StreamHandler()
     ch.setFormatter(formatter)
     logger.addHandler(ch)
@@ -158,7 +175,9 @@ if __name__ == "__main__":
     aq = AutopkgtestQueueContents(amqp_uri, database)
     queue_contents = aq.get_queue_contents()
 
-    with tempfile.NamedTemporaryFile(mode="w", dir="/tmp", delete=False) as tf:
+    with tempfile.NamedTemporaryFile(
+        mode="w", dir=os.path.dirname(args.output), delete=False
+    ) as tf:
         try:
             json.dump(queue_contents, tf, indent=2)
             umask = os.umask(0)
diff --git a/charms/focal/autopkgtest-web/webcontrol/download-all-results b/charms/focal/autopkgtest-web/webcontrol/download-all-results
new file mode 100755
index 0000000..baf3b54
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/download-all-results
@@ -0,0 +1,227 @@
+#!/usr/bin/python3
+
+# Authors: Iain Lane, Martin Pitt
+
+# This script uses the OpenStack Swift API to list all of the contents of our
+# containers, diffs that against the local database, and then downloads any
+# missing results.
+
+# While in normal operation the download-results script is supposed to receive
+# notification of completed jobs, in case of bugs or network outages etc, this
+# script can be used to find any results which were missed and insert them.
+
+import os
+import sys
+import logging
+import sqlite3
+import io
+import tarfile
+import json
+import configparser
+import urllib.parse
+import time
+
+from distro_info import UbuntuDistroInfo
+from download.utils import get_test_id, init_db
+from urllib.request import urlopen
+
+LOGGER = logging.getLogger(__name__)
+
+config = None
+db_con = None
+
+
+def list_remote_container(container_url):
+    LOGGER.debug("Listing container %s", container_url)
+    out = []
+
+    def get_batch(start=None):
+        url = f"{container_url}/?format=json"
+        if start is not None:
+            url += f"&marker={urllib.parse.quote(start)}"
+
+        LOGGER.debug('Retrieving "%s"', url)
+        resp = urlopen(url)
+        json_string = resp.read()
+        json_data = json.loads(json_string)
+
+        if not json_data:
+            return None
+
+        out.extend([e["name"] for e in json_data])
+        name = out[-1]
+
+        return name
+
+    marker = get_batch()
+
+    while True:
+        new_marker = get_batch(marker)
+        if not new_marker or new_marker == marker:
+            break
+        marker = new_marker
+
+    out = [name for name in out if name.endswith("result.tar")]
+    LOGGER.debug("Found %d items in %s", len(out), container_url)
+    ret = {}
+    for r in out:
+        (_, _, _, _, run_id, _) = r.split("/")
+        ret[run_id] = r
+    return ret
+
+
+def list_our_results(release):
+    LOGGER.debug("Finding already recorded results for %s", release)
+    c = db_con.cursor()
+    c.execute(
+        "SELECT run_id FROM result INNER JOIN test ON test.id = result.test_id WHERE test.release=?",
+        (release,),
+    )
+    return {run_id for (run_id,) in c.fetchall()}
+
+
+def fetch_one_result(url):
+    """Download one result URL from swift and add it to the DB"""
+    (release, arch, _, src, run_id, _) = url.split("/")[-6:]
+    test_id = get_test_id(db_con, release, arch, src)
+
+    try:
+        f = urlopen(url, timeout=30)
+        if f.getcode() == 200:
+            tar_bytes = io.BytesIO(f.read())
+            f.close()
+        else:
+            raise NotImplementedError(
+                "fetch_one_result(%s): cannot handle HTTP code %i"
+                % (url, f.getcode())
+            )
+    except IOError as e:
+        LOGGER.error("Failure to fetch %s: %s", url, str(e))
+        # we tolerate "not found" (something went wrong on uploading the
+        # result), but other things indicate infrastructure problems
+        if hasattr(e, "code") and e.code == 404:
+            return
+        sys.exit(1)
+
+    try:
+        with tarfile.open(None, "r", tar_bytes) as tar:
+            exitcode = int(tar.extractfile("exitcode").read().strip())
+            try:
+                srcver = (
+                    tar.extractfile("testpkg-version").read().decode().strip()
+                )
+            except KeyError as e:
+                # not found
+                if exitcode in (4, 12, 20):
+                    # repair it
+                    srcver = "%s unknown" % (src)
+                else:
+                    raise
+            (ressrc, ver) = srcver.split()
+            testinfo = json.loads(
+                tar.extractfile("testinfo.json").read().decode()
+            )
+            duration = int(tar.extractfile("duration").read().strip())
+            # KeyError means the file is not there, i.e. there isn't a human
+            # requester
+            try:
+                requester = tar.extractfile("requester").read().decode().strip()
+            except KeyError as e:
+                requester = ""
+    except (KeyError, ValueError, tarfile.TarError) as e:
+        LOGGER.debug("%s is damaged, ignoring: %s", url, str(e))
+        return
+
+    if src != ressrc:
+        LOGGER.error(
+            "%s is a result for package %s, but expected package %s",
+            url,
+            ressrc,
+            src,
+        )
+        return
+
+    # parse recorded triggers in test result
+    for e in testinfo.get("custom_environment", []):
+        if e.startswith("ADT_TEST_TRIGGERS="):
+            test_triggers = e.split("=", 1)[1]
+            break
+    else:
+        LOGGER.error("%s result has no ADT_TEST_TRIGGERS, ignoring", url)
+        return
+
+    LOGGER.debug(
+        "Fetched test result for %s/%s/%s/%s %s (triggers: %s): exit code %i",
+        release,
+        arch,
+        src,
+        ver,
+        run_id,
+        test_triggers,
+        exitcode,
+    )
+
+    try:
+        c = db_con.cursor()
+        c.execute(
+            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?)",
+            (
+                test_id,
+                run_id,
+                ver,
+                test_triggers,
+                duration,
+                exitcode,
+                requester,
+            ),
+        )
+        db_con.commit()
+    except sqlite3.IntegrityError:
+        LOGGER.info("%s was already recorded - skipping", run_id)
+
+
+def fetch_container(release, container_url):
+    """Download new results from a swift container"""
+
+    marker = ""
+
+    our_results = list_our_results(release)
+    known_results = list_remote_container(container_url)
+
+    need_to_fetch = set(known_results.keys()) - our_results
+
+    LOGGER.debug("Need to download %d items", len(need_to_fetch))
+
+    for run_id in need_to_fetch:
+        fetch_one_result(os.path.join(container_url, known_results[run_id]))
+
+
+if __name__ == "__main__":
+    LOGGER.setLevel(logging.DEBUG)
+    ch = logging.StreamHandler()
+    if "DEBUG" in os.environ:
+        ch.setLevel(logging.DEBUG)
+    formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
+    ch.setFormatter(formatter)
+    LOGGER.addHandler(ch)
+
+    releases = list(
+        set(UbuntuDistroInfo().supported() + UbuntuDistroInfo().supported_esm())
+    )
+    releases.sort(key=UbuntuDistroInfo().all.index)
+
+    config = configparser.ConfigParser()
+    config.read(os.path.expanduser("~ubuntu/autopkgtest-cloud.conf"))
+
+    try:
+        for release in releases:
+            db_con = init_db(config["web"]["database"])
+            fetch_container(
+                release,
+                os.path.join(
+                    config["web"]["SwiftURL"], f"autopkgtest-{release}"
+                ),
+            )
+    finally:
+        if db_con:
+            db_con.close()
diff --git a/charms/focal/autopkgtest-web/webcontrol/download-results b/charms/focal/autopkgtest-web/webcontrol/download-results
new file mode 100755
index 0000000..a62babe
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/download-results
@@ -0,0 +1,119 @@
+#!/usr/bin/python3
+
+import configparser
+import json
+import logging
+import os
+import socket
+import sqlite3
+import urllib.parse
+
+from download.utils import get_test_id, init_db
+from urllib.request import urlopen
+
+import amqplib.client_0_8 as amqp
+
+EXCHANGE_NAME = "testcomplete.fanout"
+
+
+def amqp_connect():
+    """Connect to AMQP server"""
+
+    cp = configparser.ConfigParser()
+    cp.read(os.path.expanduser("~ubuntu/autopkgtest-cloud.conf"))
+    amqp_uri = cp["amqp"]["uri"]
+    parts = urllib.parse.urlsplit(amqp_uri, allow_fragments=False)
+    amqp_con = amqp.Connection(
+        parts.hostname, userid=parts.username, password=parts.password
+    )
+    logging.info(
+        "Connected to AMQP server at %s@%s" % (parts.username, parts.hostname)
+    )
+
+    return amqp_con
+
+
+def db_connect():
+    """Connect to SQLite DB"""
+    cp = configparser.ConfigParser()
+    cp.read(os.path.expanduser("~ubuntu/autopkgtest-cloud.conf"))
+
+    db_con = init_db(cp["web"]["database"])
+
+    return db_con
+
+
+def process_message(msg, db_con):
+    body = msg.body
+    if isinstance(body, bytes):
+        body = body.decode("UTF-8", errors="replace")
+    info = json.loads(body)
+    logging.info("Received notification of completed test {}".format(info))
+
+    arch = info["architecture"]
+    container = info["container"]
+    duration = info["duration"]
+    exitcode = info["exitcode"]
+    package = info["package"]
+    release = info["release"]
+    requester = (info["requester"] or "").strip()
+    version = info["testpkg_version"]
+    (_, _, _, _, run_id) = info["swift_dir"].split("/")
+
+    # we don't handle PPA requests
+    if container != ("autopkgtest-{}".format(release)):
+        logging.debug("Ignoring non-distro request: {}".format(info))
+        msg.channel.basic_ack(msg.delivery_tag)
+        return
+
+    try:
+        triggers = info["triggers"]
+        if not triggers:
+            raise KeyError
+    except KeyError:
+        logging.error(
+            "%s/%s/%s result has no ADT_TEST_TRIGGERS, ignoring",
+            package,
+            arch,
+            release,
+        )
+        msg.channel.basic_ack(msg.delivery_tag)
+        return
+
+    test_id = get_test_id(db_con, release, arch, package)
+
+    try:
+        c = db_con.cursor()
+        c.execute(
+            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?)",
+            (test_id, run_id, version, triggers, duration, exitcode, requester),
+        )
+        db_con.commit()
+    except sqlite3.IntegrityError:
+        logging.info("...which was already recorded - skipping")
+
+    msg.channel.basic_ack(msg.delivery_tag)
+
+
+if __name__ == "__main__":
+    logging.basicConfig(
+        level=("DEBUG" in os.environ and logging.DEBUG or logging.INFO)
+    )
+
+    db_con = db_connect()
+    amqp_con = amqp_connect()
+    status_ch = amqp_con.channel()
+    status_ch.access_request("/complete", active=True, read=True, write=False)
+    status_ch.exchange_declare(
+        EXCHANGE_NAME, "fanout", durable=True, auto_delete=False
+    )
+    queue_name = "complete-listener-%s" % socket.getfqdn()
+    status_ch.queue_declare(queue_name, durable=True, auto_delete=False)
+    status_ch.queue_bind(queue_name, EXCHANGE_NAME, queue_name)
+
+    logging.info("Listening to requests on %s" % queue_name)
+    status_ch.basic_consume(
+        "", callback=lambda msg: process_message(msg, db_con)
+    )
+    while status_ch.callbacks:
+        status_ch.wait()
diff --git a/deployment/charms/xenial/bootstrap-node/exec.d/.gitkeep b/charms/focal/autopkgtest-web/webcontrol/download/__init__.py
similarity index 100%
rename from deployment/charms/xenial/bootstrap-node/exec.d/.gitkeep
rename to charms/focal/autopkgtest-web/webcontrol/download/__init__.py
diff --git a/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/__init__.cpython-37.pyc b/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/__init__.cpython-37.pyc
new file mode 100644
index 0000000..3998d24
Binary files /dev/null and b/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/__init__.cpython-37.pyc differ
diff --git a/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/utils.cpython-37.pyc b/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/utils.cpython-37.pyc
new file mode 100644
index 0000000..723b94d
Binary files /dev/null and b/charms/focal/autopkgtest-web/webcontrol/download/__pycache__/utils.cpython-37.pyc differ
diff --git a/charms/focal/autopkgtest-web/webcontrol/download/utils.py b/charms/focal/autopkgtest-web/webcontrol/download/utils.py
new file mode 100644
index 0000000..70ec37d
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/download/utils.py
@@ -0,0 +1,59 @@
+import logging
+import sqlite3
+
+def init_db(path):
+    '''Create DB if it does not exist, and connect to it'''
+
+    db = sqlite3.connect(path)
+    c = db.cursor()
+    try:
+        c.execute('CREATE TABLE IF NOT EXISTS test ('
+                  '  id INTEGER PRIMARY KEY, '
+                  '  release CHAR[20], '
+                  '  arch CHAR[20], '
+                  '  package char[120])')
+        c.execute('CREATE TABLE IF NOT EXISTS result ('
+                  '  test_id INTEGER, '
+                  '  run_id CHAR[30], '
+                  '  version VARCHAR[200], '
+                  '  triggers TEXT, '
+                  '  duration INTEGER, '
+                  '  exitcode INTEGER, '
+                  '  requester TEXT, '
+                  '  PRIMARY KEY(test_id, run_id), '
+                  '  FOREIGN KEY(test_id) REFERENCES test(id))')
+        db.commit()
+        logging.debug('database %s created', path)
+    except sqlite3.OperationalError as e:
+        if 'already exists' not in str(e):
+            raise
+        logging.debug('database %s already exists', path)
+
+    return db
+
+def get_test_id(db_con, release, arch, src):
+    if not get_test_id._cache:
+        # prime the cache with all test IDs; much more efficient than doing
+        # thousands of individual queries
+        c = db_con.cursor()
+        c.execute('SELECT * FROM test')
+        while True:
+            row = c.fetchone()
+            if row is None:
+                break
+            get_test_id._cache[row[1] + '/' + row[2] + '/' + row[3]] = row[0]
+
+    cache_idx = release + '/' + arch + '/' + src
+    try:
+        return get_test_id._cache[cache_idx]
+    except KeyError:
+        # create new ID
+        c = db_con.cursor()
+        c.execute('INSERT INTO test VALUES (NULL, ?, ?, ?)', (release, arch, src))
+        test_id = c.lastrowid
+        db_con.commit()
+        get_test_id._cache[cache_idx] = test_id
+        return test_id
+
+
+get_test_id._cache = {}
diff --git a/webcontrol/db-update-current-versions b/charms/focal/autopkgtest-web/webcontrol/publish-db
similarity index 55%
rename from webcontrol/db-update-current-versions
rename to charms/focal/autopkgtest-web/webcontrol/publish-db
index 05f3799..c6b02ba 100755
--- a/webcontrol/db-update-current-versions
+++ b/charms/focal/autopkgtest-web/webcontrol/publish-db
@@ -1,6 +1,7 @@
 #!/usr/bin/python3
 # download/read Sources.gz for all known releases and write them into
-# autopkgtest.db. This is being used for statistics.
+# a copy of autopkgtest.db, which is then published to the public location.
+# This is being used for statistics.
 
 import os
 import logging
@@ -8,6 +9,9 @@ import sqlite3
 import configparser
 import urllib.request
 import gzip
+import fcntl
+import shutil
+import tempfile
 
 import apt_pkg
 
@@ -18,11 +22,13 @@ archive_url = 'http://archive.ubuntu.com/ubuntu'
 components = ['main', 'restricted', 'universe', 'multiverse']
 
 
-def init_db(path):
+def init_db(path, path_current):
     '''Create DB if it does not exist, and connect to it'''
 
     db = sqlite3.connect(path)
+    db_current = sqlite3.connect(path_current)
     c = db.cursor()
+    current_c = db_current.cursor()
     try:
         c.execute('CREATE TABLE current_version('
                   '  release CHAR[20], '
@@ -36,27 +42,71 @@ def init_db(path):
             raise
         logging.debug('table already exists')
 
+    try:
+        c.execute('CREATE TABLE url_last_checked('
+                  '  url CHAR[100], '
+                  '  timestamp CHAR[50], '
+                  '  PRIMARY KEY(url))')
+        db.commit()
+        try:
+            c.executemany('INSERT INTO url_last_checked (url, timestamp) VALUES (?, ?)',
+                          current_c.execute ('SELECT url, timestamp FROM url_last_checked'))
+            logging.debug('database table url_last_checked created')
+        except sqlite3.OperationalError as e:
+            if 'no such table' not in str(e):
+                raise
+            logging.debug('no url_last_checked yet, first run probably')
+    except sqlite3.OperationalError as e:
+        if 'already exists' not in str(e):
+            raise
+        logging.debug('table already exists')
+
+    db_current.close()
     return db
 
 
+def get_last_checked(db_con, url):
+    c = db_con.cursor()
+    c.execute('SELECT timestamp FROM url_last_checked WHERE url=?', (url, ))
+
+    try:
+        (ts, ) = c.fetchone()
+        return ts
+    except TypeError:  # not found
+        return None
+
+
 def get_sources(db_con, release):
     c = db_con.cursor()
     for component in components:
         for pocket in (release, release + '-updates'):
             logging.debug('Processing %s/%s', pocket, component)
             try:
+                url = os.path.join(archive_url, 'dists', pocket, component, 'source/Sources.gz')
+                request = urllib.request.Request(url)
+                last_checked = get_last_checked(db_con, url)
+                if last_checked:
+                    request.add_header('If-Modified-Since', last_checked)
+
                 # meh, apt_pkg.TagFile() apparently cannot directly read  from an urllib stream
-                f = urllib.request.urlretrieve(os.path.join(archive_url, 'dists', pocket, component, 'source/Sources.gz'))[0]
-                with gzip.open(f) as fd:
-                    for section in apt_pkg.TagFile(fd):
-                        c.execute('INSERT OR REPLACE INTO current_version VALUES (?, ?, ?)',
-                                  (release, section['Package'], section['Version']))
-            except urllib.error.HTTPError:
-                # EOLed release; no need to update.
-                f = None
-            finally:
-                if f:
-                    os.unlink(f)
+                with tempfile.TemporaryFile() as temp_file:
+                    with urllib.request.urlopen(request) as response:
+                        temp_file.write(response.read())
+                        last_modified = response.getheader('Last-Modified')
+                        if last_modified:
+                            c.execute('INSERT OR REPLACE INTO url_last_checked (url, timestamp) '
+                                           'VALUES (?, ?)', (url, last_modified))
+                            db_con.commit()
+
+                    temp_file.seek(0)
+                    with gzip.open(temp_file) as fd:
+                        for section in apt_pkg.TagFile(fd):
+                            c.execute('INSERT OR REPLACE INTO current_version VALUES (?, ?, ?)',
+                                      (release, section['Package'], section['Version']))
+            except urllib.error.HTTPError as e:
+                if e.code == 304:
+                    logging.debug('url {} not modified'.format(url))
+                    pass
 
 
 if __name__ == '__main__':
@@ -66,9 +116,22 @@ if __name__ == '__main__':
     config = configparser.ConfigParser()
     config.read(os.path.expanduser('~ubuntu/autopkgtest-cloud.conf'))
 
-    db_con = init_db(config['web']['database'])
+    target = config['web']['database_ro']
+    target_new = '{}.new'.format(target)
+
+    with open(config['web']['database'], 'rb') as src:
+        with open(target_new, 'wb') as dest:
+            fcntl.flock(dest, fcntl.LOCK_EX | fcntl.LOCK_NB)
+            shutil.copyfileobj(src, dest)
+            dest.flush()
+            os.fsync(dest.fileno())
+
+    db_con = init_db(target_new, target)
 
     db_con.execute('DELETE FROM current_version')
     for row in db_con.execute('SELECT DISTINCT release FROM test'):
         get_sources(db_con, row[0])
     db_con.commit()
+    db_con.close()
+
+    os.rename(target_new, target)
diff --git a/webcontrol/request-test.cgi b/charms/focal/autopkgtest-web/webcontrol/request-test.cgi
similarity index 100%
rename from webcontrol/request-test.cgi
rename to charms/focal/autopkgtest-web/webcontrol/request-test.cgi
diff --git a/webcontrol/request.cgi b/charms/focal/autopkgtest-web/webcontrol/request.cgi
similarity index 100%
rename from webcontrol/request.cgi
rename to charms/focal/autopkgtest-web/webcontrol/request.cgi
diff --git a/webcontrol/request/__init__.py b/charms/focal/autopkgtest-web/webcontrol/request/__init__.py
similarity index 100%
rename from webcontrol/request/__init__.py
rename to charms/focal/autopkgtest-web/webcontrol/request/__init__.py
diff --git a/webcontrol/request/app.py b/charms/focal/autopkgtest-web/webcontrol/request/app.py
similarity index 97%
rename from webcontrol/request/app.py
rename to charms/focal/autopkgtest-web/webcontrol/request/app.py
index 6813cd0..b19945f 100644
--- a/webcontrol/request/app.py
+++ b/charms/focal/autopkgtest-web/webcontrol/request/app.py
@@ -7,7 +7,7 @@ from collections import ChainMap
 from html import escape as _escape
 
 from flask import Flask, request, session, redirect
-from flask.ext.openid import OpenID
+from flask_openid import OpenID
 
 from request.submit import Submit
 
@@ -110,7 +110,7 @@ def maybe_escape(value):
 
 
 # Initialize app
-PATH = os.path.join(os.getenv('TMPDIR', '/tmp'), 'autopkgtest_webcontrol')
+PATH = os.path.join(os.path.sep, 'run', 'autopkgtest_webcontrol')
 os.makedirs(PATH, exist_ok=True)
 app = Flask('request')
 # keep secret persistent between CGI invocations
diff --git a/webcontrol/request/submit.py b/charms/focal/autopkgtest-web/webcontrol/request/submit.py
similarity index 94%
rename from webcontrol/request/submit.py
rename to charms/focal/autopkgtest-web/webcontrol/request/submit.py
index 3f15a8b..e1326b1 100644
--- a/webcontrol/request/submit.py
+++ b/charms/focal/autopkgtest-web/webcontrol/request/submit.py
@@ -16,6 +16,7 @@ from urllib.error import HTTPError
 from datetime import datetime
 
 import amqplib.client_0_8 as amqp
+from distro_info import UbuntuDistroInfo
 
 # Launchpad REST API base
 LP = 'https://api.launchpad.net/1.0/'
@@ -36,14 +37,9 @@ class Submit:
         cp = configparser.ConfigParser()
         cp.read(os.path.expanduser('~ubuntu/autopkgtest-cloud.conf'))
 
-        cfg = configparser.ConfigParser()
-        cfg.read(os.path.expanduser('~ubuntu/autopkgtest-cloud/worker-config-production/worker.conf'))
-
         # read valid releases and architectures from DB
         self.db_con = sqlite3.connect('file:%s?mode=ro' % cp['web']['database_ro'], uri=True)
-        self.releases = []
-        for release in cfg['autopkgtest']['releases'].split():
-            self.releases.append(release)
+        self.releases = set(UbuntuDistroInfo().supported() + UbuntuDistroInfo().supported_esm())
         logging.debug('Valid releases: %s' % self.releases)
 
         self.architectures = set()
diff --git a/webcontrol/request/tests/__init__.py b/charms/focal/autopkgtest-web/webcontrol/request/tests/__init__.py
similarity index 100%
rename from webcontrol/request/tests/__init__.py
rename to charms/focal/autopkgtest-web/webcontrol/request/tests/__init__.py
diff --git a/webcontrol/request/tests/test_app.py b/charms/focal/autopkgtest-web/webcontrol/request/tests/test_app.py
similarity index 100%
rename from webcontrol/request/tests/test_app.py
rename to charms/focal/autopkgtest-web/webcontrol/request/tests/test_app.py
diff --git a/webcontrol/request/tests/test_submit.py b/charms/focal/autopkgtest-web/webcontrol/request/tests/test_submit.py
similarity index 100%
rename from webcontrol/request/tests/test_submit.py
rename to charms/focal/autopkgtest-web/webcontrol/request/tests/test_submit.py
diff --git a/webcontrol/setup.py b/charms/focal/autopkgtest-web/webcontrol/setup.py
similarity index 100%
rename from webcontrol/setup.py
rename to charms/focal/autopkgtest-web/webcontrol/setup.py
diff --git a/charms/focal/autopkgtest-web/webcontrol/static/jquery b/charms/focal/autopkgtest-web/webcontrol/static/jquery
new file mode 120000
index 0000000..7e2f826
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/static/jquery
@@ -0,0 +1 @@
+/usr/share/javascript/jquery
\ No newline at end of file
diff --git a/webcontrol/static/style.css b/charms/focal/autopkgtest-web/webcontrol/static/style.css
similarity index 100%
rename from webcontrol/static/style.css
rename to charms/focal/autopkgtest-web/webcontrol/static/style.css
diff --git a/webcontrol/templates/browse-error.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-error.html
similarity index 100%
rename from webcontrol/templates/browse-error.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-error.html
diff --git a/webcontrol/templates/browse-home.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-home.html
similarity index 100%
rename from webcontrol/templates/browse-home.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-home.html
diff --git a/webcontrol/templates/browse-layout.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-layout.html
similarity index 96%
rename from webcontrol/templates/browse-layout.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-layout.html
index 5c0d57f..b47203f 100644
--- a/webcontrol/templates/browse-layout.html
+++ b/charms/focal/autopkgtest-web/webcontrol/templates/browse-layout.html
@@ -31,6 +31,7 @@
               <li><a href="{{base_url}}running">Running</a></li>
               <li><a href="{{base_url}}statistics">Statistics</a></li>
               <li><a href="https://wiki.ubuntu.com/ProposedMigration#autopkgtests">Documentation</a></li>
+              <li><a href="https://autopkgtest-cloud.readthedocs.io/">Docs for admins</a></li>
             </ul>
           </div><!--/.nav-collapse -->
         </div>
diff --git a/webcontrol/templates/browse-package.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-package.html
similarity index 100%
rename from webcontrol/templates/browse-package.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-package.html
diff --git a/webcontrol/templates/browse-results.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html
similarity index 100%
rename from webcontrol/templates/browse-results.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html
diff --git a/webcontrol/templates/browse-running.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-running.html
similarity index 100%
rename from webcontrol/templates/browse-running.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-running.html
diff --git a/webcontrol/templates/browse-statistics.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-statistics.html
similarity index 100%
rename from webcontrol/templates/browse-statistics.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-statistics.html
diff --git a/webcontrol/templates/browse-testlist.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-testlist.html
similarity index 100%
rename from webcontrol/templates/browse-testlist.html
rename to charms/focal/autopkgtest-web/webcontrol/templates/browse-testlist.html
diff --git a/webcontrol/update-github-jobs b/charms/focal/autopkgtest-web/webcontrol/update-github-jobs
similarity index 94%
rename from webcontrol/update-github-jobs
rename to charms/focal/autopkgtest-web/webcontrol/update-github-jobs
index 97a1cbd..f2bb430 100755
--- a/webcontrol/update-github-jobs
+++ b/charms/focal/autopkgtest-web/webcontrol/update-github-jobs
@@ -16,9 +16,9 @@ from urllib.error import HTTPError
 from request.submit import Submit
 
 
-PENDING_DIR = '/tmp/autopkgtest_webcontrol/github-pending'
+PENDING_DIR = '/run/autopkgtest_webcontrol/github-pending'
 swift_url = None
-swift_url_public = None
+external_url = None
 
 
 def result_matches_job(result_url, params):
@@ -126,7 +126,7 @@ def process_job(jobfile):
                 code = result_matches_job(result_url, params)
                 if code is not None:
                     finish_job(jobfile, params, code,
-                               result_url.replace(swift_url, swift_url_public) + '/log.gz')
+                               result_url.replace(swift_url, external_url) + '/log.gz')
                     break
     except HTTPError as e:
         logging.error('job %s URL %s failed: %s', os.path.basename(jobfile), query_url, e)
@@ -139,14 +139,15 @@ if __name__ == '__main__':
         logging.basicConfig(level='DEBUG')
     if not os.path.isdir(PENDING_DIR):
         logging.info('%s does not exist, nothing to do', PENDING_DIR)
+        sys.exit(0)
 
     config = configparser.ConfigParser()
     config.read(os.path.expanduser('~ubuntu/autopkgtest-cloud.conf'))
     swift_url = config['web']['SwiftURL']
     try:
-        swift_url_public = config['web']['SwiftURLExternal']
+        external_url = config['web']['ExternalURL']
     except KeyError:
-        swift_url_public = swift_url
+        external_url = swift_url
 
     jobs = sys.argv[1:]
 
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/config.yaml b/deployment/charms/xenial/autopkgtest-cloud-worker/config.yaml
deleted file mode 100644
index 8732343..0000000
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/config.yaml
+++ /dev/null
@@ -1,15 +0,0 @@
-options:
-  swift-password:
-    type: string
-    description: "password for swift"
-    default:
-  nova-rcs:
-    type: string
-    description: "base64 encoded tarball of nova OpenStack credential .rc files"
-  lxd-remotes:
-    type: string
-    description: "lines 'arch IP numworkers' for workers on LXD remotes"
-  mail-notify:
-    type: string
-    description: "email addresses (space separated) to notify on worker failure"
-    default:
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-broken b/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-broken
deleted file mode 100755
index 3d26678..0000000
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-broken
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/bin/sh
-set -e
-
-CONFIG=/home/ubuntu/rabbitmq.cred
-
-juju-log "amqp-relation-broken: Removing rabbit config"
-rm -f $CONFIG
-
-hooks/stop
diff --git a/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-changed b/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-changed
deleted file mode 100755
index 4a5b045..0000000
--- a/deployment/charms/xenial/autopkgtest-cloud-worker/hooks/amqp-relation-changed
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/bin/sh
-set -e
-
-RABBIT_USER=autopkgtest-worker
-CONFIG=/home/ubuntu/rabbitmq.cred
-
-juju-log "amqp-relation-changed: requesting credentials for $RABBIT_USER"
-relation-set username=$RABBIT_USER
-relation-set vhost=/
-RABBIT_HOST=`relation-get private-address`
-RABBIT_PASSWORD=`relation-get password`
-if [ -z "$RABBIT_HOST" -o -z "$RABBIT_PASSWORD" ]; then