Merge ~paride/autopkgtest:nova-reboot into ~ubuntu-release/autopkgtest/+git/autopkgtest:master

[Paride Legovini]

Unmerged commits

a3de103... by Paride Legovini on 2022-12-16

nova script: force nova reboot for now (NOVA_REBOOT=1)

This is needed to workaround arm64 VMs that fail to reboot. We need to
figure out why that is happening and revert this commit.

One side effect of this change is that we end up with VMs rebooting
*twice*: once because the reboot command has been called in the VM, and
one more time from wait_reboot(). This interferes with some kernel
testing relying on setting kernel cmdline parameters.

56f13dc... by Paride Legovini on 2022-12-06

doc: mention the default Pin-Priority for pockets

c92df58... by Paride Legovini on 2022-12-06

Pin pockets with Pin-Priority: 500

Ubuntu is switching to NotAutomatic: yes for the proposed pocket:
the change is already operative on >= Lunar and under consideration
for older stable releases. This means that by default packages from
-proposed won't be installed unless explicitly requested.

The change is meant to make it easier for end users to keep -proposed
enabled (for SRU verification or individual package installation)
without manually adding a pin to avoid pulling in all of -proposed.

However when running autopkgtests with --apt-pocket=proposed we do want
to pull in all the pocket, so let's always give it Pin-Priority: 500.
This is similar to what Launchpad does with the buildds.

Note: This change does not really treat -proposed in a special way:
any pocket will get Pin-Priority: 500. It makes sense to keep the tool
behavior consistent, and we're not aware of cases where this could be
cause issues.

f7b890b... by Paride Legovini on 2022-09-19

autopkgtest-build-lxd: umount lxcfs before snapd is purged

One preparatory step for autopkgtest LXD images is purging snapd,
however this currently fails while removing the lxd snap (LP: #1989603).
As as workaround manually umount the problematic mountpoint.

e568cbc... by Paride Legovini on 2022-09-14

autopkgtest-build-lxd: mask snapd.seeded.service

autopkgtest-build-lxd: mask snapd.seeded.service as a workaround for
LP: #1952084. Can be reverted once that bug is properly fixed or if
we switch to minimal images, which do not ship with snapd, but aren't
available for armhf as of today.

55cf0ad... by Brian Murray on 2022-09-14

Reset the timeout-ssh period back to its default

7c0bddf... by Brian Murray on 2022-09-09

Wait even longer for ssh to work out

5db4fa8... by Brian Murray on 2022-09-08

Increase default value of timeout-ssh to 900

4edddb8... by Brian Murray on 2022-09-08

Increase the wait_for_ssh timeout to 900

d9d863f... by Brian Murray on 2022-06-02

kill processes with SIGKILL if SIGTERM fails

When trying to kill long running test processes on armhf they don't
always get killed so if the process is still running kill it again with
SIGKILL.

diff --git a/lib/adt_testbed.py b/lib/adt_testbed.py
index 90e2009..9ff0475 100644
--- a/lib/adt_testbed.py
+++ b/lib/adt_testbed.py
@@ -54,7 +54,7 @@ class Testbed:
                  setup_commands=[], setup_commands_boot=[], add_apt_pockets=[],
                  copy_files=[], pin_packages=[], add_apt_sources=[],
                  add_apt_releases=[], apt_default_release=None,
-                 enable_apt_fallback=True, shell_fail=False, needs_internet='run'):
+                 enable_apt_fallback=True, shell_fail=False, needs_internet='run', cross_arch=None):
         self.sp = None
         self.lastsend = None
         self.scratch = None
@@ -62,6 +62,8 @@ class Testbed:
         self._need_reset_apt = False
         self.stop_sent = False
         self.dpkg_arch = None
+        self.cross_arch = cross_arch
+        self.cross_env = []
         self.exec_cmd = None
         self.output_dir = output_dir
         self.shared_downtmp = None  # testbed's downtmp on the host, if supported
@@ -302,6 +304,23 @@ class Testbed:
         self.dpkg_arch = self.check_exec(['dpkg', '--print-architecture'], True).strip()
         adtlog.info('testbed dpkg architecture: ' + self.dpkg_arch)
 
+        # set up environment for cross-architecture if needed
+        if self.cross_arch:
+            self.cross_env = []
+            argv = ['dpkg-architecture', '-a', self.cross_arch]
+            # ignore stderr
+            (code, vars, err) = self.execute(argv,
+                                             stdout=subprocess.PIPE,
+                                             stderr=subprocess.PIPE)
+            if code != 0:
+                self.bomb('"%s" failed with status %i' % (' '.join(argv), code),
+                          adtlog.AutopkgtestError)
+
+            for var in vars.split('\n'):
+                if var.startswith('DEB_HOST'):
+                    self.cross_env.append(var)
+            adtlog.info('testbed target architecture: ' + self.cross_arch)
+
         # do we have eatmydata?
         (code, out, err) = self.execute(
             ['sh', '-ec', 'command -v eatmydata'],
@@ -391,15 +410,22 @@ class Testbed:
             ''' % (pocket, pocket, pocket)
             self.check_exec(['sh', '-ec', script])
 
-        # create apt pinning for --apt-pocket with package list
+        # create apt pinning for --apt-pocket
         for pocket in self.add_apt_pockets:
             # do we have a package list?
-            try:
+            pkglist = ''
+            if '=' in pocket:
                 (pocket, pkglist) = pocket.split('=', 1)
-            except ValueError:
-                continue
-            self._create_apt_pinning_for_packages(self._get_default_release() + '-' + pocket, pkglist,
-                                                  [self._get_default_release() + '-updates'])
+
+            # set priority for the specified pocket.
+            # this is useful to override NotAutomatic: yes.
+            self._create_apt_pinning_for_pocket(self._get_default_release() + '-' + pocket)
+
+            # set priorities for individual packages if specified
+            if pkglist:
+                self._create_apt_pinning_for_packages(self._get_default_release() + '-' + pocket, pkglist,
+                                                      [self._get_default_release() + '-updates'])
+
         if self.add_apt_releases:
             get_mirror_and_components = '''
             sed -rn 's/^(deb|deb-src) +(\\[.*\\] *)?((http|https|file):[^ ]*) +([^ ]+) +(.*)$/\\2\\3 \\6/p' \\
@@ -474,6 +500,7 @@ class Testbed:
         self.deps_installed = deps_new
         if not deps_new:
             return
+<<<<<<< lib/adt_testbed.py
         self.satisfy_dependencies_string(', '.join(deps_new), 'install-deps', shell_on_failure=shell_on_failure, synth_deps=synth_deps)
 
     def _provide_sudo(self) -> str:
@@ -569,6 +596,25 @@ class Testbed:
 
             if self._tried_provide_sudo is not True:
                 raise Unsupported(name, self._tried_provide_sudo)
+=======
+        if self.cross_arch:
+            # mock up a .dsc with our dependencies string as our build-deps
+            dsc = TempPath(self, 'autopkgtest-satdep.dsc')
+            with open(dsc.host, 'w', encoding='UTF-8') as f_dsc:
+                f_dsc.write("Source: autopkgtest-satdep\n"
+                            "Binary: autopkgtest-satdep\n"
+                            "Architecture: any all\n"
+                            "Build-Depends: " + ', '.join(deps_new) + "\n")
+            dsc.copydown()
+
+            # feed the result to apt-get build-dep
+            self.satisfy_build_deps(dsc.tb, recommends, shell_on_failure)
+            # Now handle the synthetic deps as well
+            self.install_apt('', recommends, shell_on_failure, synth_deps)
+
+        else:
+            self.satisfy_dependencies_string(', '.join(deps_new), 'install-deps', recommends, shell_on_failure=shell_on_failure, synth_deps=synth_deps)
+>>>>>>> lib/adt_testbed.py
 
     def needs_reset(self):
         # show what caused a reset
@@ -671,6 +717,11 @@ class Testbed:
             env.append('DEBIAN_FRONTEND=noninteractive')
             env.append('APT_LISTBUGS_FRONTEND=none')
             env.append('APT_LISTCHANGES_FRONTEND=none')
+<<<<<<< lib/adt_testbed.py
+=======
+        env += self.install_tmp_env
+        env += self.cross_env
+>>>>>>> lib/adt_testbed.py
 
         adtlog.debug('testbed command %s, kind %s, sout %s, serr %s, env %s' %
                      (argv, kind, stdout and 'pipe' or 'raw',
@@ -709,8 +760,13 @@ class Testbed:
         adtlog.debug('testbed command exited with code %i' % proc.returncode)
 
         if proc.returncode in (254, 255):
+<<<<<<< lib/adt_testbed.py
             self.debug_fail()
             self.bomb('testbed auxverb failed with exit code %i' % proc.returncode)
+=======
+            self.command('auxverb_debug_fail')
+            self.bomb('testbed auxverb failed with exit code %i, argv: %r\nstdout: %s\nstderr: %s' % (proc.returncode, argv, out, err))
+>>>>>>> lib/adt_testbed.py
 
         return (proc.returncode, out, err)
 
@@ -941,7 +997,205 @@ Description: satisfy autopkgtest test dependencies
 
         self.execute(['dpkg', '--purge', 'autopkgtest-satdep'])
 
+<<<<<<< lib/adt_testbed.py
     def satisfy_dependencies_string(self, deps, what,
+=======
+    def install_click(self, clickpath):
+        # copy click into testbed
+        tp = Path(self, clickpath, os.path.join(
+            self.scratch, os.path.basename(clickpath)))
+        tp.copydown()
+        # install it
+        clickopts = ['--all-users']
+        if 'AUTOPKGTEST_CLICK_NO_FRAMEWORK_CHECK' in os.environ:
+            # this is mostly for testing
+            clickopts.append('--force-missing-framework')
+        if 'root-on-testbed' in self.caps:
+            rc = self.execute(['click', 'install', '--allow-unauthenticated'] +
+                              clickopts + [tp.tb], kind='install')[0]
+        else:
+            rc = self.execute(['pkcon', 'install-local', '--allow-untrusted',
+                               tp.tb], kind='install')[0]
+        if rc != 0:
+            self.badpkg('click install failed with status %i' % rc)
+
+        # work around https://launchpad.net/bugs/1333215
+        # we don't want su -l here which resets the environment from
+        # self.execute(); so emulate the parts that we want
+        # FIXME: move "run as user" as an argument of execute()/check_exec() and run with -l
+        self.check_exec(['su', '--shell=/bin/sh', self.su_user, '-c',
+                         ('export USER=%s;' % self.user) +
+                         '. /etc/profile >/dev/null 2>&1 || true; '
+                         ' . ~/.profile >/dev/null 2>&1 || true; '
+                         '[ -z "$UPSTART_SESSION" ] || /sbin/initctl --user start click-user-hooks'])
+
+    def apparmor_click(self, clickpkgs, installed_clicks):
+        '''Update AppArmor rules for click tests
+
+        Return True if anything was modified and apparmor_restore_click()
+        needs to be called.
+        '''
+        # check if we are in a click+AppArmor environment
+        if self.execute(['sh', '-ec',
+                         '[ -d /var/cache/apparmor -a -d /var/lib/apparmor/clicks -a ! -e /var/cache/apparmor/click-ap.rules ] && '
+                         'type aa-clickhook >/dev/null 2>&1'])[0] != 0:
+            adtlog.debug('testbed does not have AppArmor/click or already has Autopilot click rules, no need to adjust rules')
+            return False
+        adtlog.debug('testbed has AppArmor/click')
+
+        if 'root-on-testbed' not in self.caps:
+            adtlog.warning('Cannot adjust AppArmor rules without root/sudo '
+                           'privileges; Autopilot tests will fail and test '
+                           'dependencies will not be available!')
+            return False
+
+        rules = 'dbus (receive, send) bus=session path=/com/canonical/Autopilot/**,'
+        for e in self.install_tmp_env:
+            if e.startswith('QT_PLUGIN_PATH='):
+                for p in e.split('=', 1)[1].split(':'):
+                    p = p.strip()
+                    if p:
+                        rules += ' %s/** r,' % p
+                break
+
+        script = '''echo '%s' > /var/cache/apparmor/click-ap.rules; ''' % rules
+
+        if clickpkgs or installed_clicks:
+            adtlog.info('Updating AppArmor rules to allow autopilot introspection for tested clicks')
+            script += 'for c in %s; do ' \
+                '   info=$(click info %s %s/$(basename "$c")); ' \
+                ''' name=$(echo "$info" | sed -rn '/"name"/ {s/^.*: *"([^"]+)",/\\1/; p}'); ''' \
+                ''' version=$(echo "$info" | sed -rn '/"version"/ {s/^.*: *"([^"]+)",/\\1/; p}'); ''' \
+                '   touch -h /var/lib/apparmor/clicks/${name}_*_${version}.json >/dev/null || true; '\
+                'done; ' \
+                'for c in %s; do ' \
+                '    touch -h /var/lib/apparmor/clicks/${c}_*.json 2>/dev/null || true; ' \
+                'done; ' \
+                'aa-clickhook --include=/var/cache/apparmor/click-ap.rules' % (
+                    ' '.join(clickpkgs),
+                    self.user and ('--user ' + self.user) or '',
+                    self.scratch,
+                    ' '.join(installed_clicks))
+        else:
+            adtlog.info('Updating AppArmor rules to allow autopilot introspection for all clicks (will take a minute)...')
+            script += 'aa-clickhook --force --include=/var/cache/apparmor/click-ap.rules'
+
+        if self.execute(['sh', adtlog.verbosity >= 2 and '-exc' or '-ec', script], kind='install')[0] != 0:
+            self.bomb('Failed to update click AppArmor rules')
+
+        return True
+
+    def apparmor_restore_click(self, clickpkgs, installed_clicks):
+        '''Restore AppArmor rules after click tests'''
+
+        adtlog.info('Restoring click package AppArmor rules')
+        # if we only modified some clicks above, --force will be fast, so it's
+        # ok to always do that
+        script = 'rm -f /var/cache/apparmor/click-ap.rules; aa-clickhook --force'
+        if self.execute(['sh', adtlog.verbosity >= 2 and '-exc' or '-ec', script], kind='install')[0] != 0:
+            self.bomb('Failed to update click AppArmor rules')
+
+    def _run_apt_build_dep(self, what, prefix, recommends, ignorerc=False):
+        '''actually run apt-get build-dep'''
+
+        # capture status-fd to stderr
+        (rc, _, serr) = self.execute(['/bin/sh', '-ec', '%s apt-get build-dep '
+                                      '--assume-yes %s '
+                                      '-o APT::Status-Fd=3 '
+                                      '-o APT::Install-Recommends=%s '
+                                      '-o Dpkg::Options::=--force-confnew '
+                                      '-o Debug::pkgProblemResolver=true 3>&2 2>&1' %
+                                      (prefix, what, recommends)],
+                                     kind='install', stderr=subprocess.PIPE)
+        if not ignorerc and rc != 0:
+            adtlog.debug('apt-get build-dep %s failed; status-fd:\n%s' % (what, serr))
+            # check if apt failed during package download, which might be a
+            # transient error, so retry
+            if 'dlstatus:' in serr and 'pmstatus:' not in serr:
+                raise adtlog.AptDownloadError
+            else:
+                raise adtlog.AptPermanentError
+
+    def satisfy_build_deps(self, source_pkg, recommends=False,
+                           shell_on_failure=False):
+        '''Install build-dependencies into testbed with apt-get build-dep
+
+        This requires root privileges and a writable file system.
+        '''
+        # check if we can use apt-get
+        can_apt_get = False
+        if 'root-on-testbed' in self.caps:
+            rc = self.execute(['test', '-w', '/var/lib/dpkg/status'])[0]
+            if rc == 0:
+                can_apt_get = True
+        adtlog.debug('can use apt-get on testbed: %s' % can_apt_get)
+
+        if not can_apt_get:
+            self.bomb('no root on testbed, unsupported when specifying target architecture')
+
+        what = source_pkg
+        if self.cross_arch:
+            what += ' -a ' + self.cross_arch
+        # install the build-dependencies for the specified source (which can
+        # be a source package name, or a path); our apt pinning is not
+        # very clever wrt. resolving transitional dependencies in the pocket,
+        # so we might need to retry without pinning
+        download_fail_retries = 3
+        while True:
+            rc = 0
+            try:
+                self._run_apt_build_dep(what, ' '.join(self.eatmydata_prefix), recommends)
+
+            # check if apt failed during package download, which might be a
+            # transient error, so retry
+            except adtlog.AptDownloadError:
+                download_fail_retries -= 1
+                if download_fail_retries > 0:
+                    adtlog.warning('apt failed to download packages, retrying in 10s...')
+                    time.sleep(10)
+                    continue
+                else:
+                    self.bomb('apt repeatedly failed to download packages')
+
+            except adtlog.AptPermanentError:
+                rc = -1
+                if shell_on_failure:
+                    self.run_shell()
+
+            if rc != 0:
+                if self.apt_pin_for_releases and self.enable_apt_fallback:
+                    release = self.apt_pin_for_releases.pop()
+                    adtlog.warning('Test dependencies are unsatisfiable using apt pinning. '
+                                   'Retrying using all packages from %s' % release)
+                    self.check_exec(['/bin/sh', '-ec', 'rm /etc/apt/preferences.d/autopkgtest-' + release])
+                    if not self.apt_pin_for_releases:
+                        self.check_exec(['/bin/sh', '-ec', 'rm -f /etc/apt/preferences.d/autopkgtest-default-release'])
+                    continue
+
+                adtlog.warning('Test dependencies are unsatisfiable - calling '
+                               'apt install on test deps directly for further '
+                               'data about failing dependencies in test logs')
+                self._run_apt_build_dep('--simulate ' + what,
+                                        ' '.join(self.eatmydata_prefix),
+                                        recommends, ignorerc=True)
+
+                if shell_on_failure:
+                    self.run_shell()
+                if self.enable_apt_fallback:
+                    self.badpkg('Test dependencies are unsatisfiable. A common reason is '
+                                'that your testbed is out of date with respect to the '
+                                'archive, and you need to use a current testbed or run '
+                                'apt-get update or use -U.')
+                else:
+                    self.badpkg('Test dependencies are unsatisfiable. A common reason is '
+                                'that the requested apt pinning prevented dependencies '
+                                'from the non-default suite to be installed. In that '
+                                'case you need to add those dependencies to the pinning '
+                                'list.')
+            break
+
+    def satisfy_dependencies_string(self, deps, what, recommends=False,
+>>>>>>> lib/adt_testbed.py
                                     build_dep=False, shell_on_failure=False, synth_deps=[]):
         '''Install dependencies from a string into the testbed'''
 
@@ -1266,6 +1520,14 @@ Description: satisfy autopkgtest test dependencies
     # helper methods
     #
 
+    def _create_apt_pinning_for_pocket(self, release):
+        '''Create apt pinning for --apt-pocket=pocket[=pkglist] with
+           Pin-Priority: 500 to undo the effect of NotAutomatic: yes'''
+
+        script = 'printf "Package: *\\nPin: release a=%(release)s\\nPin-Priority: 500\\n" > /etc/apt/preferences.d/autopkgtest-%(release)s-baseline; ' % \
+            {'release': release}
+        self.check_exec(['sh', '-ec', script])
+
     def _create_apt_pinning_for_packages(self, release, pkglist, default_releases=[]):
         '''Create apt pinning for --apt-pocket=pocket=pkglist and
            --pin-packages=release=pkglist'''
@@ -1286,7 +1548,7 @@ Description: satisfy autopkgtest test dependencies
         # translate src:name entries into binaries of that source
         if srcpkgs:
             script += 'PKGS="$PKGS $(apt-cache showsrc %s | ' \
-                '''awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1; next } { show=0 }' |''' \
+                '''awk '/^Package-List:/ { show=1; next } (/^ / && show==1) { print $1 ":*"; next } { show=0 }' |''' \
                 '''sort -u | tr '\\n' ' ')"; ''' % \
                 ' '.join(srcpkgs)
 
@@ -1495,10 +1757,13 @@ def killtree(pid):
 
     for child in child_ps(pid):
         killtree(child)
+    with open('/proc/%s/cmdline' % pid, 'rb') as fd:
+        cmd = fd.read()
     try:
         os.kill(pid, signal.SIGTERM)
     except OSError:
         pass
+<<<<<<< lib/adt_testbed.py
 
 
 def locate_setup_command(name: str) -> str:
@@ -1510,3 +1775,21 @@ def locate_setup_command(name: str) -> str:
             return shipped
 
     return name
+=======
+    for _ in range(21):
+        if not os.path.exists('/proc/%s' % pid):
+            return
+        else:
+            time.sleep(0.2)
+    try:
+        with open('/proc/%s/cmdline' % pid, 'rb') as fd:
+            if cmd != fd.read():
+                return
+    except FileNotFoundError:
+        return
+    try:
+        adtlog.info('kill with SIGTERM did not work sending SIGKILL')
+        os.kill(pid, signal.SIGKILL)
+    except OSError:
+        pass
+>>>>>>> lib/adt_testbed.py
diff --git a/lib/adtlog.py b/lib/adtlog.py
index 4405c76..061f883 100644
--- a/lib/adtlog.py
+++ b/lib/adtlog.py
@@ -67,7 +67,17 @@ def log(message, level, prefix='', timestamp=False, color=None):
                 time.sleep(0.05)
             else:
                 raise
-    sys.stderr.buffer.flush()
+    retries = 10
+    while retries >= 0:
+        try:
+            sys.stderr.buffer.flush()
+            break
+        except IOError as e:
+            if e.errno == errno.EAGAIN:
+                retries -= 1
+                time.sleep(0.05)
+            else:
+                raise
 
 
 def error(message):
diff --git a/lib/autopkgtest_args.py b/lib/autopkgtest_args.py
index b55f683..0bcc018 100644
--- a/lib/autopkgtest_args.py
+++ b/lib/autopkgtest_args.py
@@ -186,6 +186,18 @@ for details.'''
     g_test.add_argument('-B', '--no-built-binaries', dest='built_binaries',
                         action='store_false', default=True,
                         help='do not build/use binaries from .dsc, git source, or unbuilt tree')
+<<<<<<< lib/autopkgtest_args.py
+=======
+    g_test.add_argument('--installed-click', metavar='CLICKNAME',
+                        help='Run tests from already installed click package '
+                        '(e. g. "com.example.myapp"), from specified click '
+                        'source directory or manifest\'s x-source.')
+    g_test.add_argument('-a', '--architecture', metavar='ARCH',
+                        help='run tests for (and when asked, build binaries '
+                        'for) ARCH instead of the testbed host architecture. '
+                        'Assumes ARCH is available as a foreign architecture '
+                        'on the testbed.')
+>>>>>>> lib/autopkgtest_args.py
     g_test.add_argument('packages', nargs='*',
                         help='testsrc source package and testbinary packages as above')
 
diff --git a/lib/testdesc.py b/lib/testdesc.py
index af3209f..cc0b9c8 100644
--- a/lib/testdesc.py
+++ b/lib/testdesc.py
@@ -283,7 +283,7 @@ def _debian_check_unknown_fields(name, record):
         raise Unsupported(name, 'unknown field %s' % unknown_keys.pop())
 
 
-def _debian_packages_from_source(srcdir):
+def _debian_packages_from_source(srcdir, cross_arch=None):
     packages = []
     packages_no_arch = []
 
@@ -296,10 +296,17 @@ def _debian_packages_from_source(srcdir):
                 st.get('Package-type', 'deb') != 'deb':
             continue
         arch = st['Architecture']
-        if arch in ('all', 'any'):
+        qual_pkg = st['Package']
+        # take care to emit an arch qualifier only for arch-dependent
+        # packages, not for arch: all ones
+        if cross_arch:
+            qual_pkg += ':' + cross_arch
+        if arch == 'all':
             packages.append(st['Package'])
+        elif arch == 'any':
+            packages.append(qual_pkg)
         else:
-            packages.append('%s [%s]' % (st['Package'], arch))
+            packages.append('%s [%s]' % (qual_pkg, arch))
         packages_no_arch.append(st['Package'])
 
     return (packages, packages_no_arch)
@@ -380,7 +387,7 @@ def _debian_build_deps_from_source(srcdir, testbed_arch):
     deps = [d.strip() for d in deps.split(',')]
 
     # @builddeps@ should always imply build-essential
-    deps.append('build-essential')
+    deps.append('build-essential:native')
     return deps
 
 
@@ -442,7 +449,8 @@ def _synthesize_deps(dep, testbed_arch):
         return None
 
 
-def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch):
+def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch,
+                          cross_arch=None):
     '''Parse Depends: line in a Debian package
 
     Split dependencies (comma separated), validate their syntax, and expand @,
@@ -453,7 +461,12 @@ def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch):
     '''
     deps = []
     synthdeps = []
-    (my_packages, my_packages_no_arch) = _debian_packages_from_source(srcdir)
+    (my_packages, my_packages_no_arch) = _debian_packages_from_source(srcdir,
+                                                                      cross_arch=cross_arch)
+    if cross_arch:
+        target_arch = cross_arch
+    else:
+        target_arch = testbed_arch
     for alt_group_str in dep_str.split(','):
         alt_group_str = alt_group_str.strip()
         if not alt_group_str:
@@ -464,17 +477,32 @@ def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch):
             for d in my_packages:
                 adtlog.debug('synthesised dependency %s' % d)
                 deps.append(d)
-                s = _synthesize_deps(d, testbed_arch)
+                s = _synthesize_deps(d, target_arch)
                 if s:
                     synthdeps.append(s)
         elif alt_group_str == '@builddeps@':
             for d in _debian_build_deps_from_source(srcdir, testbed_arch):
                 adtlog.debug('synthesised dependency %s' % d)
                 deps.append(d)
+<<<<<<< lib/testdesc.py
         elif alt_group_str == '@recommends@':
             for d in _debian_recommends_from_source(srcdir, testbed_arch):
                 adtlog.debug('synthesised dependency %s' % d)
                 deps.append(d)
+=======
+        elif alt_group_str == 'build-essential':
+            # special case; this is how packages declare they want to build
+            # code during the test (but not necessarily doing a full package
+            # build), but for cross-architecture testing this declaration is
+            # wrong because it tries to pull in build-essential for the wrong
+            # arch.  So we apply the same fix-up here that we do in
+            # runner/autopkgtest.  We can't expect the package maintainer to
+            # do this because they don't know what crossbuild-essential-$arch
+            # to pull in.
+            deps.append('build-essential:native')
+            if cross_arch:
+                deps.append('crossbuild-essential-%s:native' % cross_arch)
+>>>>>>> lib/testdesc.py
         else:
             synthdep_alternatives = []
             for dep in alt_group_str.split('|'):
@@ -482,8 +510,10 @@ def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch):
                 if pkg not in my_packages_no_arch:
                     synthdep_alternatives = []
                     break
-                s = _synthesize_deps(dep, testbed_arch)
+                s = _synthesize_deps(dep, target_arch)
                 if s:
+                    if cross_arch:
+                        s += ':' + cross_arch
                     synthdep_alternatives.append(s)
             if synthdep_alternatives:
                 adtlog.debug('marked alternatives %s as a synthesised dependency' % synthdep_alternatives)
@@ -491,6 +521,13 @@ def _parse_debian_depends(testname, dep_str, srcdir, testbed_arch):
                     synthdeps.append(synthdep_alternatives)
                 else:
                     synthdeps.append(synthdep_alternatives[0])
+            if cross_arch:
+                for mine in my_packages:
+                    # ignore [arch] filters for matching packages
+                    if alt_group_str + ':' + cross_arch == mine.split(' ')[0]:
+                        adtlog.debug('%s is from our source package, adding arch qualifier for cross-testing' % s)
+                        alt_group_str += ':' + cross_arch
+                        break
             deps.append(alt_group_str)
 
     return (deps, synthdeps)
@@ -582,8 +619,13 @@ def _check_architecture(name, testbed_arch, architectures):
 
 
 def parse_debian_source(srcdir, testbed_caps, testbed_arch, control_path=None,
+<<<<<<< lib/testdesc.py
                         auto_control=True, ignore_restrictions=(),
                         only_tests=()):
+=======
+                        auto_control=True, cross_arch=None, ignore_restrictions=(),
+                        testname=None):
+>>>>>>> lib/testdesc.py
     '''Parse test descriptions from a Debian DEP-8 source dir
 
     @ignore_restrictions: If we would skip the test due to these restrictions,
@@ -662,6 +704,11 @@ def parse_debian_source(srcdir, testbed_caps, testbed_arch, control_path=None,
             architectures = record.get('Architecture', '').replace(
                 ',', ' ').split()
 
+            if cross_arch:
+                target_arch = cross_arch
+            else:
+                target_arch = testbed_arch
+
             if 'Tests' in record:
                 test_names = record['Tests'].replace(',', ' ').split()
                 if len(test_names) == 0:
@@ -670,7 +717,7 @@ def parse_debian_source(srcdir, testbed_caps, testbed_arch, control_path=None,
                     test_names[0],
                     record.get('Depends', '@'),
                     srcdir,
-                    testbed_arch)
+                    testbed_arch, cross_arch=cross_arch)
                 if 'Test-command' in record:
                     raise InvalidControl('*', 'Only one of "Tests" or '
                                          '"Test-Command" may be given')
@@ -682,7 +729,7 @@ def parse_debian_source(srcdir, testbed_caps, testbed_arch, control_path=None,
                 for n in test_names:
                     try:
                         _debian_check_unknown_fields(n, record)
-                        _check_architecture(n, testbed_arch, architectures)
+                        _check_architecture(n, target_arch, architectures)
 
                         test = Test(n, os.path.join(test_dir, n), None,
                                     restrictions, features, depends, synth_depends)
@@ -699,14 +746,14 @@ def parse_debian_source(srcdir, testbed_caps, testbed_arch, control_path=None,
                     command,
                     record.get('Depends', '@'),
                     srcdir,
-                    testbed_arch)
+                    testbed_arch, cross_arch=cross_arch)
                 if feature_test_name is None:
                     command_counter += 1
                     name = 'command%i' % command_counter
                 else:
                     name = feature_test_name
                 _debian_check_unknown_fields(name, record)
-                _check_architecture(name, testbed_arch, architectures)
+                _check_architecture(name, target_arch, architectures)
                 test = Test(name, None, command, restrictions, features,
                             depends, synth_depends)
                 test.check_testbed_compat(testbed_caps, ignore_restrictions)
diff --git a/runner/autopkgtest b/runner/autopkgtest
index 9eb01fe..0ca7446 100755
--- a/runner/autopkgtest
+++ b/runner/autopkgtest
@@ -117,6 +117,8 @@ def setup_trace():
         adtlog.enable_colors = False
         os.dup2(out_tee.stdin.fileno(), sys.stdout.fileno())
         os.dup2(err_tee.stdin.fileno(), sys.stderr.fileno())
+        os.set_blocking(sys.stdout.fileno(), True)
+        os.set_blocking(sys.stderr.fileno(), True)
 
         def cleanup():
             os.close(sys.stdout.fileno())
@@ -160,7 +162,13 @@ def run_tests(tests, tree):
         doTest = True
 
         try:
+<<<<<<< runner/autopkgtest
             testbed.install_deps(t.depends, opts.shell_fail, t.synth_depends)
+=======
+            testbed.install_deps(t.depends,
+                                 'needs-recommends' in t.restrictions,
+                                 opts.shell_fail, t.synth_depends)
+>>>>>>> runner/autopkgtest
         except adtlog.BadPackageError as e:
             if 'skip-not-installable' in t.restrictions:
                 errorcode |= 2
@@ -392,6 +400,12 @@ def build_source(kind, arg, built_binaries):
         return tests_tree
 
     elif kind == 'apt-source':
+        # Make sure we are selecting the binaries based on the actual target
+        # architecture, not necessarily the testbed architecture
+        if opts.architecture:
+            arch = opts.architecture
+        else:
+            arch = testbed.dpkg_arch
         # The default is to determine the version for "apt-get source
         # pkg=version" that conforms to the current apt pinning. We only
         # consider binaries which are shipped in all available versions,
@@ -447,7 +461,7 @@ pkgs=$(echo "$pkgs\n" | awk "
     if (foundarch == 0 || archmatch == 1) thissrc[\\$1] = 1;
     next }
   { if (!inlist) next;
-    inlist=0;''' % {'src': arg, 'arch': testbed.dpkg_arch}
+    inlist=0;''' % {'src': arg, 'arch': arch}
 
         create_command_part2_check_all_pkgs = '''
     remaining=0;
@@ -609,24 +623,8 @@ dpkg-source -x %(src)s_*.dsc src >/dev/null''' % {'src': arg}
         if kind not in ['dsc', 'apt-source']:
             testbed.install_deps([])
 
-        if kind in ('apt-source', 'git-source'):
-            # we need to get the downloaded debian/control from the testbed, so
-            # that we can avoid calling "apt-get build-dep" and thus
-            # introducing a second mechanism for installing build deps
-            pkg_control = adt_testbed.Path(testbed,
-                                           os.path.join(tmp, 'apt-control'),
-                                           os.path.join(result_pwd, 'debian/control'), False)
-            pkg_control.copyup()
-            dsc = pkg_control.host
-
-        with open(dsc, encoding='UTF-8') as f:
-            d = deb822.Deb822(sequence=f)
-            bd = d.get('Build-Depends', '')
-            bdi = d.get('Build-Depends-Indep', '')
-            bda = d.get('Build-Depends-Arch', '')
-
         # determine build command and build-essential packages
-        build_essential = ['build-essential']
+        build_essential = ['build-essential:native']
         assert testbed.nproc
         dpkg_buildpackage = 'DEB_BUILD_OPTIONS="parallel=%s $DEB_BUILD_OPTIONS" dpkg-buildpackage -us -uc -b' % (
             opts.build_parallel or testbed.nproc)
@@ -636,8 +634,46 @@ dpkg-source -x %(src)s_*.dsc src >/dev/null''' % {'src': arg}
             if testbed.user or 'root-on-testbed' not in testbed.caps:
                 build_essential += ['fakeroot']
 
-        testbed.satisfy_dependencies_string(bd + ', ' + bdi + ', ' + bda + ', ' + ', '.join(build_essential), arg,
-                                            build_dep=True, shell_on_failure=opts.shell_fail)
+        if opts.architecture:
+            dpkg_buildpackage += ' -a' + opts.architecture
+            build_essential += ['crossbuild-essential-%s:native' % opts.architecture]
+            # apt-get build-dep is the best option here, but we don't call
+            # it unconditionally because it doesn't take a path to a source
+            # package as an option in very old releases; so for compatibility
+            # only use it when we need its multiarch build-dep resolution
+            # support.
+            # This is supported in apt 1.1~exp2 and newer, which covers
+            # Debian oldstable (at time of writing) and Ubuntu 16.04 and
+            # newer, so once we can rely on this version of apt everywhere,
+            # the old build-dep resolver code should be deprecated.
+            all_build_deps = ''
+            testbed.satisfy_build_deps(result_pwd,
+                                       shell_on_failure=opts.shell_fail)
+        else:
+            if kind in ('apt-source', 'git-source'):
+                # we need to get the downloaded debian/control from the
+                # testbed, so that we can avoid calling "apt-get build-dep"
+                # and thus introducing a second mechanism for installing
+                # build deps
+                pkg_control = adt_testbed.Path(testbed,
+                                               os.path.join(tmp, 'apt-control'),
+                                               os.path.join(result_pwd, 'debian/control'), False)
+                pkg_control.copyup()
+                dsc = pkg_control.host
+
+            with open(dsc, encoding='UTF-8') as f:
+                d = deb822.Deb822(sequence=f)
+                bd = d.get('Build-Depends', '')
+                bdi = d.get('Build-Depends-Indep', '')
+                bda = d.get('Build-Depends-Arch', '')
+
+            all_build_deps = bd + ', ' + bdi + ', ' + bda + ', '
+
+        testbed.satisfy_dependencies_string(all_build_deps +
+                                            ', '.join(build_essential),
+                                            arg,
+                                            build_dep=True,
+                                            shell_on_failure=opts.shell_fail)
 
         # keep patches applied for tests
         source_rules_command([dpkg_buildpackage, 'dpkg-source --before-build .'], 'build', cwd=result_pwd)
@@ -709,6 +745,7 @@ def process_actions():
             adtlog.debug('cleaning up previous tests tree %s on testbed' % tests_tree.tb)
             testbed.execute(['rm', '-rf', tests_tree.tb])
 
+<<<<<<< runner/autopkgtest
         tests_tree = build_source(kind, arg, built_binaries)
         try:
             (tests, skipped) = testdesc.parse_debian_source(
@@ -719,6 +756,56 @@ def process_actions():
                 only_tests=only_tests)
         except testdesc.InvalidControl as e:
             adtlog.badpkg(str(e))
+=======
+        if kind == 'click':
+            if control_override:
+                # locally specified manifest
+                with open(control_override) as f:
+                    manifest = f.read()
+                clicks = []
+                use_installed = False
+                if os.path.exists(arg):
+                    clicks.append(arg)
+                    use_installed = True
+                (srcdir, tests, skipped) = testdesc.parse_click_manifest(
+                    manifest, testbed.caps, clicks, use_installed, pending_click_source,
+                    opts.ignore_restrictions, testname=testname)
+
+            elif os.path.exists(arg):
+                # local .click package file
+                (srcdir, tests, skipped) = testdesc.parse_click(
+                    arg, testbed.caps, srcdir=pending_click_source)
+            else:
+                # already installed click package name
+                if testbed.user:
+                    u = ['--user', testbed.user]
+                else:
+                    u = []
+                manifest = testbed.check_exec(['click', 'info'] + u + [arg], stdout=True)
+                (srcdir, tests, skipped) = testdesc.parse_click_manifest(
+                    manifest, testbed.caps, [], True, pending_click_source,
+                    opts.ignore_restrictions, testname=testname)
+
+            if not srcdir:
+                adtlog.bomb('No click source available for %s' % arg)
+
+            tests_tree = adt_testbed.Path(
+                testbed, srcdir, os.path.join(testbed.scratch, 'tree'),
+                is_dir=True)
+            pending_click_source = None
+        else:
+            tests_tree = build_source(kind, arg, built_binaries)
+            try:
+                (tests, skipped) = testdesc.parse_debian_source(
+                    tests_tree.host, testbed.caps, testbed.dpkg_arch,
+                    control_path=control_override,
+                    auto_control=opts.auto_control,
+                    ignore_restrictions=opts.ignore_restrictions,
+                    testname=testname,
+                    cross_arch=opts.architecture)
+            except testdesc.InvalidControl as e:
+                adtlog.badpkg(str(e))
+>>>>>>> runner/autopkgtest
 
         if opts.validate:
             adtlog.report("*", "Test specification is valid")
@@ -768,6 +855,8 @@ def main():
     signal.signal(signal.SIGTERM, signal_handler)
     signal.signal(signal.SIGQUIT, signal_handler)
 
+    os.set_blocking(sys.stderr.fileno(), True)
+
     try:
         setup_trace()
         testbed = adt_testbed.Testbed(vserver_argv=vserver_args,
@@ -783,7 +872,8 @@ def main():
                                       add_apt_sources=getattr(opts, 'add_apt_sources', []),
                                       add_apt_releases=getattr(opts, 'add_apt_releases', []),
                                       pin_packages=opts.pin_packages,
-                                      apt_default_release=opts.apt_default_release)
+                                      apt_default_release=opts.apt_default_release,
+                                      cross_arch=opts.architecture)
         testbed.start()
         testbed.open()
         process_actions()
diff --git a/runner/autopkgtest.1 b/runner/autopkgtest.1
index 3fe795a..63c83e6 100644
--- a/runner/autopkgtest.1
+++ b/runner/autopkgtest.1
@@ -103,6 +103,13 @@ is implied.
 .SH TEST OPTIONS
 
 .TP
+.BR -a " ARCH" " | --architecture=" ARCH
+
+Run tests for the specified architecture, rather than for the host
+architecture as defined by dpkg \-\-print-architecture. When building
+packages from source, cross-build for the target architecture as well.
+
+.TP
 .BR -B " | " --no-built-binaries
 Binaries from unbuilt source packages (see above)
 will not be built or ignored, and dependencies are satisfied with packages from
@@ -262,7 +269,9 @@ that pocket to
 .B /etc/apt/sources.list.d/\fIpocket\fB.list\fR.
 This also calls
 .B apt-get update
-for the new pocket (but not for anything else).
+for the new pocket (but not for anything else). The pocket will be pinned with
+Pin-Priority: 500, so the "NotAutomatic: yes" setting will have no effect on
+the testbed system.
 
 If a package list is given after =, set up apt pinning to use only those
 packages from
diff --git a/setup-commands/setup-testbed b/setup-commands/setup-testbed
index 1b801d5..87645d6 100755
--- a/setup-commands/setup-testbed
+++ b/setup-commands/setup-testbed
@@ -315,6 +315,7 @@ fi
 
 # go-faster apt/dpkg
 echo "Acquire::Languages \"none\";" > "$root"/etc/apt/apt.conf.d/90nolanguages
+echo "Acquire::Retries \"10\";" > "$root"/etc/apt/apt.conf.d/90retry
 echo 'force-unsafe-io' > "$root"/etc/dpkg/dpkg.cfg.d/autopkgtest
 
 # avoid failures do to transient timeouts
@@ -427,10 +428,19 @@ if [ -x /sbin/init ]; then
         chroot "$root" apt-get install -y libpam-systemd </dev/null
     fi
 
+<<<<<<< setup-commands/setup-testbed
     # some tests use a lot of /dev/random, avoid hangs
     if ! systemd-detect-virt --quiet --container; then
         chroot "$root" apt-get install -y rng-tools </dev/null
     fi
+=======
+if ! systemd-detect-virt --quiet --container; then
+    chroot "$root" apt-get install -y haveged </dev/null
+fi
+
+if [ ! -e "$root/usr/share/doc/libpam-systemd" ] && chroot "$root" apt-cache show libpam-systemd >/dev/null 2>&1; then
+    chroot "$root" apt-get install -y libpam-systemd </dev/null
+>>>>>>> setup-commands/setup-testbed
 fi
 
 # optimization as we need to install it for most tests anyway
@@ -456,7 +466,11 @@ if [ -z "${AUTOPKGTEST_IS_SETUP_COMMAND:-}" ]; then
              unattended-upgrades update-notifier-common ureadahead debootstrap \
              lxcfs ppp pppconfig pppoeconf snapd snap-confine ubuntu-core-launcher \
              thermald xdg-user-dirs zerofree xml-core needrestart; do
+<<<<<<< setup-commands/setup-testbed
         if package_is_present "$p"; then
+=======
+        if [ -d "$root/usr/share/doc/$p" ]; then
+>>>>>>> setup-commands/setup-testbed
             purge_list="$purge_list $p"
         fi
     done
@@ -522,7 +536,10 @@ echo 'APT::Periodic::Enable "0";' > "$root/etc/apt/apt.conf.d/02periodic"
 
 # always include phased updates, so that the output is what we expect.
 echo 'APT::Get::Always-Include-Phased-Updates "true";' > "$root/etc/apt/apt.conf.d/90always-include-phased-updates"
+<<<<<<< setup-commands/setup-testbed
 
 if [ -n "$need_update_initramfs" ]; then
     chroot "$root" update-initramfs -u
 fi
+=======
+>>>>>>> setup-commands/setup-testbed
diff --git a/ssh-setup/nova b/ssh-setup/nova
index ad1c5f2..b828b1c 100755
--- a/ssh-setup/nova
+++ b/ssh-setup/nova
@@ -78,7 +78,7 @@ FLOATING_IP=""
 SECURITY_GROUPS=""
 EXTRA_ENV=""
 MIRROR=""
-NOVA_REBOOT=""
+NOVA_REBOOT="1"
 
 DEBUG=""
 
@@ -227,6 +227,9 @@ EOF
     imageuuid=$(echo "$last" | cut -f1)
     image=$(echo "$last" | cut -f2)
 
+    echo "Waiting..." >&2
+    flock -x /home/ubuntu/$OS_REGION_NAME.boot.lock sleep 30
+
     # Boot a nova instance and returns its connection parameters
     [ -n "$SRVNAME" ] || SRVNAME=`mktemp -u autopkgtest-nova-XXXXXX`
     echo "Creating nova instance $SRVNAME from image ${image} (UUID $imageuuid)..." >&2
@@ -253,7 +256,7 @@ EOF
     EXTRA_OPTS=''
     if [ -n "$NET_ID" ]; then
         # translate a name into a UUID
-        OUT=$(nova network-show $NET_ID)
+        OUT=$(openstack network show $NET_ID 2>/dev/null)
         NET_ID="$(echo "$OUT"| awk -F'|' '/ id / {gsub(" ", "", $3); print $3}')"
         EXTRA_OPTS="$EXTRA_OPTS --nic net-id=$NET_ID"
     fi
@@ -315,7 +318,7 @@ EOF
             OUT=$(nova show --minimal $SRVUUID)
             ipaddr=$(echo "$OUT" | awk 'BEGIN {FS="|"} $2 ~ /network/ {n=split($3,i,/,\s*/); gsub(" ", "", i[n]); print i[n]}')
             retry=$(( retry - 1 ))
-            if [ $retry -le 0 ]; then
+            if [ $retry -le 0 ] || echo "$OUT" | grep -q 'network\s*|.*,'; then
                 error "Failed to acquire an IP address. Aborting!"
                 error "$OUT"
                 cleanup
diff --git a/tests/autopkgtest b/tests/autopkgtest
index 985d07b..95ef393 100755
--- a/tests/autopkgtest
+++ b/tests/autopkgtest
@@ -2256,7 +2256,7 @@ Restrictions: needs-root
         self.assertIn('synthesised dependency bdep4\n', err)
         self.assertIn('synthesised dependency bdep5\n', err)
         self.assertIn('synthesised dependency bdep6\n', err)
-        self.assertIn('synthesised dependency build-essential\n', err)
+        self.assertIn('synthesised dependency build-essential:native\n', err)
         self.assertIn('processing dependency testdep2\n', err)
 
     def test_build_deps_profiles(self):
@@ -2276,7 +2276,7 @@ Restrictions: needs-root
         self.assertRegex(out, r'pass\s+PASS')
 
         self.assertIn('synthesised dependency bdepyes\n', err)
-        self.assertIn('synthesised dependency build-essential\n', err)
+        self.assertIn('synthesised dependency build-essential:native\n', err)
 
         dpkg_deps_ver = subprocess.check_output(['perl', '-MDpkg::Deps', '-e', 'print $Dpkg::Deps::VERSION'],
                                                 universal_newlines=True)
diff --git a/tests/testdesc b/tests/testdesc
index 236df41..48e29c4 100755
--- a/tests/testdesc
+++ b/tests/testdesc
@@ -401,7 +401,7 @@ Recommends: ${package:one}, two (<= 10) <!nocheck>, three (<= ${ver:three}~), fo
             'Package: one\nArchitecture: any')
         self.assertEqual(ts[0].depends, ['one', 'bd1', 'bd3:native (>= 7) | bd4',
                                          'bdi1', 'bdi2', 'bda1', 'bda2',
-                                         'build-essential', 'foo (>= 7)'])
+                                         'build-essential:native', 'foo (>= 7)'])
         self.assertFalse(skipped)
 
     @unittest.skipUnless(have_dpkg_build_profiles,
@@ -414,7 +414,7 @@ Recommends: ${package:one}, two (<= 10) <!nocheck>, three (<= ${ver:three}~), fo
             'Source: nums\nBuild-Depends: bd1, bd2 <!check>, bd3 <!cross>, bdnotme <stage1> <cross>\n'
             '\n'
             'Package: one\nArchitecture: any')
-        self.assertEqual(ts[0].depends, ['one', 'bd1', 'bd2', 'bd3', 'build-essential'])
+        self.assertEqual(ts[0].depends, ['one', 'bd1', 'bd2', 'bd3', 'build-essential:native'])
         self.assertFalse(skipped)
 
     def test_complex_deps(self):
@@ -463,7 +463,7 @@ Recommends: ${package:one}, two (<= 10) <!nocheck>, three (<= ${ver:three}~), fo
             ' , bd2\n'
             '\n'
             'Package: one\nArchitecture: any')
-        self.assertEqual(ts[0].depends, ['one', 'bd1', 'bd2', 'build-essential'])
+        self.assertEqual(ts[0].depends, ['one', 'bd1', 'bd2', 'build-essential:native'])
         self.assertFalse(skipped)
 
     @patch('adtlog.report')
diff --git a/tools/autopkgtest-build-lxd b/tools/autopkgtest-build-lxd
index 14601db..a0a3c88 100755
--- a/tools/autopkgtest-build-lxd
+++ b/tools/autopkgtest-build-lxd
@@ -69,6 +69,12 @@ setup() {
     fi
 
     sleep 5
+
+    # Workaround for LP: #1952084
+    # Drop if fixed or if we switch to minimal images in armhf-lxd.userdata.
+    # This has to go before the "we get a numeric runlevel" check below.
+    lxc exec "$CONTAINER" -- systemctl mask snapd.seeded.service --now
+
     if lxc exec "$CONTAINER" -- systemctl mask [e-mail]; then
        lxc exec "$CONTAINER" -- reboot
     fi
@@ -86,6 +92,10 @@ setup() {
         exit 1
     }
 
+    # Workaround for LP: #1989603
+    # Drop if fixed or if we switch to minimal images in armhf-lxd.userdata.
+    lxc exec "$CONTAINER" -- umount /var/snap/lxd/common/var/lib/lxcfs || true
+
     # wait until a systemd based container has networking
     if ! echo '[ ! -d /run/systemd/system ] || systemctl start network-online.target' | timeout 60 lxc exec "$CONTAINER" -- sh -e; then
         echo "Timed out waiting for container to start network-online.target" >&2
diff --git a/virt/autopkgtest-virt-lxc b/virt/autopkgtest-virt-lxc
index 61e7eed..015ac4f 100755
--- a/virt/autopkgtest-virt-lxc
+++ b/virt/autopkgtest-virt-lxc
@@ -303,7 +303,11 @@ def hook_revert():
     hook_open()
 
 
+<<<<<<< virt/autopkgtest-virt-lxc
 def hook_wait_reboot(*func_args, **kwargs):
+=======
+def hook_wait_reboot(*args, **kwargs):
+>>>>>>> virt/autopkgtest-virt-lxc
     adtlog.debug('hook_wait_reboot: waiting for container to shut down...')
     VirtSubproc.execute_timeout(None, 65, sudoify(
         ['lxc-wait', '-n', lxc_container_name, '-s', 'STOPPED', '-t', '60']))
diff --git a/virt/autopkgtest-virt-lxd b/virt/autopkgtest-virt-lxd
index 373f2ed..cb4a10b 100755
--- a/virt/autopkgtest-virt-lxd
+++ b/virt/autopkgtest-virt-lxd
@@ -81,7 +81,7 @@ def get_available_container_name():
         rnd = [random.choice(string.ascii_lowercase) for i in range(6)]
         candidate = 'autopkgtest-lxd-' + ''.join(rnd)
 
-        rc = VirtSubproc.execute_timeout(None, 10, ['lxc', 'info', candidate],
+        rc = VirtSubproc.execute_timeout(None, 100, ['lxc', 'info', candidate],
                                          stdout=subprocess.DEVNULL,
                                          stderr=subprocess.STDOUT)[0]
         if rc != 0:
@@ -89,7 +89,35 @@ def get_available_container_name():
 
 
 def wait_booted():
+<<<<<<< virt/autopkgtest-virt-lxd
     VirtSubproc.wait_booted(['lxc', 'exec', container_name, '--'], timeout=180)
+=======
+    '''Wait until the container has sufficiently booted to interact with it
+
+    Do this by checking that the runlevel is someting numeric, i. e. not
+    "unknown" or "S".
+    '''
+    timeout = 60
+    while timeout > 0:
+        timeout -= 1
+        time.sleep(1)
+        (rc, out, _) = VirtSubproc.execute_timeout(
+            None, 100, ['lxc', 'exec', container_name, 'runlevel'],
+            stdout=subprocess.PIPE)
+        if rc != 0:
+            adtlog.debug('wait_booted: lxc exec failed, retrying...')
+            continue
+        out = out.strip()
+        if out.split()[-1].isdigit():
+            adtlog.debug('waiting for network')
+            VirtSubproc.check_exec(['lxc', 'exec', container_name, '--', 'sh', '-ec', r'if [ -d /run/systemd/system ]; then systemctl start network-online.target; else while ps -ef | grep -q "/etc/init\.d/rc"; do sleep 1; done; fi'], timeout=60)
+            return
+
+        adtlog.debug('wait_booted: runlevel "%s", retrying...' % out)
+
+    VirtSubproc.bomb('timed out waiting for container %s to start; '
+                     'last runlevel "%s"' % (container_name, out))
+>>>>>>> virt/autopkgtest-virt-lxd
 
 
 def determine_normal_user():
@@ -102,7 +130,7 @@ def determine_normal_user():
     cmd = ['lxc', 'exec', container_name, '--', 'sh', '-c',
            'getent passwd | sort -t: -nk3 | '
            "awk -F: '{if ($3 >= 1000 && $3 <= 59999) { print $1; exit } }'"]
-    out = VirtSubproc.execute_timeout(None, 10, cmd,
+    out = VirtSubproc.execute_timeout(None, 100, cmd,
                                       stdout=subprocess.PIPE)[1].strip()
     if out:
         normal_user = out
@@ -184,7 +212,7 @@ def hook_revert():
 def get_uptime():
     try:
         (rc, out, _) = VirtSubproc.execute_timeout(
-            None, 10, ['lxc', 'exec', container_name, '--', 'cat', '/proc/uptime'],
+            None, 100, ['lxc', 'exec', container_name, '--', 'cat', '/proc/uptime'],
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
         if rc != 0:
diff --git a/virt/autopkgtest-virt-qemu b/virt/autopkgtest-virt-qemu
index 2b5b3fe..838c076 100755
--- a/virt/autopkgtest-virt-qemu
+++ b/virt/autopkgtest-virt-qemu
@@ -641,10 +641,14 @@ def hook_prepare_reboot() -> None:
         monitor.close()
 
 
+<<<<<<< virt/autopkgtest-virt-qemu
 def hook_wait_reboot(*func_args: Any, **kwargs: Any) -> None:
     assert args is not None
     assert qemu is not None
 
+=======
+def hook_wait_reboot(*func_args, **kwargs):
+>>>>>>> virt/autopkgtest-virt-qemu
     os.unlink(os.path.join(qemu.shareddir, 'done_shared'))
     wait_boot()
     tty = setup_shell()
diff --git a/virt/autopkgtest-virt-ssh b/virt/autopkgtest-virt-ssh
index 186ebe6..230bb74 100755
--- a/virt/autopkgtest-virt-ssh
+++ b/virt/autopkgtest-virt-ssh
@@ -29,6 +29,7 @@
 import sys
 import os
 import argparse
+import errno
 import tempfile
 import shlex
 import shutil
@@ -58,6 +59,7 @@ sshconfig = {'identity': None,
 # Note: Running in jenkins might require -tt
 sshopts = '-q -o BatchMode=yes -o UserKnownHostsFile=/dev/null '\
           '-o StrictHostKeyChecking=no -o CheckHostIP=no '\
+          '-o ConnectionAttempts=20 '\
           '-o ControlMaster=auto -o ControlPersist=60 '\
           '-o ControlPath=%s/ssh_control-%%r@%%h:%%p'
 
@@ -205,7 +207,17 @@ def execute_setup_script(command, fail_ok=False, print_stderr=True):
             stderr=subprocess.PIPE)
         if print_stderr:
             # Keep outputting the error on stderr as well as capturing it
-            sys.stderr.write(err)
+            retries = 10
+            while retries >= 0:
+                try:
+                    sys.stderr.write(err)
+                    break
+                except IOError as e:
+                    if e.errno == errno.EAGAIN:
+                        retries -= 1
+                        time.sleep(0.05)
+                    else:
+                        raise
         if status != 0:
             err = 'setup script failed with code %i: %s' % (status,
                                                             ' '.join(cmd))
@@ -346,16 +358,16 @@ def build_auxverb():
     # create remote wrapper
     rc = VirtSubproc.execute_timeout(
         terminal_kill_wrapper % extra_cmd, 30, sshcmd +
-        ['rm -f /tmp/autopkgtest-run-wrapper; set -C; cat > /tmp/autopkgtest-run-wrapper; chmod 755 /tmp/autopkgtest-run-wrapper'],
+        ['rm -f /var/tmp/autopkgtest-run-wrapper; set -C; cat > /var/tmp/autopkgtest-run-wrapper; chmod 755 /var/tmp/autopkgtest-run-wrapper'],
         stdin=subprocess.PIPE)[0]
     if rc != 0:
-        VirtSubproc.bomb('Failed to create /tmp/autopkgtest-run-wrapper')
+        VirtSubproc.bomb('Failed to create /var/tmp/autopkgtest-run-wrapper')
 
     # create local auxverb script
     auxverb = os.path.join(workdir, 'runcmd')
     with open(auxverb, 'w') as f:
         f.write('''#!/bin/bash
-exec %s -- %s /tmp/autopkgtest-run-wrapper $(printf '%%q ' "${@%% }")
+exec %s -- %s /var/tmp/autopkgtest-run-wrapper $(printf '%%q ' "${@%% }")
 ''' % (" ".join(sshcmd), sudocmd or ''))
     os.chmod(auxverb, 0o755)
     VirtSubproc.auxverb = [auxverb]
@@ -517,6 +529,7 @@ def hook_cleanup():
 def hook_capabilities():
     return capabilities
 
-
+# Somehow it ends up non-blocking
+os.set_blocking(sys.stderr.fileno(), True)
 parse_args()
 VirtSubproc.main()