Author: Steve Langasek
Author Date: 2021-03-03 19:21:01 UTC

unblacklist rust-onig, https://bugs.debian.org/973657 is fixed in the package

Author: Iain Lane
Author Date: 2019-06-12 11:50:24 UTC

WIP: Move to Mojo & Juju 2 reactive charms

The main aim of this is to reduce the amount of direct administration
that admins perform on the machine itself.

All configuration changes should now be performed by editing Juju
configuration, by editing the Mojo 'service-bundle' file.

autopkgtest-web is now a subordinate of the standard apache2 charm, and
uses its interface. Multiple units can be deployed. The shipped
configuration will deploy 2 backend units and a haproxy unit to act as
the gateway.

Another highlight (which might move to a separate commit) is that web
updates are now handled much faster. The worker submits a message to
AMQP when it completes a job, and web units pick that up immediately.
(The 'live' database is updated every minute.)

The race condition when adding new LXD worker machines and restaring the
controller instances is fixed. The charm now writes a systemd drop-in
with the appropriate Wants=/After= to ensure the corresponding socat is
started up.

cloud worker units now ensure that they have an image available before
starting up, also via a drop-in to a new build-adt-image@ service. There
is also a build-adt-image@ timer to regenerate these daily.

All of the cron jobs have moved to systemd timers.

Author: Iain Lane
Author Date: 2020-11-16 11:40:03 UTC

worker: Properly retry on failures we think might be temporary

We're seeing a test run currently looping with this trace:

WARNING: Saw Temporary failure resolving in log, which is a sign of a temporary failure.
WARNING: Retrying in 5 minutes. Log follows:
[ ... log ... ]
gzip: /tmp/autopkgtest-work.j5qth004/out/log: No such file or directory
Traceback (most recent call last):
  [ ... cut some bits of the trace ... ]
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 645, in request
    process_output_dir(out_dir, pkgname, code)
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 172, in process_output_dir
    subprocess.check_call(['gzip', '-9', os.path.join(dir, 'log')])
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gzip', '-9', '/tmp/autopkgtest-work.j5qth004/out/log']' returned non-zero exit status 1

But we should not be calling process_output_dir() when we're retrying.
That is to be called when we are about to upload the directory to swift.

What's happening is, we have this logic:

  for retry in range(3):
     <run the test>
     <did it permanently fail?> { /* 1 */
             <grep the log, to see if we think this might be transient>
             <break if not, otherwise print a warning, *delete the
              output directory* and retry>
     }

     <did it temporarily fail?> { /* 2 */
             <grep the log, to see if we think this might be permanent>
             <print a warning, delete the output directory and retry if
              not, otherwise break>
     } else { /* 3, passed */
             <break, no more retries, upload the result>
     }

We think it might be transient, so we clean up the output directory and
try to retry. But since we have two *separate* if statement here, the
second's else clause is entered - which is supposed to be the case for
if the run has passed cleanly - and we break out the loop, then go on to
try to upload the result. This fails, because we cleaned up the
directory.

Instead, we should have one if statement here. If we enter the first
case, for 'permanent' failures, we should never go on to enter any of
the others.

Author: Steve Langasek
Author Date: 2019-12-10 16:47:00 UTC

Run i386 autopkgtests on amd64 cloud images

This requires changing our templating engine to know the difference between
the testbed arch and the test arch.

Author: Steve Langasek
Author Date: 2019-05-18 20:05:07 UTC

Purge systemd journal after generating our base images

Now that we ensure daily up-to-date base images by upgrading the previous
one when no generic image is available, we might wind up continuously
upgrading the same image for a very long time (e.g. right now, when there
is no generic eoan image published yet after 3 weeks). This leads to an
increasingly cluttered filesystem over time, specifically with respect to
the systemd journal growing over time.

This was noticed because of a failing test in systemd itself that dumps the
complete systemd journal in the logs. It's possible the journal bloat is
the cause of the test failure, but even if it isn't, we're better off not
carrying this bloat.

This is not the only source of logs that would increase over time in these
images - e.g. apt and dpkg logs will also grow - but those should be
ignorable for now.

Author: Iain Lane
Author Date: 2019-04-03 16:11:41 UTC

WIP: worker: Support 'private' tests

If private is passed as a parameter, don't send status updates and hide
queue items.

TODO: How to receive requests from a private PPA?
      Make private PPAs automatically imply a private job?
      ACL for the resulting swift container (then how are the results
      gathered? up to the consumer?)

Author: Iain Lane
Author Date: 2019-02-28 09:39:25 UTC

download-results: Retry a few times when downloading the result tarball

We're seeing 404 errors occasionally. It seems that when the underlying
swift store is performing a rebalance objects can go missing for a
little while. Let's give them a chance to pop back up.

Author: Steve Langasek
Author Date: 2019-02-23 03:48:08 UTC

Bail early if the cloud image rebuild script is passed an invalid argument

When trying to manually rerun this script to fix performance problems caused
by stale images, we instead managed to generate broken images because
nothing minds if the argument passed as $1 doesn't exist on the filesystem.
This should be treated as an error early to guard against human error.

Author: Steve Langasek
Author Date: 2019-02-22 07:21:41 UTC

Always regenerate adt cloud images daily.

Don't make image regeneration conditional on whether there is a new imported
image. This allows us to remain current even in the face of problems with
the cloud image production pipeline, preventing us unnecessarily paying a
per-test startup cost as the system gets dist-upgraded regardless.

Author: Steve Langasek
Author Date: 2019-01-17 13:03:05 UTC

Update armhf lxd slave metadata for bionic.

Don't try to install lxd from xenial-backports on a bionic system.

Author: Łukasz Zemczak
Author Date: 2018-05-01 13:21:29 UTC

Propagate the requester information to the environment of the runner.

Author: Łukasz Zemczak
Author Date: 2018-04-30 11:49:02 UTC

Fix typo, add per-series logging for build-adt-image-all-clouds.

Author: Iain Lane
Author Date: 2018-03-08 11:18:34 UTC

Use lxd from xenial-backports, and configure MTU/DNS/network appropriately

We're fixing a bunch of configuration problems here, which hopefully
should add up to DNS working reliably enough for snapcraft to finish its
tests.

Author: Steve Langasek
Author Date: 2017-12-18 21:30:22 UTC

run tools/build-adt-image-all-clouds for all supported releases by default, not just the devel release

Author: Steve Langasek
Author Date: 2017-10-25 05:38:44 UTC

Support using the last image from the latest stable release for series opening