lp:~hyask/autopkgtest-cloud

Author: Skia
Author Date: 2024-12-18 15:24:46 UTC

feat: cloud-worker: support riscv64

This is all that was necessary to support the `riscv64` architecture in
the autopkgtest-cloud infrastructure.

We won't support `riscv64` on releases older than Jammy so this
implements the necessary charm rules to have the systemd units correctly
setup for that situation.

Also, the existing images coming from IS (auto-sync/ubuntu-....img)
don't always have the right OpenStack properties by default, and our
credentials don't have the rights to tweak those properties ourselve.
That means the kickstart involves something like the following:
  - openstack image save --file noble.qcow2 auto-sync/ubuntu-noble-24.10-riscv64-server-20241218-disk1.img
    This save a local file called `noble.qcow2`, for subsequent upload.
  - openstack image create --file noble.qcow2 --disk-format qcow2 --property hw_emulation_architecture=riscv64 --property hw_machine_type=virt --property hw_video_model=virtio --property kernel_id=61d82457-f888-4999-9c4c-687cbd135367 adt/ubuntu-noble-riscv64-server-19700101.img
    This uploads the local file to a new image controlled by us, with the right properties.
    The kernel_id property is an OpenStack image ID that contains QEMU's
    uboot part. Check with the riscv64 experts if you need to mess with
    that. For this time, there was an existing `noble-qemu-riscv64-uboot`
    image that just worked™.
With the previous kickstart, `build-adt-image` with this commit's tweak
is able to successfully prepare an up-to-date base image.

Author: Skia
Author Date: 2024-11-29 14:55:23 UTC

fix: worker: use new ppa interface of autopkgtest

Autopkgtest 5.40 brought the following change:
  * Implement support for adding PPAs via --add-apt-source=ppa:OWNER/PPA
This change makes use of that new interface, because for some reason
the old way is broken, and this was just faster to implement the new way
than debug the old one. This simplified the code a lot along the way,
because the heavy-lifting is now done by autopkgtest.
In the corner-case where a test request specifies a private PPA without
giving the fingerprint, the error is now cleanly written by autopkgtest,
so no need for the worker to check that case and log a line that nobody
will every see: the developer will have access to the error in the
autopkgtest log.

Author: Skia
Author Date: 2024-11-21 15:14:54 UTC

fix: web: check membership including sub-teams

When submitting a test request, if user is not a direct uploader of
the package, the code checks membership in a defined set of teams.
Until now, this check would only include direct team membership, but
Launchpad's model allows teams to be part of other teams, meaning people
can be indirectly member of a team. This situation is actually very
common, and users are used to it, meaning it should be supported by
autopkgtest-cloud.

Author: Skia
Author Date: 2024-11-20 16:41:52 UTC

mojo: fix staging flavors and network names

Author: Skia
Author Date: 2024-09-26 10:22:59 UTC

refactor: worker: change to depend on 'python3-amqp'

For all scripts:
- `python3-amqplib` is deprecated, replaced by `python3-amqp`.

For worker:
- The amqp `access_request` API was completely deprecated, and didn't
  seem required anyway.

For filter-amqp:
- Make it work with RabbitMQ creds from the environment too.
- Use `argparse` instead of the deprecated `optparse`.
- Fix the deletion_count_history growing indefinitely.

For filter-amqp-dupes-upstream:
- Make it work with RabbitMQ creds from the environment too.
- Use `argparse` instead of the deprecated `optparse`.
- Fix AMQP message spliting on '\n' instead of ' '.

For pull/push-amqp:
- Make those work with RabbitMQ creds from the environment too.

For run-autopkgtest:
- Modernize the datetime usage to remove a Python warning.

For with-distributed-lock:
- Fix the wrong usage of 'x-single-active-consumer', making the whole
  script useless because it did not work.

Author: Skia
Author Date: 2024-11-20 13:53:25 UTC

doc: explain how to compute the quota we need

Author: Skia
Author Date: 2024-10-21 11:10:29 UTC

fix: web: publish-db: repair use of url_last_checked

Due to the table not being copied correctly, the script would spend
10 seconds every run updating everything. With the timestamp correctly
copied, most runs are now actually saved from processing the Sources.gz.
This is roughly a 30% time saving on the production machines, going from
about 30s for each run to 20s.

Author: Brian Murray
Author Date: 2024-07-11 20:45:35 UTC

webcontrol: use a space between triggers in Exceptions

Author: Skia
Author Date: 2024-10-01 11:23:57 UTC

mojo: upgrade HAProxy to run on Jammy

Following cc2beca6f45c9fa292af4e492e239fc32d763cf7, the `service-bundle`
configuration required having a more up to date version of HAProxy.
Indeed, the `rsprep` and `reqrep` got deprecated in HAProxy 2.1 [1], and
`focal` is shipping 2.0 while `jammy` has 2.4.
The production unit has been upgraded in place, but this needs to be
reflected in the `service-bundle` for future deployments.
Also fix the configuration to remove all warnings and errors (correct
use of http-request, drop of useless http-response, and right ordering
of options).

[1]: https://www.haproxy.com/blog/haproxy-2-1#deprecated-configuration-options

Author: Skia
Author Date: 2024-09-24 09:16:52 UTC

fix: github: distinguish on 'testname' too

For upstream tests triggered with a `testname`, `update-github-jobs`
would sometimes confuse some results because of the lack of distinction
on `testname`. This makes the worker to always add `testname` to
`testinfo.json`, and `update-github-jobs` to check on that key.
`testname` can be `null` when not specified, but the distinction still
happens (`testname` was simply not specified).

Author: Skia
Author Date: 2024-08-07 07:29:25 UTC

feat: worker: make local development easier

This mostly sets working defaults, and documents the process to go from
`git clone` to having a local worker able to run some jobs.

Changes:
- Allow working without swift, in which case it keeps the local results.
- Make the delay between retries configurable instead of hardcoded 5min.
- Make the systemd log handler to only activate when run by systemd.
- Overhaul the default `worker.conf` to have sane development defaults and comments.
- Add a README next to the worker to explain how to run it.
- Embed the default RabbitMQ credentials for easier use.

Author: Skia
Author Date: 2024-09-18 10:27:26 UTC

fix: web: fix Github integration to take into account the 'testname'

Author: Skia
Author Date: 2024-09-13 10:44:29 UTC

feat: web: improve context sent back to Github

For upstream project integration, the reply to Github gets displayed on
the Pull Request interface. Improving the 'context' parameter here to
include the `testname` when one was specified, to help differenciate the
various jobs when multiple are running in parallel.

Author: Skia
Author Date: 2024-09-11 13:03:04 UTC

fix: add a special iptables rule to make 'rust-reqwest' to pass

LP: #2080346

These IP addresses are supposed to be unreachable in most networks. For
some reason they are routed differently on Canonical's infrastructure
and sent to the proxy, which makes some tests to fail because they
expect a timeout instead when trying to reach the IPs.

Blocking these addresses might have unintended side effects since
a quick search in Debian Code Search shows that many packages uses
10.255.255.1 in their test suite. However, it seems to me that most
packages indeed expect it to be unreachable, so I guess making that a
reality should in fact help more than one package, but it's difficult to
be certain that it won't break one particular package somewhere.

Author: Skia
Author Date: 2024-08-13 13:31:01 UTC

feat: web charm: use code from a git repo

See previous commit for the context. This is the same thing, but for the
web part of autopkgtest-cloud.

Also, the action `update-sources` is here an addition, as there were no
actions at all in this charm.

Author: Skia
Author Date: 2024-08-16 09:07:52 UTC

fix: web: fix IndexError on user page

In case a `private job` is queued, the user page failed to parse it and
ended up erroring.

Author: Skia
Author Date: 2024-01-25 12:25:10 UTC

c-w/tools: OS_REGION_NAME may not exist in devel

Author: Skia
Author Date: 2024-07-29 16:36:37 UTC

fix: worker: pin LXD to 5.21 branch

Following the release of LXD 6.1, and the removal of
`core.trust_password`, we need to rethink how we add the LXD remote.
In the meantime, pinning to the 5.21 branch is the easiest way to keep
being able to replace our workers.
See #2074381 for more details.

Author: Skia
Author Date: 2024-07-16 12:25:56 UTC

doc/administration: improve the EoL section

Author: Skia
Author Date: 2024-07-24 13:40:43 UTC

fix: web: improve exception handling on the custom page

werkzeug default HTTPException have a code that is best to use. Prior
to that, all exception outside of our own would return a 500. Even, for
example, when it's an obvious routing error that should show 404.

Author: Skia
Author Date: 2024-07-24 10:45:08 UTC

feat: web: implement some memoization in 'update-github-jobs'

Since we don't seem to have a choice in looping over all the results in
the container to find the right one, at least memoize these results and
their processing for much faster run.

Improvement are very noticeable:
before: about 2min15s of runtime for about 70MB of peak RAM usage
after: about 30s of runtime for about 75MB of peak RAM usage
That's more or less a x4 in speed for a barely visible memory increase.

`result_cache` grows to about 2500 items, which isn't much to store, but
is apparently long to compute and get from the network.

Author: Skia
Author Date: 2024-07-15 16:33:22 UTC

fix: worker: charm: fix 'NameError' with `region_arch_conf`

Full error was: `NameError: free variable 'region_arch_conf' referenced before assignment in enclosing scope`.
This happened only on the LXD worker, because 'region_arch_conf' isn't
necessary for it.

Author: Skia
Author Date: 2024-07-11 16:13:46 UTC

fix: web: stop eating tracebacks and print them

When an exception occured anywhere in the web code, the global exception
handler would catch it and display a nice error page to the user, but
then the traceback would be eaten by this process, and we end up unable
to do anything about this.
This fixes the issue by both printing the traceback on the error page,
so that users can report them, but also print the traceback on stderr,
so that it also ends up in the logs.

Author: Skia
Author Date: 2024-07-03 11:30:47 UTC

docs: add a section about blackholing harmful IP ranges

Author: Skia
Author Date: 2024-07-02 13:27:50 UTC

fix: web: package template was broken for some reason

Author: Skia
Author Date: 2024-06-27 10:23:51 UTC

mojo: replace lxd-armhf-bos03-06

Author: Skia
Author Date: 2024-06-27 08:56:44 UTC

fix: web: close channel to free resources

In case an exception in thrown, the channel would be left opened. If
this happens in a loop, that script can quickly have a big number of
dangling channels. I've observed more than 2k in a single night.

See previous commit for additional context.

Author: Skia
Author Date: 2024-06-26 10:57:30 UTC

fix: worker: wait for server deletion

See LP #2067628.

The goal is to ensure the worker always waits for server deletion before
taking the next job, to avoid over quotas.

The `nova` script called to spawn the instances is already taking care
of normal clean up and is already waiting for instance deletion before
handing back to the worker, but that is only true for normal operation.
When the worker needs to exit earlier, nothing was forcing it to wait
the deletion.

Author: Skia
Author Date: 2024-06-19 15:42:28 UTC

feat: web: improve running page

* Self document that the information is refreshed every 30 seconds.
* Reorder the queue length section and the running packages one so that
  we very quickly have the info of the queue length, without needing to
  scroll down when too much packages are running.
* Some semantic improvements (h2 vs h3)

Author: Skia
Author Date: 2024-06-19 14:52:33 UTC

stats: show basic stats on the database

Showing those when connecting to the database gives the analyst some
indsight on what data is available to analyze.

Author: Skia
Author Date: 2024-06-19 11:49:13 UTC

feat: web: improve logs viewer

- better parse each test section:
  - preparing testbed
  - test run
  - test results
- icon is now the Ubuntu logo, without a request to Google

Author: Skia
Author Date: 2024-06-06 13:28:12 UTC

feat: web: adding some tools to analyze log files

This adds two things:
* stats.py, which helps collecting data across log file for a given time
  range, and store those in a new table of the autopkgtest.db database.
* stats.ipynb, which is a Jupyter notebook with some starting points for
  graphing and analyzing the collected data.

This was necessary when the infra experienced some issue with some
specific architectures taking a long time to get into the VMs. This also
helped evaluate some solutions cowboyed on only one worker unit, and
comparing stats after a week-end.

Author: Skia
Author Date: 2024-06-11 12:26:03 UTC

feat: worker: armhf LXD remotes: set a password for console login

This makes armhf LXD remotes to have a password for the `ubuntu` user,
useful for console login, but still rejecting password login on ssh.

Sometimes, a remote gets unreachable through ssh, but still displays a
normal output in `openstack console log show`. Since we now have access
to the spice console, it would be interesting to be able to login to the
machine and investigate the issue further instead of just trashing it
and spawning a new remote.

Author: Skia
Author Date: 2024-06-07 09:35:25 UTC

feat: web: log User-Agent directly in HAProxy too

Since this is our exposed frontend, better log the User-Agent in here
too, at the closest possible place from the clients.

Author: Skia
Author Date: 2024-05-31 11:20:28 UTC

feat: improve admin page

Multiple improvements to make the admin page more useful:
* don't bail out if a running test triggers an exception somewhere
* self-document each heuristic in the rendered HTML
* add a new heuristic based on the delta between recorded duration and last log line printed

Author: Skia
Author Date: 2024-05-29 09:05:18 UTC

doc/admin: explain how to access a testbed spice console

Author: Skia
Author Date: 2024-05-20 13:55:05 UTC

feat: worker: metrics: remove unused cluster part

Author: Skia
Author Date: 2024-05-20 10:00:21 UTC

docs: explain how to access RabbitMQ web UI

Author: Skia
Author Date: 2024-05-06 10:32:12 UTC

fix: web: cache-amqp: allow '--refresh-semaphores' even on non-leader

Since this is an admin operation, and it is documented that every
instance of `cache-amqp` should be stopped before running this, we don't
need to restrict this on the leader mechanism.

Author: Skia
Author Date: 2024-04-30 15:52:17 UTC

fix: web: cache-amqp: avoid pushing additional messages to the semaphore queues

The whole point of the semaphore queue is to always contain no more than
a single message. If the queue is already empty, that usually just mean
that another instance of the script is running, not that a new message
should be pushed to the queue.
If at some point the synchronization messages get lost by amqp
server, then this is a problem than can be fixed with the newly added
`--refresh-semaphores` flag (see documentation about it), but this can
only be decided by an admin having a look at the problem, and is in no
way decidable from a single script point of view (because of the way
distributed systems work).

Author: Skia
Author Date: 2024-04-29 10:02:54 UTC

fix: armhf userdata: explicitely set the list of images to build

With the killing of armhf in future Ubuntu releases, we now have a fixed
set of releases for which we build LXD remotes. Computing this set with
`distro-info` becomes complicated because `distro-info` is unaware of
the architectures, and will keep displaying us future releases that we
don't want.

Author: Skia
Author Date: 2024-04-29 09:54:50 UTC

fix: worker: avoid hanging on early 'request' return

When the `request` function returns early, explicitely ack or reject
the message to avoid hanging on the queue.wait() indefinitely. According
to Tim, there should be some mechanism for having this automatically,
but since we don't know them yet, this is the code that already works
in production.

Author: Skia
Author Date: 2024-04-25 12:52:09 UTC

fix: worker: remove shenanigans with '/etc/hosts' since we have a proper fix

This has been properly fixed by setting an explicit `fqdn` in `autopkgtest/ssh-setup/nova`.
https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/334

Those were introduced to work around LP #2019472.

Author: Skia
Author Date: 2024-04-24 09:14:56 UTC

feat: web: add sha256 checksum for public autopkgtest.db

This comes along side a Britney change to make it use that checksum,
instead of the old `Content-Length` check that was used for the same
purpose.

Author: Skia
Author Date: 2024-04-21 07:07:44 UTC

fix: worker: better implement upstream jobs prioritization

This makes rolling the dice to know whether or not to run an upstream
job for every request, and not only when defining the queues to connect
to.

The main consequence is that every worker process will now be connected
to the `-upstream-` queues. This speads the dice rolls over a far bigger
ensemble (every request), instead of just the "worker start" events.
Consequently the deviation from the targeted percentage of upstream jobs
to run is greatly reduced, and the case where a badly distributed dice
roll can last for days (until next restart of all workers) is avoided.

Closes LP #2062537

Author: Skia
Author Date: 2024-04-18 14:21:01 UTC

fix: armhf userdata: make Xenial containers to work on Jammy host

This cmdline option allows the use of cgroupv1 on a Jammy host, that has
it disabled by default, but is required to run Xenial containers.
Trick is better explained in these discussions:
https://discuss.linuxcontainers.org/t/does-ubuntu-16-04-lxc-container-work-on-ubuntu-22-04-lxd-host/15250
https://discuss.linuxcontainers.org/t/error-the-image-used-by-this-instance-requires-a-cgroupv1-host-system-when-using-clustering/13885

Author: Skia
Author Date: 2024-04-16 07:14:57 UTC

fix: mojo: add keyserver.u.c to no_proxy

With the new mechanism replacing `apt-key`, we need to teach `wget` to
ignore the proxy when fetching the key.

Author: Skia
Author Date: 2024-04-16 10:08:25 UTC

fix: worker: unbreak the 'bytes' detection heuristic

Messages coming from `readline()` always end with a `\n`, so the
heuristic needs to take that into account.

Author: Skia
Author Date: 2024-04-11 11:54:12 UTC

feat: worker: improve logtail by always getting the first 5 lines of a log

The first 5 lines look like this:
```
  0s autopkgtest [23:08:03]: starting date and time: 2024-04-07 23:08:03+0000
  0s autopkgtest [23:08:03]: git checkout: 31124158 autopkgtest: take Paride's WIP change regarding wrong src pkg selection
  0s autopkgtest [23:08:03]: host juju-7f2275-prod-proposed-migration-environment-3; command line: /home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.22jn7d5i/out --timeout-copy=6000 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --apt-pocket=proposed --apt-upgrade libreoffice --timeout-short=300 --timeout-copy=20000 --timeout-test=20000 --timeout-build=20000 '--env=ADT_TEST_TRIGGERS=mysql-8.0/8.0.36-2ubuntu3 redland/1.0.17-3.1ubuntu3' -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest-big --security-groups autopkgtest-juju-7f2275-prod-proposed-migration-environment-3@lcy02-58.secgroup --name adt-noble-amd64-libreoffice-20240407-230803-juju-7f2275-prod-proposed-migration-environment-3-510113db-c395-457e-93e4-190cd04588df --image adt/ubuntu-noble-amd64-server --keyname testbed-juju-7f2275-prod-proposed-migration-environment-3 --net-id=net_prod-proposed-migration -e TERM=linux -e ''"'"'http_proxy=http://squid.internal:3128'"'"'' -e ''"'"'https_proxy=http://squid.internal:3128'"'"'' -e ''"'"'no_proxy=127.0.0.1,127.0.1.1,login.ubuntu.com,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpadlibrarian.net,launchpadcontent.net,launchpad.net,10.24.0.0/24,keystone.ps5.canonical.com,objectstorage.prodstack5.canonical.com'"'"'' --mirror=http://ftpmaster.internal/ubuntu/
 61s autopkgtest [23:09:04]: testbed dpkg architecture: amd64
 62s autopkgtest [23:09:05]: testbed apt version: 2.7.14build2
```
They are thus very valuable in many different situation:
* a job seems stuck, any developer can grab the command line and start
  to debug this locally without waiting for the job to unstuck.
* the starting date and time can help autopkgtest admin to detect jobs
  behaving weirdly, like looping.
* displaying the autopkgtest version will help developers know exactly
  what the infra is running, and help admins quickly ensure that all
  workers are running the intended version, for example.
* also dumping the testbed architecture and apt version can help detect
  issues with testbeds, particularly with foreign architecture testing.

Author: Skia
Author Date: 2024-04-11 17:05:34 UTC

feat: pull/push-amqp: improve self-documentation

Author: Skia
Author Date: 2024-04-09 13:35:27 UTC

fix: cloud: push-amqp: evaluate the messages to convert them from bytes to str

`pull-amqp` outputs messages as `bytes`, meaning the usual input for
`push-amqp` will look like this:
```
b'apt\n{"triggers": ["base-passwd/3.6.3build1"], "submit-time": "2024-04-08 18:38:17"}'
b'binutils\n{"triggers": ["bison/2:3.8.2+dfsg-1build2"], "submit-time": "2024-04-08 18:38:42"}'
```
Those lines will correctly be read as string, but will still contain the
`b'`, the `\n` (as two chars `\` and `n`, not a single `\n`), and the
trailing `'`, plus some other possible bytes escaping, and will thus be
pushed like this to the queue, which is wrong.

This patch adds an heuristic (detecting the starting `b'` and trailing
`'`) to evaluate the string as bytes and then decode it again.

As our messages contain the `\n` character, having an interface between
`pull-amqp` and `push-amqp` that relies on lines too maybe is not such
a great idea, since we need to keep printing the escaped bytes in
`pull-amqp`.
We may want to change this in the future to work with something like
`\0`, or even better: change the message format to be easier to handle.

Author: Skia
Author Date: 2024-04-10 08:20:58 UTC

fix: worker: ignore errors when deleting folders

Sometime on production, the removal of folders end up with the following kind of errors:
```
[Errno 39] Directory not empty: 'tests-tree'
```
or
```
[Errno 2] No such file or directory: 'rifle.json'
```
This makes sure really clean those folder, whatever happens.

Author: Skia
Author Date: 2024-04-08 13:36:52 UTC

feat: add a default global vimrc in the charms

This will ease the admin's work, and is also less error-prone with the
display of special-characters and highlight of extra whitespaces.

Author: Skia
Author Date: 2024-04-03 16:00:58 UTC

feat: cloud: metrics: add some basic logging

This will help gather more precise feedback when something odd appears
in the KPI.

Author: Skia
Author Date: 2024-03-29 11:38:41 UTC

docs: improve LXD documentation when deploying in a new environment

There are some one-time steps that need to be performed when deploying
LXD remotes in a new environment that were lacking documentation. This
is based on the experience of deploying armhf remotes on bos03.

Author: Skia
Author Date: 2024-03-29 09:59:35 UTC

fix: worker: 'fails' should be of type 'list'

This fixes the following error in production:
```
  File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1229, in request
    " and ".join(fails),
TypeError: can only join an iterable
```

Author: Skia
Author Date: 2024-03-28 17:30:08 UTC

WIP merge flask apps

Author: Skia
Author Date: 2024-03-28 11:39:03 UTC

fix: web: cache-amqp: improve split heuristic

Requests format has been made more consistant recently, so the split
heuristic can be made more precise, and more likely to do what it's
supposed to.

Author: Skia
Author Date: 2024-03-26 10:37:59 UTC

feat: web: add logs-viewer userscript

This adds a userscript, usable client-side in browser through an
extension like TamperMonkey, to ease the browsing of autopkgtest log
files by adding line number and foldable sections, along with some shiny
colors.

Kudos to Cajuteq for implementing this while not even working on Ubuntu,
thanks bro' \o/

Author: Skia
Author Date: 2024-03-21 11:28:12 UTC

fix: mojo: 'dump-lxd-remotes' now uses 'yaml-dump', which produces usable output

Author: Skia
Author Date: 2024-03-21 10:28:00 UTC

fix: service-bundle: lxd: remove 'sed' line setting apt sources

That line is not needed, because the `armhf-lxd.userdata` cloud-init
file already sets up that mirror in the sources of the autopkgtest LXD
images, in a much more reliable way.

Author: Skia
Author Date: 2024-03-14 12:15:35 UTC

fix: web: make queued message format consistent

All other ways of pushing test requests to the queues separate the
package name from the params with a '\n'.
The consequence of not doing that consistently was that manually
requested tests would not appear in a package's filtered queue.

Author: Skia
Author Date: 2024-03-08 17:02:43 UTC

feat: web: display running and queued jobs on package result page

This adds the data from running.json and queued.json to the list of
rows to display on the results page. Obviously, it's by design lacking
some columns, but this at least allows a better display of the current
non-finished jobs.

Thanks Adrien for the feedback and review :-)

Author: Skia
Author Date: 2024-03-04 16:59:51 UTC

fix: web: add additional indexes to the database

`/admin` took about 80s to load locally on my laptop, and `/packages/
<name>` was around 14s. Adding those two indexes makes those pages go
down to a few dozen ms for `/admin` and less than 10ms for `/packages`
on the same machine.

Author: Skia
Author Date: 2024-03-04 12:04:49 UTC

feat: web: browse-test: add options to use existing non-dummy data files

This is useful to quickly run the web UI locally on an existing database
with a bit more useful data than the procedural dummy values.
For example, with the following, I can easily reproduce the very slow
`/admin` page locally:
`./browse-test.py --database /tmp/autopkgtest-slow-from-prod.db --running /tmp/running-slow-from-prod.json`

Author: Skia
Author Date: 2024-02-29 12:28:19 UTC

feat: charms: improve status reporting to avoid misleading messages in 'juju status'

If an error happens after the last update, then the displayed status
has the potential of being in another completely unrelated function.
It's important to be as precise as possible with that status to avoid
misleading the user into debugging unrelated matters.

Author: Skia
Author Date: 2024-02-28 16:08:32 UTC

fix: web: make download-all-results handle 'all-proposed' correctly

`testinfo.json` may now contain `"all-proposed": null` in most recent
results, so the value behind the `all-proposed` key needs to be checked
before appending that blindly in the `env` column.

Author: Skia
Author Date: 2024-02-28 12:18:27 UTC

feat: web: templates: add Launchpad and excuses links from packages pages

Author: Skia
Author Date: 2024-02-28 11:17:29 UTC

feat: mojo: dump the LXD remotes in a file

This allows easy modification and import with
juju config autopkgtest-lxd-worker --file=~/lxd-remotes.yaml

Author: Skia
Author Date: 2024-02-23 13:51:01 UTC

feat: web: throw an error when no running.json file

With the exception handler added previously, this enables a clear error
message visible to the users, showing the infrastructure issue, instead
of defaulting to an empty list that could be confusing if users expect
to find something there.

Author: Skia
Author Date: 2024-02-23 11:54:12 UTC

feat: web: browse: display queued tests on /package page

Author: Skia
Author Date: 2024-02-22 14:53:16 UTC

feat: web: template: improve the display of running tests in various pages

Author: Skia
Author Date: 2024-02-22 11:21:28 UTC

feat: ease by far the local development of browse.cgi

This commit bring two main things tied together:
  - the helpers/tests.py, with functions helping populating databases
and cache files with dummy data
  - the browse-test.py file, which is able to run the Flask app in
standalone debug mode, for easy local development, and leverages the
dummy data from the helpers/tests.py file.

All of that is also documented in a new README.md file that has the
necessary dependencies and instructions on how to run the app locally.

Author: Skia
Author Date: 2024-02-22 11:56:58 UTC

feat: display currently running jobs on each package's page

Author: Skia
Author Date: 2024-02-21 10:25:27 UTC

web: mention 'qa-help' highlight for IRC

Author: Skia
Author Date: 2024-02-19 11:17:11 UTC

FIX: d-r+d-a-r: make sure to release the lock

This makes sure that the database updates are performed in a
transaction, which explicitely releases the table lock when the
transaction complete. This allow both `download-results` and
`download-all-results` to work at the same time with one blocking this
other.

Author: Skia
Author Date: 2024-02-08 11:08:28 UTC

feat: worker: p_o_d: always dump ADDITIONAL_PARAMS

This makes the output post processing always include the `all-proposed`
key in the `testinfo.json` file. This is useful when scripting around
that file, so that you always know exactly what happens:
* all-proposed: 1
  --> obvious, it was given when triggering the test.
* all-proposed: null
  --> all-proposed wasn't found when post-processing, so wasn't given in
  the request.
* all-proposed not present in the file
  --> you are looking at some old results that didn't have that feature
  in place, so you cannot know if all-proposed=1 was given unless you
  parse the logs.

Author: Steve Langasek
Author Date: 2021-11-04 20:25:45 UTC

Record an 'exit_invalid' tag when the instance failed

Influx doesn't let us usefully aggregate different exit codes to calculate
a failure rate. Set an explicit tag for a failed job so we can graph it.

Author: Steve Langasek
Author Date: 2021-10-14 00:33:01 UTC

Drop references to precise in the code

Precise is no more, so remove references to it from the code for
simplification.

Author: Julian Andres Klode
Author Date: 2021-05-05 14:24:49 UTC

Do not coalesce minutely timers, use 30s randomized delay

This avoids all the things being started at the same time, reducing
peak load.

Author: Skia
Author Date: 2024-01-12 16:40:17 UTC

webcontrol: fix some hardcoded URLs hostnames

For example, calling the following URL:
https://autopkgtest.staging.ubuntu.com/request.cgi/?release=noble&arch=amd64&package=gzip&trigger=gzip/1.12-1ubuntu1
gives the following output:
```
...
Test request submitted.

Result history
    https://autopkgtest.ubuntu.com/packages/gzip/noble/amd64
...
```
which is obviously wrong, as it should point us to a staging page.

Author: Skia
Author Date: 2024-01-05 17:26:56 UTC

WIP w/r/submit: put architectures in configuration

Author: Skia
Author Date: 2024-01-03 14:12:40 UTC

w/download-all-results: handle non-existing containers

In a fresh infrastructure, some containers may not exist, and that should
not prevent the download of results for the next containers that may
exist.

Author: Skia
Author Date: 2023-12-15 16:16:49 UTC

m/service-bundle: split flavors definition for each stage

Author: Skia
Author Date: 2023-11-28 08:47:29 UTC

docs: improve architecture diagram

Author: Skia
Author Date: 2023-12-01 16:38:23 UTC

cloud/tools: don't use hardcoded flavor to ensure autopkgtest images

mojo/service-bundle already has a configuration for the flavors to use,
but some scripts didn't make use of it, and had hardcoded values.

Author: Skia
Author Date: 2023-11-23 14:35:28 UTC

readthedocs: graphviz is needed to build the doc

Author: Skia
Author Date: 2023-11-14 13:40:28 UTC

autopkgtest-web: link to requester LP page when available

Author: Brian Murray
Author Date: 2023-11-02 14:05:42 UTC

dos: the spec is running on focal now

Author: Brian Murray
Author Date: 2023-11-01 14:03:03 UTC

docs: update how to add the sec group

Author: Brian Murray
Author Date: 2023-11-01 12:38:33 UTC

service-bundle: bump number of cloud-workers from zero

Author: Brian Murray
Author Date: 2023-10-26 23:16:36 UTC

autopkgtest-web: add jq to installed packages

This way you can parse the json files on the server

Author: Brian Murray
Author Date: 2023-10-26 16:52:48 UTC

mojo: lgw01 is in 3FP and going away

Author: Brian Murray
Author Date: 2023-10-24 00:40:36 UTC

Add noble to production, drop kinetic from staging.

Author: Brian Murray
Author Date: 2023-10-04 16:26:40 UTC

We should only need to a login.u.c once not twice

Author: Brian Murray
Author Date: 2023-09-29 16:53:01 UTC

Only create a new container if we'll copy results to it

Author: Brian Murray
Author Date: 2023-09-29 16:48:15 UTC

Clarify the name of swift containers for PPA results

Author: Brian Murray
Author Date: 2023-09-18 21:22:14 UTC

launchpad.yaml: drop proxy env settings per conversation with cjwatson

Author: Brian Murray
Author Date: 2023-08-28 20:53:12 UTC

webcontrol: if the queue is empty put a message in it

Author: Brian Murray
Author Date: 2023-08-24 20:59:17 UTC

mojo: update add-floating-ip