Merge ~andersson123/autopkgtest-cloud:docs-add-queue-cleanup-section into autopkgtest-cloud:master

[Tim Andersson]

[Skia (hyask)]

review: Needs Fixing

[Tim Andersson (andersson123)]

Hey Skia, this is now amended based on your previous comments, please re-review

[Brian Murray (brian-murray)]

Regular queue cleanup shouldn't be regularly necessary rather it is just help when the queues are quite long. I've added an inline comment about that.

I think it might also be worth mentioning that the package name passed to filter-amqp is a regex so you'll need to escape a '+' symbol which occurs commonly in package names. Or more cleverly throwing a sed call in the shell pipeline would avoid one needing to remember that.

review: Needs Fixing

[Brian Murray (brian-murray)]

I wanted to test the shell pipeline so tried adding in the sed command. This worked for me.

uniq -c | sed 's/+/\\+/' | less

[Brian Murray (brian-murray)]

The pipeline doesn't seem to work with the ubuntu queue for one reason or another. I tracked it down to the this call to jq:

 $ cat queues.json | jq '.ubuntu.noble.armhf' | jq -c '.[]' -r | grep '"triggers"' | jq -c '.triggers' --sort-keys
parse error: Invalid numeric literal at line 1, column 15

[Skia (hyask)]

Yes, for now it doesn't work, due to this, which is already deployed: https://git.launchpad.net/autopkgtest-cloud/commit/?id=91f6d022f17c0659c57978a9cdaecdacc0d26584

That's why I didn't raise it in this MP, since as soon as existing inconsistent jobs in queue will be gone, the oneliner will magically work.

[Tim Andersson (andersson123)]

I've added the sed command to escape the +'s, and I'll leave the command as is since skia has given a good reason for it currently not working.

[Tim Andersson (andersson123)]

Amended and ready for re-review.

[Skia (hyask)]

review: Needs Fixing

[Tim Andersson (andersson123)]

Good catch, amended and ready for re-review

[Brian Murray (brian-murray)]

See my in-line comments.

review: Needs Fixing

[Tim Andersson (andersson123)]

Amended and ready for re-review

[Skia (hyask)]

review: Needs Fixing

[Tim Andersson (andersson123)]

amended, sorry about that!

[Skia (hyask)]

review: Approve

diff --git a/docs/administration.rst b/docs/administration.rst
index adb27b9..3bc80b9 100644
--- a/docs/administration.rst
+++ b/docs/administration.rst
@@ -212,8 +212,7 @@ Requests can be requested by using an API key instead of authenticating using SS
 To do so, attach a cookie to whatever script is making the test request, with the name
 "X-Api-Key". The value should look like this:
 
-.. code-block::
-  user:api-key
+``user:api-key``
 
 Where the user and api-key fields are provided by the Ubuntu Release Management team.
 
@@ -337,3 +336,38 @@ Test the setup with some dummy PR that changes some README or similar. You can
 then re-trigger new tests by force-pushing to the branch. Once everything works,
 you can add more web hooks with different test parameters to e. g. trigger tests
 on multiple architectures or multiple Ubuntu releases.
+
+
+Queue Cleanup
+^^^^^^^^^^^^^
+Regular queue cleanup can become necessary when the queues are quite large.
+The best way to go about doing this is by first downloading the queues.json:
+
+``curl https://autopkgtest.ubuntu.com/queues.json | jq '.huge.noble.amd64' | jq -c '.[]' -r | grep '"triggers"' | jq -c '.triggers' --sort-keys | sort | uniq -c | sed 's/+/\\+/' | less``
+
+And what this does, is filter the queue, and returns a list of all the unique
+triggering packages currently queued, with a count of how many queue items per
+package, and pipes it to less. You can then check this output, and look for
+obsoleted package versions, e.g.:
+
+.. code-block::
+
+  117 [\"dpkg/1.22.6ubuntu2\"],
+  117 [\"dpkg/1.22.6ubuntu4\"],
+
+Here, you can see that ``dpkg/1.22.6ubuntu2`` has been obsoleted by ``dpkg/1.22.6ubuntu4``.
+So, the workflow now would be to remove said package from the queue::
+
+.. code-block::
+
+  ./filter-amqp -v debci-huge-noble-$arch "dpkg/1.22.6ubuntu2\b"
+
+However, this gets tedious with lots of obsoleted packages in the queue. So an approach,
+when you have lots of obsoleted packages, would be like so:
+
+.. code-block::
+
+  packages="package1/trigger-2.3.0 package2/trigger-2.4.3..." # obviously with more packages
+  for pkg in $packages; do for arch in amd64 arm64 s390x ppc64el armhf i386; do ./filter-amqp -v debci-huge-noble-$arch "$pkg\b"; done; done
+
+This way you can remove all the packages in one command on every architecture.