Merge ~andersson123/autopkgtest-cloud:docs-add-queue-cleanup-section into autopkgtest-cloud:master

Proposed by Tim Andersson
Status: Merged
Merged at revision: 23500468fd94c0e6b4ff71e4a34171474515b4c8
Proposed branch: ~andersson123/autopkgtest-cloud:docs-add-queue-cleanup-section
Merge into: autopkgtest-cloud:master
Diff against target: 53 lines (+36/-2)
1 file modified
docs/administration.rst (+36/-2)
Reviewer Review Type Date Requested Status
Skia Approve
Brian Murray Needs Fixing
Review via email: mp+462692@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Skia (hyask) :
review: Needs Fixing
Revision history for this message
Tim Andersson (andersson123) wrote :

Hey Skia, this is now amended based on your previous comments, please re-review

Revision history for this message
Brian Murray (brian-murray) wrote :

Regular queue cleanup shouldn't be regularly necessary rather it is just help when the queues are quite long. I've added an inline comment about that.

I think it might also be worth mentioning that the package name passed to filter-amqp is a regex so you'll need to escape a '+' symbol which occurs commonly in package names. Or more cleverly throwing a sed call in the shell pipeline would avoid one needing to remember that.

review: Needs Fixing
Revision history for this message
Brian Murray (brian-murray) wrote :

I wanted to test the shell pipeline so tried adding in the sed command. This worked for me.

uniq -c | sed 's/+/\\+/' | less

Revision history for this message
Brian Murray (brian-murray) wrote :

The pipeline doesn't seem to work with the ubuntu queue for one reason or another. I tracked it down to the this call to jq:

 $ cat queues.json | jq '.ubuntu.noble.armhf' | jq -c '.[]' -r | grep '"triggers"' | jq -c '.triggers' --sort-keys
parse error: Invalid numeric literal at line 1, column 15

Revision history for this message
Skia (hyask) wrote :

Yes, for now it doesn't work, due to this, which is already deployed: https://git.launchpad.net/autopkgtest-cloud/commit/?id=91f6d022f17c0659c57978a9cdaecdacc0d26584

That's why I didn't raise it in this MP, since as soon as existing inconsistent jobs in queue will be gone, the oneliner will magically work.

Revision history for this message
Tim Andersson (andersson123) wrote :

I've added the sed command to escape the +'s, and I'll leave the command as is since skia has given a good reason for it currently not working.

Revision history for this message
Tim Andersson (andersson123) wrote :

Amended and ready for re-review.

Revision history for this message
Skia (hyask) :
review: Needs Fixing
Revision history for this message
Tim Andersson (andersson123) wrote :

Good catch, amended and ready for re-review

Revision history for this message
Brian Murray (brian-murray) wrote :

See my in-line comments.

review: Needs Fixing
Revision history for this message
Tim Andersson (andersson123) wrote :

Amended and ready for re-review

Revision history for this message
Skia (hyask) :
review: Needs Fixing
Revision history for this message
Tim Andersson (andersson123) wrote :

amended, sorry about that!

Revision history for this message
Skia (hyask) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/docs/administration.rst b/docs/administration.rst
index adb27b9..3bc80b9 100644
--- a/docs/administration.rst
+++ b/docs/administration.rst
@@ -212,8 +212,7 @@ Requests can be requested by using an API key instead of authenticating using SS
212To do so, attach a cookie to whatever script is making the test request, with the name212To do so, attach a cookie to whatever script is making the test request, with the name
213"X-Api-Key". The value should look like this:213"X-Api-Key". The value should look like this:
214214
215.. code-block::215``user:api-key``
216 user:api-key
217216
218Where the user and api-key fields are provided by the Ubuntu Release Management team.217Where the user and api-key fields are provided by the Ubuntu Release Management team.
219218
@@ -337,3 +336,38 @@ Test the setup with some dummy PR that changes some README or similar. You can
337then re-trigger new tests by force-pushing to the branch. Once everything works,336then re-trigger new tests by force-pushing to the branch. Once everything works,
338you can add more web hooks with different test parameters to e. g. trigger tests337you can add more web hooks with different test parameters to e. g. trigger tests
339on multiple architectures or multiple Ubuntu releases.338on multiple architectures or multiple Ubuntu releases.
339
340
341Queue Cleanup
342^^^^^^^^^^^^^
343Regular queue cleanup can become necessary when the queues are quite large.
344The best way to go about doing this is by first downloading the queues.json:
345
346``curl https://autopkgtest.ubuntu.com/queues.json | jq '.huge.noble.amd64' | jq -c '.[]' -r | grep '"triggers"' | jq -c '.triggers' --sort-keys | sort | uniq -c | sed 's/+/\\+/' | less``
347
348And what this does, is filter the queue, and returns a list of all the unique
349triggering packages currently queued, with a count of how many queue items per
350package, and pipes it to less. You can then check this output, and look for
351obsoleted package versions, e.g.:
352
353.. code-block::
354
355 117 [\"dpkg/1.22.6ubuntu2\"],
356 117 [\"dpkg/1.22.6ubuntu4\"],
357
358Here, you can see that ``dpkg/1.22.6ubuntu2`` has been obsoleted by ``dpkg/1.22.6ubuntu4``.
359So, the workflow now would be to remove said package from the queue::
360
361.. code-block::
362
363 ./filter-amqp -v debci-huge-noble-$arch "dpkg/1.22.6ubuntu2\b"
364
365However, this gets tedious with lots of obsoleted packages in the queue. So an approach,
366when you have lots of obsoleted packages, would be like so:
367
368.. code-block::
369
370 packages="package1/trigger-2.3.0 package2/trigger-2.4.3..." # obviously with more packages
371 for pkg in $packages; do for arch in amd64 arm64 s390x ppc64el armhf i386; do ./filter-amqp -v debci-huge-noble-$arch "$pkg\b"; done; done
372
373This way you can remove all the packages in one command on every architecture.

Subscribers

People subscribed via source and target branches