Merge ~andersson123/autopkgtest-cloud:delete_old_github_requests into autopkgtest-cloud:master

[Tim Andersson]

[Paride Legovini (paride)]

One inline comment (Needs fixing) on the implemention.

In general I'm not a fan of this kind of "hiding leaks by periodic cleanup" approach, unless the problem is fairly well understood. Do we have an idea of why github jobs "get lost" and then "forgotten about"? What does it mean that they "get lost"? If they're in queue, why aren't them processed at some point? What do you mean by "forgotten about"?

review: Needs Information

[Tim Andersson (andersson123)]

I don't have a good understanding of what happens. We have a ticket from a long time ago without much information to go off of.

[Tim Andersson (andersson123)]

I agree it's not a great approach, but I'm not personally sure investigating deep into why this happens is a particularly efficient use of time, especially at the moment, but I'm open to doing so if others disagree with me.

[Tim Andersson (andersson123)]

I've amended, please re-review

[Paride Legovini (paride)]

Technically LGTM, but I'm still uncomfortable with the vagueness of the issue. The commit message says

  github jobs sometimes get lost in the queue

what does it mean that they "get lost"? Are them in the queue or not? Where or how does these stale jobs show up?

[Tim Andersson (andersson123)]

I don't know. We don't have much information on the issue. The old card I picked up simply stated something similar to the commit message (ADT-64)

[Paride Legovini (paride)]

@Brian do you happen to know/remember more?

I am fine with merging this, but if we are able to explain the problem better in the commit message, we should.

review: Needs Information

[Tim Andersson (andersson123)]

I agree. I will contact Julian regarding this since he reported the card

[Tim Andersson (andersson123)]

as per juliank:

I think it's that the rabbitmq is not fully reliably persistent
But we add those files to the directory
So when rabbitmq needs to be restarted because it fails you end up with files you no longer have items queued for
Oh yeah and then I added the 24h TTL to the upstream queue

From 2022:
In light of the systemd upstream tests hanging and not leaving the queues on arm64, I have also applied a 24 hour message TTL on the upstream testing queues.

Hope that gives some context, I'll amend the commit message

[Tim Andersson (andersson123)]

I've amended the commit message, let me know what you think

[Paride Legovini (paride)]

This is much better, thanks for investigating!

review: Approve

diff --git a/charms/focal/autopkgtest-web/units/delete-old-github-requests.service b/charms/focal/autopkgtest-web/units/delete-old-github-requests.service
new file mode 100644
index 0000000..e4bec05
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/delete-old-github-requests.service
@@ -0,0 +1,7 @@
+[Unit]
+Description=Delete github requests older than 1 day
+
+[Service]
+Type=oneshot
+User=ubuntu
+ExecStart=/home/ubuntu/webcontrol/delete-old-github-requests
diff --git a/charms/focal/autopkgtest-web/units/delete-old-github-requests.timer b/charms/focal/autopkgtest-web/units/delete-old-github-requests.timer
new file mode 100644
index 0000000..993c77f
--- /dev/null
+++ b/charms/focal/autopkgtest-web/units/delete-old-github-requests.timer
@@ -0,0 +1,8 @@
+[Unit]
+Description=Delete github requests older than 1 day (timer)
+
+[Timer]
+OnCalendar=hourly
+
+[Install]
+WantedBy=autopkgtest-web.target
diff --git a/charms/focal/autopkgtest-web/webcontrol/delete-old-github-requests b/charms/focal/autopkgtest-web/webcontrol/delete-old-github-requests
new file mode 100755
index 0000000..adf87f3
--- /dev/null
+++ b/charms/focal/autopkgtest-web/webcontrol/delete-old-github-requests
@@ -0,0 +1,2 @@
+#!/bin/sh
+find /run/autopkgtest_webcontrol/github-pending/ -mtime +0 -exec echo "Deleting stale Github job: {}" \; -delete