Merge ubuntu-debuginfod:systemd into ubuntu-debuginfod:master

[Sergio Durigan Junior]

One more before bedtime? :-)

This should be the last one before I can fully deploy the new poller/getter. It extends the current celery application to be systemd-aware, creates new systemd services/timer, and finally creates a simple poller application that will be executed every 30 minutes to fetch new ddebs (and source code).

This should be simpler to review than the other MPs, I hope. Let me know what you think, please.

[Sergio Durigan Junior (sergiodj)]

Note that various parameters chosen below (number of threads to spawn, $HOME directory, etc.) reflect what we have in the current deployment.

[Lena Voytek (lvoytek)]

Just need to make sure that sys is imported in poll_launchpad.py, then I approve. I'm not as familiar with creating systemd service files, but at least for the current deployment they look good to me!

review: Needs Fixing

[Sergio Durigan Junior (sergiodj)]

Thanks for the review, Lena.

I've addressed your comments and merged the branch, in preparation for the rework that I did (which will require yet another big MP).

[Lena Voytek (lvoytek)]

Looks good, thanks!

review: Approve

diff --git a/conf/ubuntu-debuginfod-celery.default b/conf/ubuntu-debuginfod-celery.default
new file mode 100644
index 0000000..9f915b0
--- /dev/null
+++ b/conf/ubuntu-debuginfod-celery.default
@@ -0,0 +1,12 @@
+# App instance to use.
+CELERY_APP="debuginfod"
+
+# Extra command-line arguments to the worker.
+#
+# We chose 6 workers here because it seemed like a good compromise
+# between not hammering the network and processing as many things in
+# parallel as possible.
+CELERYD_OPTS="--concurrency=6"
+
+# Log level to use.
+CELERYD_LOG_LEVEL="INFO"
diff --git a/conf/ubuntu-debuginfod-celery.service b/conf/ubuntu-debuginfod-celery.service
new file mode 100644
index 0000000..f9331ce
--- /dev/null
+++ b/conf/ubuntu-debuginfod-celery.service
@@ -0,0 +1,26 @@
+[Unit]
+Description=Ubuntu Debuginfod Celery
+After=network.target rabbitmq-server.service
+Requires=rabbitmq-server.service
+
+[Service]
+Type=notify
+User=mirror
+Group=mirror
+EnvironmentFile=/etc/default/ubuntu-debuginfod-celery
+WorkingDirectory=/home/mirror/ubuntu-debuginfod/
+ExecStart=/usr/bin/celery -A ${CELERY_APP} worker --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}
+Restart=always
+TimeoutStopSec=600
+PrivateDevices=true
+PrivateUsers=true
+ProtectKernelTunables=true
+ProtectControlGroups=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+MemoryDenyWriteExecute=true
+RestrictRealtime=true
+ReadWritePaths=/srv/debug-mirror/
+
+[Install]
+WantedBy=multi-user.target
diff --git a/conf/ubuntu-debuginfod-poll-lp.service b/conf/ubuntu-debuginfod-poll-lp.service
new file mode 100644
index 0000000..5c9e13a
--- /dev/null
+++ b/conf/ubuntu-debuginfod-poll-lp.service
@@ -0,0 +1,23 @@
+[Unit]
+Description=Poll Launchpad for new ddebs
+After=network.target ubuntu-debuginfod-celery.service
+Requires=ubuntu-debuginfod-celery.service
+
+[Service]
+Type=oneshot
+User=mirror
+Group=mirror
+WorkingDirectory=/home/mirror/ubuntu-debuginfod/
+ExecStart=/usr/bin/python3 /home/mirror/ubuntu-debuginfod/poll_launchpad.py
+PrivateDevices=true
+PrivateUsers=true
+ProtectKernelTunables=true
+ProtectControlGroups=true
+ProtectKernelLogs=true
+ProtectKernelModules=true
+MemoryDenyWriteExecute=true
+RestrictRealtime=true
+ReadWritePaths=/home/mirror/.config/ubuntu-debuginfod/
+
+[Install]
+WantedBy=multi-user.target
diff --git a/conf/ubuntu-debuginfod-poll-lp.timer b/conf/ubuntu-debuginfod-poll-lp.timer
new file mode 100644
index 0000000..c61fdcb
--- /dev/null
+++ b/conf/ubuntu-debuginfod-poll-lp.timer
@@ -0,0 +1,10 @@
+[Unit]
+Description=Poll Launchpad for new ddebs
+
+[Timer]
+# Every 30 minutes
+OnCalendar=*:0/30
+Persistent=true
+
+[Install]
+WantedBy=timers.target
diff --git a/debuginfod.py b/debuginfod.py
index d64efd4..c0d0357 100644
--- a/debuginfod.py
+++ b/debuginfod.py
@@ -18,11 +18,14 @@
 # Authors: Sergio Durigan Junior <[e-mail]>
 
 from celery import Celery
+from celery.signals import worker_ready, worker_shutting_down
 
 from debuggetter import DebugGetterTimeout, DebugGetterRetry
 
 from ddebgetter import DdebGetter, DdebSourceCodeGetter
 
+import sdnotify
+
 app = Celery("debuginfod", broker="pyamqp://guest@localhost//")
 
 app.conf.update(
@@ -34,9 +37,13 @@ app.conf.update(
     task_acks_late=True,
     # Only one task per worker.
     worker_prefetch_multiplier=1,
-    worker_log_format="[%(asctime)s: %(levelname)s/%(processName)s] %(funcName)s: %(message)s",
+    # Rewrite the log format.  We will most likely be running under
+    # systemd, which already includes a timestamp on every log line.
+    worker_log_format="[%(levelname)s/%(processName)s] %(funcName)s: %(message)s",
 )
 
+sdnotifier = sdnotify.SystemdNotifier()
+
 
 @app.task(
     name="grab_ddebs",
@@ -76,5 +83,17 @@ def grab_ddebs_sources(msg):
     g.process_request(msg)
 
 
+@worker_ready.connect
+def notify_worker_ready(**kwargs):
+    """Notify systemd that the worker initialization has finished."""
+    sdnotifier.notify("READY=1")
+
+
+@worker_shutting_down.connect
+def notify_worker_shutting_down(**kwargs):
+    """Notify systemd that the worker will start its process to shut down."""
+    sdnotifier.notify("STOPPING=1")
+
+
 if __name__ == "__main__":
     app.start()
diff --git a/poll_launchpad.py b/poll_launchpad.py
new file mode 100644
index 0000000..161aa77
--- /dev/null
+++ b/poll_launchpad.py
@@ -0,0 +1,34 @@
+#!/usr/bin/python3
+
+# Copyright (C) 2022 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Authors: Sergio Durigan Junior <[e-mail]>
+
+from ddebpoller import DdebPoller
+from debuginfod import grab_ddebs, grab_ddebs_sources
+
+from celery import chain
+
+def poll_launchpad():
+    """Poll Launchpad."""
+    poller = DdebPoller()
+    for msg in poller.get_ddebs():
+        # We have to use a chain here because the ddebs need to be
+        # downloaded before we can start fetching the sources.
+        chain(grab_ddebs.si(msg), grab_ddebs_sources.si(msg))()
+
+if __name__ == '__main__':
+    poll_launchpad()