Merge ubuntu-debuginfod:overhaul into ubuntu-debuginfod:master

Proposed by Sergio Durigan Junior
Status: Merged
Merged at revision: 1603dca0a3eee2c59dd5a0aabeee3588ae6fdee0
Proposed branch: ubuntu-debuginfod:overhaul
Merge into: ubuntu-debuginfod:master
Diff against target: 5729 lines (+3614/-332)
59 files modified
README.md (+151/-1)
conf/ubuntu-debuginfod-celery.default (+1/-1)
conf/ubuntu-debuginfod-celery.service (+0/-1)
conf/ubuntu-debuginfod-launchpad-dispatcher.service (+24/-0)
conf/ubuntu-debuginfod-launchpad-dispatcher.timer (+10/-0)
conf/ubuntu-debuginfod-launchpad-poller.service (+2/-3)
debian/changelog (+5/-0)
debian/control (+50/-0)
debian/copyright (+29/-0)
debian/rules (+19/-0)
debian/source/format (+1/-0)
debian/source/options (+1/-0)
debian/ubuntu-debuginfod.install (+3/-0)
debian/ubuntu-debuginfod.postinst (+168/-0)
debian/ubuntu-debuginfod.postrm (+39/-0)
debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service (+1/-0)
debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service (+1/-0)
debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer (+1/-0)
debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service (+1/-0)
debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer (+1/-0)
dev/null (+0/-191)
pyproject.toml (+40/-0)
services/launchpad-dispatcher.py (+451/-0)
services/launchpad-poller.py (+156/-0)
setup.cfg (+14/-0)
tests/test_baselp.py (+58/-0)
tests/test_common_lp_getter.py (+91/-0)
tests/test_common_ppa_getter.py (+112/-0)
tests/test_ddebgetter.py (+64/-0)
tests/test_exceptions.py (+33/-0)
tests/test_getter.py (+38/-0)
tests/test_ppaddebgetter.py (+94/-0)
tests/test_ppasourcepackagegetter.py (+94/-0)
tests/test_sourcepackagegetter.py (+69/-0)
tests/test_utils.py (+114/-0)
ubuntu_debuginfod/__init__.py (+10/-0)
ubuntu_debuginfod/debuginfod.py (+87/-76)
ubuntu_debuginfod/getter.py (+57/-0)
ubuntu_debuginfod/getters/__init__.py (+1/-0)
ubuntu_debuginfod/getters/common_lp_getter.py (+104/-0)
ubuntu_debuginfod/getters/common_ppa_getter.py (+65/-0)
ubuntu_debuginfod/getters/ddebgetter.py (+70/-0)
ubuntu_debuginfod/getters/ppaddebgetter.py (+65/-0)
ubuntu_debuginfod/getters/ppasourcepackagegetter.py (+70/-0)
ubuntu_debuginfod/getters/sourcepackagegetter.py (+255/-0)
ubuntu_debuginfod/poller.py (+28/-57)
ubuntu_debuginfod/pollers/__init__.py (+1/-0)
ubuntu_debuginfod/pollers/common_lp_poller.py (+86/-0)
ubuntu_debuginfod/pollers/common_ppa_poller.py (+121/-0)
ubuntu_debuginfod/pollers/ddebpoller.py (+69/-0)
ubuntu_debuginfod/pollers/ppaddebpoller.py (+73/-0)
ubuntu_debuginfod/pollers/ppasourcepackagepoller.py (+73/-0)
ubuntu_debuginfod/pollers/sourcepackagepoller.py (+69/-0)
ubuntu_debuginfod/utils/__init__.py (+1/-0)
ubuntu_debuginfod/utils/baselp.py (+45/-0)
ubuntu_debuginfod/utils/debugdb.py (+11/-2)
ubuntu_debuginfod/utils/exceptions.py (+27/-0)
ubuntu_debuginfod/utils/utils.py (+235/-0)
utils/obtain-launchpad-credentials.py (+55/-0)
Reviewer Review Type Date Requested Status
Athos Ribeiro (community) Approve
Canonical Server Reporter Pending
Review via email: mp+441906@code.launchpad.net

Description of the change

This MP is the big overhaul/rewrite of ubuntu-debuginfod that we talked about.

There are many new things being done here, but most of the diff is basically code reorganization to make better use of Python's capabilities.

I'll try to make a list of important things to consider when reviewing this code:

1) I'm make heavy use of base classes for pollers and getters. Using pollers and as example:

- You will see that there is the most basic class, Poller, which implements only the necessary/common bits for a generic poller to operate.

- On top of it, there is a CommonLaunchpadPoller class, which extends Poller and implements generic methods for both a ddeb and a source package poller to work. The class also inherits from BaseLaunchpadClass, which follows the mixin design pattern and is useful because it provides a "launchpad_login" methods to classes that need this feature.

- On top of CommonLaunchpadPoller you will find the DdebPoller and SourcePackagePoller classes, which finally implement the necessary method (in this case, poll_artifacts) that will do the polling of their respective artifacts.

- PPAs are kind of a special beast, so they have their own base class (CommonPPAPoller) which is actually crafted to be used specifically in multiple-inheritance scenarios. This base class is used alongside both DdebPoller and SourcePackagePoller to implement the PPA methods to fetch ddebs and source packages (for public and private PPAs).

The design of getters follows exactly the same rationale.

2) You will find that I broke down the "poll_lp.py" service in two:

- The first one, "launchpad-poller.py", only performs the polling of artifacts from LP and adds them to a local database named "jobs2dispatch". It's important to say that this poller does *not* do any complex filtering to determine if the artifact is actually interesting for debuginfod or not; it simply dumps the contents of a getPublished{Sources,Binaries} call into the local table. This was required because the service was hitting some strange "Error 400" problems with LP, and part of the reason seemed to be the fact that "poll_lp.py" was performing too many calls to the API.

- There is a second service now, called "launchpad-dispatcher.py". This service is responsible for iterating over the entries of the "jobs2dispatch" table and determine if they are interesting to us. If positive, the service dispatches the job to Celery.

Currently, the dispatcher is a single service that runs sequentially over the table. Until now I haven't found any issues with this approach, but eventually I intend to transform "launchpad-dispatcher.py" into a Celery service as well, so that the processing & dispatching operations can be done in parallel.

3) This is now a Python package. You should be able to use virtualenv and pip to install it locally. I also revamped the README file and included instructions on how to set things up locally.

4) I disabled a bunch of pylint warnings that were, IMO, false positives in my case. pylint is happy with the code, as well as flake8 and mypy. But I'm open to suggestions/comments, of course.

5) The package now has a testsuite. It is aimed at unittesting so far, but it did help me catch a few silly mistakes when I was developing things. I consider the testsuite to be a WIP; I still want to add more tests and, eventually, work towards having a proper CI/CD setup (thanks for the idea, Athos!).

I think that's it. I'm not in a big hurry to get this approved/deployed, so I'm fine to wait if you would like to talk in person about the MP during the upcoming sprint.

To post a comment you must log in.
Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

Hi Sergio,

As we discussed before, I will review this one :)

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks, Athos.

I'm marking this MP as WIP for now because I decided that it actually makes sense to work on debianizing the package now. I should have something ready by tomorrow.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

This is ready for review.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

A few more notes here:

1) The package has been debianized. It needs a version of setuptools that isn't available in Jammy, so I set up a special PPA where I keep "extra dependencies" needed to build the package. It's here:

  https://launchpad.net/~ubuntu-debuginfod-devs/+archive/ubuntu/ubuntu-debuginfod-deps/+packages

This means that you'll need to pass this PPA as an extra repository to sbuild locally.

2) I reorganized the way the services are located in the source tree. It should be clearer to determine what is part of the Python library and what is not.

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

Hi Sergio, as we discussed before, I injected my reviews as comments in https://code.launchpad.net/~athos-ribeiro/ubuntu-debuginfod/+git/ubuntu-debuginfod/+ref/overhaul.

This should now be complete.

Nice work refactoring the whole code base to make it way more pythonic!

As we discussed offline, I have a few suggestions in my comments, but overall, the work looks great and I see no reason to block deploying it due to any of the review points I raised!

LGTM!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/README.md b/README.md
2index a0f5d14..c74b164 100644
3--- a/README.md
4+++ b/README.md
5@@ -12,8 +12,11 @@ python3-git
6 python3-celery
7 python3-launchpadlib
8 python3-requests
9+python3-requests-oauthlib
10 python3-sdnotify
11 python3-psycopg2
12+python3-pytest (test-only)
13+python3-freezegun (test-only)
14 ```
15
16 * Applications
17@@ -27,10 +30,157 @@ postgresql*
18 support Jammy.
19 ```
20
21-## Setup
22+## Installation
23+
24+You can install the package from the following PPA:
25+
26+ https://launchpad.net/~ubuntu-debuginfod-devs/+archive/ubuntu/ubuntu-debuginfod
27+
28+## Manual setup
29+
30+Nowadays, it is strongly recommended to install the package directly
31+from the PPA:
32+
33+ https://launchpad.net/~ubuntu-debuginfod-devs/+archive/ubuntu/ubuntu-debuginfod
34+
35+The `.deb` package will take care of setting (almost) everything up,
36+and will provide instructions on how to configure anything that might
37+be missing. However, if you would like to set the package up
38+manually, follow the instructions below.
39+
40+### Create user & clone repository
41+
42+First, create a user. Currently the scripts are hardcoded to assume
43+that the username is `mirror`, so let's create it:
44+
45+```
46+# adduser --disabled-password mirror
47+```
48+
49+For the sake of keeping things simple, we will assume that the
50+`ubuntu-debuginfod` repository is cloned at
51+`/home/mirror/ubuntu-debuginfod/`.
52+
53+`ubuntu-debuginfod` is a proper Python package, so you can install it
54+using `virtualenv` and `pip`. Make sure to install all the required
55+dependencies before proceeding.
56+
57+Create the `ubuntu-debuginfod` directory under `~mirror/.config`:
58+
59+```
60+# su - mirror
61+$ mkdir -p $HOME/.config/ubuntu-debuginfod
62+$ chmod -R 0700 $HOME/.config/
63+```
64+
65+### Create mirror directory
66+
67+The scripts assume by default that the mirror directory (i.e., the
68+directory where the debuginfo files will be saved) will be
69+`/srv/debug-mirror/`. Let's create it:
70+
71+```
72+# mkdir -p /srv/debug-mirror/
73+# chown -R mirror:mirror /srv/debug-mirror/
74+```
75+
76+### Create a database
77+
78+Now, create the PostgreSQL database for the service:
79
80 ```
81 # su - postgres
82 $ createuser -d mirror
83 $ createdb -O mirror ubuntu-debuginfod
84 ```
85+
86+### Configure RabbitMQ
87+
88+We will also need to tweak RabbitMQ's `consumer_timeout` to a higher
89+value.
90+
91+```
92+# cat >> /etc/rabbitmq/rabbitmq.conf << __EOF__
93+# 3 hours
94+consumer_timeout = 10800000
95+__EOF__
96+```
97+
98+Let's restart the service so that the new setting takes effect:
99+
100+```
101+# systemctl restart rabbitmq.service
102+```
103+
104+### Install the configuration files
105+
106+For now, you will need to manually install the configuration files.
107+Let's do it:
108+
109+```
110+# cd /etc/systemd/system/
111+# ln -s /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-celery.service
112+# for file in dispatcher poller; do \
113+ ln -s /home/mirror/ubuntu-debuginfod/conf/launchpad-${file}.service; \
114+ ln -s /home/mirror/ubuntu-debuginfod/conf/launchpad-${file}.timer; \
115+ done
116+# cd /etc/default/
117+# ln -s \
118+ /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-celery.default \
119+ ubuntu-debuginfod-celery
120+# ln -s \
121+ /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-launchpad-poller.default \
122+ ubuntu-debuginfod-launchpad-poller
123+```
124+
125+Feel free to make any adjustments to the `.default` files.
126+
127+### Start the service
128+
129+Now, let's reload the systemd daemon and start the Celery service:
130+
131+```
132+# systemctl daemon-reload
133+# systemctl start ubuntu-debuginfod-celery.service
134+```
135+
136+Take a look at the logs to see if everything is OK.
137+
138+### Run the Launchpad poller for the first time
139+
140+To run the Launchpad poller, simply do:
141+
142+```
143+systemctl start ubuntu-debuginfod-launchpad-poller.service
144+```
145+
146+You might want to open another terminal and follow the service logs
147+there.
148+
149+If you want to enable the systemd timer and have the poller be
150+executed every 30 minutes, do:
151+
152+```
153+systemctl enable ubuntu-debuginfod-launchpad-poller.timer
154+```
155+
156+### Run the dispatcher
157+
158+Now that the initial polling has been done, you can run the
159+dispatcher.
160+
161+```
162+systemctl start ubuntu-debuginfod-launchpad-dispatcher.service
163+```
164+
165+Again, you may want to check the logs from another terminal.
166+
167+If the poller was able to find any valid jobs to be dispatched, you
168+should start seeing the `/srv/debug-mirror/` directory be populated.
169+
170+If you want to enable the systemd timer and have the dispatcher be
171+executed every 30 minutes, do:
172+
173+```
174+systemctl enable ubuntu-debuginfod-launchpad-dispatcher.timer
175+```
176diff --git a/conf/ubuntu-debuginfod-celery.default b/conf/ubuntu-debuginfod-celery.default
177index 9f915b0..6741b04 100644
178--- a/conf/ubuntu-debuginfod-celery.default
179+++ b/conf/ubuntu-debuginfod-celery.default
180@@ -1,5 +1,5 @@
181 # App instance to use.
182-CELERY_APP="debuginfod"
183+CELERY_APP="ubuntu_debuginfod.debuginfod"
184
185 # Extra command-line arguments to the worker.
186 #
187diff --git a/conf/ubuntu-debuginfod-celery.service b/conf/ubuntu-debuginfod-celery.service
188index 37b1b9c..edbb388 100644
189--- a/conf/ubuntu-debuginfod-celery.service
190+++ b/conf/ubuntu-debuginfod-celery.service
191@@ -10,7 +10,6 @@ User=mirror
192 Group=mirror
193 Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred
194 EnvironmentFile=/etc/default/ubuntu-debuginfod-celery
195-WorkingDirectory=/home/mirror/ubuntu-debuginfod/
196 ExecStart=/usr/bin/celery -A ${CELERY_APP} worker --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}
197 Restart=always
198 TimeoutStopSec=600
199diff --git a/conf/ubuntu-debuginfod-launchpad-dispatcher.service b/conf/ubuntu-debuginfod-launchpad-dispatcher.service
200new file mode 100644
201index 0000000..169accf
202--- /dev/null
203+++ b/conf/ubuntu-debuginfod-launchpad-dispatcher.service
204@@ -0,0 +1,24 @@
205+[Unit]
206+Description=Dispatcher for ubuntu-debuginfod
207+After=network.target ubuntu-debuginfod-celery.service
208+Requires=ubuntu-debuginfod-celery.service
209+ConditionPathExists=/home/mirror/.config/ubuntu-debuginfod/lp.cred
210+
211+[Service]
212+Type=oneshot
213+User=mirror
214+Group=mirror
215+Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred
216+ExecStart=/usr/bin/python3 /usr/share/ubuntu-debuginfod/services/launchpad-dispatcher.py
217+PrivateDevices=true
218+PrivateUsers=true
219+ProtectKernelTunables=true
220+ProtectControlGroups=true
221+ProtectKernelLogs=true
222+ProtectKernelModules=true
223+MemoryDenyWriteExecute=true
224+RestrictRealtime=true
225+ReadOnlyPaths=/home/mirror/.config/ubuntu-debuginfod/
226+
227+[Install]
228+WantedBy=multi-user.target
229diff --git a/conf/ubuntu-debuginfod-launchpad-dispatcher.timer b/conf/ubuntu-debuginfod-launchpad-dispatcher.timer
230new file mode 100644
231index 0000000..e5cabd4
232--- /dev/null
233+++ b/conf/ubuntu-debuginfod-launchpad-dispatcher.timer
234@@ -0,0 +1,10 @@
235+[Unit]
236+Description=Dispatch jobs for ubuntu-debuginfod
237+
238+[Timer]
239+# Every 30 minutes
240+OnCalendar=*:0/30
241+Persistent=true
242+
243+[Install]
244+WantedBy=timers.target
245diff --git a/conf/ubuntu-debuginfod-poll-lp.default b/conf/ubuntu-debuginfod-launchpad-poller.default
246similarity index 100%
247rename from conf/ubuntu-debuginfod-poll-lp.default
248rename to conf/ubuntu-debuginfod-launchpad-poller.default
249diff --git a/conf/ubuntu-debuginfod-poll-lp.service b/conf/ubuntu-debuginfod-launchpad-poller.service
250similarity index 87%
251rename from conf/ubuntu-debuginfod-poll-lp.service
252rename to conf/ubuntu-debuginfod-launchpad-poller.service
253index a97ffb6..dd98d14 100644
254--- a/conf/ubuntu-debuginfod-poll-lp.service
255+++ b/conf/ubuntu-debuginfod-launchpad-poller.service
256@@ -9,9 +9,8 @@ Type=oneshot
257 User=mirror
258 Group=mirror
259 Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred
260-EnvironmentFile=/etc/default/ubuntu-debuginfod-poll-lp
261-WorkingDirectory=/home/mirror/ubuntu-debuginfod/
262-ExecStart=/usr/bin/python3 /home/mirror/ubuntu-debuginfod/poll_launchpad.py
263+EnvironmentFile=/etc/default/ubuntu-debuginfod-launchpad-poller
264+ExecStart=/usr/bin/python3 /usr/share/ubuntu-debuginfod/services/launchpad-poller.py
265 PrivateDevices=true
266 PrivateUsers=true
267 ProtectKernelTunables=true
268diff --git a/conf/ubuntu-debuginfod-poll-lp.timer b/conf/ubuntu-debuginfod-launchpad-poller.timer
269similarity index 100%
270rename from conf/ubuntu-debuginfod-poll-lp.timer
271rename to conf/ubuntu-debuginfod-launchpad-poller.timer
272diff --git a/ddebgetter.py b/ddebgetter.py
273deleted file mode 100644
274index c441bc6..0000000
275--- a/ddebgetter.py
276+++ /dev/null
277@@ -1,356 +0,0 @@
278-#!/usr/bin/python3
279-
280-# Copyright (C) 2022 Canonical Ltd.
281-
282-# This program is free software: you can redistribute it and/or modify
283-# it under the terms of the GNU General Public License as published by
284-# the Free Software Foundation, either version 3 of the License, or
285-# (at your option) any later version.
286-
287-# This program is distributed in the hope that it will be useful,
288-# but WITHOUT ANY WARRANTY; without even the implied warranty of
289-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
290-# GNU General Public License for more details.
291-
292-# You should have received a copy of the GNU General Public License
293-# along with this program. If not, see <https://www.gnu.org/licenses/>.
294-
295-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
296-
297-import os
298-import lzma
299-import tarfile
300-import subprocess
301-
302-import tempfile
303-
304-from git import Git, Repo
305-from git.exc import GitCommandError
306-
307-from debuggetter import DebugGetter, DebugGetterRetry, DEFAULT_MIRROR_DIR
308-
309-
310-def _validate_generic_request(request):
311- """Validate a request coming from Celery.
312-
313- :param dict[str, str] request: The dictionary containing the
314- request to be validated."""
315- if request is None:
316- raise TypeError("Invalid request (None)")
317- if request.get("architecture") is None:
318- raise TypeError("No 'architecture' in request")
319-
320-def validate_ddeb_request(request):
321- """Validate a request to fetch a ddeb from the main archive.
322-
323- :param dict[str, str] request: The dictionary containing the
324- request to be validated."""
325- _validate_generic_request(request)
326- if request.get("ddeb_url") is None:
327- raise TypeError("No 'ddeb_url' in request")
328- if request["architecture"] == "source":
329- raise ValueError("Wrong request: source fetch")
330-
331-def validate_source_code_request(request):
332- """Validate a request to fetch a source package from the main
333- archive.
334-
335- :param dict[str, str] request: The dictionary containing the
336- request to be validated."""
337- _validate_generic_request(request)
338- if request.get("source_urls") is None:
339- raise TypeError("No 'source_urls' in request")
340- if request["architecture"] != "source":
341- raise ValueError("Wrong request: ddeb fetch")
342-
343-
344-class DdebGetter(DebugGetter):
345- """Get (fetch) a ddeb."""
346-
347- def __init__(self, subdir="ddebs", mirror_dir=DEFAULT_MIRROR_DIR):
348- """Initialize the object.
349-
350- See DebugGetter's __init__ for an explanation of the arguments."""
351- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
352-
353- def process_request(self, request):
354- """Process a request, usually coming from Celery.
355-
356- :param dict[str, str] request: The dictionary containing the
357- information necessary to fetch this ddeb."""
358- validate_ddeb_request(request)
359- self.do_process_request(request)
360-
361- def do_process_request(self, request, credentials=None):
362- """Perform the actual processing of the request.
363-
364- :param dict[str, str] request: The dictionary containing the
365- information necessary to fetch this ddeb.
366-
367- :param requests_oauthlib.OAuth1 credentials: The credentials
368- to be used when downloading the artifact from Launchpad.
369- Default is None, which means anonymous."""
370- self._logger.debug(f"Processing request to download ddeb: {request}")
371- self._download_ddeb(
372- request["source_package"],
373- request["component"],
374- request["ddeb_url"],
375- credentials=credentials,
376- )
377-
378- def _download_ddeb(self, source_package, component, ddeb_url, credentials=None):
379- """Download a ddeb associated with a package.
380-
381- :param str source_package: Source package name.
382-
383- :param str component: Source package component.
384-
385- :param str ddeb_url: The ddeb URL.
386-
387- :param requests_oauthlib.OAuth1 credentials: The credentials
388- to be used when downloading the artifact from Launchpad.
389- Default is None, which means anonymous."""
390- savepath = self._make_savepath(source_package, component)
391- self._logger.debug(f"Downloading '{ddeb_url}' into '{savepath}'")
392- if credentials is not None:
393- self._logger.debug("Downloading with credentials")
394- self._download_from_lp(ddeb_url, savepath, credentials=credentials)
395-
396-
397-# The function below was taken from git-ubuntu. We need to make
398-# sure we perform the same version -> tag transformation as it
399-# does.
400-def git_dep14_tag(version):
401- """Munge a version string according to http://dep.debian.net/deps/dep14/
402-
403- :param str version: The version to be adjusted."""
404- version = version.replace("~", "_").replace(":", "%").replace("..", ".#.")
405- if version.endswith("."):
406- version = version + "#"
407- if version.endswith(".lock"):
408- pre, _, _ = version.partition(".lock")
409- version = pre + ".#lock"
410- return version
411-
412-
413-def adjust_tar_filepath(tarinfo):
414- """Adjust the filepath for a TarInfo file.
415-
416- This function is needed because TarFile.add strips the leading
417- slash from the filenames, so we have to workaround it by
418- re-adding the slash ourselves.
419-
420- This function is intended to be used as a callback provided to
421- TarFile.add.
422-
423- :param TarInfo tarinfo: The tarinfo."""
424- tarinfo.name = os.path.join("/", tarinfo.name)
425- return tarinfo
426-
427-
428-class DdebSourceCodeGetter(DebugGetter):
429- """Get (fetch) the source code associated with a ddeb."""
430-
431- def __init__(self, subdir="ddebs", mirror_dir=DEFAULT_MIRROR_DIR):
432- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
433-
434- def process_request(self, request):
435- """Process a request, usually coming from Celery.
436-
437- :param dict[str, str] request: The dictionary containing the
438- information necessary to fetch this source code."""
439- validate_source_code_request(request)
440- self.do_process_request(request)
441-
442- def do_process_request(self, request, fallback_to_git=True, credentials=None):
443- """Perform the actual processing of the request.
444-
445- :param dict[str, str] request: The dictionary containing the
446- information necessary to fetch this source code.
447-
448- :param requests_oauthlib.OAuth1 credentials: The credentials
449- to be used when downloading the artifact from Launchpad.
450- Default is None, which means anonymous."""
451- self._logger.debug(f"Processing request to download source code: {request}")
452- self._download_source_code(
453- request["source_package"],
454- request["version"],
455- request["component"],
456- request["source_urls"],
457- fallback_to_git=fallback_to_git,
458- credentials=credentials,
459- )
460-
461- def _download_source_code_from_git(self, source_package, version, filepath):
462- """Download the source code using Launchpad's git repository.
463-
464- :param str source_package: Source package name.
465-
466- :param str version: Source package version.
467-
468- :param str filepath: The full pathname where the resulting
469- source code tarball should be saved.
470-
471- :rtype: bool
472-
473- This method returns True when the operation succeeds, or False
474- *iff* the "git clone" command fails to run. Otherwise, this
475- function will throw an exception."""
476- with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
477- g = Git()
478- git_dir = os.path.join(source_dir, f"{source_package}")
479- tag = "applied/" + git_dep14_tag(f"{version}")
480- self._logger.debug(
481- f"Cloning '{source_package}' git repo into '{git_dir}' (tag: '{tag}')"
482- )
483- try:
484- g.clone(
485- f"https://git.launchpad.net/ubuntu/+source/{source_package}",
486- git_dir,
487- depth=1,
488- branch=tag,
489- )
490- except GitCommandError as e:
491- # Couldn't perform the download. Let's signal and
492- # bail out.
493- self._logger.warning(
494- f"Could not clone git repo for '{source_package}': {e}"
495- )
496- return False
497- repo = Repo(git_dir)
498- prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
499- self._logger.debug(
500- f"Archiving git repo for '{source_package}-{version}' as '{filepath}'"
501- )
502- with tempfile.NamedTemporaryFile() as tmpfile:
503- with lzma.open(tmpfile.name, "w") as xzfile:
504- repo.archive(xzfile, prefix=prefix_path, format="tar")
505- self._try_to_move_atomically(tmpfile.name, filepath)
506-
507- return True
508-
509- def _download_source_code_from_dsc(
510- self, source_package, version, filepath, source_urls, credentials=None
511- ):
512- """Download the source code using the .dsc file.
513-
514- :param str source_package: Source package name.
515-
516- :param str version: Source package version.
517-
518- :param str filepath: The full pathname where the resulting
519- source code tarball should be saved.
520-
521- :param list source_urls: List of URLs used to fetch the source
522- package. This is usually the list returned by the
523- sourceFileUrls() Launchpad API call.
524-
525- :param requests_oauthlib.OAuth1 credentials: The credentials
526- to be used when downloading the artifact from Launchpad.
527- Default is None, which means anonymous.
528-
529- :rtype: bool
530- This method returns True when the operation succeeds, or False
531- *iff* the "dpkg-source -x" command fails to run. Otherwise,
532- this function will throw an exception."""
533- with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
534- for url in source_urls:
535- self._download_from_lp(url, source_dir, credentials=credentials)
536-
537- dscfile = None
538- for f in os.listdir(source_dir):
539- newf = os.path.join(source_dir, f)
540- if os.path.isfile(newf) and f.endswith(".dsc"):
541- dscfile = newf
542- break
543-
544- if dscfile is None:
545- self._logger.warning(
546- "Could not find .dsc file, even though it should exist."
547- )
548- return False
549-
550- outdir = os.path.join(source_dir, "outdir")
551- self._logger.debug(f"Will call 'dpkg-source -x {dscfile} {outdir}'")
552- try:
553- subprocess.run(
554- ["/usr/bin/dpkg-source", "-x", dscfile, outdir],
555- cwd=source_dir,
556- check=True,
557- )
558- except subprocess.CalledProcessError:
559- self._logger.warning(f"Call to 'dpkg-source -x' failed.")
560- return False
561-
562- if not os.path.isdir(outdir):
563- self._logger.warning(
564- f"'{outdir}' has not been created by 'dpkg-source -x'."
565- )
566- return False
567-
568- prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
569-
570- with tempfile.NamedTemporaryFile() as tmpfile:
571- with tarfile.open(tmpfile.name, "w:xz") as tfile:
572- tfile.add(outdir, arcname=prefix_path, filter=adjust_tar_filepath)
573- self._try_to_move_atomically(tmpfile.name, filepath)
574-
575- return True
576-
577- def _download_source_code(
578- self,
579- source_package,
580- version,
581- component,
582- source_urls,
583- fallback_to_git=True,
584- credentials=None,
585- ):
586-
587- """Download the source code for a package.
588-
589- :param str source_package: Source package name.
590-
591- :param str version: Source package version.
592-
593- :param str component: Source package component.
594-
595- :param list source_urls: List of source file URLS.
596-
597- :param boolean fallback_to_git: Whether we should try to fetch
598- the source code using git if the regular approach (via
599- dget) fails. Default to True.
600-
601- :param requests_oauthlib.OAuth1 credentials: The credentials
602- to be used when downloading the artifact from Launchpad.
603- Default is None, which means anonymous."""
604- savepath = self._make_savepath(source_package, component)
605- txzfilepath = os.path.join(savepath, f"{source_package}-{version}.tar.xz")
606- if os.path.exists(txzfilepath):
607- self._logger.debug(f"'{txzfilepath}' already exists; doing nothing.")
608- return
609-
610- if self._download_source_code_from_dsc(
611- source_package,
612- version,
613- txzfilepath,
614- source_urls,
615- credentials=credentials
616- ):
617- self._logger.info(
618- f"Downloaded source code from dsc for '{source_package}-{version}' as '{txzfilepath}'"
619- )
620- return
621-
622- if fallback_to_git and self._download_source_code_from_git(
623- source_package, version, txzfilepath
624- ):
625- self._logger.info(
626- f"Downloaded source code from git for '{source_package}-{version}' as '{txzfilepath}'"
627- )
628- return
629-
630- # In the (likely?) event that there is a problem with
631- # Launchpad, let's raise an exception signalling that we'd
632- # like to retry the task.
633- raise DebugGetterRetry()
634diff --git a/ddebpoller.py b/ddebpoller.py
635deleted file mode 100644
636index 9883782..0000000
637--- a/ddebpoller.py
638+++ /dev/null
639@@ -1,200 +0,0 @@
640-#!/usr/bin/python3
641-
642-# Copyright (C) 2022 Canonical Ltd.
643-
644-# This program is free software: you can redistribute it and/or modify
645-# it under the terms of the GNU General Public License as published by
646-# the Free Software Foundation, either version 3 of the License, or
647-# (at your option) any later version.
648-
649-# This program is distributed in the hope that it will be useful,
650-# but WITHOUT ANY WARRANTY; without even the implied warranty of
651-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
652-# GNU General Public License for more details.
653-
654-# You should have received a copy of the GNU General Public License
655-# along with this program. If not, see <https://www.gnu.org/licenses/>.
656-
657-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
658-
659-import os
660-from urllib import parse
661-import datetime
662-from debugpoller import DebugPoller
663-
664-
665-class DdebPoller(DebugPoller):
666- """Perform the Launchpad polling and obtain a list of ddebs and source
667- packages."""
668-
669- def __init__(
670- self,
671- initial_interval=1,
672- force_initial_interval=False,
673- fetch_all_on_first_run=False,
674- dry_run=False,
675- anonymous=True,
676- ):
677- """Initialize the object using 'ddeb' as its name.
678-
679- Look at DebugPoller's docstring for an explanation about the arguments."""
680- super().__init__(
681- initial_interval=initial_interval,
682- force_initial_interval=force_initial_interval,
683- fetch_all_on_first_run=fetch_all_on_first_run,
684- dry_run=dry_run,
685- anonymous=anonymous,
686- )
687-
688- def get_ddebs(self, archive=None, ppainfo=None):
689- """Get the list of ddebs that have been published since the last
690- timestamp from Launchpad.
691-
692- :param archive Launchpad archive: The Launchpad archive that
693- should be queried for new binaries. Defaults to
694- self._main_archive.
695-
696- :param ppainfo dict(str, str): A dictionary containing three
697- keys: "ppauser", "ppaname" and "isprivateppa". It should
698- only be provided when dealing with a PPA; otherwise, the
699- default is None.
700-
701- :rtype: dict, datetime.datetime
702-
703- Return a dictionary containing all ddebs found, and also the
704- new timestamp that should then be recorded by calling
705- record_timestamp.
706-
707- """
708- timestamp, real_timestamp = self._get_normal_and_real_timestamps()
709-
710- if archive is None:
711- archive = self._main_archive
712-
713- if ppainfo is None:
714- archive_label = f"'{archive.displayname}' archive"
715- else:
716- archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})"
717-
718- self._logger.info(f"Polling ddebs created since '{real_timestamp}' from {archive_label}")
719-
720- result = []
721- latest_timestamp_created = timestamp
722- for pkg in archive.getPublishedBinaries(
723- order_by_date=True, created_since_date=real_timestamp
724- ):
725- if pkg.status not in ("Pending", "Published"):
726- continue
727- if not pkg.is_debug:
728- continue
729-
730- ddeb_urls = pkg.binaryFileUrls()
731- if len(ddeb_urls) == 0:
732- # Safety check.
733- continue
734-
735- if (
736- latest_timestamp_created is None
737- or pkg.date_created > latest_timestamp_created
738- ):
739- latest_timestamp_created = pkg.date_created
740-
741- srcname = pkg.source_package_name
742- binname = pkg.binary_package_name
743- binver = pkg.binary_package_version
744- component = pkg.component_name
745-
746- # We create one message (which will eventually become a
747- # Celery task) per ddeb. This makes it easier later to do
748- # deduplication, and also has the benefit of making the
749- # downloading process more granular for multiple ddebs.
750- for url in ddeb_urls:
751- # Obtain the ddeb filename and its architecture.
752- ddeb = os.path.basename(parse.urlparse(url).path)
753- _, _, arch_and_extension = ddeb.split("_")
754- arch, _ = arch_and_extension.split(".")
755-
756- msg = {
757- "source_package": srcname,
758- "binary_package": binname,
759- "version": binver,
760- "component": component,
761- "ddeb_url": url,
762- "ddeb_filename": ddeb,
763- "architecture": arch,
764- }
765- if ppainfo is not None:
766- msg.update(ppainfo)
767-
768- result.append(msg)
769-
770- return result, latest_timestamp_created
771-
772- def get_sources(self, archive=None, ppainfo=None):
773- """Get the list of source packages that have been published since the
774- last timestamp from Launchpad.
775-
776- :param archive Launchpad archive: The Launchpad archive that
777- should be queried for new sources. Defaults to
778- self._main_archive.
779-
780- :param ppainfo dict(str, str): A dictionary containing three
781- keys: "ppauser", "ppaname" and "isprivateppa". It should
782- only be provided when dealing with a PPA; otherwise, the
783- default is None.
784-
785- :rtype: dict, datetime.datetime
786-
787- Return a dictionary containing all source packages found, and
788- also the new timestamp that should then be recorded by calling
789- record_timestamp.
790-
791- """
792- timestamp, real_timestamp = self._get_normal_and_real_timestamps()
793-
794- if archive is None:
795- archive = self._main_archive
796-
797- if ppainfo is None:
798- archive_label = f"'{archive.displayname}' archive"
799- else:
800- archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})"
801-
802- self._logger.info(f"Polling source packages created since '{real_timestamp}' from {archive_label}")
803-
804- result = []
805- latest_timestamp_created = timestamp
806- for pkg in archive.getPublishedSources(
807- order_by_date=True, created_since_date=real_timestamp
808- ):
809- if pkg.status not in ("Pending", "Published"):
810- continue
811-
812- src_urls = pkg.sourceFileUrls()
813- if len(src_urls) == 0:
814- # Safety check.
815- continue
816-
817- if (
818- latest_timestamp_created is None
819- or pkg.date_created > latest_timestamp_created
820- ):
821- latest_timestamp_created = pkg.date_created
822-
823- srcname = pkg.source_package_name
824- srcver = pkg.source_package_version
825- component = pkg.component_name
826-
827- msg = {
828- "source_package": srcname,
829- "version": srcver,
830- "component": component,
831- "source_urls": src_urls,
832- "architecture": "source",
833- }
834- if ppainfo is not None:
835- msg.update(ppainfo)
836-
837- result.append(msg)
838-
839- return result, latest_timestamp_created
840diff --git a/debian/changelog b/debian/changelog
841new file mode 100644
842index 0000000..ca2645a
843--- /dev/null
844+++ b/debian/changelog
845@@ -0,0 +1,5 @@
846+ubuntu-debuginfod (0.1.0) UNRELEASED; urgency=medium
847+
848+ * Initial Release.
849+
850+ -- Sergio Durigan Junior <sergio.durigan@canonical.com> Wed, 26 Apr 2023 15:20:29 -0400
851diff --git a/debian/control b/debian/control
852new file mode 100644
853index 0000000..e2e8a88
854--- /dev/null
855+++ b/debian/control
856@@ -0,0 +1,50 @@
857+Source: ubuntu-debuginfod
858+Section: devel
859+Priority: optional
860+Maintainer: Ubuntu Server <ubuntu-server@lists.ubuntu.com>
861+Uploaders: Sergio Durigan Junior <sergio.durigan@canonical.com>
862+Build-Depends: debhelper-compat (= 13),
863+ dh-python,
864+ python3-setuptools,
865+ pybuild-plugin-pyproject,
866+ python3-all,
867+ python3-git,
868+ python3-celery,
869+ python3-launchpadlib,
870+ python3-requests,
871+ python3-requests-oauthlib,
872+ python3-sdnotify,
873+ python3-psycopg2,
874+ python3-pytest <!nocheck>,
875+ python3-freezegun <!nocheck>,
876+Standards-Version: 4.6.0
877+Homepage: https://launchpad.net/ubuntu-debuginfod
878+Vcs-Browser: https://git.launchpad.net/ubuntu-debuginfod
879+Vcs-Git: https://git.launchpad.net/ubuntu-debuginfod
880+#Testsuite: autopkgtest-pkg-python
881+Rules-Requires-Root: no
882+
883+Package: ubuntu-debuginfod
884+Architecture: all
885+Pre-Depends: adduser, ucf
886+Depends: ${python3:Depends},
887+ ${misc:Depends},
888+ python3-ubuntu-debuginfod (= ${source:Version}),
889+Description: Scripts to poll/fetch artifacts for Ubuntu's debuginfod instance
890+ This package offers a set of scripts and utilities that are used to
891+ manage Ubuntu's debuginfod instance (<https://debuginfod.ubuntu.com>).
892+
893+Package: python3-ubuntu-debuginfod
894+Section: python
895+Architecture: all
896+Depends: ${python3:Depends},
897+ ${misc:Depends},
898+ rabbitmq-server,
899+ celery,
900+ postgresql,
901+Suggests: ubuntu-debuginfod (= ${source:Version})
902+Description: Scripts to poll/fetch artifacts for Ubuntu's debuginfod instance (Python library)
903+ This package offers a set of scripts and utilities that are used to
904+ manage Ubuntu's debuginfod instance (<https://debuginfod.ubuntu.com>).
905+ .
906+ This is the Python library.
907diff --git a/debian/copyright b/debian/copyright
908new file mode 100644
909index 0000000..7aec2cb
910--- /dev/null
911+++ b/debian/copyright
912@@ -0,0 +1,29 @@
913+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
914+Upstream-Name: ubuntu-debuginfod
915+Upstream-Contact: Sergio Durigan Junior <sergio.durigan@canonical.com>
916+Source: https://launchpad.net/ubuntu-debuginfod
917+
918+Files: *
919+Copyright: 2022-2023 Canonical Ltd.
920+License: GPL-3.0+
921+
922+Files: debian/*
923+Copyright: 2022-2023 Canonical Ltd.
924+License: GPL-3.0+
925+
926+License: GPL-3.0+
927+ This program is free software: you can redistribute it and/or modify
928+ it under the terms of the GNU General Public License as published by
929+ the Free Software Foundation, either version 3 of the License, or
930+ (at your option) any later version.
931+ .
932+ This package is distributed in the hope that it will be useful,
933+ but WITHOUT ANY WARRANTY; without even the implied warranty of
934+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
935+ GNU General Public License for more details.
936+ .
937+ You should have received a copy of the GNU General Public License
938+ along with this program. If not, see <https://www.gnu.org/licenses/>.
939+ .
940+ On Debian/Ubuntu systems, the complete text of the GNU General
941+ Public License version 3 can be found in "/usr/share/common-licenses/GPL-3".
942diff --git a/debian/rules b/debian/rules
943new file mode 100755
944index 0000000..f31c23e
945--- /dev/null
946+++ b/debian/rules
947@@ -0,0 +1,19 @@
948+#!/usr/bin/make -f
949+
950+export PYBUILD_NAME = ubuntu-debuginfod
951+
952+%:
953+ dh $@ --with python3 --buildsystem=pybuild
954+
955+override_dh_auto_test:
956+ifeq (,$(findstring nocheck,$(DEB_BUILD_OPTIONS)))
957+ PYBUILD_SYSTEM=custom \
958+ PYBUILD_TEST_ARGS="PYTHONPATH={build_dir} \
959+ {interpreter} -m pytest -v" \
960+ dh_auto_test
961+endif
962+
963+override_dh_installsystemd:
964+ dh_installsystemd --name=ubuntu-debuginfod-celery
965+ dh_installsystemd --name=ubuntu-debuginfod-launchpad-poller --no-start --no-enable
966+ dh_installsystemd --name=ubuntu-debuginfod-launchpad-dispatcher --no-start --no-enable
967diff --git a/debian/source/format b/debian/source/format
968new file mode 100644
969index 0000000..89ae9db
970--- /dev/null
971+++ b/debian/source/format
972@@ -0,0 +1 @@
973+3.0 (native)
974diff --git a/debian/source/options b/debian/source/options
975new file mode 100644
976index 0000000..cb61fa5
977--- /dev/null
978+++ b/debian/source/options
979@@ -0,0 +1 @@
980+extend-diff-ignore = "^[^/]*[.]egg-info/"
981diff --git a/debian/ubuntu-debuginfod.install b/debian/ubuntu-debuginfod.install
982new file mode 100644
983index 0000000..52b779e
984--- /dev/null
985+++ b/debian/ubuntu-debuginfod.install
986@@ -0,0 +1,3 @@
987+services/ usr/share/ubuntu-debuginfod/
988+conf/*.default usr/share/ubuntu-debuginfod/etc-default/
989+utils/ usr/share/ubuntu-debuginfod/
990diff --git a/debian/ubuntu-debuginfod.postinst b/debian/ubuntu-debuginfod.postinst
991new file mode 100644
992index 0000000..e01120c
993--- /dev/null
994+++ b/debian/ubuntu-debuginfod.postinst
995@@ -0,0 +1,168 @@
996+#!/bin/sh
997+# postinst script for ubuntu-debuginfod
998+#
999+# see: dh_installdeb(1)
1000+
1001+set -e
1002+
1003+# summary of how this script can be called:
1004+# * <postinst> `configure' <most-recently-configured-version>
1005+# * <old-postinst> `abort-upgrade' <new version>
1006+# * <conflictor's-postinst> `abort-remove' `in-favour' <package>
1007+# <new-version>
1008+# * <postinst> `abort-remove'
1009+# * <deconfigured's-postinst> `abort-deconfigure' `in-favour'
1010+# <failed-install-package> <version> `removing'
1011+# <conflicting-package> <version>
1012+# for details, see https://www.debian.org/doc/debian-policy/ or
1013+# the debian-policy package
1014+
1015+# The user we will run the services as.
1016+UD_USER=$(grep '^User=' /lib/systemd/system/ubuntu-debuginfod-celery.service | cut -d= -f2)
1017+# The mirror directory.
1018+UD_MIRROR_DIR="/srv/debug-mirror"
1019+# The database name.
1020+UD_DB="ubuntu-debuginfod"
1021+# The location of the Launchpad credential file.
1022+UD_LP_CREDENTIAL_FILE=$(grep '^Environment=LP_CREDENTIALS_FILE' /lib/systemd/system/ubuntu-debuginfod-celery.service | cut -d= -f3)
1023+
1024+# Install the *.default files.
1025+install_etc_default_files()
1026+{
1027+ for file in celery launchpad-poller; do
1028+ ucf --three-way \
1029+ "/usr/share/ubuntu-debuginfod/etc-default/ubuntu-debuginfod-${file}.default" \
1030+ "/etc/default/ubuntu-debuginfod-${file}"
1031+ ucfr ubuntu-debuginfod \
1032+ "/etc/default/ubuntu-debuginfod-${file}"
1033+ done
1034+}
1035+
1036+# Create a user if it doesn't exist.
1037+create_user()
1038+{
1039+ if ! getent passwd "${UD_USER}" > /dev/null; then
1040+ adduser --quiet --disabled-password \
1041+ --gecos "ubuntu-debuginfod mirror user" \
1042+ "${UD_USER}"
1043+ fi
1044+
1045+ lpcreddir=$(dirname "${UD_LP_CREDENTIAL_FILE}")
1046+ if [ ! -d "${lpcreddir}" ]; then
1047+ mkdir -p "${lpcreddir}"
1048+ chown "${UD_USER}":"${UD_USER}" "${lpcreddir}"
1049+ fi
1050+}
1051+
1052+# Helper function to run commands as the postgres user.
1053+runuser_postgres()
1054+{
1055+ cd /tmp && runuser -u postgres -- "$@" && cd -
1056+}
1057+
1058+# Create the mirror directory if it doesn't exist.
1059+create_mirror_dir()
1060+{
1061+ if [ ! -d "${UD_MIRROR_DIR}" ]; then
1062+ mkdir -p "${UD_MIRROR_DIR}"
1063+ chown "${UD_USER}":"${UD_USER}" "${UD_MIRROR_DIR}"
1064+ fi
1065+}
1066+
1067+# Create the database user if it doesn't exist.
1068+create_database_user()
1069+{
1070+ if ! runuser_postgres psql postgres -tAc \
1071+ "SELECT 1 FROM pg_roles WHERE rolname='${UD_USER}'" | \
1072+ grep -qFx 1; then
1073+ runuser -u postgres -- createuser -d "${UD_USER}"
1074+ fi
1075+}
1076+
1077+# Create the database if it doesn't exist.
1078+create_database()
1079+{
1080+ create_database_user
1081+
1082+ if ! runuser_postgres psql -ltq | \
1083+ cut -d \| -f 1 | \
1084+ grep -qFw "${UD_DB}"; then
1085+ runuser_postgres createdb -O "${UD_USER}" "${UD_DB}"
1086+ fi
1087+}
1088+
1089+# Check that the recommended RabbitMQ configuration is present. If it
1090+# isn't, display a warning.
1091+check_rabbitmq_configuration()
1092+{
1093+ rabconf="/etc/rabbitmq/rabbitmq.conf"
1094+ if [ -f "${rabconf}" ] && \
1095+ grep -q '^consumer_timeout' "${rabconf}"; then
1096+ return
1097+ fi
1098+
1099+ cat << __EOF__
1100+
1101+ * It is suggested that RabbitMQ be configured with the following option:
1102+
1103+ \$ cat ${rabconf}
1104+ consumer_timeout = 10800000
1105+__EOF__
1106+}
1107+
1108+# Obtain the Launchpad credentials.
1109+#
1110+# TODO: See if there's a way to allow the user to bail out.
1111+# obtain_launchpad_credentials()
1112+# {
1113+# if [ ! -f "${UD_LP_CREDENTIAL_FILE}" ]; then
1114+# /usr/share/ubuntu-debuginfod/utils/obtain-launchpad-credentials.py \
1115+# "${UD_LP_CREDENTIAL_FILE}"
1116+# chown "${UD_USER}":"${UD_USER}" "${UD_LP_CREDENTIAL_FILE}"
1117+# fi
1118+# }
1119+
1120+# Check if the Launchpad credentials exist, and warn the user
1121+# otherwise.
1122+check_launchpad_credentials()
1123+{
1124+ if [ ! -f "${UD_LP_CREDENTIAL_FILE}" ]; then
1125+ cat << __EOF__
1126+
1127+ * The Launchpad credentials have not been obtained yet. You need to
1128+ obtain them in order to properly start the services. To do that,
1129+ please run:
1130+
1131+ \$ runuser -u ${UD_USER} -- \\
1132+ /usr/share/ubuntu-debuginfod/utils/obtain-launchpad-credentials.py \\
1133+ ${UD_LP_CREDENTIAL_FILE}
1134+__EOF__
1135+ fi
1136+}
1137+
1138+case "$1" in
1139+ configure)
1140+ install_etc_default_files
1141+ create_user
1142+ create_mirror_dir
1143+ create_database
1144+ check_rabbitmq_configuration
1145+ #obtain_launchpad_credentials
1146+ check_launchpad_credentials
1147+ ;;
1148+
1149+ abort-upgrade|abort-remove|abort-deconfigure)
1150+ ;;
1151+
1152+ *)
1153+ echo "postinst called with unknown argument \`$1'" >&2
1154+ exit 1
1155+ ;;
1156+esac
1157+
1158+# dh_installdeb will replace this with shell code automatically
1159+# generated by other debhelper scripts.
1160+
1161+#DEBHELPER#
1162+
1163+exit 0
1164diff --git a/debian/ubuntu-debuginfod.postrm b/debian/ubuntu-debuginfod.postrm
1165new file mode 100644
1166index 0000000..366eee5
1167--- /dev/null
1168+++ b/debian/ubuntu-debuginfod.postrm
1169@@ -0,0 +1,39 @@
1170+#!/bin/sh
1171+
1172+set -e
1173+
1174+purge_etc_default_files()
1175+{
1176+ for file in celery launchpad-poller; do
1177+ if command -v ucf > /dev/null; then
1178+ ucf --purge \
1179+ "/etc/default/ubuntu-debuginfod-${file}"
1180+ fi
1181+
1182+ if command -v ucfr > /dev/null; then
1183+ ucfr --purge ubuntu-debuginfod \
1184+ "/etc/default/ubuntu-debuginfod-${file}"
1185+ fi
1186+ done
1187+}
1188+
1189+case "$1" in
1190+ purge)
1191+ purge_etc_default_files
1192+ ;;
1193+
1194+ remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear)
1195+ ;;
1196+
1197+ *)
1198+ echo "postrm called with unknown argument \`$1'" >&2
1199+ exit 1
1200+ ;;
1201+esac
1202+
1203+# dh_installdeb will replace this with shell code automatically
1204+# generated by other debhelper scripts.
1205+
1206+#DEBHELPER#
1207+
1208+exit 0
1209diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service
1210new file mode 120000
1211index 0000000..6537f37
1212--- /dev/null
1213+++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service
1214@@ -0,0 +1 @@
1215+../conf/ubuntu-debuginfod-celery.service
1216\ No newline at end of file
1217diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service
1218new file mode 120000
1219index 0000000..9f7c034
1220--- /dev/null
1221+++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service
1222@@ -0,0 +1 @@
1223+../conf/ubuntu-debuginfod-launchpad-dispatcher.service
1224\ No newline at end of file
1225diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer
1226new file mode 120000
1227index 0000000..a921a02
1228--- /dev/null
1229+++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer
1230@@ -0,0 +1 @@
1231+../conf/ubuntu-debuginfod-launchpad-dispatcher.timer
1232\ No newline at end of file
1233diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service
1234new file mode 120000
1235index 0000000..aae0131
1236--- /dev/null
1237+++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service
1238@@ -0,0 +1 @@
1239+../conf/ubuntu-debuginfod-launchpad-poller.service
1240\ No newline at end of file
1241diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer
1242new file mode 120000
1243index 0000000..253a01c
1244--- /dev/null
1245+++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer
1246@@ -0,0 +1 @@
1247+../conf/ubuntu-debuginfod-launchpad-poller.timer
1248\ No newline at end of file
1249diff --git a/debuggetter.py b/debuggetter.py
1250deleted file mode 100644
1251index e5500a7..0000000
1252--- a/debuggetter.py
1253+++ /dev/null
1254@@ -1,149 +0,0 @@
1255-#!/usr/bin/python3
1256-
1257-# Copyright (C) 2022 Canonical Ltd.
1258-
1259-# This program is free software: you can redistribute it and/or modify
1260-# it under the terms of the GNU General Public License as published by
1261-# the Free Software Foundation, either version 3 of the License, or
1262-# (at your option) any later version.
1263-
1264-# This program is distributed in the hope that it will be useful,
1265-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1266-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1267-# GNU General Public License for more details.
1268-
1269-# You should have received a copy of the GNU General Public License
1270-# along with this program. If not, see <https://www.gnu.org/licenses/>.
1271-
1272-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
1273-
1274-import os
1275-import requests
1276-from urllib import parse
1277-import logging
1278-import tempfile
1279-import shutil
1280-import re
1281-
1282-
1283-class DebugGetterTimeout(Exception):
1284- """Default exception to raise when there's a timeout issue."""
1285-
1286-
1287-class DebugGetterRetry(Exception):
1288- """Default exception to raise when we'd like to signal that the task
1289- should be retried."""
1290-
1291-
1292-DEFAULT_MIRROR_DIR = "/srv/debug-mirror"
1293-
1294-
1295-class DebugGetter:
1296- """Base class for a Debug Getter."""
1297-
1298- def __init__(self, subdir, mirror_dir=DEFAULT_MIRROR_DIR):
1299- """Initialize the object.
1300-
1301- :param str mirror_dir: The directory we use to save the
1302- mirrored files from Launchpad.
1303-
1304- :param str subdir: The subdirectory (insider mirror_dir) where
1305- the module will save its files. For example, a ddeb
1306- getter module should specify "ddebs" here."""
1307- self._mirror_dir = mirror_dir
1308- self._subdir = subdir
1309- self._logger = logging.getLogger(__name__)
1310-
1311- def _make_savepath(self, source_package, component):
1312- """Return the full save path for a package.
1313-
1314- :param str source_package: The package name.
1315- :param str component: The component name (main, universe, etc.)."""
1316- if source_package.startswith("lib"):
1317- pkgname_initials = source_package[:4]
1318- else:
1319- pkgname_initials = source_package[0]
1320-
1321- return os.path.join(
1322- self._mirror_dir,
1323- self._subdir,
1324- component,
1325- pkgname_initials,
1326- source_package,
1327- )
1328-
1329- def _try_to_move_atomically(self, src, dst):
1330- """Try to move SRC to DST atomically.
1331-
1332- This function assumes that SRC and DST have different filenames.
1333-
1334- :param str src: The source file (full path).
1335-
1336- :param str dst: The destination file (full path)."""
1337- self._logger.debug(f"Trying to move '{src}' to '{dst}' atomically")
1338- savepath = os.path.dirname(dst)
1339- os.makedirs(savepath, mode=0o755, exist_ok=True)
1340- newtmpfile = os.path.join(savepath, os.path.basename(src))
1341- shutil.copyfile(src, newtmpfile)
1342- try:
1343- shutil.move(newtmpfile, dst)
1344- except shutil.SameFileError:
1345- os.remove(newtmpfile)
1346- self._logger.warning(
1347- f"Could not create '{dst}' atomically: same file already exists"
1348- )
1349-
1350- def _download_from_lp(self, url, savepath, credentials=None):
1351- """Download a file from Launchpad.
1352-
1353- This method tries to download the file in chunks into a
1354- temporary file and then does an atomic move to the final
1355- destination.
1356-
1357- :param str url: The URL that should be downloaded.
1358-
1359- :param str savepath: The full path (minus the filename) where
1360- the file should be saved.
1361-
1362- :param requests_oauthlib.OAuth1 credentials: The credentials
1363- to be used when downloading the artifact from Launchpad.
1364- Default is None, which means anonymous."""
1365- filepath = os.path.join(savepath, os.path.basename(parse.urlparse(url).path))
1366- if os.path.exists(filepath):
1367- self._logger.debug(f"'{filepath}' exists, doing nothing")
1368- return
1369-
1370- # Launchpad only accepts authenticated requests if we use
1371- # 'https://api.launchpad.net...'
1372- if credentials is not None:
1373- url = re.sub(r"^https://launchpad.net/", "https://api.launchpad.net/devel/", url)
1374-
1375- try:
1376- with requests.Session() as s:
1377- with s.get(
1378- url, allow_redirects=True, timeout=10, stream=True, auth=credentials
1379- ) as r:
1380- r.raise_for_status()
1381- with tempfile.NamedTemporaryFile(mode="wb") as tmpfile:
1382- shutil.copyfileobj(r.raw, tmpfile)
1383- tmpfile.flush()
1384- self._try_to_move_atomically(tmpfile.name, filepath)
1385- except (
1386- requests.ConnectionError,
1387- requests.HTTPError,
1388- requests.ConnectTimeout,
1389- requests.ReadTimeout,
1390- requests.Timeout,
1391- ) as e:
1392- self._logger.warning(f"Timeout while downloading '{url}': '{e}'")
1393- raise DebugGetterTimeout()
1394-
1395- self._logger.debug(f"Saved '{filepath}'")
1396-
1397- @property
1398- def subdir(self):
1399- return self._subdir
1400-
1401- @subdir.setter
1402- def subdir(self, subdir):
1403- self._subdir = subdir
1404diff --git a/poll_launchpad.py b/poll_launchpad.py
1405deleted file mode 100644
1406index d8645a8..0000000
1407--- a/poll_launchpad.py
1408+++ /dev/null
1409@@ -1,198 +0,0 @@
1410-#!/usr/bin/python3
1411-
1412-# Copyright (C) 2022 Canonical Ltd.
1413-
1414-# This program is free software: you can redistribute it and/or modify
1415-# it under the terms of the GNU General Public License as published by
1416-# the Free Software Foundation, either version 3 of the License, or
1417-# (at your option) any later version.
1418-
1419-# This program is distributed in the hope that it will be useful,
1420-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1421-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1422-# GNU General Public License for more details.
1423-
1424-# You should have received a copy of the GNU General Public License
1425-# along with this program. If not, see <https://www.gnu.org/licenses/>.
1426-
1427-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
1428-
1429-import os
1430-from ddebpoller import DdebPoller
1431-from ppapoller import PPAPoller, PrivatePPAPoller
1432-from debuginfod import (
1433- grab_ddebs,
1434- grab_ddebs_sources,
1435- grab_ppa_ddebs,
1436- grab_ppa_sources,
1437- grab_private_ppa_ddebs,
1438- grab_private_ppa_sources,
1439-)
1440-from utils import DebugDB
1441-
1442-import psycopg2
1443-
1444-
1445-def _ensure_task(msg, conn, cursor):
1446- """Return True if the task described by MSG should be processed.
1447-
1448- Also, if True, then insert the task information into the database.
1449-
1450- :param DebugDB conn: The database connection handler.
1451-
1452- :param psycopg2.cursor cursor: The cursor to the SQL database.
1453-
1454- :param dict msg: The message describing a task.
1455-
1456- :rtype: int
1457- :return: The taskid if the task should be processed (or None otherwise)."""
1458- if msg["architecture"] == "source":
1459- pkgname = msg["source_package"]
1460- else:
1461- pkgname = msg["binary_package"]
1462- pkgver = msg["version"]
1463- architecture = msg["architecture"]
1464-
1465- if "ppauser" in msg.keys():
1466- sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = TRUE AND ppauser = %s AND ppaname = %s"
1467- sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, ppauser, ppaname, created_date) VALUES (%s, %s, %s, TRUE, %s, %s, NOW()) RETURNING id"
1468- sqlparams = (pkgname, pkgver, architecture, msg["ppauser"], msg["ppaname"])
1469- else:
1470- sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = FALSE"
1471- sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, created_date) VALUES (%s, %s, %s, FALSE, NOW()) RETURNING id"
1472- sqlparams = (pkgname, pkgver, architecture)
1473-
1474- cursor.execute(sql_select, sqlparams)
1475-
1476- if cursor.rowcount > 0:
1477- return None
1478-
1479- cursor.execute(sql_insert, sqlparams)
1480-
1481- taskid = cursor.fetchone()[0]
1482- conn.commit()
1483- return taskid
1484-
1485-
1486-def _process_msg(msg, conn, cursor):
1487- """Add a task to the database to download the source or binary ddebs,
1488- if it isn't there already.
1489-
1490- :param str msg: The message to be processed.
1491-
1492- :param DebugDB conn: The database connection handler.
1493-
1494- :param psycopg2.cursor cursor: The cursor to the database.
1495-
1496- """
1497- taskid = _ensure_task(msg, conn, cursor)
1498- if taskid is not None:
1499- if msg.get("ppaname") is None:
1500- ddeb_fn = grab_ddebs
1501- src_fn = grab_ddebs_sources
1502- elif msg["isprivateppa"] == "no":
1503- ddeb_fn = grab_ppa_ddebs
1504- src_fn = grab_ppa_sources
1505- else:
1506- ddeb_fn = grab_private_ppa_ddebs
1507- src_fn = grab_private_ppa_sources
1508-
1509- # We append the ID to the message so that we can efficiently
1510- # remove the task once it's been processed.
1511- msg["taskid"] = str(taskid)
1512- if msg["architecture"] == "source":
1513- # Source packages have "source" as their architecture.
1514- src_fn.delay(msg)
1515- else:
1516- ddeb_fn.delay(msg)
1517-
1518-
1519-def _poll_launchpad_generic(poller, conn, cursor):
1520- """Poll Launchpad and process the list of ddebs and source packages.
1521-
1522- :param DebugDB conn: The SQL connection handler.
1523-
1524- :param psycopg2.cursor cursor: The cursor to the SQL database."""
1525-
1526- # Process ddebs.
1527- messages, new_timestamp_ddebs = poller.get_ddebs()
1528- for msg in messages:
1529- _process_msg(msg, conn, cursor)
1530-
1531- # Process source packages.
1532- messages, new_timestamp_src = poller.get_sources()
1533- for msg in messages:
1534- _process_msg(msg, conn, cursor)
1535-
1536- # Choose the earliest timestamp.
1537- new_timestamp = min(new_timestamp_ddebs, new_timestamp_src)
1538-
1539- poller.record_timestamp(new_timestamp)
1540-
1541-
1542-def _should_poll_archive(archive):
1543- """Whether we should poll a certain archive.
1544-
1545- This function relies on the value of environment variables named
1546- "POLL_{archive}". See the /etc/default/ubuntu-debuginfod-poll-lp
1547- file for more details.
1548-
1549- :param archive string: The archive name, in uppercase.
1550-
1551- :rtype boolean: True if we should poll the archive, False
1552- otherwise.
1553- """
1554- return os.environ.get(f"POLL_{archive.upper()}") == "yes"
1555-
1556-
1557-def poll_launchpad(conn, cursor):
1558- """Poll Launchpad and process the list of ddebs and source packages
1559- from the main archive.
1560-
1561- :param DebugDB conn: The SQL connection handler.
1562-
1563- :param psycopg2.cursor cursor: The cursor to the SQL database.
1564-
1565- """
1566- if _should_poll_archive("MAIN_ARCHIVE"):
1567- _poll_launchpad_generic(DdebPoller(), conn, cursor)
1568-
1569-
1570-def poll_launchpad_ppas(conn, cursor):
1571- """Poll Launchpad and process the list of ddebs and source packages
1572- from PPAs.
1573-
1574- :param DebugDB conn: The SQL connection handler.
1575-
1576- :param psycopg2.cursor cursor: The cursor to the SQL database.
1577-
1578- """
1579- if _should_poll_archive("PPAS"):
1580- _poll_launchpad_generic(PPAPoller(), conn, cursor)
1581-
1582-
1583-def poll_launchpad_private_ppas(conn, cursor):
1584- """Poll Launchpad and process the list of ddebs and source packages
1585- from private PPAs.
1586-
1587- :param DebugDB conn: The SQL connection handler.
1588-
1589- :param psycopg2.cursor cursor: The cursor to the SQL database.
1590-
1591- """
1592- if _should_poll_archive("PRIVATE_PPAS"):
1593- _poll_launchpad_generic(PrivatePPAPoller(), conn, cursor)
1594-
1595-
1596-def main():
1597- # Connect to the database.
1598- with DebugDB() as db:
1599- db.create_tables()
1600- with db.cursor() as cursor:
1601- poll_launchpad(db, cursor)
1602- poll_launchpad_ppas(db, cursor)
1603- poll_launchpad_private_ppas(db, cursor)
1604-
1605-
1606-if __name__ == "__main__":
1607- main()
1608diff --git a/ppagetter.py b/ppagetter.py
1609deleted file mode 100644
1610index 0f0f323..0000000
1611--- a/ppagetter.py
1612+++ /dev/null
1613@@ -1,227 +0,0 @@
1614-#!/usr/bin/python3
1615-
1616-# Copyright (C) 2023 Canonical Ltd.
1617-
1618-# This program is free software: you can redistribute it and/or modify
1619-# it under the terms of the GNU General Public License as published by
1620-# the Free Software Foundation, either version 3 of the License, or
1621-# (at your option) any later version.
1622-
1623-# This program is distributed in the hope that it will be useful,
1624-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1625-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1626-# GNU General Public License for more details.
1627-
1628-# You should have received a copy of the GNU General Public License
1629-# along with this program. If not, see <https://www.gnu.org/licenses/>.
1630-
1631-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
1632-
1633-import os
1634-import configparser
1635-from ddebgetter import (
1636- DdebGetter,
1637- DdebSourceCodeGetter,
1638- validate_ddeb_request,
1639- validate_source_code_request,
1640-)
1641-from debuggetter import DebugGetterRetry, DEFAULT_MIRROR_DIR
1642-
1643-
1644-import requests_oauthlib
1645-from launchpadlib.launchpad import Credentials
1646-
1647-
1648-def _validate_generic_ppa_request(request):
1649- """Validate a request to fetch an artifact from a PPA (private or
1650- not).
1651-
1652- :param dict[str, str] request: The dictionary containing the
1653- request to be validated."""
1654- if request.get("ppauser") is None:
1655- raise TypeError("No 'ppauser' in request")
1656- if request.get("ppaname") is None:
1657- raise TypeError("No 'ppaname' in request")
1658- if request.get("isprivateppa") is None:
1659- raise TypeError("No 'isprivateppa' in request")
1660-
1661-
1662-def validate_ppa_ddeb_request(request):
1663- """Validate a request to fetch a ddeb from a PPA.
1664-
1665- :param dict[str, str] request: The dictionary containing the
1666- request to be validated."""
1667-
1668- validate_ddeb_request(request)
1669- _validate_generic_ppa_request(request)
1670- if request["isprivateppa"] == "yes":
1671- raise ValueError("Wrong request: from private PPA")
1672-
1673-
1674-def validate_ppa_source_code_request(request):
1675- """Validate a request to fetch a source package from a PPA.
1676-
1677- :param dict[str, str] request: The dictionary containing the
1678- request to be validated."""
1679-
1680- validate_source_code_request(request)
1681- _validate_generic_ppa_request(request)
1682- if request["isprivateppa"] == "yes":
1683- raise ValueError("Wrong request: from private PPA")
1684-
1685-
1686-def validate_private_ppa_ddeb_request(request):
1687- """Validate a request to fetch a ddeb from a private PPA.
1688-
1689- :param dict[str, str] request: The dictionary containing the
1690- request to be validated."""
1691-
1692- validate_ddeb_request(request)
1693- _validate_generic_ppa_request(request)
1694- if request["isprivateppa"] == "no":
1695- raise ValueError("Wrong request: not from private PPA")
1696-
1697-
1698-def validate_private_ppa_source_code_request(request):
1699- """Validate a request to fetch a source package from a private
1700- PPA.
1701-
1702- :param dict[str, str] request: The dictionary containing the
1703- request to be validated."""
1704- validate_source_code_request(request)
1705- _validate_generic_ppa_request(request)
1706- if request["isprivateppa"] == "no":
1707- raise ValueError("Wrong request: not from private PPA")
1708-
1709-
1710-class PPADdebGetter(DdebGetter):
1711- """Get a ddeb from a Launchpad PPA."""
1712-
1713- def __init__(self, subdir="ppas", mirror_dir=DEFAULT_MIRROR_DIR):
1714- """Initialize the object.
1715-
1716- See DebugGetter's __init__ for an explanation of the arguments."""
1717- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
1718-
1719- def process_request(self, request):
1720- """Process a request, usually coming from Celery.
1721-
1722- :param request: The dictionary containing the information
1723- necessary to fetch this ddeb.
1724- :type request: dict(str : str)"""
1725- validate_ppa_ddeb_request(request)
1726-
1727- # Adjust the subdirectory to account for the PPA name.
1728- new_subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"])
1729- self.subdir = new_subdir
1730-
1731- self.do_process_request(request)
1732-
1733-
1734-class PPASourceCodeGetter(DdebSourceCodeGetter):
1735- """Get source code from a Launchpad PPA."""
1736-
1737- def __init__(self, subdir="ppas", mirror_dir=DEFAULT_MIRROR_DIR):
1738- """Initialize the object.
1739-
1740- See DebugGetter's __init__ for an explanation of the arguments."""
1741- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
1742-
1743- def process_request(self, request):
1744- """Process a request, usually coming from Celery.
1745-
1746- :param request: The dictionary containing the information
1747- necessary to fetch this source code.
1748- :type request: dict(str : str)"""
1749- validate_ppa_source_code_request(request)
1750-
1751- # Adjust the subdirectory to account for the PPA name.
1752- new_subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"])
1753- self.subdir = new_subdir
1754-
1755- self.do_process_request(request, fallback_to_git=False)
1756-
1757-
1758-def obtain_lp_oauth_credentials():
1759- """Obtain the necessary OAuth credentials to perform authenticated
1760- operations against Launchpad.
1761-
1762- :rtype requests_oauthlib.OAuth1: The OAuth1 object containing the
1763- Launchpad credentials."""
1764- lpcredfile = os.environ.get("LP_CREDENTIALS_FILE")
1765- if lpcredfile is None:
1766- raise ValueError(
1767- "Could not obtain Launchpad credentials: environment variable LP_CREDENTIALS_FILE must be set"
1768- )
1769- if not os.path.isfile(lpcredfile):
1770- raise FileNotFoundError(
1771- f"Could not obtain Launchpad credentials: file {lpcredfile} does not exist"
1772- )
1773-
1774- lpcred = Credentials()
1775- with open(lpcredfile, "r", encoding="UTF-8") as f:
1776- lpcred.load(f)
1777-
1778- return requests_oauthlib.OAuth1(
1779- lpcred.consumer.key,
1780- client_secret=lpcred.consumer.secret,
1781- resource_owner_key=lpcred.access_token.key,
1782- resource_owner_secret=lpcred.access_token.secret,
1783- signature_method="PLAINTEXT",
1784- )
1785-
1786-
1787-class PrivatePPADdebGetter(PPADdebGetter):
1788- """Get a ddeb from a private Launchpad PPA."""
1789-
1790- def __init__(self, subdir="private-ppas", mirror_dir=DEFAULT_MIRROR_DIR):
1791- """Initialize the object.
1792-
1793- See DebugGetter's __init__ for an explanation of the arguments."""
1794- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
1795-
1796- def process_request(self, request):
1797- """Process a request, usually coming from Celery.
1798-
1799- :param request: The dictionary containing the information
1800- necessary to fetch this ddeb.
1801- :type request: dict(str : str)"""
1802- validate_private_ppa_ddeb_request(request)
1803-
1804- ppauser = request["ppauser"]
1805- ppaname = request["ppaname"]
1806- creds = obtain_lp_oauth_credentials()
1807-
1808- # Adjust the subdirectory to account for the PPA name.
1809- new_subdir = os.path.join(self._subdir, ppauser, ppaname)
1810- self.subdir = new_subdir
1811-
1812- self.do_process_request(request, credentials=creds)
1813-
1814-
1815-class PrivatePPASourceCodeGetter(PPASourceCodeGetter):
1816- """Get source code from a private Launchpad PPA."""
1817-
1818- def __init__(self, subdir="private-ppas", mirror_dir=DEFAULT_MIRROR_DIR):
1819- """Initialize the object.
1820-
1821- See DebugGetter's __init__ for an explanation of the arguments."""
1822- super().__init__(subdir=subdir, mirror_dir=mirror_dir)
1823-
1824- def process_request(self, request):
1825- """Process a request, usually coming from Celery.
1826-
1827- :param request: The dictionary containing the information
1828- necessary to fetch this source code.
1829- :type request: dict(str : str)"""
1830- validate_private_ppa_source_code_request(request)
1831-
1832- ppauser = request["ppauser"]
1833- ppaname = request["ppaname"]
1834- creds = obtain_lp_oauth_credentials()
1835-
1836- # Adjust the subdirectory to account for the PPA name.
1837- new_subdir = os.path.join(self._subdir, ppauser, ppaname)
1838- self.subdir = new_subdir
1839-
1840- self.do_process_request(request, fallback_to_git=False, credentials=creds)
1841diff --git a/ppapoller.py b/ppapoller.py
1842deleted file mode 100644
1843index 27f844c..0000000
1844--- a/ppapoller.py
1845+++ /dev/null
1846@@ -1,191 +0,0 @@
1847-#!/usr/bin/python3
1848-
1849-# Copyright (C) 2023 Canonical Ltd.
1850-
1851-# This program is free software: you can redistribute it and/or modify
1852-# it under the terms of the GNU General Public License as published by
1853-# the Free Software Foundation, either version 3 of the License, or
1854-# (at your option) any later version.
1855-
1856-# This program is distributed in the hope that it will be useful,
1857-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1858-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1859-# GNU General Public License for more details.
1860-
1861-# You should have received a copy of the GNU General Public License
1862-# along with this program. If not, see <https://www.gnu.org/licenses/>.
1863-
1864-# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
1865-
1866-import os
1867-import lazr.restfulclient.errors
1868-from ddebpoller import DdebPoller
1869-
1870-
1871-class PPAPoller(DdebPoller):
1872- """Perform the Launchpad polling and obtain a list of ddebs and source
1873- packages from a PPA."""
1874-
1875- def __init__(
1876- self,
1877- initial_interval=1,
1878- force_initial_interval=False,
1879- fetch_all_on_first_run=True,
1880- dry_run=False,
1881- anonymous=True,
1882- ):
1883- """Initialize the object using 'ppa' as its name.
1884-
1885- Look at DebugPoller's docstring for an explanation about the arguments."""
1886- super().__init__(
1887- initial_interval=initial_interval,
1888- force_initial_interval=force_initial_interval,
1889- fetch_all_on_first_run=fetch_all_on_first_run,
1890- dry_run=dry_run,
1891- anonymous=anonymous,
1892- )
1893-
1894- def _get_list_of_ppas(self, private=False):
1895- """Get the list of PPAs to be polled.
1896-
1897- :param private boolean: Whether the list will contain private
1898- or non-private PPAs. Default is False.
1899-
1900- :rtype list: The list of PPAs. This list will contain one
1901- dictionary per PPA, with the following keys:
1902- { "ppauser" : string,
1903- "ppaname" : string,
1904- "isprivateppa" : string }
1905- """
1906- ppafile = os.path.expanduser("~/.config/ubuntu-debuginfod/ppalist")
1907- if private:
1908- ppafile += "-private"
1909-
1910- self._logger.debug(f"Processing '{ppafile}'")
1911-
1912- if not os.path.isfile(ppafile):
1913- self._logger.info(f"{ppafile} does not exist; doing nothing")
1914- return []
1915-
1916- ppas = []
1917- with open(ppafile, "r", encoding="UTF-8") as f:
1918- for ppa in f.readlines():
1919- ppa = ppa.rstrip()
1920- if ppa == "" or ppa.startswith("#"):
1921- continue
1922- ppa = ppa.split(":")[1].split("/")
1923- ppa_user = ppa[0]
1924- ppa_name = ppa[1]
1925- self._logger.info(f"Processing ppa:{ppa_user}/{ppa_name}")
1926- ppas.append(
1927- {
1928- "ppauser": ppa_user,
1929- "ppaname": ppa_name,
1930- "isprivateppa": "yes" if private else "no",
1931- }
1932- )
1933- return ppas
1934-
1935- def get_ddebs(self, private=False):
1936- """Get the list of ddebs from Launchpad PPAs.
1937-
1938- :param private boolean: Whether the PPAs are private or not.
1939- Default is False.
1940-
1941- :rtype dict, datetime.datetime:
1942- Return a dictionary containing all source packages found, and
1943- also the new timestamp that should then be recorded by calling
1944- record_timestamp."""
1945- messages = []
1946- new_timestamp = self._generate_timestamp()
1947- for ppa in self._get_list_of_ppas(private=private):
1948- ppauser = ppa["ppauser"]
1949- ppaname = ppa["ppaname"]
1950- try:
1951- lpuser = self._lp.people[ppauser]
1952- except KeyError:
1953- self._logger.error(f"Launchpad user {ppauser} does not exist")
1954- continue
1955-
1956- try:
1957- lpppa = lpuser.getPPAByName(name=ppaname)
1958- except lazr.restfulclient.errors.NotFound:
1959- self._logger.error(
1960- f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist"
1961- )
1962- continue
1963-
1964- self._logger.info(f"Getting list of ddebs from ppa:{ppauser}/{ppaname}")
1965- msg, ts = super().get_ddebs(archive=lpppa, ppainfo=ppa)
1966- messages += msg
1967-
1968- return messages, new_timestamp
1969-
1970- def get_sources(self, private=False):
1971- """Get the list of source packages from Launchpad PPAs.
1972-
1973- :param private boolean: Whether the PPAs are private or not.
1974- Default is False.
1975-
1976- :rtype dict, datetime.datetime:
1977- Return a dictionary containing all source packages found, and
1978- also the new timestamp that should then be recorded by calling
1979- record_timestamp.
1980-
1981- """
1982- messages = []
1983- new_timestamp = self._generate_timestamp()
1984- for ppa in self._get_list_of_ppas(private=private):
1985- ppauser = ppa["ppauser"]
1986- ppaname = ppa["ppaname"]
1987- try:
1988- lpuser = self._lp.people[ppauser]
1989- except KeyError:
1990- self._logger.error(f"Launchpad user {ppauser} does not exist")
1991- continue
1992-
1993- try:
1994- lpppa = lpuser.getPPAByName(name=ppaname)
1995- except lazr.restfulclient.errors.NotFound:
1996- self._logger.error(
1997- f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist"
1998- )
1999- continue
2000-
2001- self._logger.info(f"Getting list of source packages from ppa:{ppauser}/{ppaname}")
2002- msg, ts = super().get_sources(archive=lpppa, ppainfo=ppa)
2003- messages += msg
2004-
2005- return messages, new_timestamp
2006-
2007-
2008-class PrivatePPAPoller(PPAPoller):
2009- """Perform the Launchpad polling and obtain a list of ddebs and source
2010- packages from a private PPA."""
2011-
2012- def __init__(
2013- self,
2014- initial_interval=1,
2015- force_initial_interval=False,
2016- fetch_all_on_first_run=True,
2017- dry_run=False,
2018- anonymous=False,
2019- ):
2020- """Initialize the object using 'privateppa' as its name.
2021-
2022- Look at DebugPoller's docstring for an explanation about the arguments."""
2023- super().__init__(
2024- initial_interval=initial_interval,
2025- force_initial_interval=force_initial_interval,
2026- fetch_all_on_first_run=fetch_all_on_first_run,
2027- dry_run=dry_run,
2028- anonymous=anonymous,
2029- )
2030-
2031- def get_ddebs(self):
2032- """Get the list of ddebs from private Launchpad PPAs."""
2033- return super().get_ddebs(private=True)
2034-
2035- def get_sources(self):
2036- """Get the list of source packages from private Launchpad PPAs."""
2037- return super().get_sources(private=True)
2038diff --git a/pyproject.toml b/pyproject.toml
2039new file mode 100644
2040index 0000000..b16fcbd
2041--- /dev/null
2042+++ b/pyproject.toml
2043@@ -0,0 +1,40 @@
2044+[build-system]
2045+requires = [ "setuptools >= 61.0.0" ]
2046+build-backend = "setuptools.build_meta"
2047+
2048+[project]
2049+name = "ubuntu_debuginfod"
2050+version = "0.1.0"
2051+authors = [
2052+ { name="Sergio Durigan Junior", email="sergio.durigan@canonical.com" },
2053+]
2054+keywords = [ "ubuntu", "debuginfod" ]
2055+readme = "README.md"
2056+requires-python = ">= 3.7"
2057+classifiers = [
2058+ "Programming Language :: Python :: 3",
2059+ "Development Status :: 4 - Beta",
2060+ "Framework :: Celery",
2061+ "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
2062+ "Operating System :: POSIX :: Linux",
2063+ "Topic :: Software Development :: Debuggers",
2064+]
2065+dependencies = [
2066+ "GitPython",
2067+ "celery",
2068+ "launchpadlib",
2069+ "requests",
2070+ "requests-oauthlib",
2071+ "sdnotify",
2072+ "psycopg2",
2073+]
2074+
2075+[project.optional-dependencies]
2076+tests = [
2077+ "pytest",
2078+ "freezegun",
2079+]
2080+
2081+[project.urls]
2082+"Homepage" = "https://launchpad.net/ubuntu-debuginfod"
2083+"Bug Tracker" = "https://bugs.launchpad.net/ubuntu-debuginfod"
2084diff --git a/services/launchpad-dispatcher.py b/services/launchpad-dispatcher.py
2085new file mode 100755
2086index 0000000..81bb754
2087--- /dev/null
2088+++ b/services/launchpad-dispatcher.py
2089@@ -0,0 +1,451 @@
2090+#!/usr/bin/python3
2091+
2092+# Copyright (C) 2022 Canonical Ltd.
2093+
2094+# This program is free software: you can redistribute it and/or modify
2095+# it under the terms of the GNU General Public License as published by
2096+# the Free Software Foundation, either version 3 of the License, or
2097+# (at your option) any later version.
2098+
2099+# This program is distributed in the hope that it will be useful,
2100+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2101+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2102+# GNU General Public License for more details.
2103+
2104+# You should have received a copy of the GNU General Public License
2105+# along with this program. If not, see <https://www.gnu.org/licenses/>.
2106+
2107+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
2108+
2109+import os
2110+import time
2111+import logging
2112+from urllib import parse
2113+import lazr.restfulclient.errors
2114+import tempfile
2115+import gzip
2116+import re
2117+from ubuntu_debuginfod.utils.debugdb import DebugDB
2118+from ubuntu_debuginfod.utils.utils import download_from_lp, obtain_lp_oauth_credentials
2119+from ubuntu_debuginfod.utils.baselp import BaseLaunchpadClass
2120+from ubuntu_debuginfod.debuginfod import (
2121+ grab_ddebs,
2122+ grab_sources,
2123+ grab_ppa_ddebs,
2124+ grab_ppa_sources,
2125+ grab_private_ppa_ddebs,
2126+ grab_private_ppa_sources,
2127+)
2128+
2129+
2130+def ensure_task(msg, db, cursor):
2131+ """Return the task id if the task described by MSG should be
2132+ processed.
2133+
2134+ Also, if true, then insert the task information into the database.
2135+
2136+ :param DebugDB db: The database connection handler.
2137+
2138+ :param psycopg2.cursor cursor: The cursor to the SQL database.
2139+
2140+ :param dict[str -> str] msg: The message describing a task.
2141+
2142+ :rtype: int
2143+ :return: The taskid if the task should be processed (or None otherwise)."""
2144+ if msg["architecture"] == "source":
2145+ pkgname = msg["source_package"]
2146+ else:
2147+ pkgname = msg["binary_package"]
2148+ pkgver = msg["version"]
2149+ architecture = msg["architecture"]
2150+
2151+ if "ppauser" in msg.keys():
2152+ sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = TRUE AND ppauser = %s AND ppaname = %s"
2153+ sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, ppauser, ppaname, created_date) VALUES (%s, %s, %s, TRUE, %s, %s, NOW()) RETURNING id"
2154+ sqlparams = (pkgname, pkgver, architecture, msg["ppauser"], msg["ppaname"])
2155+ else:
2156+ sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = FALSE"
2157+ sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, created_date) VALUES (%s, %s, %s, FALSE, NOW()) RETURNING id"
2158+ sqlparams = (pkgname, pkgver, architecture)
2159+
2160+ cursor.execute(sql_select, sqlparams)
2161+
2162+ if cursor.rowcount > 0:
2163+ return None
2164+
2165+ cursor.execute(sql_insert, sqlparams)
2166+
2167+ taskid = cursor.fetchone()[0]
2168+ db.commit()
2169+ return taskid
2170+
2171+
2172+class LaunchpadDispatcher(BaseLaunchpadClass):
2173+ """Dispatch Launchpad jobs to be processed by the
2174+ ubuntu-debuginfod getters."""
2175+
2176+ SOURCE_PACKAGE_INDEXABLE_REGEX = re.compile(r"-fdebug-prefix-map=[^=]+=/usr/src/")
2177+
2178+ def __init__(self, anonymous):
2179+ """Initialize class.
2180+
2181+ :param bool anonymous: Whether we should log into Launchpad
2182+ anonymously or not."""
2183+ self._anonymous = anonymous
2184+ self._logger = logging.getLogger(__name__)
2185+ self.launchpad_login()
2186+
2187+ def _get_pkg(self, uri):
2188+ """Obtain the Launchpad entry associated with URI.
2189+
2190+ This function handles possible exceptions that might come from
2191+ Launchpad, and performs retries accordingly.
2192+
2193+ :param str uri: The URI associated with the package.
2194+
2195+ :rtype: lazr.restfulclient.resource.Entry
2196+ :return: The Launchpad object representing the package."""
2197+ retry = 0
2198+ while True:
2199+ try:
2200+ self._logger.debug(f"Obtaining package from URI '{uri}'")
2201+ pkg = self._lp.load(uri)
2202+ except (
2203+ lazr.restfulclient.errors.BadRequest,
2204+ lazr.restfulclient.errors.HTTPError,
2205+ lazr.restfulclient.errors.ServerError,
2206+ lazr.restfulclient.errors.UnexpectedResponseError,
2207+ lazr.restfulclient.errors.ResponseError,
2208+ IndexError,
2209+ ) as e:
2210+ self._logger.warning(
2211+ f"Exception when obtaining information from URI '{uri}': {e}"
2212+ )
2213+ if retry >= 4:
2214+ self._logger.warning(
2215+ f"Maximum number of retries exceeded. Raising {e}"
2216+ )
2217+ raise e
2218+ retry += 1
2219+ time.sleep(2)
2220+ continue
2221+ self._logger.debug(f"Obtained package information: '{pkg}'")
2222+ return pkg
2223+
2224+ def _remove_id_from_database(self, jobid, db, cursor):
2225+ """Helper function to remove a job ID from the database.
2226+
2227+ This function is invoked when the job is either invalid or
2228+ complete.
2229+
2230+ :param str jobid: The job ID.
2231+
2232+ :param DebugDB db: The database handler.
2233+
2234+ :param psycopg2.cursor cursor: The cursor to the database."""
2235+ self._logger.debug(f"Removing {jobid} from dispatcher table")
2236+ cursor.execute("DELETE FROM jobs2dispatch WHERE id = %s", (jobid,))
2237+ db.commit()
2238+
2239+ def _dispatch_binpkg_job(self, pkg, jobid, uri, ppainfo, db, cursor):
2240+ """Dispatch a job for a binary package package.
2241+
2242+ :param lazr.restfulclient.resource.Entry pkg: The binary
2243+ package object.
2244+
2245+ :param str jobid: The job ID.
2246+
2247+ :param str uri: The URI for this artifact.
2248+
2249+ :param dict[str -> str] ppainfo: The PPA information, if
2250+ applicable.
2251+
2252+ :param DebugDB db: The database handler.
2253+
2254+ :param psycopg2.cursor cursor: The cursor to the database.
2255+
2256+ :rtype bool:
2257+ :return: True if the package should be removed from the
2258+ database, False otherwise."""
2259+ self._logger.debug(
2260+ f"Processing binary package {pkg}, with ID {jobid}, ppainfo = {ppainfo}"
2261+ )
2262+ if not pkg.is_debug:
2263+ self._logger.debug(
2264+ f"Package {pkg} is not a debug package; nothing to be done"
2265+ )
2266+ return True
2267+
2268+ ddeb_urls = pkg.binaryFileUrls()
2269+ if len(ddeb_urls) == 0:
2270+ self._logger.debug(
2271+ f"Package {pkg} does not have binaryFileUrls associated with it"
2272+ )
2273+ return True
2274+
2275+ msg = {
2276+ "uri": uri,
2277+ "source_package": pkg.source_package_name,
2278+ "binary_package": pkg.binary_package_name,
2279+ "version": pkg.binary_package_version,
2280+ "component": pkg.component_name,
2281+ "ddeb_url": "",
2282+ "ddeb_filename": "",
2283+ "architecture": "",
2284+ }
2285+ if ppainfo is not None:
2286+ msg.update(ppainfo)
2287+
2288+ for url in ddeb_urls:
2289+ # Obtain the ddeb filename and its architecture.
2290+ ddeb = os.path.basename(parse.urlparse(url).path)
2291+ _, _, arch_and_extension = ddeb.split("_")
2292+ arch, _ = arch_and_extension.split(".")
2293+
2294+ msg["ddeb_url"] = url
2295+ msg["ddeb_filename"] = ddeb
2296+ msg["architecture"] = arch
2297+
2298+ taskid = ensure_task(msg, db, cursor)
2299+ if taskid is not None:
2300+ msg["taskid"] = str(taskid)
2301+ self._logger.debug(f"Dispatching the following message: {msg}")
2302+ # Do the actual dispatching here.
2303+ if ppainfo is None:
2304+ grab_ddebs.delay(msg)
2305+ elif ppainfo["isprivateppa"] == "no":
2306+ grab_ppa_ddebs.delay(msg)
2307+ else:
2308+ grab_private_ppa_ddebs.delay(msg)
2309+
2310+ return True
2311+
2312+ def _srcpkg_is_indexable(self, binpkg):
2313+ """Check if the source package associated with the binary
2314+ package PKG is indexable, and mark it as such in the database.
2315+
2316+ This approach has a few downsides. Namely, it relies on the
2317+ verboseness of build log files. If the source package
2318+ "prettifies" the output of "make", we're doomed. However,
2319+ this will have to do for now until we come up with a better
2320+ way to examine the ddebs.
2321+
2322+ :param lazr.restfulclient.resource.Entry binpkg: A Launchpad
2323+ object to a binary package associated with the source
2324+ package to be examined.
2325+
2326+ :rtype: bool
2327+ :return: True is the source package is indexable by
2328+ debuginfod, False otherwise."""
2329+ if not self._anonymous:
2330+ # It's currently impossible for us to check build logs of
2331+ # anonymous PPAs, so we just assume the source is
2332+ # indexable. This should not be a big problem.
2333+ return True
2334+
2335+ credentials = None if self._anonymous else obtain_lp_oauth_credentials()
2336+ log_url = self._lp.load(binpkg.build_link).build_log_url
2337+ if not log_url:
2338+ return False
2339+ with tempfile.TemporaryDirectory() as tempdir:
2340+ logfile = download_from_lp(log_url, tempdir, credentials=credentials)
2341+ if not logfile.endswith(".txt.gz"):
2342+ return False
2343+ try:
2344+ with gzip.open(logfile, "rt") as f:
2345+ for line in f:
2346+ if re.search(self.SOURCE_PACKAGE_INDEXABLE_REGEX, line):
2347+ return True
2348+ except gzip.BadGzipFile:
2349+ self._logger.warning(f"Could not un-gzip {logfile} (URL: {log_url})")
2350+ return False
2351+
2352+ def _dispatch_srcpkg_job(self, pkg, jobid, uri, ppainfo, db, cursor):
2353+ """Process a job for a source package.
2354+
2355+ :param lazr.restfulclient.resource.Entry pkg: The source
2356+ package object.
2357+
2358+ :param str jobid: The job ID.
2359+
2360+ :param str uri: The URI for this artifact.
2361+
2362+ :param dict[str -> str] ppainfo: The PPA information, if
2363+ applicable.
2364+
2365+ :param DebugDB db: The database handler.
2366+
2367+ :param psycopg2.cursor cursor: The cursor to the database.
2368+
2369+ :rtype bool:
2370+ :return: True if the package should be removed from the
2371+ database, False otherwise."""
2372+ self._logger.debug(
2373+ f"Processing source package {pkg}, with ID {jobid}, ppainfo = {ppainfo}"
2374+ )
2375+
2376+ srcpkgname = pkg.source_package_name
2377+ if srcpkgname.startswith("linux"):
2378+ # We know Linux tarballs are not indexable, so let's save
2379+ # time.
2380+ return True
2381+
2382+ src_urls = pkg.sourceFileUrls()
2383+ if len(src_urls) == 0:
2384+ self._logger.debug(
2385+ f"Package {pkg} does not have sourceFileUrls associated with it"
2386+ )
2387+ return True
2388+
2389+ # Check if the source package is actually indexable by
2390+ # debuginfod.
2391+ binpkgs = pkg.getPublishedBinaries()
2392+ if len(binpkgs) == 0:
2393+ # This means that no binary has been published yet. We
2394+ # return without removing the ID from the database in
2395+ # order to make sure this source package will be
2396+ # re-evaluated later.
2397+ return False
2398+ if not self._srcpkg_is_indexable(binpkgs[0]):
2399+ # The package is not indexable. Just remove it from the
2400+ # DB.
2401+ return True
2402+
2403+ msg = {
2404+ "uri": uri,
2405+ "source_package": srcpkgname,
2406+ "version": pkg.source_package_version,
2407+ "component": pkg.component_name,
2408+ "source_urls": src_urls,
2409+ "architecture": "source",
2410+ }
2411+
2412+ taskid = ensure_task(msg, db, cursor)
2413+ if taskid is not None:
2414+ msg["taskid"] = str(taskid)
2415+ self._logger.debug(f"Dispatching the following message: {msg}")
2416+ if ppainfo is None:
2417+ grab_sources.delay(msg)
2418+ else:
2419+ msg.update(ppainfo)
2420+ if ppainfo["isprivateppa"] == "no":
2421+ grab_ppa_sources.delay(msg)
2422+ else:
2423+ grab_private_ppa_sources.delay(msg)
2424+
2425+ return True
2426+
2427+ def _dispatch_pkg_job(self, pkg, jobid, uri, ppainfo, db, cursor):
2428+ """Process a job for a package.
2429+
2430+ This is a generic function that will call the respective
2431+ _dispatch_{srcpkg,binpkg}_job methods when applicable.
2432+
2433+ :param lazr.restfulclient.resource.Entry pkg: The package
2434+ object.
2435+
2436+ :param str jobid: The job ID.
2437+
2438+ :param str uri: The URI for this job.
2439+
2440+ :param dict[str -> str] ppainfo: The PPA information, if
2441+ applicable.
2442+
2443+ :param DebugDB db: The database handler.
2444+
2445+ :param psycopg2.cursor cursor: The cursor to the database."""
2446+ # We're only interested in packages that have been published
2447+ # or are pending publication.
2448+ if pkg.status in (
2449+ "Published",
2450+ "Pending",
2451+ ):
2452+ if "/+binarypub/" in uri:
2453+ self._logger.debug(f"Binary package {pkg} (ID {jobid}) detected")
2454+ if not self._dispatch_binpkg_job(pkg, jobid, uri, ppainfo, db, cursor):
2455+ return
2456+ elif "/+sourcepub/" in uri:
2457+ self._logger.debug(f"Source package {pkg} (ID {jobid}) detected")
2458+ if not self._dispatch_srcpkg_job(pkg, jobid, uri, ppainfo, db, cursor):
2459+ return
2460+ else:
2461+ raise ValueError(f"Invalid URI when processing package '{pkg}': {uri}")
2462+
2463+ self._remove_id_from_database(jobid, db, cursor)
2464+
2465+ def dispatch_jobs(self, db, cursor):
2466+ """Dispatch jobs.
2467+
2468+ This function goes over the existing jobs in the
2469+ "jobs2dispatch" table and filters out those jobs that are
2470+ interesting to debuginfod. This means that the job must:
2471+
2472+ * Be related to debuginfo (ddebs) or source packages.
2473+
2474+ * Be public or private, depending on which dispatcher called
2475+ us.
2476+
2477+ * Be in a valid publication state (published or pending).
2478+
2479+ :param DebugDB db: The database handler.
2480+
2481+ :param psycopg2.cursor cursor: The cursor to the database."""
2482+ cursor.execute(
2483+ "SELECT id, uri, isfromppa, ppauser, ppaname FROM jobs2dispatch WHERE private = %s",
2484+ (not self._anonymous,),
2485+ )
2486+ results = cursor.fetchall()
2487+ self._logger.info(f"Starting to dispatch {len(results)} entries")
2488+ for (jobid, uri, isfromppa, ppauser, ppaname) in results:
2489+ try:
2490+ pkg = self._get_pkg(uri)
2491+ except Exception as e:
2492+ # Record the error and continue processing. We don't
2493+ # delete the entry from the table in case we decide to
2494+ # retry it later.
2495+ cursor.execute(
2496+ "INSERT INTO errors VALUES (%s, %s, NOW())", (uri, str(e))
2497+ )
2498+ continue
2499+
2500+ ppainfo = None
2501+ if isfromppa:
2502+ ppainfo = {
2503+ "ppauser": ppauser,
2504+ "ppaname": ppaname,
2505+ "isprivateppa": "no" if self._anonymous else "yes",
2506+ }
2507+ self._dispatch_pkg_job(pkg, jobid, uri, ppainfo, db, cursor)
2508+
2509+
2510+class PublicLaunchpadDispatcher(LaunchpadDispatcher):
2511+ """Dispatch Launchpad public jobs.
2512+
2513+ Public here means that we don't need to log into Launchpad using
2514+ credentials."""
2515+
2516+ def __init__(self):
2517+ super().__init__(anonymous=True)
2518+
2519+
2520+class PrivateLaunchpadDispatcher(LaunchpadDispatcher):
2521+ """Dispatch Launchpad private jobs.
2522+
2523+ Private here means that we need to log into Launchpad using
2524+ credentials."""
2525+
2526+ def __init__(self):
2527+ super().__init__(anonymous=False)
2528+
2529+
2530+def main():
2531+ # Connect to the database.
2532+ with DebugDB() as db:
2533+ db.create_tables()
2534+ with db.cursor() as cursor:
2535+ PublicLaunchpadDispatcher().dispatch_jobs(db, cursor)
2536+ PrivateLaunchpadDispatcher().dispatch_jobs(db, cursor)
2537+
2538+
2539+if __name__ == "__main__":
2540+ main()
2541diff --git a/services/launchpad-poller.py b/services/launchpad-poller.py
2542new file mode 100755
2543index 0000000..e38f549
2544--- /dev/null
2545+++ b/services/launchpad-poller.py
2546@@ -0,0 +1,156 @@
2547+#!/usr/bin/python3
2548+
2549+# Copyright (C) 2022 Canonical Ltd.
2550+
2551+# This program is free software: you can redistribute it and/or modify
2552+# it under the terms of the GNU General Public License as published by
2553+# the Free Software Foundation, either version 3 of the License, or
2554+# (at your option) any later version.
2555+
2556+# This program is distributed in the hope that it will be useful,
2557+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2558+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2559+# GNU General Public License for more details.
2560+
2561+# You should have received a copy of the GNU General Public License
2562+# along with this program. If not, see <https://www.gnu.org/licenses/>.
2563+
2564+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
2565+
2566+import os
2567+from ubuntu_debuginfod.pollers.ddebpoller import DdebPoller
2568+from ubuntu_debuginfod.pollers.sourcepackagepoller import SourcePackagePoller
2569+from ubuntu_debuginfod.pollers.ppaddebpoller import PPADdebPoller, PrivatePPADdebPoller
2570+from ubuntu_debuginfod.pollers.ppasourcepackagepoller import (
2571+ PPASourcePackagePoller,
2572+ PrivatePPASourcePackagePoller,
2573+)
2574+from ubuntu_debuginfod.utils.debugdb import DebugDB
2575+
2576+
2577+def _process_msg(msg, db, cursor):
2578+ """Add a task to the database to download the source or binary ddebs,
2579+ if it isn't there already.
2580+
2581+ :param str msg: The message to be processed.
2582+
2583+ :param DebugDB db: The database connection handler.
2584+
2585+ :param psycopg2.cursor cursor: The cursor to the database.
2586+
2587+ """
2588+ if "ppauser" in msg.keys():
2589+ sqlinsert = "INSERT INTO jobs2dispatch (uri, private, isfromppa, ppauser, ppaname, date) VALUES (%s, %s, %s, %s, %s, NOW())"
2590+ sqlvalues = (
2591+ msg["uri"],
2592+ msg["isprivateppa"] == "yes",
2593+ True,
2594+ msg["ppauser"],
2595+ msg["ppaname"],
2596+ )
2597+ else:
2598+ sqlinsert = "INSERT INTO jobs2dispatch (uri, private, isfromppa, date) VALUES (%s, %s, %s, NOW())"
2599+ sqlvalues = (msg["uri"], False, False)
2600+
2601+ cursor.execute(sqlinsert, sqlvalues)
2602+ db.commit()
2603+
2604+
2605+def _poll_launchpad_generic(poller, db, cursor):
2606+ """Poll Launchpad and process the list of ddebs and source packages.
2607+
2608+ :param DebugDB db: The SQL connection handler.
2609+
2610+ :param psycopg2.cursor cursor: The cursor to the SQL database."""
2611+
2612+ messages, timestamp = poller.poll_artifacts()
2613+ for msg in messages:
2614+ _process_msg(msg, db, cursor)
2615+
2616+ if timestamp is not None:
2617+ poller.record_timestamp(timestamp)
2618+ # # Process ddebs.
2619+ # messages, new_timestamp_ddebs = poller.get_ddebs()
2620+ # for msg in messages:
2621+ # _process_msg(msg, db, cursor)
2622+
2623+ # # Process source packages.
2624+ # messages, new_timestamp_src = poller.get_sources()
2625+ # for msg in messages:
2626+ # _process_msg(msg, db, cursor)
2627+
2628+ # # Choose the earliest timestamp.
2629+ # new_timestamp = min(new_timestamp_ddebs, new_timestamp_src)
2630+
2631+ # poller.record_timestamp(new_timestamp)
2632+
2633+
2634+def _should_poll_archive(archive):
2635+ """Whether we should poll a certain archive.
2636+
2637+ This function relies on the value of environment variables named
2638+ "POLL_{archive}". See the /etc/default/ubuntu-debuginfod-poll-lp
2639+ file for more details.
2640+
2641+ :param archive string: The archive name, in uppercase.
2642+
2643+ :rtype: boolean
2644+
2645+ :return: True if we should poll the archive, False otherwise."""
2646+ return os.environ.get(f"POLL_{archive.upper()}") == "yes"
2647+
2648+
2649+def poll_launchpad(db, cursor):
2650+ """Poll Launchpad and process the list of ddebs and source packages
2651+ from the main archive.
2652+
2653+ :param DebugDB db: The SQL connection handler.
2654+
2655+ :param psycopg2.cursor cursor: The cursor to the SQL database.
2656+
2657+ """
2658+ if _should_poll_archive("MAIN_ARCHIVE"):
2659+ _poll_launchpad_generic(DdebPoller(), db, cursor)
2660+ _poll_launchpad_generic(SourcePackagePoller(), db, cursor)
2661+
2662+
2663+def poll_launchpad_ppas(db, cursor):
2664+ """Poll Launchpad and process the list of ddebs and source packages
2665+ from PPAs.
2666+
2667+ :param DebugDB db: The SQL connection handler.
2668+
2669+ :param psycopg2.cursor cursor: The cursor to the SQL database.
2670+
2671+ """
2672+ if _should_poll_archive("PPAS"):
2673+ _poll_launchpad_generic(PPADdebPoller(), db, cursor)
2674+ _poll_launchpad_generic(PPASourcePackagePoller(), db, cursor)
2675+
2676+
2677+def poll_launchpad_private_ppas(db, cursor):
2678+ """Poll Launchpad and process the list of ddebs and source packages
2679+ from private PPAs.
2680+
2681+ :param DebugDB db: The SQL connection handler.
2682+
2683+ :param psycopg2.cursor cursor: The cursor to the SQL database.
2684+
2685+ """
2686+ if _should_poll_archive("PRIVATE_PPAS"):
2687+ _poll_launchpad_generic(PrivatePPADdebPoller(), db, cursor)
2688+ _poll_launchpad_generic(PrivatePPASourcePackagePoller(), db, cursor)
2689+
2690+
2691+def main():
2692+ # Connect to the database.
2693+ with DebugDB() as db:
2694+ db.create_tables()
2695+ with db.cursor() as cursor:
2696+ poll_launchpad(db, cursor)
2697+ poll_launchpad_ppas(db, cursor)
2698+ poll_launchpad_private_ppas(db, cursor)
2699+
2700+
2701+if __name__ == "__main__":
2702+ main()
2703diff --git a/setup.cfg b/setup.cfg
2704new file mode 100644
2705index 0000000..2f0184a
2706--- /dev/null
2707+++ b/setup.cfg
2708@@ -0,0 +1,14 @@
2709+[options]
2710+packages = find:
2711+include_package_data = True
2712+
2713+[flake8]
2714+# Ignore line-too-long
2715+extend-ignore = E501
2716+
2717+[mypy]
2718+# Ignore warning about missing packages when assessing imports.
2719+ignore_missing_imports = True
2720+
2721+[pylint.'MESSAGES CONTROL']
2722+disable = logging-fstring-interpolation,line-too-long,missing-module-docstring,too-many-arguments,import-error,invalid-name,too-few-public-methods,duplicate-code
2723diff --git a/tests/test_baselp.py b/tests/test_baselp.py
2724new file mode 100644
2725index 0000000..ba27670
2726--- /dev/null
2727+++ b/tests/test_baselp.py
2728@@ -0,0 +1,58 @@
2729+#!/usr/bin/python3
2730+
2731+# Copyright (C) 2023 Canonical Ltd.
2732+
2733+# This program is free software: you can redistribute it and/or modify
2734+# it under the terms of the GNU General Public License as published by
2735+# the Free Software Foundation, either version 3 of the License, or
2736+# (at your option) any later version.
2737+
2738+# This program is distributed in the hope that it will be useful,
2739+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2740+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2741+# GNU General Public License for more details.
2742+
2743+# You should have received a copy of the GNU General Public License
2744+# along with this program. If not, see <https://www.gnu.org/licenses/>.
2745+
2746+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
2747+
2748+from unittest.mock import patch
2749+import unittest
2750+from ubuntu_debuginfod.utils import baselp
2751+
2752+
2753+class FakeArchive:
2754+ """A fake Launchpad archive object."""
2755+
2756+ def __init__(self):
2757+ self.main_archive = []
2758+
2759+
2760+class FakeLaunchpad:
2761+ """A fake Launchpad object."""
2762+
2763+ def __init__(self):
2764+ self.distributions = {"ubuntu": FakeArchive()}
2765+
2766+
2767+@patch(
2768+ "launchpadlib.launchpad.Launchpad.login_anonymously", return_value=FakeLaunchpad()
2769+)
2770+@patch("launchpadlib.launchpad.Launchpad.login_with", return_value=FakeLaunchpad())
2771+class TestBaseLP(baselp.BaseLaunchpadClass):
2772+ def test_anonymous_login(self, mock_login_with, mock_login_anon):
2773+ self._anonymous = True
2774+ self.launchpad_login()
2775+ assert mock_login_anon.called is True
2776+ assert mock_login_with.called is False
2777+
2778+ def test_non_anonymous_login(self, mock_login_with, mock_login_anon):
2779+ self._anonymous = False
2780+ self.launchpad_login()
2781+ assert mock_login_anon.called is False
2782+ assert mock_login_with.called is True
2783+
2784+
2785+if __name__ == "__main__":
2786+ unittest.main()
2787diff --git a/tests/test_common_lp_getter.py b/tests/test_common_lp_getter.py
2788new file mode 100644
2789index 0000000..5663dc0
2790--- /dev/null
2791+++ b/tests/test_common_lp_getter.py
2792@@ -0,0 +1,91 @@
2793+#!/usr/bin/python3
2794+
2795+# Copyright (C) 2023 Canonical Ltd.
2796+
2797+# This program is free software: you can redistribute it and/or modify
2798+# it under the terms of the GNU General Public License as published by
2799+# the Free Software Foundation, either version 3 of the License, or
2800+# (at your option) any later version.
2801+
2802+# This program is distributed in the hope that it will be useful,
2803+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2804+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2805+# GNU General Public License for more details.
2806+
2807+# You should have received a copy of the GNU General Public License
2808+# along with this program. If not, see <https://www.gnu.org/licenses/>.
2809+
2810+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
2811+
2812+import os
2813+import re
2814+import unittest
2815+import pytest
2816+from ubuntu_debuginfod.getters import common_lp_getter
2817+
2818+
2819+class TestCommonLaunchpadGetter:
2820+
2821+ VALID_GENERIC_REQ = {
2822+ "architecture": "amd64",
2823+ "uri": "URL",
2824+ }
2825+
2826+ def test_make_savepath_no_lib(self):
2827+ subdir = "test"
2828+ d = common_lp_getter.CommonLaunchpadGetter(subdir=subdir)
2829+ source_package = "foo"
2830+ component = "bar"
2831+ expected_path = os.path.join(
2832+ d.DEFAULT_MIRROR_DIR, subdir, component, source_package[0], source_package
2833+ )
2834+ assert d.make_savepath(source_package, component) == expected_path
2835+
2836+ def test_make_savepath_lib(self):
2837+ subdir = "test"
2838+ d = common_lp_getter.CommonLaunchpadGetter(subdir=subdir)
2839+ source_package = "libfoo"
2840+ component = "bar"
2841+ expected_path = os.path.join(
2842+ d.DEFAULT_MIRROR_DIR, subdir, component, source_package[:4], source_package
2843+ )
2844+ assert d.make_savepath(source_package, component) == expected_path
2845+
2846+ @pytest.mark.parametrize(
2847+ "req, exception, ex_message",
2848+ [
2849+ (None, TypeError, "Invalid request"),
2850+ ({"foo": "bar"}, ValueError, "No 'architecture' in request"),
2851+ (
2852+ {"architecture": None},
2853+ ValueError,
2854+ "Invalid 'architecture' (None) in request",
2855+ ),
2856+ ({"architecture": "amd64"}, ValueError, "No 'uri' in request"),
2857+ (
2858+ {"architecture": "amd64", "uri": None},
2859+ ValueError,
2860+ "Invalid 'uri' (None) in request",
2861+ ),
2862+ ],
2863+ )
2864+ def test_validate_request_invalid_request(self, req, exception, ex_message):
2865+ d = common_lp_getter.CommonLaunchpadGetter(subdir="test")
2866+ with pytest.raises(exception, match=re.escape(ex_message)):
2867+ d.validate_request(req)
2868+
2869+ def test_validate_request_valid_request(self):
2870+ d = common_lp_getter.CommonLaunchpadGetter(subdir="test")
2871+ assert d.validate_request(self.VALID_GENERIC_REQ) is None
2872+
2873+ def test_download_artifact_raises_notimplemented(self):
2874+ d = common_lp_getter.CommonLaunchpadGetter(subdir="test")
2875+ with pytest.raises(
2876+ NotImplementedError,
2877+ match=re.escape("Wrong call to 'download_artifact' from base class"),
2878+ ):
2879+ d.download_artifact()
2880+
2881+
2882+if __name__ == "__main__":
2883+ unittest.main()
2884diff --git a/tests/test_common_ppa_getter.py b/tests/test_common_ppa_getter.py
2885new file mode 100644
2886index 0000000..8eb736a
2887--- /dev/null
2888+++ b/tests/test_common_ppa_getter.py
2889@@ -0,0 +1,112 @@
2890+#!/usr/bin/python3
2891+
2892+# Copyright (C) 2023 Canonical Ltd.
2893+
2894+# This program is free software: you can redistribute it and/or modify
2895+# it under the terms of the GNU General Public License as published by
2896+# the Free Software Foundation, either version 3 of the License, or
2897+# (at your option) any later version.
2898+
2899+# This program is distributed in the hope that it will be useful,
2900+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2901+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2902+# GNU General Public License for more details.
2903+
2904+# You should have received a copy of the GNU General Public License
2905+# along with this program. If not, see <https://www.gnu.org/licenses/>.
2906+
2907+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
2908+
2909+import re
2910+import pytest
2911+import unittest
2912+from ubuntu_debuginfod.getters import common_ppa_getter
2913+
2914+
2915+class MockValidateRequest:
2916+ def validate_request(self, request):
2917+ pass
2918+
2919+
2920+class MockCommonPPAGetter(common_ppa_getter.CommonPPAGetter, MockValidateRequest):
2921+ def __init__(self):
2922+ super().__init__()
2923+
2924+
2925+class TestCommonPPAGetter:
2926+ @pytest.mark.parametrize(
2927+ "req, exception, ex_message",
2928+ [
2929+ (
2930+ {"architecture": "amd64", "uri": "URL", "ddeb_url": "URL"},
2931+ TypeError,
2932+ "No 'ppauser' in request",
2933+ ),
2934+ (
2935+ {
2936+ "architecture": "amd64",
2937+ "ddeb_url": "URL",
2938+ "uri": "URL",
2939+ "ppauser": "foo",
2940+ },
2941+ TypeError,
2942+ "No 'ppaname' in request",
2943+ ),
2944+ (
2945+ {
2946+ "architecture": "amd64",
2947+ "uri": "URL",
2948+ "ddeb_url": "URL",
2949+ "ppauser": "foo",
2950+ "ppaname": "bar",
2951+ },
2952+ TypeError,
2953+ "No 'isprivateppa' in request",
2954+ ),
2955+ (
2956+ {
2957+ "architecture": "amd64",
2958+ "uri": "URL",
2959+ "ddeb_url": "URL",
2960+ "ppauser": None,
2961+ "ppaname": "bar",
2962+ "isprivateppa": "no",
2963+ },
2964+ ValueError,
2965+ "Invalid 'ppauser' (None) in request",
2966+ ),
2967+ (
2968+ {
2969+ "architecture": "amd64",
2970+ "uri": "URL",
2971+ "ddeb_url": "URL",
2972+ "ppauser": "foo",
2973+ "ppaname": None,
2974+ "isprivateppa": "no",
2975+ },
2976+ ValueError,
2977+ "Invalid 'ppaname' (None) in request",
2978+ ),
2979+ (
2980+ {
2981+ "architecture": "amd64",
2982+ "uri": "URL",
2983+ "ddeb_url": "URL",
2984+ "ppauser": "foo",
2985+ "ppaname": "bar",
2986+ "isprivateppa": None,
2987+ },
2988+ ValueError,
2989+ "Invalid 'isprivateppa' (None) in request",
2990+ ),
2991+ ],
2992+ )
2993+ def test_validate_generic_ppa_request_invalid_request(
2994+ self, req, exception, ex_message
2995+ ):
2996+ with pytest.raises(exception, match=re.escape(ex_message)):
2997+ MockCommonPPAGetter().validate_request(req)
2998+
2999+
3000+if __name__ == "__main__":
3001+ unittest.main()
3002diff --git a/tests/test_ddebgetter.py b/tests/test_ddebgetter.py
3003new file mode 100644
3004index 0000000..3a67894
3005--- /dev/null
3006+++ b/tests/test_ddebgetter.py
3007@@ -0,0 +1,64 @@
3008+#!/usr/bin/python3
3009+
3010+# Copyright (C) 2023 Canonical Ltd.
3011+
3012+# This program is free software: you can redistribute it and/or modify
3013+# it under the terms of the GNU General Public License as published by
3014+# the Free Software Foundation, either version 3 of the License, or
3015+# (at your option) any later version.
3016+
3017+# This program is distributed in the hope that it will be useful,
3018+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3019+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3020+# GNU General Public License for more details.
3021+
3022+# You should have received a copy of the GNU General Public License
3023+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3024+
3025+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3026+
3027+import re
3028+import pytest
3029+import unittest
3030+from ubuntu_debuginfod.getters import ddebgetter
3031+
3032+
3033+class TestDdebGetter:
3034+
3035+ VALID_DDEB_REQ = {
3036+ "architecture": "amd64",
3037+ "ddeb_url": "URL",
3038+ "uri": "URL",
3039+ }
3040+
3041+ def test_validate_ddeb_request_valid_request(self):
3042+ assert ddebgetter.DdebGetter().validate_request(self.VALID_DDEB_REQ) is None
3043+
3044+ @pytest.mark.parametrize(
3045+ "req, exception, ex_message",
3046+ [
3047+ (None, TypeError, "Invalid request"),
3048+ (
3049+ {"architecture": "amd64", "uri": "URL", "foo": "bar"},
3050+ TypeError,
3051+ "No 'ddeb_url' in request",
3052+ ),
3053+ (
3054+ {"architecture": "amd64", "uri": "URL", "ddeb_url": None},
3055+ ValueError,
3056+ "Invalid 'ddeb_url' (None) in request",
3057+ ),
3058+ (
3059+ {"architecture": "source", "uri": "URL", "ddeb_url": "URL"},
3060+ ValueError,
3061+ "Wrong request: source fetch",
3062+ ),
3063+ ],
3064+ )
3065+ def test_validate_ddeb_request_invalid_request(self, req, exception, ex_message):
3066+ with pytest.raises(exception, match=re.escape(ex_message)):
3067+ ddebgetter.DdebGetter().validate_request(req)
3068+
3069+
3070+if __name__ == "__main__":
3071+ unittest.main()
3072diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py
3073new file mode 100644
3074index 0000000..0a64163
3075--- /dev/null
3076+++ b/tests/test_exceptions.py
3077@@ -0,0 +1,33 @@
3078+#!/usr/bin/python3
3079+
3080+# Copyright (C) 2023 Canonical Ltd.
3081+
3082+# This program is free software: you can redistribute it and/or modify
3083+# it under the terms of the GNU General Public License as published by
3084+# the Free Software Foundation, either version 3 of the License, or
3085+# (at your option) any later version.
3086+
3087+# This program is distributed in the hope that it will be useful,
3088+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3089+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3090+# GNU General Public License for more details.
3091+
3092+# You should have received a copy of the GNU General Public License
3093+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3094+
3095+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3096+
3097+import unittest
3098+from ubuntu_debuginfod.utils import exceptions
3099+
3100+
3101+class TestExceptions:
3102+ def test_timeout_exception_inherits_ok(self):
3103+ assert issubclass(exceptions.UbuntuDebuginfodTimeout, Exception)
3104+
3105+ def test_retry_exception_inherits_ok(self):
3106+ assert issubclass(exceptions.UbuntuDebuginfodRetry, Exception)
3107+
3108+
3109+if __name__ == "__main__":
3110+ unittest.main()
3111diff --git a/tests/test_getter.py b/tests/test_getter.py
3112new file mode 100644
3113index 0000000..933370c
3114--- /dev/null
3115+++ b/tests/test_getter.py
3116@@ -0,0 +1,38 @@
3117+#!/usr/bin/python3
3118+
3119+# Copyright (C) 2023 Canonical Ltd.
3120+
3121+# This program is free software: you can redistribute it and/or modify
3122+# it under the terms of the GNU General Public License as published by
3123+# the Free Software Foundation, either version 3 of the License, or
3124+# (at your option) any later version.
3125+
3126+# This program is distributed in the hope that it will be useful,
3127+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3128+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3129+# GNU General Public License for more details.
3130+
3131+# You should have received a copy of the GNU General Public License
3132+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3133+
3134+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3135+
3136+import unittest
3137+from ubuntu_debuginfod import getter
3138+
3139+
3140+class TestGetter:
3141+ def test_getter_get_subdir(self):
3142+ subdir = "test"
3143+ d = getter.Getter(subdir=subdir)
3144+ assert d.subdir == subdir
3145+
3146+ def test_getter_set_subdir(self):
3147+ d = getter.Getter(subdir="test")
3148+ newsubdir = "newtest"
3149+ d.subdir = newsubdir
3150+ assert d.subdir == newsubdir
3151+
3152+
3153+if __name__ == "__main__":
3154+ unittest.main()
3155diff --git a/tests/test_ppaddebgetter.py b/tests/test_ppaddebgetter.py
3156new file mode 100644
3157index 0000000..3ca3dd1
3158--- /dev/null
3159+++ b/tests/test_ppaddebgetter.py
3160@@ -0,0 +1,94 @@
3161+#!/usr/bin/python3
3162+
3163+# Copyright (C) 2023 Canonical Ltd.
3164+
3165+# This program is free software: you can redistribute it and/or modify
3166+# it under the terms of the GNU General Public License as published by
3167+# the Free Software Foundation, either version 3 of the License, or
3168+# (at your option) any later version.
3169+
3170+# This program is distributed in the hope that it will be useful,
3171+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3172+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3173+# GNU General Public License for more details.
3174+
3175+# You should have received a copy of the GNU General Public License
3176+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3177+
3178+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3179+
3180+import re
3181+import pytest
3182+import unittest
3183+from ubuntu_debuginfod.getters import ppaddebgetter
3184+
3185+
3186+class TestPPADdebGetter:
3187+
3188+ VALID_PUBLIC_PPA_DDEB_REQ = {
3189+ "architecture": "amd64",
3190+ "ddeb_url": "URL",
3191+ "ppauser": "foo",
3192+ "ppaname": "bar",
3193+ "isprivateppa": "no",
3194+ "uri": "URL",
3195+ }
3196+
3197+ VALID_PRIVATE_PPA_DDEB_REQ = {
3198+ "architecture": "amd64",
3199+ "ddeb_url": "URL",
3200+ "ppauser": "foo",
3201+ "ppaname": "bar",
3202+ "isprivateppa": "yes",
3203+ "uri": "URL",
3204+ }
3205+
3206+ def test_validate_ppa_generic_valid_request(self):
3207+ assert (
3208+ ppaddebgetter.PPADdebGetter().validate_request(
3209+ self.VALID_PUBLIC_PPA_DDEB_REQ
3210+ )
3211+ is None
3212+ )
3213+ assert (
3214+ ppaddebgetter.PrivatePPADdebGetter().validate_request(
3215+ self.VALID_PRIVATE_PPA_DDEB_REQ
3216+ )
3217+ is None
3218+ )
3219+
3220+ def test_validate_ppa_ddeb_request_valid_request(self):
3221+ assert (
3222+ ppaddebgetter.PPADdebGetter().validate_request(
3223+ self.VALID_PUBLIC_PPA_DDEB_REQ
3224+ )
3225+ is None
3226+ )
3227+
3228+ def test_validate_ppa_ddeb_request_invalid_request(self):
3229+ with pytest.raises(
3230+ ValueError, match=re.escape("Wrong request: from private PPA")
3231+ ):
3232+ ppaddebgetter.PPADdebGetter().validate_request(
3233+ self.VALID_PRIVATE_PPA_DDEB_REQ
3234+ )
3235+
3236+ def test_validate_private_ppa_ddeb_request_valid_request(self):
3237+ assert (
3238+ ppaddebgetter.PrivatePPADdebGetter().validate_request(
3239+ self.VALID_PRIVATE_PPA_DDEB_REQ
3240+ )
3241+ is None
3242+ )
3243+
3244+ def test_validate_private_ddeb_request_invalid_request(self):
3245+ with pytest.raises(
3246+ ValueError, match=re.escape("Wrong request: not from private PPA")
3247+ ):
3248+ ppaddebgetter.PrivatePPADdebGetter().validate_request(
3249+ self.VALID_PUBLIC_PPA_DDEB_REQ
3250+ )
3251+
3252+
3253+if __name__ == "__main__":
3254+ unittest.main()
3255diff --git a/tests/test_ppasourcepackagegetter.py b/tests/test_ppasourcepackagegetter.py
3256new file mode 100644
3257index 0000000..aec5b95
3258--- /dev/null
3259+++ b/tests/test_ppasourcepackagegetter.py
3260@@ -0,0 +1,94 @@
3261+#!/usr/bin/python3
3262+
3263+# Copyright (C) 2023 Canonical Ltd.
3264+
3265+# This program is free software: you can redistribute it and/or modify
3266+# it under the terms of the GNU General Public License as published by
3267+# the Free Software Foundation, either version 3 of the License, or
3268+# (at your option) any later version.
3269+
3270+# This program is distributed in the hope that it will be useful,
3271+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3272+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3273+# GNU General Public License for more details.
3274+
3275+# You should have received a copy of the GNU General Public License
3276+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3277+
3278+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3279+
3280+import re
3281+import pytest
3282+import unittest
3283+from ubuntu_debuginfod.getters import ppasourcepackagegetter
3284+
3285+
3286+class TestPPASourcePackageGetter:
3287+
3288+ VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ = {
3289+ "architecture": "source",
3290+ "source_urls": ["URL1", "URL2"],
3291+ "ppauser": "foo",
3292+ "ppaname": "bar",
3293+ "isprivateppa": "no",
3294+ "uri": "URL",
3295+ }
3296+
3297+ VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ = {
3298+ "architecture": "source",
3299+ "source_urls": ["URL1", "URL2"],
3300+ "ppauser": "foo",
3301+ "ppaname": "bar",
3302+ "isprivateppa": "yes",
3303+ "uri": "URL",
3304+ }
3305+
3306+ def test_validate_request_valid_request(self):
3307+ assert (
3308+ ppasourcepackagegetter.PPASourcePackageGetter().validate_request(
3309+ self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ
3310+ )
3311+ is None
3312+ )
3313+ assert (
3314+ ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request(
3315+ self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ
3316+ )
3317+ is None
3318+ )
3319+
3320+ def test_validate_ppa_source_code_request_valid_request(self):
3321+ assert (
3322+ ppasourcepackagegetter.PPASourcePackageGetter().validate_request(
3323+ self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ
3324+ )
3325+ is None
3326+ )
3327+
3328+ def test_validate_ppa_source_code_request_invalid_request(self):
3329+ with pytest.raises(
3330+ ValueError, match=re.escape("Wrong request: from private PPA")
3331+ ):
3332+ ppasourcepackagegetter.PPASourcePackageGetter().validate_request(
3333+ self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ
3334+ )
3335+
3336+ def test_validate_private_ppa_source_code_request_valid_request(self):
3337+ assert (
3338+ ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request(
3339+ self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ
3340+ )
3341+ is None
3342+ )
3343+
3344+ def test_validate_private_ppa_source_code_request_invalid_request(self):
3345+ with pytest.raises(
3346+ ValueError, match=re.escape("Wrong request: not from private PPA")
3347+ ):
3348+ ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request(
3349+ self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ
3350+ )
3351+
3352+
3353+if __name__ == "__main__":
3354+ unittest.main()
3355diff --git a/tests/test_sourcepackagegetter.py b/tests/test_sourcepackagegetter.py
3356new file mode 100644
3357index 0000000..f82c4f7
3358--- /dev/null
3359+++ b/tests/test_sourcepackagegetter.py
3360@@ -0,0 +1,69 @@
3361+#!/usr/bin/python3
3362+
3363+# Copyright (C) 2023 Canonical Ltd.
3364+
3365+# This program is free software: you can redistribute it and/or modify
3366+# it under the terms of the GNU General Public License as published by
3367+# the Free Software Foundation, either version 3 of the License, or
3368+# (at your option) any later version.
3369+
3370+# This program is distributed in the hope that it will be useful,
3371+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3372+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3373+# GNU General Public License for more details.
3374+
3375+# You should have received a copy of the GNU General Public License
3376+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3377+
3378+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3379+
3380+import re
3381+import pytest
3382+import unittest
3383+from ubuntu_debuginfod.getters import sourcepackagegetter
3384+
3385+
3386+class TestSourcePackageGetter:
3387+
3388+ VALID_SOURCE_PACKAGE_REQ = {
3389+ "architecture": "source",
3390+ "source_urls": [],
3391+ "uri": "URL",
3392+ }
3393+
3394+ def test_validate_request_valid_request(self):
3395+ assert (
3396+ sourcepackagegetter.SourcePackageGetter().validate_request(
3397+ self.VALID_SOURCE_PACKAGE_REQ
3398+ )
3399+ is None
3400+ )
3401+
3402+ @pytest.mark.parametrize(
3403+ "req, exception, ex_message",
3404+ [
3405+ (None, TypeError, "Invalid request"),
3406+ (
3407+ {"architecture": "source", "uri": "URL", "foo": "bar"},
3408+ TypeError,
3409+ "No 'source_urls' in request",
3410+ ),
3411+ (
3412+ {"architecture": "source", "uri": "URL", "source_urls": None},
3413+ ValueError,
3414+ "Invalid 'source_urls' (None) in request",
3415+ ),
3416+ (
3417+ {"architecture": "amd64", "uri": "URL", "source_urls": "URL"},
3418+ ValueError,
3419+ "Wrong request: ddeb fetch",
3420+ ),
3421+ ],
3422+ )
3423+ def test_validate_request_invalid_request(self, req, exception, ex_message):
3424+ with pytest.raises(exception, match=re.escape(ex_message)):
3425+ sourcepackagegetter.SourcePackageGetter().validate_request(req)
3426+
3427+
3428+if __name__ == "__main__":
3429+ unittest.main()
3430diff --git a/tests/test_utils.py b/tests/test_utils.py
3431new file mode 100644
3432index 0000000..7cac497
3433--- /dev/null
3434+++ b/tests/test_utils.py
3435@@ -0,0 +1,114 @@
3436+#!/usr/bin/python3
3437+
3438+# Copyright (C) 2023 Canonical Ltd.
3439+
3440+# This program is free software: you can redistribute it and/or modify
3441+# it under the terms of the GNU General Public License as published by
3442+# the Free Software Foundation, either version 3 of the License, or
3443+# (at your option) any later version.
3444+
3445+# This program is distributed in the hope that it will be useful,
3446+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3447+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3448+# GNU General Public License for more details.
3449+
3450+# You should have received a copy of the GNU General Public License
3451+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3452+
3453+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3454+
3455+import os
3456+import re
3457+import unittest
3458+import datetime
3459+import tempfile
3460+import filecmp
3461+import pytest
3462+from freezegun import freeze_time
3463+from ubuntu_debuginfod.utils import utils
3464+
3465+
3466+class TestUtils:
3467+ def test_generate_timestamp_without_interval(self, *args):
3468+ curdate = datetime.datetime.now(datetime.timezone.utc)
3469+ with freeze_time(curdate):
3470+ ret = utils.generate_timestamp()
3471+ assert isinstance(ret, datetime.datetime)
3472+ assert ret == curdate
3473+
3474+ def test_generate_timestamp_with_interval(self, *args):
3475+ curdate = datetime.datetime.now(datetime.timezone.utc)
3476+ interval = 8
3477+ with freeze_time(curdate):
3478+ ret = utils.generate_timestamp(interval=interval)
3479+ assert isinstance(ret, datetime.datetime)
3480+ assert ret == curdate - datetime.timedelta(hours=interval)
3481+
3482+ @pytest.mark.skip(
3483+ reason="currently broken when dealing with files in the same directory"
3484+ )
3485+ def test_move_atomically_same_dir(self):
3486+ with tempfile.TemporaryDirectory() as tmp:
3487+ file1 = os.path.join(tmp, "file1")
3488+ file2 = os.path.join(tmp, "file2")
3489+
3490+ with open(file1, "w") as f:
3491+ f.write("This is a test")
3492+ f.flush()
3493+
3494+ utils.move_atomically(file1, file2)
3495+ filecmp.clear_cache()
3496+ assert filecmp.cmp(file1, file2, shallow=False)
3497+
3498+ def test_move_atomically_different_dirs(self):
3499+ with tempfile.TemporaryDirectory() as tmp1:
3500+ file1 = os.path.join(tmp1, "file1")
3501+ with open(file1, "w") as f:
3502+ f.write("This is a test")
3503+
3504+ with tempfile.TemporaryDirectory() as tmp2:
3505+ file2 = os.path.join(tmp2, "file2")
3506+
3507+ utils.move_atomically(file1, file2)
3508+ filecmp.clear_cache()
3509+ assert filecmp.cmp(file1, file2, shallow=False)
3510+
3511+ @pytest.mark.parametrize(
3512+ "version,expected_result",
3513+ [
3514+ ("1.0-1", "1.0-1"),
3515+ ("1.0-1~ppa1", "1.0-1_ppa1"),
3516+ ("2:1.0-1", "2%1.0-1"),
3517+ ("1.0..1", "1.0.#.1"),
3518+ ("1.0..1.", "1.0.#.1.#"),
3519+ ("1.0-1.lock", "1.0-1.#lock"),
3520+ ("3:1.0-1..~ppa1.lock", "3%1.0-1.#._ppa1.#lock"),
3521+ ],
3522+ )
3523+ def test_git_dep14_tag(self, version, expected_result):
3524+ assert utils.git_dep14_tag(version) == expected_result
3525+
3526+ def test_obtain_lp_oauth_credentials_no_LP_CREDENTIALS_FILE_set(self):
3527+ if "LP_CREDENTIALS_FILE" in os.environ:
3528+ del os.environ["LP_CREDENTIALS_FILE"]
3529+ with pytest.raises(
3530+ ValueError,
3531+ match=re.escape(
3532+ "Could not obtain Launchpad credentials: environment variable LP_CREDENTIALS_FILE must be set"
3533+ ),
3534+ ):
3535+ utils.obtain_lp_oauth_credentials()
3536+
3537+ def test_obtain_lp_oauth_credentials_LP_CREDENTIALS_FILE_invalid_file(self):
3538+ os.environ["LP_CREDENTIALS_FILE"] = "/path/to/nonexistent/file"
3539+ with pytest.raises(
3540+ FileNotFoundError,
3541+ match=re.escape(
3542+ "Could not obtain Launchpad credentials: file /path/to/nonexistent/file does not exist"
3543+ ),
3544+ ):
3545+ utils.obtain_lp_oauth_credentials()
3546+
3547+
3548+if __name__ == "__main__":
3549+ unittest.main()
3550diff --git a/ubuntu_debuginfod/__init__.py b/ubuntu_debuginfod/__init__.py
3551new file mode 100644
3552index 0000000..c63b042
3553--- /dev/null
3554+++ b/ubuntu_debuginfod/__init__.py
3555@@ -0,0 +1,10 @@
3556+"""Init."""
3557+
3558+import logging
3559+
3560+logging.basicConfig(
3561+ level=logging.INFO,
3562+ format=(
3563+ "%(levelname) -10s %(asctime)s %(name) -20s %(funcName) " "-25s : %(message)s"
3564+ ),
3565+)
3566diff --git a/debuginfod.py b/ubuntu_debuginfod/debuginfod.py
3567similarity index 61%
3568rename from debuginfod.py
3569rename to ubuntu_debuginfod/debuginfod.py
3570index ef39a95..54d73c9 100644
3571--- a/debuginfod.py
3572+++ b/ubuntu_debuginfod/debuginfod.py
3573@@ -24,18 +24,24 @@ from celery.signals import (
3574 task_postrun,
3575 celeryd_init,
3576 )
3577+import sdnotify
3578
3579-from debuggetter import DebugGetterTimeout, DebugGetterRetry
3580-from ddebgetter import DdebGetter, DdebSourceCodeGetter
3581-from ppagetter import (
3582+from ubuntu_debuginfod.utils.exceptions import (
3583+ UbuntuDebuginfodRetry,
3584+ UbuntuDebuginfodTimeout,
3585+)
3586+from ubuntu_debuginfod.getters.ddebgetter import DdebGetter
3587+from ubuntu_debuginfod.getters.sourcepackagegetter import SourcePackageGetter
3588+from ubuntu_debuginfod.getters.ppaddebgetter import (
3589 PPADdebGetter,
3590- PPASourceCodeGetter,
3591 PrivatePPADdebGetter,
3592- PrivatePPASourceCodeGetter,
3593 )
3594-from utils import DebugDB
3595-
3596-import sdnotify
3597+from ubuntu_debuginfod.getters.ppasourcepackagegetter import (
3598+ PPASourcePackageGetter,
3599+ PrivatePPASourcePackageGetter,
3600+)
3601+from ubuntu_debuginfod.utils.debugdb import DebugDB
3602+from ubuntu_debuginfod.utils.utils import record_error_into_db
3603
3604 app = Celery("debuginfod", broker="pyamqp://guest@localhost//")
3605
3606@@ -55,29 +61,14 @@ app.conf.update(
3607
3608 sdnotifier = sdnotify.SystemdNotifier()
3609
3610-
3611-def _record_error_into_db(exc, msg):
3612- """Record an error into the database.
3613-
3614- :param Exception exc: The Exception that was triggered.
3615-
3616- :param dict msg: The message that triggere the error."""
3617- with DebugDB() as db:
3618- with db.cursor() as cur:
3619- cur.execute(
3620- "INSERT INTO errors (task, exception, date) VALUES (%s, %s, NOW())",
3621- (
3622- str(msg),
3623- str(exc),
3624- ),
3625- )
3626+# 1 day
3627+DEFAULT_RETRY_DELAY = 24 * 60 * 60
3628
3629
3630 @app.task(
3631 name="grab_ddebs",
3632- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3633- # 1 day
3634- default_retry_delay=24 * 60 * 60,
3635+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3636+ default_retry_delay=DEFAULT_RETRY_DELAY,
3637 # Launchpad can be problematic, so we retry every day for a
3638 # week.
3639 retry_kwargs={"max_retries": 7},
3640@@ -91,41 +82,40 @@ def grab_ddebs(msg):
3641 g = DdebGetter()
3642 g.process_request(msg)
3643 except Exception as e:
3644- if not isinstance(e, DebugGetterRetry) and not isinstance(
3645- e, DebugGetterTimeout
3646+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3647+ e, UbuntuDebuginfodTimeout
3648 ):
3649 # This is some other kind of error that we need to deal
3650 # with. Mark it as such.
3651- _record_error_into_db(e, msg)
3652+ record_error_into_db(e, msg)
3653 # We still need to raise the exception here. Celery will
3654 # reschedule the task if applicable.
3655 raise e
3656
3657
3658 @app.task(
3659- name="grab_ddebs_sources",
3660- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3661- # 1 day
3662- default_retry_delay=24 * 60 * 60,
3663+ name="grab_sources",
3664+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3665+ default_retry_delay=DEFAULT_RETRY_DELAY,
3666 # Launchpad can be problematic, so we retry every day for a
3667 # week.
3668 retry_kwargs={"max_retries": 7},
3669 )
3670-def grab_ddebs_sources(msg):
3671- """Dispatch the DdebSourceCodeGetter task.
3672+def grab_sources(msg):
3673+ """Dispatch the SourceCodeGetter task.
3674
3675 :param dict[str, str] msg: The dictionary containing the message
3676 that will be processed by the getter."""
3677 try:
3678- g = DdebSourceCodeGetter()
3679+ g = SourcePackageGetter()
3680 g.process_request(msg)
3681 except Exception as e:
3682- if not isinstance(e, DebugGetterRetry) and not isinstance(
3683- e, DebugGetterTimeout
3684+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3685+ e, UbuntuDebuginfodTimeout
3686 ):
3687 # This is some other kind of error that we need to deal
3688 # with. Mark it as such.
3689- _record_error_into_db(e, msg)
3690+ record_error_into_db(e, msg)
3691 # We still need to raise the exception here. Celery will
3692 # reschedule the task if applicable.
3693 raise e
3694@@ -133,9 +123,8 @@ def grab_ddebs_sources(msg):
3695
3696 @app.task(
3697 name="grab_ppa_ddebs",
3698- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3699- # 1 day
3700- default_retry_delay=24 * 60 * 60,
3701+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3702+ default_retry_delay=DEFAULT_RETRY_DELAY,
3703 # Launchpad can be problematic, so we retry every day for a
3704 # week.
3705 retry_kwargs={"max_retries": 7},
3706@@ -149,12 +138,12 @@ def grab_ppa_ddebs(msg):
3707 g = PPADdebGetter()
3708 g.process_request(msg)
3709 except Exception as e:
3710- if not isinstance(e, DebugGetterRetry) and not isinstance(
3711- e, DebugGetterTimeout
3712+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3713+ e, UbuntuDebuginfodTimeout
3714 ):
3715 # This is some other kind of error that we need to deal
3716 # with. Mark it as such.
3717- _record_error_into_db(e, msg)
3718+ record_error_into_db(e, msg)
3719 # We still need to raise the exception here. Celery will
3720 # reschedule the task if applicable.
3721 raise e
3722@@ -162,28 +151,27 @@ def grab_ppa_ddebs(msg):
3723
3724 @app.task(
3725 name="grab_ppa_sources",
3726- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3727- # 1 day
3728- default_retry_delay=24 * 60 * 60,
3729+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3730+ default_retry_delay=DEFAULT_RETRY_DELAY,
3731 # Launchpad can be problematic, so we retry every day for a
3732 # week.
3733 retry_kwargs={"max_retries": 7},
3734 )
3735 def grab_ppa_sources(msg):
3736- """Dispatch the PPASourceCodeGetter task.
3737+ """Dispatch the PPASourcePackageGetter task.
3738
3739 :param dict[str, str] msg: The dictionary containing the message
3740 that will be processed by the getter."""
3741 try:
3742- g = PPASourceCodeGetter()
3743+ g = PPASourcePackageGetter()
3744 g.process_request(msg)
3745 except Exception as e:
3746- if not isinstance(e, DebugGetterRetry) and not isinstance(
3747- e, DebugGetterTimeout
3748+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3749+ e, UbuntuDebuginfodTimeout
3750 ):
3751 # This is some other kind of error that we need to deal
3752 # with. Mark it as such.
3753- _record_error_into_db(e, msg)
3754+ record_error_into_db(e, msg)
3755 # We still need to raise the exception here. Celery will
3756 # reschedule the task if applicable.
3757 raise e
3758@@ -191,9 +179,8 @@ def grab_ppa_sources(msg):
3759
3760 @app.task(
3761 name="grab_private_ppa_ddebs",
3762- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3763- # 1 day
3764- default_retry_delay=24 * 60 * 60,
3765+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3766+ default_retry_delay=DEFAULT_RETRY_DELAY,
3767 # Launchpad can be problematic, so we retry every day for a
3768 # week.
3769 retry_kwargs={"max_retries": 7},
3770@@ -207,12 +194,12 @@ def grab_private_ppa_ddebs(msg):
3771 g = PrivatePPADdebGetter()
3772 g.process_request(msg)
3773 except Exception as e:
3774- if not isinstance(e, DebugGetterRetry) and not isinstance(
3775- e, DebugGetterTimeout
3776+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3777+ e, UbuntuDebuginfodTimeout
3778 ):
3779 # This is some other kind of error that we need to deal
3780 # with. Mark it as such.
3781- _record_error_into_db(e, msg)
3782+ record_error_into_db(e, msg)
3783 # We still need to raise the exception here. Celery will
3784 # reschedule the task if applicable.
3785 raise e
3786@@ -220,28 +207,27 @@ def grab_private_ppa_ddebs(msg):
3787
3788 @app.task(
3789 name="grab_private_ppa_sources",
3790- autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
3791- # 1 day
3792- default_retry_delay=24 * 60 * 60,
3793+ autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry),
3794+ default_retry_delay=DEFAULT_RETRY_DELAY,
3795 # Launchpad can be problematic, so we retry every day for a
3796 # week.
3797 retry_kwargs={"max_retries": 7},
3798 )
3799 def grab_private_ppa_sources(msg):
3800- """Dispatch the PrivatePPASourceCodeGetter task.
3801+ """Dispatch the PrivatePPASourcePackageGetter task.
3802
3803 :param dict[str, str] msg: The dictionary containing the message
3804 that will be processed by the getter."""
3805 try:
3806- g = PrivatePPASourceCodeGetter()
3807+ g = PrivatePPASourcePackageGetter()
3808 g.process_request(msg)
3809 except Exception as e:
3810- if not isinstance(e, DebugGetterRetry) and not isinstance(
3811- e, DebugGetterTimeout
3812+ if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance(
3813+ e, UbuntuDebuginfodTimeout
3814 ):
3815 # This is some other kind of error that we need to deal
3816 # with. Mark it as such.
3817- _record_error_into_db(e, msg)
3818+ record_error_into_db(e, msg)
3819 # We still need to raise the exception here. Celery will
3820 # reschedule the task if applicable.
3821 raise e
3822@@ -259,13 +245,42 @@ def notify_worker_shutting_down(**kwargs):
3823 sdnotifier.notify("STOPPING=1")
3824
3825
3826+def _get_msg_field(msg, field):
3827+ """Return a specific field from msg.
3828+
3829+ :param dict[str -> str] msg: The dictionary received by this
3830+ Celery job.
3831+
3832+ :param str field: The field name that we want.
3833+
3834+ :rtype str or list:
3835+ :return: The value for field.
3836+
3837+ :raises: RuntimeError if field does not exist or if it doesn't
3838+ have a non-None value."""
3839+ if field not in msg.keys():
3840+ e = RuntimeError(f"{field} could not be found in {msg}")
3841+ record_error_into_db(e, "Failed during _get_msg_field")
3842+ raise e
3843+ val = msg[field]
3844+ if val is None:
3845+ e = RuntimeError(f"{field} is None")
3846+ record_error_into_db(e, "Failed during _get_msg_field")
3847+ raise e
3848+ return val
3849+
3850+
3851 @task_postrun.connect
3852-def task_postrun(sender, args, state, **kwargs):
3853+def debuginfod_task_postrun(sender, args, state, **kwargs):
3854 """Process a task that's just finished.
3855
3856 We keep the state of each task in a SQL database, and this
3857 function is responsible for cleaning up the state for tasks that have
3858- succeeded."""
3859+ succeeded.
3860+
3861+ Moreover, it also records information about the downloaded
3862+ artifact so that we can keep track of it later (i.e., when we need
3863+ to clean things up)."""
3864 if state not in ("SUCCESS",):
3865 return
3866
3867@@ -274,13 +289,9 @@ def task_postrun(sender, args, state, **kwargs):
3868 msg = args[0]
3869 if not msg:
3870 e = RuntimeError("Could not obtain message to be processed.")
3871- _record_error_into_db(e, "Failed during task_postrun")
3872- raise e
3873- if not msg.get("taskid"):
3874- e = RuntimeError("Could not obtain taskid.")
3875- _record_error_into_db(e, "Failed during task_postrun")
3876+ record_error_into_db(e, "Failed during task_postrun")
3877 raise e
3878- taskid = msg["taskid"]
3879+ taskid = _get_msg_field(msg, "taskid")
3880 cur.execute("DELETE FROM tasks WHERE id = %s", (taskid,))
3881
3882
3883diff --git a/ubuntu_debuginfod/getter.py b/ubuntu_debuginfod/getter.py
3884new file mode 100644
3885index 0000000..7ced991
3886--- /dev/null
3887+++ b/ubuntu_debuginfod/getter.py
3888@@ -0,0 +1,57 @@
3889+#!/usr/bin/python3
3890+
3891+# Copyright (C) 2022 Canonical Ltd.
3892+
3893+# This program is free software: you can redistribute it and/or modify
3894+# it under the terms of the GNU General Public License as published by
3895+# the Free Software Foundation, either version 3 of the License, or
3896+# (at your option) any later version.
3897+
3898+# This program is distributed in the hope that it will be useful,
3899+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3900+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3901+# GNU General Public License for more details.
3902+
3903+# You should have received a copy of the GNU General Public License
3904+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3905+
3906+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3907+
3908+import logging
3909+
3910+
3911+class Getter:
3912+ """Base class for a Debug Getter."""
3913+
3914+ # The default directory where the debug artifacts will be saved.
3915+ DEFAULT_MIRROR_DIR = "/srv/debug-mirror"
3916+
3917+ def __init__(self, subdir, mirror_dir=None, anonymous=True):
3918+ """Initialize the object.
3919+
3920+ :param str mirror_dir: The directory we use to save the
3921+ mirrored files from Launchpad.
3922+
3923+ :param str subdir: The subdirectory (insider mirror_dir) where
3924+ the module will save its files. For example, a ddeb
3925+ getter module should specify "ddebs"here. If None, use
3926+ DEFAULT_MIRROR_DIR.
3927+
3928+ :param bool anonymous: Whether this getter handles anonymous
3929+ (public) artifacts. Default to True."""
3930+ self._mirror_dir = self.DEFAULT_MIRROR_DIR if mirror_dir is None else mirror_dir
3931+ self._subdir = subdir
3932+ self._anonymous = anonymous
3933+ self._logger = logging.getLogger(__name__)
3934+ self._logger.debug(
3935+ f"Initializing Getter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}"
3936+ )
3937+
3938+ @property
3939+ def subdir(self):
3940+ return self._subdir
3941+
3942+ @subdir.setter
3943+ def subdir(self, subdir):
3944+ self._logger.debug(f"Setting {type(self).__name__} subdir to '{subdir}'")
3945+ self._subdir = subdir
3946diff --git a/ubuntu_debuginfod/getters/__init__.py b/ubuntu_debuginfod/getters/__init__.py
3947new file mode 100644
3948index 0000000..14e8999
3949--- /dev/null
3950+++ b/ubuntu_debuginfod/getters/__init__.py
3951@@ -0,0 +1 @@
3952+"""Init."""
3953diff --git a/ubuntu_debuginfod/getters/common_lp_getter.py b/ubuntu_debuginfod/getters/common_lp_getter.py
3954new file mode 100644
3955index 0000000..4852678
3956--- /dev/null
3957+++ b/ubuntu_debuginfod/getters/common_lp_getter.py
3958@@ -0,0 +1,104 @@
3959+#!/usr/bin/python3
3960+
3961+# Copyright (C) 2022-2023 Canonical Ltd.
3962+
3963+# This program is free software: you can redistribute it and/or modify
3964+# it under the terms of the GNU General Public License as published by
3965+# the Free Software Foundation, either version 3 of the License, or
3966+# (at your option) any later version.
3967+
3968+# This program is distributed in the hope that it will be useful,
3969+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3970+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3971+# GNU General Public License for more details.
3972+
3973+# You should have received a copy of the GNU General Public License
3974+# along with this program. If not, see <https://www.gnu.org/licenses/>.
3975+
3976+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
3977+
3978+import os
3979+from ubuntu_debuginfod.getter import Getter
3980+from ubuntu_debuginfod.utils.debugdb import DebugDB
3981+from ubuntu_debuginfod.utils.utils import obtain_lp_oauth_credentials
3982+
3983+
3984+class CommonLaunchpadGetter(Getter):
3985+ """Common getter class for dealing with Launchpad."""
3986+
3987+ def __init__(self, subdir, mirror_dir=None, anonymous=True):
3988+ """Initialize the object.
3989+
3990+ See Getter's __init__ for an explanation of the arguments."""
3991+ super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=anonymous)
3992+ self._logger.debug(
3993+ f"Initializing CommonLaunchpadGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}"
3994+ )
3995+
3996+ def make_savepath(self, source_package, component):
3997+ """Return the full save path for a package.
3998+
3999+ :param str source_package: The package name.
4000+
4001+ :param str component: The component name (main, universe,
4002+ etc.)."""
4003+ if source_package.startswith("lib"):
4004+ pkgname_initials = source_package[:4]
4005+ else:
4006+ pkgname_initials = source_package[0]
4007+
4008+ return os.path.join(
4009+ self._mirror_dir,
4010+ self._subdir,
4011+ component,
4012+ pkgname_initials,
4013+ source_package,
4014+ )
4015+
4016+ def process_request(self, request):
4017+ """Process a request.
4018+
4019+ :param dict[str, str] request: The dictionary containing the
4020+ information necessary to fetch this ddeb.
4021+
4022+ :param requests_oauthlib.OAuth1 credentials: The credentials
4023+ to be used when downloading the artifact from Launchpad.
4024+ Default is None, which means anonymous."""
4025+ self._logger.debug(
4026+ f"Processing request from {type(self).__name__} to download artifact: {request}"
4027+ )
4028+ credentials = None if self._anonymous else obtain_lp_oauth_credentials()
4029+ fpath = self.download_artifact(
4030+ request,
4031+ credentials=credentials,
4032+ )
4033+ if fpath is not None:
4034+ with DebugDB() as db:
4035+ with db.cursor() as cur:
4036+ cur.execute(
4037+ "INSERT INTO artifacts (fullpath, uri, private, date) VALUES (%s, %s, %s, NOW())",
4038+ (fpath, request["uri"], not self._anonymous),
4039+ )
4040+
4041+ def validate_request(self, request):
4042+ """Validate a request coming from Celery.
4043+
4044+ :param dict[str, str] request: The dictionary containing the
4045+ request to be validated."""
4046+ if request is None:
4047+ raise TypeError("Invalid request (None)")
4048+
4049+ if "architecture" not in request.keys():
4050+ raise ValueError("No 'architecture' in request")
4051+ if request.get("architecture") is None:
4052+ raise ValueError("Invalid 'architecture' (None) in request")
4053+
4054+ if "uri" not in request.keys():
4055+ raise ValueError("No 'uri' in request")
4056+ if request.get("uri") is None:
4057+ raise ValueError("Invalid 'uri' (None) in request")
4058+
4059+ def download_artifact(self, *args):
4060+ """Interface for download_artifact. Subclasses of
4061+ CommonLaunchpadGetter are expected to implement it."""
4062+ raise NotImplementedError("Wrong call to 'download_artifact' from base class")
4063diff --git a/ubuntu_debuginfod/getters/common_ppa_getter.py b/ubuntu_debuginfod/getters/common_ppa_getter.py
4064new file mode 100644
4065index 0000000..290bbe7
4066--- /dev/null
4067+++ b/ubuntu_debuginfod/getters/common_ppa_getter.py
4068@@ -0,0 +1,65 @@
4069+#!/usr/bin/python3
4070+
4071+# Copyright (C) 2023 Canonical Ltd.
4072+
4073+# This program is free software: you can redistribute it and/or modify
4074+# it under the terms of the GNU General Public License as published by
4075+# the Free Software Foundation, either version 3 of the License, or
4076+# (at your option) any later version.
4077+
4078+# This program is distributed in the hope that it will be useful,
4079+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4080+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4081+# GNU General Public License for more details.
4082+
4083+# You should have received a copy of the GNU General Public License
4084+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4085+
4086+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4087+
4088+import os
4089+import logging
4090+
4091+
4092+class CommonPPAGetter:
4093+ """A common class for PPA getters."""
4094+
4095+ def __init__(self, *args, **kwargs):
4096+ """Initialize the object.
4097+
4098+ We're not interested in doing anything here, since this is a
4099+ base class that should be used in multiple-inheritance
4100+ scenarios."""
4101+ super().__init__(*args, **kwargs)
4102+ self._logger = logging.getLogger(__name__)
4103+ self._logger.debug("Initializing CommonPPAGetter")
4104+
4105+ def validate_request(self, request):
4106+ """Validate a request to fetch an artifact from a PPA (private or
4107+ not).
4108+
4109+ :param dict[str, str] request: The dictionary containing the
4110+ request to be validated."""
4111+ super().validate_request(request)
4112+ keys = request.keys()
4113+ if "ppauser" not in keys:
4114+ raise TypeError("No 'ppauser' in request")
4115+ if "ppaname" not in keys:
4116+ raise TypeError("No 'ppaname' in request")
4117+ if "isprivateppa" not in keys:
4118+ raise TypeError("No 'isprivateppa' in request")
4119+
4120+ if request.get("ppauser") is None:
4121+ raise ValueError("Invalid 'ppauser' (None) in request")
4122+ if request.get("ppaname") is None:
4123+ raise ValueError("Invalid 'ppaname' (None) in request")
4124+ if request.get("isprivateppa") is None:
4125+ raise ValueError("Invalid 'isprivateppa' (None) in request")
4126+
4127+ def process_request(self, request, credentials=None):
4128+ """Process a request, usually coming from Celery.
4129+
4130+ :param dict[str -> str] request: The dictionary containing the
4131+ information necessary to fetch this ddeb."""
4132+ self.subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"])
4133+ super().process_request(request, credentials=credentials)
4134diff --git a/ubuntu_debuginfod/getters/ddebgetter.py b/ubuntu_debuginfod/getters/ddebgetter.py
4135new file mode 100644
4136index 0000000..633ed93
4137--- /dev/null
4138+++ b/ubuntu_debuginfod/getters/ddebgetter.py
4139@@ -0,0 +1,70 @@
4140+#!/usr/bin/python3
4141+
4142+# Copyright (C) 2022 Canonical Ltd.
4143+
4144+# This program is free software: you can redistribute it and/or modify
4145+# it under the terms of the GNU General Public License as published by
4146+# the Free Software Foundation, either version 3 of the License, or
4147+# (at your option) any later version.
4148+
4149+# This program is distributed in the hope that it will be useful,
4150+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4151+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4152+# GNU General Public License for more details.
4153+
4154+# You should have received a copy of the GNU General Public License
4155+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4156+
4157+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4158+
4159+from ubuntu_debuginfod.getters.common_lp_getter import CommonLaunchpadGetter
4160+from ubuntu_debuginfod.utils.utils import download_from_lp
4161+
4162+
4163+class DdebGetter(CommonLaunchpadGetter):
4164+ """Get (fetch) a ddeb."""
4165+
4166+ def __init__(self, subdir="ddebs", mirror_dir=None, anonymous=True):
4167+ """Initialize the object.
4168+
4169+ See Getter's __init__ for an explanation of the arguments."""
4170+ super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=anonymous)
4171+ self._logger.debug(
4172+ f"Initializing DdebGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}"
4173+ )
4174+
4175+ def download_artifact(self, request, credentials=None):
4176+ """Download a ddeb associated with a package.
4177+
4178+ :param dict[str, str] request: The dictionary containing the
4179+ information necessary to fetch this ddeb.
4180+
4181+ :param requests_oauthlib.OAuth1 credentials: The credentials
4182+ to be used when downloading the artifact from Launchpad.
4183+ Default is None, which means anonymous.
4184+
4185+ :rtype: str
4186+ :return: The full path for the file that was saved, or None if
4187+ the file already exists."""
4188+ self.validate_request(request)
4189+ source_package = request["source_package"]
4190+ component = request["component"]
4191+ ddeb_url = request["ddeb_url"]
4192+ savepath = self.make_savepath(source_package, component)
4193+ self._logger.debug(f"Downloading '{ddeb_url}' into '{savepath}'")
4194+ if credentials is not None:
4195+ self._logger.debug("Downloading with credentials")
4196+ return download_from_lp(ddeb_url, savepath, credentials=credentials)
4197+
4198+ def validate_request(self, request):
4199+ """Validate a request to fetch a ddeb from the main archive.
4200+
4201+ :param dict[str, str] request: The dictionary containing the
4202+ request to be validated."""
4203+ super().validate_request(request)
4204+ if "ddeb_url" not in request.keys():
4205+ raise TypeError("No 'ddeb_url' in request")
4206+ if request.get("ddeb_url") is None:
4207+ raise ValueError("Invalid 'ddeb_url' (None) in request")
4208+ if request["architecture"] == "source":
4209+ raise ValueError("Wrong request: source fetch")
4210diff --git a/ubuntu_debuginfod/getters/ppaddebgetter.py b/ubuntu_debuginfod/getters/ppaddebgetter.py
4211new file mode 100644
4212index 0000000..4d86232
4213--- /dev/null
4214+++ b/ubuntu_debuginfod/getters/ppaddebgetter.py
4215@@ -0,0 +1,65 @@
4216+#!/usr/bin/python3
4217+
4218+# Copyright (C) 2023 Canonical Ltd.
4219+
4220+# This program is free software: you can redistribute it and/or modify
4221+# it under the terms of the GNU General Public License as published by
4222+# the Free Software Foundation, either version 3 of the License, or
4223+# (at your option) any later version.
4224+
4225+# This program is distributed in the hope that it will be useful,
4226+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4227+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4228+# GNU General Public License for more details.
4229+
4230+# You should have received a copy of the GNU General Public License
4231+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4232+
4233+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4234+
4235+from ubuntu_debuginfod.getters.ddebgetter import DdebGetter
4236+from ubuntu_debuginfod.getters.common_ppa_getter import CommonPPAGetter
4237+
4238+
4239+class PPADdebGetter(CommonPPAGetter, DdebGetter):
4240+ """Get a ddeb from a Launchpad PPA."""
4241+
4242+ def __init__(self, subdir="ppas", mirror_dir=None):
4243+ """Initialize the object.
4244+
4245+ See Getter's __init__ for an explanation of the arguments."""
4246+ super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=True)
4247+ self._logger.debug(
4248+ f"Initializing PPADdebGetter with subdir={subdir}, mirror_dir={mirror_dir}"
4249+ )
4250+
4251+ def validate_request(self, request):
4252+ """Validate a request to fetch a ddeb from a PPA.
4253+
4254+ :param dict[str, str] request: The dictionary containing the
4255+ request to be validated."""
4256+ super().validate_request(request)
4257+ if request["isprivateppa"] == "yes":
4258+ raise ValueError("Wrong request: from private PPA")
4259+
4260+
4261+class PrivatePPADdebGetter(CommonPPAGetter, DdebGetter):
4262+ """Get a ddeb from a private Launchpad PPA."""
4263+
4264+ def __init__(self, subdir="private-ppas", mirror_dir=None):
4265+ """Initialize the object.
4266+
4267+ See Getter's __init__ for an explanation of the arguments."""
4268+ super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=False)
4269+ self._logger.debug(
4270+ f"Initializing PrivatePPADdebGetter with subdir={subdir}, mirror_dir={mirror_dir}"
4271+ )
4272+
4273+ def validate_request(self, request):
4274+ """Validate a request to fetch a ddeb from a private PPA.
4275+
4276+ :param dict[str, str] request: The dictionary containing the
4277+ request to be validated."""
4278+ super().validate_request(request)
4279+ if request["isprivateppa"] == "no":
4280+ raise ValueError("Wrong request: not from private PPA")
4281diff --git a/ubuntu_debuginfod/getters/ppasourcepackagegetter.py b/ubuntu_debuginfod/getters/ppasourcepackagegetter.py
4282new file mode 100644
4283index 0000000..ae488a4
4284--- /dev/null
4285+++ b/ubuntu_debuginfod/getters/ppasourcepackagegetter.py
4286@@ -0,0 +1,70 @@
4287+#!/usr/bin/python3
4288+
4289+# Copyright (C) 2023 Canonical Ltd.
4290+
4291+# This program is free software: you can redistribute it and/or modify
4292+# it under the terms of the GNU General Public License as published by
4293+# the Free Software Foundation, either version 3 of the License, or
4294+# (at your option) any later version.
4295+
4296+# This program is distributed in the hope that it will be useful,
4297+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4298+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4299+# GNU General Public License for more details.
4300+
4301+# You should have received a copy of the GNU General Public License
4302+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4303+
4304+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4305+
4306+from ubuntu_debuginfod.getters.sourcepackagegetter import SourcePackageGetter
4307+from ubuntu_debuginfod.getters.common_ppa_getter import CommonPPAGetter
4308+
4309+
4310+class PPASourcePackageGetter(CommonPPAGetter, SourcePackageGetter):
4311+ """Get a source package from a Launchpad PPA."""
4312+
4313+ def __init__(self, subdir="ppas", mirror_dir=None):
4314+ """Initialize the object.
4315+
4316+ See Getter's __init__ for an explanation of the arguments."""
4317+ super().__init__(
4318+ subdir=subdir, mirror_dir=mirror_dir, anonymous=True, fallback_to_git=False
4319+ )
4320+ self._logger.debug(
4321+ f"Initializing PPASourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}"
4322+ )
4323+
4324+ def validate_request(self, request):
4325+ """Validate a request to fetch a source package from a PPA.
4326+
4327+ :param dict[str, str] request: The dictionary containing the
4328+ request to be validated."""
4329+ super().validate_request(request)
4330+ if request["isprivateppa"] == "yes":
4331+ raise ValueError("Wrong request: from private PPA")
4332+
4333+
4334+class PrivatePPASourcePackageGetter(CommonPPAGetter, SourcePackageGetter):
4335+ """Get a source package from a private Launchpad PPA."""
4336+
4337+ def __init__(self, subdir="private-ppas", mirror_dir=None):
4338+ """Initialize the object.
4339+
4340+ See Getter's __init__ for an explanation of the arguments."""
4341+ super().__init__(
4342+ subdir=subdir, mirror_dir=mirror_dir, anonymous=False, fallback_to_git=False
4343+ )
4344+ self._logger.debug(
4345+ f"Initializing PrivatePPASourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}"
4346+ )
4347+
4348+ def validate_request(self, request):
4349+ """Validate a request to fetch a source package from a private
4350+ PPA.
4351+
4352+ :param dict[str, str] request: The dictionary containing the
4353+ request to be validated."""
4354+ super().validate_request(request)
4355+ if request["isprivateppa"] == "no":
4356+ raise ValueError("Wrong request: not from private PPA")
4357diff --git a/ubuntu_debuginfod/getters/sourcepackagegetter.py b/ubuntu_debuginfod/getters/sourcepackagegetter.py
4358new file mode 100644
4359index 0000000..2287620
4360--- /dev/null
4361+++ b/ubuntu_debuginfod/getters/sourcepackagegetter.py
4362@@ -0,0 +1,255 @@
4363+#!/usr/bin/python3
4364+
4365+# Copyright (C) 2022-2023 Canonical Ltd.
4366+
4367+# This program is free software: you can redistribute it and/or modify
4368+# it under the terms of the GNU General Public License as published by
4369+# the Free Software Foundation, either version 3 of the License, or
4370+# (at your option) any later version.
4371+
4372+# This program is distributed in the hope that it will be useful,
4373+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4374+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4375+# GNU General Public License for more details.
4376+
4377+# You should have received a copy of the GNU General Public License
4378+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4379+
4380+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4381+
4382+import os
4383+import lzma
4384+import tarfile
4385+import subprocess
4386+import tempfile
4387+from git import Git, Repo
4388+from git.exc import GitCommandError
4389+from ubuntu_debuginfod.getters.common_lp_getter import CommonLaunchpadGetter
4390+from ubuntu_debuginfod.utils.utils import (
4391+ git_dep14_tag,
4392+ download_from_lp,
4393+ move_atomically,
4394+)
4395+from ubuntu_debuginfod.utils.exceptions import UbuntuDebuginfodRetry
4396+
4397+
4398+def adjust_tar_filepath(tarinfo):
4399+ """Adjust the filepath for a TarInfo file.
4400+
4401+ This function is needed because TarFile.add strips the leading
4402+ slash from the filenames, so we have to workaround it by
4403+ re-adding the slash ourselves.
4404+
4405+ This function is intended to be used as a callback provided to
4406+ TarFile.add.
4407+
4408+ :param TarInfo tarinfo: The tarinfo."""
4409+ tarinfo.name = os.path.join("/", tarinfo.name)
4410+ return tarinfo
4411+
4412+
4413+class SourcePackageGetter(CommonLaunchpadGetter):
4414+ """Get (fetch) the source package artifact."""
4415+
4416+ def __init__(
4417+ self, subdir="ddebs", mirror_dir=None, anonymous=True, fallback_to_git=True
4418+ ):
4419+ """Initialize the object.
4420+
4421+ :param boolean fallback_to_git: Whether we should try to fetch
4422+ the source package using git if the regular approach (via
4423+ dget) fails. Default to True.
4424+
4425+ See Getter's __init__ for an explanation of the other arguments."""
4426+ super().__init__(subdir=subdir, mirror_dir=mirror_dir)
4427+ self._fallback_to_git = fallback_to_git
4428+ self._logger.debug(
4429+ f"Initializing SourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}, fallback_to_git={fallback_to_git}"
4430+ )
4431+
4432+ def _download_source_package_from_git(self, source_package, version, filepath):
4433+ """Download the source package using Launchpad's git repository.
4434+
4435+ :param str source_package: Source package name.
4436+
4437+ :param str version: Source package version.
4438+
4439+ :param str filepath: The full pathname where the resulting
4440+ source package tarball should be saved.
4441+
4442+ :rtype: bool
4443+ :return: This method returns True when the operation succeeds, or False
4444+ *iff* the "git clone" command fails to run. Otherwise, this
4445+ function will throw an exception."""
4446+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
4447+ g = Git()
4448+ git_dir = os.path.join(source_dir, f"{source_package}")
4449+ tag = "applied/" + git_dep14_tag(f"{version}")
4450+ self._logger.debug(
4451+ f"Cloning '{source_package}' git repo into '{git_dir}' (tag: '{tag}')"
4452+ )
4453+ try:
4454+ g.clone(
4455+ f"https://git.launchpad.net/ubuntu/+source/{source_package}",
4456+ git_dir,
4457+ depth=1,
4458+ branch=tag,
4459+ )
4460+ except GitCommandError as e:
4461+ # Couldn't perform the download. Let's signal and
4462+ # bail out.
4463+ self._logger.warning(
4464+ f"Could not clone git repo for '{source_package}': {e}"
4465+ )
4466+ return False
4467+ repo = Repo(git_dir)
4468+ prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
4469+ self._logger.debug(
4470+ f"Archiving git repo for '{source_package}-{version}' as '{filepath}'"
4471+ )
4472+ with tempfile.NamedTemporaryFile() as tmpfile:
4473+ with lzma.open(tmpfile.name, "w") as xzfile:
4474+ repo.archive(xzfile, prefix=prefix_path, format="tar")
4475+ move_atomically(tmpfile.name, filepath)
4476+
4477+ return True
4478+
4479+ def _download_source_package_from_dsc(
4480+ self, source_package, version, filepath, source_urls, credentials=None
4481+ ):
4482+ """Download the source package using the .dsc file.
4483+
4484+ :param str source_package: Source package name.
4485+
4486+ :param str version: Source package version.
4487+
4488+ :param str filepath: The full pathname where the resulting
4489+ source package tarball should be saved.
4490+
4491+ :param list source_urls: List of URLs used to fetch the source
4492+ package. This is usually the list returned by the
4493+ sourceFileUrls() Launchpad API call.
4494+
4495+ :param requests_oauthlib.OAuth1 credentials: The credentials
4496+ to be used when downloading the artifact from Launchpad.
4497+ Default is None, which means anonymous.
4498+
4499+ :rtype: bool
4500+ :return: This method returns True when the operation succeeds, or False
4501+ *iff* the "dpkg-source -x"command fails to run.
4502+ Otherwise, this function will throw an exception."""
4503+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
4504+ for url in source_urls:
4505+ download_from_lp(url, source_dir, credentials=credentials)
4506+
4507+ dscfile = None
4508+ for f in os.listdir(source_dir):
4509+ newf = os.path.join(source_dir, f)
4510+ if os.path.isfile(newf) and f.endswith(".dsc"):
4511+ dscfile = newf
4512+ break
4513+
4514+ if dscfile is None:
4515+ self._logger.warning(
4516+ "Could not find .dsc file, even though it should exist."
4517+ )
4518+ return False
4519+
4520+ outdir = os.path.join(source_dir, "outdir")
4521+ self._logger.debug(f"Will call 'dpkg-source -x {dscfile} {outdir}'")
4522+ try:
4523+ subprocess.run(
4524+ ["/usr/bin/dpkg-source", "-x", dscfile, outdir],
4525+ cwd=source_dir,
4526+ check=True,
4527+ )
4528+ except subprocess.CalledProcessError:
4529+ self._logger.warning("Call to 'dpkg-source -x' failed.")
4530+ return False
4531+
4532+ if not os.path.isdir(outdir):
4533+ self._logger.warning(
4534+ f"'{outdir}' has not been created by 'dpkg-source -x'."
4535+ )
4536+ return False
4537+
4538+ prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
4539+
4540+ with tempfile.NamedTemporaryFile() as tmpfile:
4541+ with tarfile.open(tmpfile.name, "w:xz") as tfile:
4542+ tfile.add(outdir, arcname=prefix_path, filter=adjust_tar_filepath)
4543+ move_atomically(tmpfile.name, filepath)
4544+
4545+ return True
4546+
4547+ def download_artifact(
4548+ self,
4549+ request,
4550+ credentials=None,
4551+ ):
4552+ """Download the source package.
4553+
4554+ :param str source_package: Source package name.
4555+
4556+ :param str version: Source package version.
4557+
4558+ :param str component: Source package component.
4559+
4560+ :param list source_urls: List of source file URLS.
4561+
4562+ :param boolean fallback_to_git: Whether we should try to fetch
4563+ the source package using git if the regular approach (via
4564+ dget) fails. Default to True.
4565+
4566+ :param requests_oauthlib.OAuth1 credentials: The credentials
4567+ to be used when downloading the artifact from Launchpad.
4568+ Default is None, which means anonymous.
4569+
4570+ :rtype: str
4571+ :return: The full path for the file that was saved, or None if
4572+ the file already exists."""
4573+ source_package = request["source_package"]
4574+ version = request["version"]
4575+ component = request["component"]
4576+ source_urls = request["source_urls"]
4577+
4578+ savepath = self.make_savepath(source_package, component)
4579+ txzfilepath = os.path.join(savepath, f"{source_package}-{version}.tar.xz")
4580+ if os.path.exists(txzfilepath):
4581+ self._logger.debug(f"'{txzfilepath}' already exists; doing nothing.")
4582+ return None
4583+
4584+ if self._download_source_package_from_dsc(
4585+ source_package, version, txzfilepath, source_urls, credentials=credentials
4586+ ):
4587+ self._logger.info(
4588+ f"Downloaded source package from dsc for '{source_package}-{version}' as '{txzfilepath}'"
4589+ )
4590+ return txzfilepath
4591+
4592+ if self._fallback_to_git and self._download_source_package_from_git(
4593+ source_package, version, txzfilepath
4594+ ):
4595+ self._logger.info(
4596+ f"Downloaded source package from git for '{source_package}-{version}' as '{txzfilepath}'"
4597+ )
4598+ return txzfilepath
4599+
4600+ # In the (likely?) event that there is a problem with
4601+ # Launchpad, let's raise an exception signalling that we'd
4602+ # like to retry the task.
4603+ raise UbuntuDebuginfodRetry()
4604+
4605+ def validate_request(self, request):
4606+ """Validate a request to fetch a source package from the main
4607+ archive.
4608+
4609+ :param dict[str, str] request: The dictionary containing the
4610+ request to be validated."""
4611+ super().validate_request(request)
4612+ if "source_urls" not in request.keys():
4613+ raise TypeError("No 'source_urls' in request")
4614+ if request.get("source_urls") is None:
4615+ raise ValueError("Invalid 'source_urls' (None) in request")
4616+ if request["architecture"] != "source":
4617+ raise ValueError("Wrong request: ddeb fetch")
4618diff --git a/debugpoller.py b/ubuntu_debuginfod/poller.py
4619similarity index 66%
4620rename from debugpoller.py
4621rename to ubuntu_debuginfod/poller.py
4622index 4e4a657..d6f32d0 100644
4623--- a/debugpoller.py
4624+++ b/ubuntu_debuginfod/poller.py
4625@@ -18,14 +18,13 @@
4626 # Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4627
4628 import os
4629-import sys
4630-from launchpadlib.launchpad import Launchpad
4631 import datetime
4632 import logging
4633+from ubuntu_debuginfod.utils.utils import generate_timestamp
4634
4635
4636-class DebugPoller:
4637- """The DebugPoller class implements the basics of a poller for the
4638+class Poller:
4639+ """The Poller class implements the basics of a poller for the
4640 debuginfod service."""
4641
4642 # The timestamp file we will save our timestamp into.
4643@@ -43,7 +42,7 @@ class DebugPoller:
4644 dry_run=False,
4645 anonymous=True,
4646 ):
4647- """Initalize the DebugPoller object.
4648+ """Initalize the Poller object.
4649
4650 :param int initial_interval: The initial interval (in hours),
4651 i.e., if no timestamp file has been found then we use this
4652@@ -65,60 +64,27 @@ class DebugPoller:
4653 the timestamp in the file when the operation finishes.
4654 Default is False.
4655
4656- :param bool anonymous: Whether we should login anonymously.
4657- Default is True.
4658-
4659- """
4660- if anonymous:
4661- self._lp = Launchpad.login_anonymously(
4662- "ubuntu-debuginfod poller", "production", version="devel"
4663- )
4664- else:
4665- self._lp = Launchpad.login_with(
4666- "ubuntu-debuginfod poller", "production", version="devel"
4667- )
4668-
4669- self._main_archive = self._lp.distributions["ubuntu"].main_archive
4670-
4671- self._logger = logging.getLogger(__name__)
4672- logging.basicConfig(
4673- level=logging.INFO,
4674- format=(
4675- "%(levelname) -10s %(asctime)s %(name) -20s %(funcName) "
4676- "-25s : %(message)s"
4677- ),
4678- )
4679-
4680+ :param bool anonymous: Whether we should poll anonymously.
4681+ Default is True."""
4682 self._initial_interval = initial_interval
4683 self._force_initial_interval = force_initial_interval
4684 self._fetch_all_on_first_run = fetch_all_on_first_run
4685 self._dry_run = dry_run
4686+ self._anonymous = anonymous
4687+ self._logger = logging.getLogger(__name__)
4688+ self._logger.debug(
4689+ f"Initializing Poller with initial_interval={initial_interval}, force_initial_interval={force_initial_interval}, fetch_all_on_first_run={fetch_all_on_first_run}, dry_run={dry_run}, anonymous={anonymous}"
4690+ )
4691
4692 @property
4693 def timestamp_filename(self):
4694 """Get the filename that contains the timestamp for this module.
4695
4696- :rtype str: The timestamp filename."""
4697- module_name = type(self).__name__.replace('Poller', '').lower()
4698+ :rtype: str
4699+ :return: The timestamp filename."""
4700+ module_name = type(self).__name__.replace("Poller", "").lower()
4701 return f"{self.TIMESTAMP_FILE}-{module_name}"
4702
4703- def _generate_timestamp(self, interval=None):
4704- """Generate a timestamp that can be used when querying Launchpad
4705- (via getPublished{Sources,Binaries}).
4706-
4707- :param interval: Specify how long ago (in hours) the timestamp must
4708- refer to. If not specified, the timestamp is generated for
4709- the current time.
4710- :type interval: int or None"""
4711- d = datetime.datetime.now(datetime.timezone.utc)
4712-
4713- if interval is not None:
4714- self._logger.debug(f"Generating timestamp with interval {interval}")
4715- d = d - datetime.timedelta(hours=interval)
4716-
4717- self._logger.debug(f"Generated timestamp '{d}'")
4718- return d
4719-
4720 def _get_timestamp(self):
4721 """Get the timestamp from the timestamp file, or generate a
4722 new one.
4723@@ -138,15 +104,17 @@ class DebugPoller:
4724 self._logger.debug(
4725 f"Or force_initial_interval = {self._force_initial_interval}"
4726 )
4727- return self._generate_timestamp(interval=self._initial_interval)
4728+ return generate_timestamp(interval=self._initial_interval)
4729
4730- with open(tfile, "r", encoding="UTF-8") as f:
4731- ts = int(f.readline().rstrip())
4732- d = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc)
4733+ with open(tfile, "r", encoding="UTF-8") as file:
4734+ ts_int = int(file.readline().rstrip())
4735+ last_timestamp = datetime.datetime.fromtimestamp(
4736+ ts_int, tz=datetime.timezone.utc
4737+ )
4738
4739- if not isinstance(d, datetime.datetime):
4740- raise RuntimeError(f"Invalid timestamp {d}")
4741- return d
4742+ if not isinstance(last_timestamp, datetime.datetime):
4743+ raise RuntimeError(f"Invalid timestamp {last_timestamp}")
4744+ return last_timestamp
4745
4746 def _get_normal_and_real_timestamps(self):
4747 """Get the previous timestamp and adjust it to account for
4748@@ -175,5 +143,8 @@ class DebugPoller:
4749 os.makedirs(dirname, mode=0o755, exist_ok=True)
4750 with open(tfile, "w", encoding="UTF-8") as f:
4751 epoch = datetime.datetime.fromtimestamp(0, tz=datetime.timezone.utc)
4752- ts = int((timestamp - epoch).total_seconds())
4753- f.write("%d" % ts)
4754+ ts_int = int((timestamp - epoch).total_seconds())
4755+ self._logger.debug(
4756+ f"Recording timestamp '{ts_int}' for {type(self).__name__}"
4757+ )
4758+ f.write(f"{ts_int}")
4759diff --git a/ubuntu_debuginfod/pollers/__init__.py b/ubuntu_debuginfod/pollers/__init__.py
4760new file mode 100644
4761index 0000000..14e8999
4762--- /dev/null
4763+++ b/ubuntu_debuginfod/pollers/__init__.py
4764@@ -0,0 +1 @@
4765+"""Init."""
4766diff --git a/ubuntu_debuginfod/pollers/common_lp_poller.py b/ubuntu_debuginfod/pollers/common_lp_poller.py
4767new file mode 100644
4768index 0000000..93a958e
4769--- /dev/null
4770+++ b/ubuntu_debuginfod/pollers/common_lp_poller.py
4771@@ -0,0 +1,86 @@
4772+#!/usr/bin/python3
4773+
4774+# Copyright (C) 2022 Canonical Ltd.
4775+
4776+# This program is free software: you can redistribute it and/or modify
4777+# it under the terms of the GNU General Public License as published by
4778+# the Free Software Foundation, either version 3 of the License, or
4779+# (at your option) any later version.
4780+
4781+# This program is distributed in the hope that it will be useful,
4782+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4783+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4784+# GNU General Public License for more details.
4785+
4786+# You should have received a copy of the GNU General Public License
4787+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4788+
4789+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4790+
4791+from ubuntu_debuginfod.utils.utils import lp_pub_iterator
4792+from ubuntu_debuginfod.poller import Poller
4793+from ubuntu_debuginfod.utils.baselp import BaseLaunchpadClass
4794+
4795+
4796+class CommonLaunchpadPoller(Poller, BaseLaunchpadClass):
4797+ """Common poller class when dealing with Launchpad archives."""
4798+
4799+ def __init__(self, *args, **kwargs):
4800+ """Initialize the object.
4801+
4802+ Look at BaseLaunchpadClass's docstring for an explanation
4803+ about the arguments."""
4804+ super().__init__(*args, **kwargs)
4805+ self.launchpad_login()
4806+ self._logger.debug("Initializing CommonLaunchpadPoller")
4807+
4808+ def poll_artifacts(self, archive, ppainfo, poller_fn):
4809+ """Poll artifacts from an archive.
4810+
4811+ :param lazr.restfulclient.resource.Entry archive: The Launchpad archive.
4812+
4813+ :param dict[str -> str] ppainfo: The dictionary containing
4814+ information about the PPA, if applicable. Otherwise,
4815+ None.
4816+
4817+ :param function poller_fn: The polling function to be called.
4818+
4819+ :rtype: dict[str -> str], datetime.datetime
4820+ :return: A dictionary containing "uri" as key, listing all
4821+ packages' self_link found. The most recent package
4822+ timestamp processed."""
4823+ timestamp, real_timestamp = self._get_normal_and_real_timestamps()
4824+
4825+ if ppainfo is None:
4826+ archive_label = f"'{archive.displayname}' archive"
4827+ else:
4828+ archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})"
4829+
4830+ self._logger.info(
4831+ f"Polling artifacts ({type(self).__name__}) created since '{real_timestamp}' from {archive_label}"
4832+ )
4833+
4834+ result = []
4835+ latest_timestamp_created = timestamp
4836+ # Create an iterator for the Launchpad publications.
4837+ pkgiter = lp_pub_iterator(
4838+ poller_fn(order_by_date=True, created_since_date=real_timestamp)
4839+ )
4840+
4841+ for pkg in pkgiter:
4842+ msg = {
4843+ "uri": pkg.self_link,
4844+ }
4845+
4846+ if (
4847+ latest_timestamp_created is None
4848+ or pkg.date_created > latest_timestamp_created
4849+ ):
4850+ latest_timestamp_created = pkg.date_created
4851+
4852+ if ppainfo is not None:
4853+ msg.update(ppainfo)
4854+
4855+ result.append(msg)
4856+
4857+ return result, latest_timestamp_created
4858diff --git a/ubuntu_debuginfod/pollers/common_ppa_poller.py b/ubuntu_debuginfod/pollers/common_ppa_poller.py
4859new file mode 100644
4860index 0000000..7f1c2ca
4861--- /dev/null
4862+++ b/ubuntu_debuginfod/pollers/common_ppa_poller.py
4863@@ -0,0 +1,121 @@
4864+#!/usr/bin/python3
4865+
4866+# Copyright (C) 2023 Canonical Ltd.
4867+
4868+# This program is free software: you can redistribute it and/or modify
4869+# it under the terms of the GNU General Public License as published by
4870+# the Free Software Foundation, either version 3 of the License, or
4871+# (at your option) any later version.
4872+
4873+# This program is distributed in the hope that it will be useful,
4874+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4875+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4876+# GNU General Public License for more details.
4877+
4878+# You should have received a copy of the GNU General Public License
4879+# along with this program. If not, see <https://www.gnu.org/licenses/>.
4880+
4881+# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com>
4882+
4883+import os
4884+import lazr.restfulclient.errors
4885+from ubuntu_debuginfod.utils.utils import generate_timestamp
4886+
4887+
4888+class CommonPPAPoller:
4889+ """A common poller for PPAs."""
4890+
4891+ PPALIST_PATH = os.path.expanduser("~/.config/ubuntu-debuginfod/ppalist")
4892+ PRIVATE_PPALIST_PATH = os.path.expanduser(
4893+ "~/.config/ubuntu-debuginfod/ppalist-private"
4894+ )
4895+
4896+ def __init__(self, *args, **kwargs):
4897+ """Initialize the object.
4898+
4899+ This class is meant to be use in a multiple-inheritance scenario."""
4900+ super().__init__(*args, **kwargs)
4901+ self._logger.debug("Initializing CommonPPAPoller")
4902+
4903+ def get_list_of_ppas(self):
4904+ """Get the list of PPAs to be polled.
4905+
4906+ :rtype: list
4907+ :return: The list of PPAs. This list will contain one
4908+ dictionary per PPA, with the following keys:
4909+
4910+ { "ppauser" : string,
4911+ "ppaname" : string,
4912+ "isprivateppa" : string }"""
4913+ ppafile = self.PPALIST_PATH if self._anonymous else self.PRIVATE_PPALIST_PATH
4914+
4915+ self._logger.debug(f"Processing '{ppafile}'")
4916+
4917+ if not os.path.isfile(ppafile):
4918+ self._logger.info(f"{ppafile} does not exist; doing nothing")
4919+ return []
4920+
4921+ ppas = []
4922+ with open(ppafile, "r", encoding="UTF-8") as f:
4923+ for ppa in f.readlines():
4924+ ppa = ppa.rstrip()
4925+ if ppa == "" or ppa.startswith("#"):
4926+ continue
4927+ ppa = ppa.split(":")[1].split("/")
4928+ ppa_user = ppa[0]
4929+ ppa_name = ppa[1]
4930+ self._logger.info(f"Processing ppa:{ppa_user}/{ppa_name}")
4931+ ppas.append(
4932+ {
4933+ "ppauser": ppa_user,
4934+ "ppaname": ppa_name,
4935+ "isprivateppa": "no" if self._anonymous else "yes",
4936+ }
4937+ )
4938+ return ppas
4939+
4940+ def poll_artifacts(self):
4941+ """Get the list of artifacts from Launchpad PPAs.
4942+
4943+ :param poller_fn function: The poller function that should be
4944+ used when polling for artifacts.
4945+
4946+ :param lp_people launchpadlib.launchpad.PersonSet: The pointer
4947+ to a Launchpad "people"set. This is usually
4948+ "self._lp.people".
4949+
4950+ :rtype: dict, datetime.datetime
4951+ :return: Return a dictionary containing all artifacts found,
4952+ and also the new timestamp that should then be recorded by
4953+ calling record_timestamp. Timestamp can be None, in which
4954+ case nothing should be recorded."""
4955+ ppalist = self.get_list_of_ppas()
4956+ if len(ppalist) == 0:
4957+ return [], None
4958+
4959+ messages = []
4960+ new_timestamp = generate_timestamp()
4961+ for ppainfo in ppalist:
4962+ ppauser = ppainfo["ppauser"]
4963+ ppaname = ppainfo["ppaname"]
4964+ try:
4965+ lpuser = self._lp.people[ppauser]
4966+ except KeyError:
4967+ self._logger.error(f"Launchpad user {ppauser} does not exist")
4968+ continue
4969+
4970+ try:
4971+ lpppa = lpuser.getPPAByName(name=ppaname)
4972+ except lazr.restfulclient.errors.NotFound:
4973+ self._logger.error(
4974+ f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist"
4975+ )
4976+ continue
4977+
4978+ self._logger.info(
4979+ f"Getting list of artifacts ({type(self).__name__}) from ppa:{ppauser}/{ppaname} (private: {not self._anonymous})"
4980+ )
4981+ msg, _ = super().poll_artifacts(archive=lpppa, ppainfo=ppainfo)
4982+ messages += msg
4983+
4984+ return messages, new_timestamp
4985diff --git a/ubuntu_debuginfod/pollers/ddebpoller.py b/ubuntu_debuginfod/pollers/ddebpoller.py
4986new file mode 100644
4987index 0000000..ea96030
4988--- /dev/null
4989+++ b/ubuntu_debuginfod/pollers/ddebpoller.py
4990@@ -0,0 +1,69 @@
4991+#!/usr/bin/python3
4992+
4993+# Copyright (C) 2022 Canonical Ltd.
4994+
4995+# This program is free software: you can redistribute it and/or modify
4996+# it under the terms of the GNU General Public License as published by
4997+# the Free Software Foundation, either version 3 of the License, or
4998+# (at your option) any later version.
4999+
5000+# This program is distributed in the hope that it will be useful,
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches

to all changes: