Merge ubuntu-debuginfod:overhaul into ubuntu-debuginfod:master
- Git
- lp:ubuntu-debuginfod
- overhaul
- Merge into master
Status: | Merged |
---|---|
Merged at revision: | 1603dca0a3eee2c59dd5a0aabeee3588ae6fdee0 |
Proposed branch: | ubuntu-debuginfod:overhaul |
Merge into: | ubuntu-debuginfod:master |
Diff against target: |
5729 lines (+3614/-332) 59 files modified
README.md (+151/-1) conf/ubuntu-debuginfod-celery.default (+1/-1) conf/ubuntu-debuginfod-celery.service (+0/-1) conf/ubuntu-debuginfod-launchpad-dispatcher.service (+24/-0) conf/ubuntu-debuginfod-launchpad-dispatcher.timer (+10/-0) conf/ubuntu-debuginfod-launchpad-poller.service (+2/-3) debian/changelog (+5/-0) debian/control (+50/-0) debian/copyright (+29/-0) debian/rules (+19/-0) debian/source/format (+1/-0) debian/source/options (+1/-0) debian/ubuntu-debuginfod.install (+3/-0) debian/ubuntu-debuginfod.postinst (+168/-0) debian/ubuntu-debuginfod.postrm (+39/-0) debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service (+1/-0) debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service (+1/-0) debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer (+1/-0) debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service (+1/-0) debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer (+1/-0) dev/null (+0/-191) pyproject.toml (+40/-0) services/launchpad-dispatcher.py (+451/-0) services/launchpad-poller.py (+156/-0) setup.cfg (+14/-0) tests/test_baselp.py (+58/-0) tests/test_common_lp_getter.py (+91/-0) tests/test_common_ppa_getter.py (+112/-0) tests/test_ddebgetter.py (+64/-0) tests/test_exceptions.py (+33/-0) tests/test_getter.py (+38/-0) tests/test_ppaddebgetter.py (+94/-0) tests/test_ppasourcepackagegetter.py (+94/-0) tests/test_sourcepackagegetter.py (+69/-0) tests/test_utils.py (+114/-0) ubuntu_debuginfod/__init__.py (+10/-0) ubuntu_debuginfod/debuginfod.py (+87/-76) ubuntu_debuginfod/getter.py (+57/-0) ubuntu_debuginfod/getters/__init__.py (+1/-0) ubuntu_debuginfod/getters/common_lp_getter.py (+104/-0) ubuntu_debuginfod/getters/common_ppa_getter.py (+65/-0) ubuntu_debuginfod/getters/ddebgetter.py (+70/-0) ubuntu_debuginfod/getters/ppaddebgetter.py (+65/-0) ubuntu_debuginfod/getters/ppasourcepackagegetter.py (+70/-0) ubuntu_debuginfod/getters/sourcepackagegetter.py (+255/-0) ubuntu_debuginfod/poller.py (+28/-57) ubuntu_debuginfod/pollers/__init__.py (+1/-0) ubuntu_debuginfod/pollers/common_lp_poller.py (+86/-0) ubuntu_debuginfod/pollers/common_ppa_poller.py (+121/-0) ubuntu_debuginfod/pollers/ddebpoller.py (+69/-0) ubuntu_debuginfod/pollers/ppaddebpoller.py (+73/-0) ubuntu_debuginfod/pollers/ppasourcepackagepoller.py (+73/-0) ubuntu_debuginfod/pollers/sourcepackagepoller.py (+69/-0) ubuntu_debuginfod/utils/__init__.py (+1/-0) ubuntu_debuginfod/utils/baselp.py (+45/-0) ubuntu_debuginfod/utils/debugdb.py (+11/-2) ubuntu_debuginfod/utils/exceptions.py (+27/-0) ubuntu_debuginfod/utils/utils.py (+235/-0) utils/obtain-launchpad-credentials.py (+55/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Athos Ribeiro (community) | Approve | ||
Canonical Server Reporter | Pending | ||
Review via email: mp+441906@code.launchpad.net |
Commit message
Description of the change
This MP is the big overhaul/rewrite of ubuntu-debuginfod that we talked about.
There are many new things being done here, but most of the diff is basically code reorganization to make better use of Python's capabilities.
I'll try to make a list of important things to consider when reviewing this code:
1) I'm make heavy use of base classes for pollers and getters. Using pollers and as example:
- You will see that there is the most basic class, Poller, which implements only the necessary/common bits for a generic poller to operate.
- On top of it, there is a CommonLaunchpad
- On top of CommonLaunchpad
- PPAs are kind of a special beast, so they have their own base class (CommonPPAPoller) which is actually crafted to be used specifically in multiple-
The design of getters follows exactly the same rationale.
2) You will find that I broke down the "poll_lp.py" service in two:
- The first one, "launchpad-
- There is a second service now, called "launchpad-
Currently, the dispatcher is a single service that runs sequentially over the table. Until now I haven't found any issues with this approach, but eventually I intend to transform "launchpad-
3) This is now a Python package. You should be able to use virtualenv and pip to install it locally. I also revamped the README file and included instructions on how to set things up locally.
4) I disabled a bunch of pylint warnings that were, IMO, false positives in my case. pylint is happy with the code, as well as flake8 and mypy. But I'm open to suggestions/
5) The package now has a testsuite. It is aimed at unittesting so far, but it did help me catch a few silly mistakes when I was developing things. I consider the testsuite to be a WIP; I still want to add more tests and, eventually, work towards having a proper CI/CD setup (thanks for the idea, Athos!).
I think that's it. I'm not in a big hurry to get this approved/deployed, so I'm fine to wait if you would like to talk in person about the MP during the upcoming sprint.
Athos Ribeiro (athos-ribeiro) wrote : | # |
Sergio Durigan Junior (sergiodj) wrote : | # |
Thanks, Athos.
I'm marking this MP as WIP for now because I decided that it actually makes sense to work on debianizing the package now. I should have something ready by tomorrow.
Sergio Durigan Junior (sergiodj) wrote : | # |
This is ready for review.
Sergio Durigan Junior (sergiodj) wrote : | # |
A few more notes here:
1) The package has been debianized. It needs a version of setuptools that isn't available in Jammy, so I set up a special PPA where I keep "extra dependencies" needed to build the package. It's here:
https:/
This means that you'll need to pass this PPA as an extra repository to sbuild locally.
2) I reorganized the way the services are located in the source tree. It should be clearer to determine what is part of the Python library and what is not.
Athos Ribeiro (athos-ribeiro) wrote : | # |
Hi Sergio, as we discussed before, I injected my reviews as comments in https:/
This should now be complete.
Nice work refactoring the whole code base to make it way more pythonic!
As we discussed offline, I have a few suggestions in my comments, but overall, the work looks great and I see no reason to block deploying it due to any of the review points I raised!
LGTM!
Preview Diff
1 | diff --git a/README.md b/README.md |
2 | index a0f5d14..c74b164 100644 |
3 | --- a/README.md |
4 | +++ b/README.md |
5 | @@ -12,8 +12,11 @@ python3-git |
6 | python3-celery |
7 | python3-launchpadlib |
8 | python3-requests |
9 | +python3-requests-oauthlib |
10 | python3-sdnotify |
11 | python3-psycopg2 |
12 | +python3-pytest (test-only) |
13 | +python3-freezegun (test-only) |
14 | ``` |
15 | |
16 | * Applications |
17 | @@ -27,10 +30,157 @@ postgresql* |
18 | support Jammy. |
19 | ``` |
20 | |
21 | -## Setup |
22 | +## Installation |
23 | + |
24 | +You can install the package from the following PPA: |
25 | + |
26 | + https://launchpad.net/~ubuntu-debuginfod-devs/+archive/ubuntu/ubuntu-debuginfod |
27 | + |
28 | +## Manual setup |
29 | + |
30 | +Nowadays, it is strongly recommended to install the package directly |
31 | +from the PPA: |
32 | + |
33 | + https://launchpad.net/~ubuntu-debuginfod-devs/+archive/ubuntu/ubuntu-debuginfod |
34 | + |
35 | +The `.deb` package will take care of setting (almost) everything up, |
36 | +and will provide instructions on how to configure anything that might |
37 | +be missing. However, if you would like to set the package up |
38 | +manually, follow the instructions below. |
39 | + |
40 | +### Create user & clone repository |
41 | + |
42 | +First, create a user. Currently the scripts are hardcoded to assume |
43 | +that the username is `mirror`, so let's create it: |
44 | + |
45 | +``` |
46 | +# adduser --disabled-password mirror |
47 | +``` |
48 | + |
49 | +For the sake of keeping things simple, we will assume that the |
50 | +`ubuntu-debuginfod` repository is cloned at |
51 | +`/home/mirror/ubuntu-debuginfod/`. |
52 | + |
53 | +`ubuntu-debuginfod` is a proper Python package, so you can install it |
54 | +using `virtualenv` and `pip`. Make sure to install all the required |
55 | +dependencies before proceeding. |
56 | + |
57 | +Create the `ubuntu-debuginfod` directory under `~mirror/.config`: |
58 | + |
59 | +``` |
60 | +# su - mirror |
61 | +$ mkdir -p $HOME/.config/ubuntu-debuginfod |
62 | +$ chmod -R 0700 $HOME/.config/ |
63 | +``` |
64 | + |
65 | +### Create mirror directory |
66 | + |
67 | +The scripts assume by default that the mirror directory (i.e., the |
68 | +directory where the debuginfo files will be saved) will be |
69 | +`/srv/debug-mirror/`. Let's create it: |
70 | + |
71 | +``` |
72 | +# mkdir -p /srv/debug-mirror/ |
73 | +# chown -R mirror:mirror /srv/debug-mirror/ |
74 | +``` |
75 | + |
76 | +### Create a database |
77 | + |
78 | +Now, create the PostgreSQL database for the service: |
79 | |
80 | ``` |
81 | # su - postgres |
82 | $ createuser -d mirror |
83 | $ createdb -O mirror ubuntu-debuginfod |
84 | ``` |
85 | + |
86 | +### Configure RabbitMQ |
87 | + |
88 | +We will also need to tweak RabbitMQ's `consumer_timeout` to a higher |
89 | +value. |
90 | + |
91 | +``` |
92 | +# cat >> /etc/rabbitmq/rabbitmq.conf << __EOF__ |
93 | +# 3 hours |
94 | +consumer_timeout = 10800000 |
95 | +__EOF__ |
96 | +``` |
97 | + |
98 | +Let's restart the service so that the new setting takes effect: |
99 | + |
100 | +``` |
101 | +# systemctl restart rabbitmq.service |
102 | +``` |
103 | + |
104 | +### Install the configuration files |
105 | + |
106 | +For now, you will need to manually install the configuration files. |
107 | +Let's do it: |
108 | + |
109 | +``` |
110 | +# cd /etc/systemd/system/ |
111 | +# ln -s /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-celery.service |
112 | +# for file in dispatcher poller; do \ |
113 | + ln -s /home/mirror/ubuntu-debuginfod/conf/launchpad-${file}.service; \ |
114 | + ln -s /home/mirror/ubuntu-debuginfod/conf/launchpad-${file}.timer; \ |
115 | + done |
116 | +# cd /etc/default/ |
117 | +# ln -s \ |
118 | + /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-celery.default \ |
119 | + ubuntu-debuginfod-celery |
120 | +# ln -s \ |
121 | + /home/mirror/ubuntu-debuginfod/conf/ubuntu-debuginfod-launchpad-poller.default \ |
122 | + ubuntu-debuginfod-launchpad-poller |
123 | +``` |
124 | + |
125 | +Feel free to make any adjustments to the `.default` files. |
126 | + |
127 | +### Start the service |
128 | + |
129 | +Now, let's reload the systemd daemon and start the Celery service: |
130 | + |
131 | +``` |
132 | +# systemctl daemon-reload |
133 | +# systemctl start ubuntu-debuginfod-celery.service |
134 | +``` |
135 | + |
136 | +Take a look at the logs to see if everything is OK. |
137 | + |
138 | +### Run the Launchpad poller for the first time |
139 | + |
140 | +To run the Launchpad poller, simply do: |
141 | + |
142 | +``` |
143 | +systemctl start ubuntu-debuginfod-launchpad-poller.service |
144 | +``` |
145 | + |
146 | +You might want to open another terminal and follow the service logs |
147 | +there. |
148 | + |
149 | +If you want to enable the systemd timer and have the poller be |
150 | +executed every 30 minutes, do: |
151 | + |
152 | +``` |
153 | +systemctl enable ubuntu-debuginfod-launchpad-poller.timer |
154 | +``` |
155 | + |
156 | +### Run the dispatcher |
157 | + |
158 | +Now that the initial polling has been done, you can run the |
159 | +dispatcher. |
160 | + |
161 | +``` |
162 | +systemctl start ubuntu-debuginfod-launchpad-dispatcher.service |
163 | +``` |
164 | + |
165 | +Again, you may want to check the logs from another terminal. |
166 | + |
167 | +If the poller was able to find any valid jobs to be dispatched, you |
168 | +should start seeing the `/srv/debug-mirror/` directory be populated. |
169 | + |
170 | +If you want to enable the systemd timer and have the dispatcher be |
171 | +executed every 30 minutes, do: |
172 | + |
173 | +``` |
174 | +systemctl enable ubuntu-debuginfod-launchpad-dispatcher.timer |
175 | +``` |
176 | diff --git a/conf/ubuntu-debuginfod-celery.default b/conf/ubuntu-debuginfod-celery.default |
177 | index 9f915b0..6741b04 100644 |
178 | --- a/conf/ubuntu-debuginfod-celery.default |
179 | +++ b/conf/ubuntu-debuginfod-celery.default |
180 | @@ -1,5 +1,5 @@ |
181 | # App instance to use. |
182 | -CELERY_APP="debuginfod" |
183 | +CELERY_APP="ubuntu_debuginfod.debuginfod" |
184 | |
185 | # Extra command-line arguments to the worker. |
186 | # |
187 | diff --git a/conf/ubuntu-debuginfod-celery.service b/conf/ubuntu-debuginfod-celery.service |
188 | index 37b1b9c..edbb388 100644 |
189 | --- a/conf/ubuntu-debuginfod-celery.service |
190 | +++ b/conf/ubuntu-debuginfod-celery.service |
191 | @@ -10,7 +10,6 @@ User=mirror |
192 | Group=mirror |
193 | Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred |
194 | EnvironmentFile=/etc/default/ubuntu-debuginfod-celery |
195 | -WorkingDirectory=/home/mirror/ubuntu-debuginfod/ |
196 | ExecStart=/usr/bin/celery -A ${CELERY_APP} worker --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS} |
197 | Restart=always |
198 | TimeoutStopSec=600 |
199 | diff --git a/conf/ubuntu-debuginfod-launchpad-dispatcher.service b/conf/ubuntu-debuginfod-launchpad-dispatcher.service |
200 | new file mode 100644 |
201 | index 0000000..169accf |
202 | --- /dev/null |
203 | +++ b/conf/ubuntu-debuginfod-launchpad-dispatcher.service |
204 | @@ -0,0 +1,24 @@ |
205 | +[Unit] |
206 | +Description=Dispatcher for ubuntu-debuginfod |
207 | +After=network.target ubuntu-debuginfod-celery.service |
208 | +Requires=ubuntu-debuginfod-celery.service |
209 | +ConditionPathExists=/home/mirror/.config/ubuntu-debuginfod/lp.cred |
210 | + |
211 | +[Service] |
212 | +Type=oneshot |
213 | +User=mirror |
214 | +Group=mirror |
215 | +Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred |
216 | +ExecStart=/usr/bin/python3 /usr/share/ubuntu-debuginfod/services/launchpad-dispatcher.py |
217 | +PrivateDevices=true |
218 | +PrivateUsers=true |
219 | +ProtectKernelTunables=true |
220 | +ProtectControlGroups=true |
221 | +ProtectKernelLogs=true |
222 | +ProtectKernelModules=true |
223 | +MemoryDenyWriteExecute=true |
224 | +RestrictRealtime=true |
225 | +ReadOnlyPaths=/home/mirror/.config/ubuntu-debuginfod/ |
226 | + |
227 | +[Install] |
228 | +WantedBy=multi-user.target |
229 | diff --git a/conf/ubuntu-debuginfod-launchpad-dispatcher.timer b/conf/ubuntu-debuginfod-launchpad-dispatcher.timer |
230 | new file mode 100644 |
231 | index 0000000..e5cabd4 |
232 | --- /dev/null |
233 | +++ b/conf/ubuntu-debuginfod-launchpad-dispatcher.timer |
234 | @@ -0,0 +1,10 @@ |
235 | +[Unit] |
236 | +Description=Dispatch jobs for ubuntu-debuginfod |
237 | + |
238 | +[Timer] |
239 | +# Every 30 minutes |
240 | +OnCalendar=*:0/30 |
241 | +Persistent=true |
242 | + |
243 | +[Install] |
244 | +WantedBy=timers.target |
245 | diff --git a/conf/ubuntu-debuginfod-poll-lp.default b/conf/ubuntu-debuginfod-launchpad-poller.default |
246 | similarity index 100% |
247 | rename from conf/ubuntu-debuginfod-poll-lp.default |
248 | rename to conf/ubuntu-debuginfod-launchpad-poller.default |
249 | diff --git a/conf/ubuntu-debuginfod-poll-lp.service b/conf/ubuntu-debuginfod-launchpad-poller.service |
250 | similarity index 87% |
251 | rename from conf/ubuntu-debuginfod-poll-lp.service |
252 | rename to conf/ubuntu-debuginfod-launchpad-poller.service |
253 | index a97ffb6..dd98d14 100644 |
254 | --- a/conf/ubuntu-debuginfod-poll-lp.service |
255 | +++ b/conf/ubuntu-debuginfod-launchpad-poller.service |
256 | @@ -9,9 +9,8 @@ Type=oneshot |
257 | User=mirror |
258 | Group=mirror |
259 | Environment=LP_CREDENTIALS_FILE=/home/mirror/.config/ubuntu-debuginfod/lp.cred |
260 | -EnvironmentFile=/etc/default/ubuntu-debuginfod-poll-lp |
261 | -WorkingDirectory=/home/mirror/ubuntu-debuginfod/ |
262 | -ExecStart=/usr/bin/python3 /home/mirror/ubuntu-debuginfod/poll_launchpad.py |
263 | +EnvironmentFile=/etc/default/ubuntu-debuginfod-launchpad-poller |
264 | +ExecStart=/usr/bin/python3 /usr/share/ubuntu-debuginfod/services/launchpad-poller.py |
265 | PrivateDevices=true |
266 | PrivateUsers=true |
267 | ProtectKernelTunables=true |
268 | diff --git a/conf/ubuntu-debuginfod-poll-lp.timer b/conf/ubuntu-debuginfod-launchpad-poller.timer |
269 | similarity index 100% |
270 | rename from conf/ubuntu-debuginfod-poll-lp.timer |
271 | rename to conf/ubuntu-debuginfod-launchpad-poller.timer |
272 | diff --git a/ddebgetter.py b/ddebgetter.py |
273 | deleted file mode 100644 |
274 | index c441bc6..0000000 |
275 | --- a/ddebgetter.py |
276 | +++ /dev/null |
277 | @@ -1,356 +0,0 @@ |
278 | -#!/usr/bin/python3 |
279 | - |
280 | -# Copyright (C) 2022 Canonical Ltd. |
281 | - |
282 | -# This program is free software: you can redistribute it and/or modify |
283 | -# it under the terms of the GNU General Public License as published by |
284 | -# the Free Software Foundation, either version 3 of the License, or |
285 | -# (at your option) any later version. |
286 | - |
287 | -# This program is distributed in the hope that it will be useful, |
288 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
289 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
290 | -# GNU General Public License for more details. |
291 | - |
292 | -# You should have received a copy of the GNU General Public License |
293 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
294 | - |
295 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
296 | - |
297 | -import os |
298 | -import lzma |
299 | -import tarfile |
300 | -import subprocess |
301 | - |
302 | -import tempfile |
303 | - |
304 | -from git import Git, Repo |
305 | -from git.exc import GitCommandError |
306 | - |
307 | -from debuggetter import DebugGetter, DebugGetterRetry, DEFAULT_MIRROR_DIR |
308 | - |
309 | - |
310 | -def _validate_generic_request(request): |
311 | - """Validate a request coming from Celery. |
312 | - |
313 | - :param dict[str, str] request: The dictionary containing the |
314 | - request to be validated.""" |
315 | - if request is None: |
316 | - raise TypeError("Invalid request (None)") |
317 | - if request.get("architecture") is None: |
318 | - raise TypeError("No 'architecture' in request") |
319 | - |
320 | -def validate_ddeb_request(request): |
321 | - """Validate a request to fetch a ddeb from the main archive. |
322 | - |
323 | - :param dict[str, str] request: The dictionary containing the |
324 | - request to be validated.""" |
325 | - _validate_generic_request(request) |
326 | - if request.get("ddeb_url") is None: |
327 | - raise TypeError("No 'ddeb_url' in request") |
328 | - if request["architecture"] == "source": |
329 | - raise ValueError("Wrong request: source fetch") |
330 | - |
331 | -def validate_source_code_request(request): |
332 | - """Validate a request to fetch a source package from the main |
333 | - archive. |
334 | - |
335 | - :param dict[str, str] request: The dictionary containing the |
336 | - request to be validated.""" |
337 | - _validate_generic_request(request) |
338 | - if request.get("source_urls") is None: |
339 | - raise TypeError("No 'source_urls' in request") |
340 | - if request["architecture"] != "source": |
341 | - raise ValueError("Wrong request: ddeb fetch") |
342 | - |
343 | - |
344 | -class DdebGetter(DebugGetter): |
345 | - """Get (fetch) a ddeb.""" |
346 | - |
347 | - def __init__(self, subdir="ddebs", mirror_dir=DEFAULT_MIRROR_DIR): |
348 | - """Initialize the object. |
349 | - |
350 | - See DebugGetter's __init__ for an explanation of the arguments.""" |
351 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
352 | - |
353 | - def process_request(self, request): |
354 | - """Process a request, usually coming from Celery. |
355 | - |
356 | - :param dict[str, str] request: The dictionary containing the |
357 | - information necessary to fetch this ddeb.""" |
358 | - validate_ddeb_request(request) |
359 | - self.do_process_request(request) |
360 | - |
361 | - def do_process_request(self, request, credentials=None): |
362 | - """Perform the actual processing of the request. |
363 | - |
364 | - :param dict[str, str] request: The dictionary containing the |
365 | - information necessary to fetch this ddeb. |
366 | - |
367 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
368 | - to be used when downloading the artifact from Launchpad. |
369 | - Default is None, which means anonymous.""" |
370 | - self._logger.debug(f"Processing request to download ddeb: {request}") |
371 | - self._download_ddeb( |
372 | - request["source_package"], |
373 | - request["component"], |
374 | - request["ddeb_url"], |
375 | - credentials=credentials, |
376 | - ) |
377 | - |
378 | - def _download_ddeb(self, source_package, component, ddeb_url, credentials=None): |
379 | - """Download a ddeb associated with a package. |
380 | - |
381 | - :param str source_package: Source package name. |
382 | - |
383 | - :param str component: Source package component. |
384 | - |
385 | - :param str ddeb_url: The ddeb URL. |
386 | - |
387 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
388 | - to be used when downloading the artifact from Launchpad. |
389 | - Default is None, which means anonymous.""" |
390 | - savepath = self._make_savepath(source_package, component) |
391 | - self._logger.debug(f"Downloading '{ddeb_url}' into '{savepath}'") |
392 | - if credentials is not None: |
393 | - self._logger.debug("Downloading with credentials") |
394 | - self._download_from_lp(ddeb_url, savepath, credentials=credentials) |
395 | - |
396 | - |
397 | -# The function below was taken from git-ubuntu. We need to make |
398 | -# sure we perform the same version -> tag transformation as it |
399 | -# does. |
400 | -def git_dep14_tag(version): |
401 | - """Munge a version string according to http://dep.debian.net/deps/dep14/ |
402 | - |
403 | - :param str version: The version to be adjusted.""" |
404 | - version = version.replace("~", "_").replace(":", "%").replace("..", ".#.") |
405 | - if version.endswith("."): |
406 | - version = version + "#" |
407 | - if version.endswith(".lock"): |
408 | - pre, _, _ = version.partition(".lock") |
409 | - version = pre + ".#lock" |
410 | - return version |
411 | - |
412 | - |
413 | -def adjust_tar_filepath(tarinfo): |
414 | - """Adjust the filepath for a TarInfo file. |
415 | - |
416 | - This function is needed because TarFile.add strips the leading |
417 | - slash from the filenames, so we have to workaround it by |
418 | - re-adding the slash ourselves. |
419 | - |
420 | - This function is intended to be used as a callback provided to |
421 | - TarFile.add. |
422 | - |
423 | - :param TarInfo tarinfo: The tarinfo.""" |
424 | - tarinfo.name = os.path.join("/", tarinfo.name) |
425 | - return tarinfo |
426 | - |
427 | - |
428 | -class DdebSourceCodeGetter(DebugGetter): |
429 | - """Get (fetch) the source code associated with a ddeb.""" |
430 | - |
431 | - def __init__(self, subdir="ddebs", mirror_dir=DEFAULT_MIRROR_DIR): |
432 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
433 | - |
434 | - def process_request(self, request): |
435 | - """Process a request, usually coming from Celery. |
436 | - |
437 | - :param dict[str, str] request: The dictionary containing the |
438 | - information necessary to fetch this source code.""" |
439 | - validate_source_code_request(request) |
440 | - self.do_process_request(request) |
441 | - |
442 | - def do_process_request(self, request, fallback_to_git=True, credentials=None): |
443 | - """Perform the actual processing of the request. |
444 | - |
445 | - :param dict[str, str] request: The dictionary containing the |
446 | - information necessary to fetch this source code. |
447 | - |
448 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
449 | - to be used when downloading the artifact from Launchpad. |
450 | - Default is None, which means anonymous.""" |
451 | - self._logger.debug(f"Processing request to download source code: {request}") |
452 | - self._download_source_code( |
453 | - request["source_package"], |
454 | - request["version"], |
455 | - request["component"], |
456 | - request["source_urls"], |
457 | - fallback_to_git=fallback_to_git, |
458 | - credentials=credentials, |
459 | - ) |
460 | - |
461 | - def _download_source_code_from_git(self, source_package, version, filepath): |
462 | - """Download the source code using Launchpad's git repository. |
463 | - |
464 | - :param str source_package: Source package name. |
465 | - |
466 | - :param str version: Source package version. |
467 | - |
468 | - :param str filepath: The full pathname where the resulting |
469 | - source code tarball should be saved. |
470 | - |
471 | - :rtype: bool |
472 | - |
473 | - This method returns True when the operation succeeds, or False |
474 | - *iff* the "git clone" command fails to run. Otherwise, this |
475 | - function will throw an exception.""" |
476 | - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir: |
477 | - g = Git() |
478 | - git_dir = os.path.join(source_dir, f"{source_package}") |
479 | - tag = "applied/" + git_dep14_tag(f"{version}") |
480 | - self._logger.debug( |
481 | - f"Cloning '{source_package}' git repo into '{git_dir}' (tag: '{tag}')" |
482 | - ) |
483 | - try: |
484 | - g.clone( |
485 | - f"https://git.launchpad.net/ubuntu/+source/{source_package}", |
486 | - git_dir, |
487 | - depth=1, |
488 | - branch=tag, |
489 | - ) |
490 | - except GitCommandError as e: |
491 | - # Couldn't perform the download. Let's signal and |
492 | - # bail out. |
493 | - self._logger.warning( |
494 | - f"Could not clone git repo for '{source_package}': {e}" |
495 | - ) |
496 | - return False |
497 | - repo = Repo(git_dir) |
498 | - prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/") |
499 | - self._logger.debug( |
500 | - f"Archiving git repo for '{source_package}-{version}' as '{filepath}'" |
501 | - ) |
502 | - with tempfile.NamedTemporaryFile() as tmpfile: |
503 | - with lzma.open(tmpfile.name, "w") as xzfile: |
504 | - repo.archive(xzfile, prefix=prefix_path, format="tar") |
505 | - self._try_to_move_atomically(tmpfile.name, filepath) |
506 | - |
507 | - return True |
508 | - |
509 | - def _download_source_code_from_dsc( |
510 | - self, source_package, version, filepath, source_urls, credentials=None |
511 | - ): |
512 | - """Download the source code using the .dsc file. |
513 | - |
514 | - :param str source_package: Source package name. |
515 | - |
516 | - :param str version: Source package version. |
517 | - |
518 | - :param str filepath: The full pathname where the resulting |
519 | - source code tarball should be saved. |
520 | - |
521 | - :param list source_urls: List of URLs used to fetch the source |
522 | - package. This is usually the list returned by the |
523 | - sourceFileUrls() Launchpad API call. |
524 | - |
525 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
526 | - to be used when downloading the artifact from Launchpad. |
527 | - Default is None, which means anonymous. |
528 | - |
529 | - :rtype: bool |
530 | - This method returns True when the operation succeeds, or False |
531 | - *iff* the "dpkg-source -x" command fails to run. Otherwise, |
532 | - this function will throw an exception.""" |
533 | - with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir: |
534 | - for url in source_urls: |
535 | - self._download_from_lp(url, source_dir, credentials=credentials) |
536 | - |
537 | - dscfile = None |
538 | - for f in os.listdir(source_dir): |
539 | - newf = os.path.join(source_dir, f) |
540 | - if os.path.isfile(newf) and f.endswith(".dsc"): |
541 | - dscfile = newf |
542 | - break |
543 | - |
544 | - if dscfile is None: |
545 | - self._logger.warning( |
546 | - "Could not find .dsc file, even though it should exist." |
547 | - ) |
548 | - return False |
549 | - |
550 | - outdir = os.path.join(source_dir, "outdir") |
551 | - self._logger.debug(f"Will call 'dpkg-source -x {dscfile} {outdir}'") |
552 | - try: |
553 | - subprocess.run( |
554 | - ["/usr/bin/dpkg-source", "-x", dscfile, outdir], |
555 | - cwd=source_dir, |
556 | - check=True, |
557 | - ) |
558 | - except subprocess.CalledProcessError: |
559 | - self._logger.warning(f"Call to 'dpkg-source -x' failed.") |
560 | - return False |
561 | - |
562 | - if not os.path.isdir(outdir): |
563 | - self._logger.warning( |
564 | - f"'{outdir}' has not been created by 'dpkg-source -x'." |
565 | - ) |
566 | - return False |
567 | - |
568 | - prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/") |
569 | - |
570 | - with tempfile.NamedTemporaryFile() as tmpfile: |
571 | - with tarfile.open(tmpfile.name, "w:xz") as tfile: |
572 | - tfile.add(outdir, arcname=prefix_path, filter=adjust_tar_filepath) |
573 | - self._try_to_move_atomically(tmpfile.name, filepath) |
574 | - |
575 | - return True |
576 | - |
577 | - def _download_source_code( |
578 | - self, |
579 | - source_package, |
580 | - version, |
581 | - component, |
582 | - source_urls, |
583 | - fallback_to_git=True, |
584 | - credentials=None, |
585 | - ): |
586 | - |
587 | - """Download the source code for a package. |
588 | - |
589 | - :param str source_package: Source package name. |
590 | - |
591 | - :param str version: Source package version. |
592 | - |
593 | - :param str component: Source package component. |
594 | - |
595 | - :param list source_urls: List of source file URLS. |
596 | - |
597 | - :param boolean fallback_to_git: Whether we should try to fetch |
598 | - the source code using git if the regular approach (via |
599 | - dget) fails. Default to True. |
600 | - |
601 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
602 | - to be used when downloading the artifact from Launchpad. |
603 | - Default is None, which means anonymous.""" |
604 | - savepath = self._make_savepath(source_package, component) |
605 | - txzfilepath = os.path.join(savepath, f"{source_package}-{version}.tar.xz") |
606 | - if os.path.exists(txzfilepath): |
607 | - self._logger.debug(f"'{txzfilepath}' already exists; doing nothing.") |
608 | - return |
609 | - |
610 | - if self._download_source_code_from_dsc( |
611 | - source_package, |
612 | - version, |
613 | - txzfilepath, |
614 | - source_urls, |
615 | - credentials=credentials |
616 | - ): |
617 | - self._logger.info( |
618 | - f"Downloaded source code from dsc for '{source_package}-{version}' as '{txzfilepath}'" |
619 | - ) |
620 | - return |
621 | - |
622 | - if fallback_to_git and self._download_source_code_from_git( |
623 | - source_package, version, txzfilepath |
624 | - ): |
625 | - self._logger.info( |
626 | - f"Downloaded source code from git for '{source_package}-{version}' as '{txzfilepath}'" |
627 | - ) |
628 | - return |
629 | - |
630 | - # In the (likely?) event that there is a problem with |
631 | - # Launchpad, let's raise an exception signalling that we'd |
632 | - # like to retry the task. |
633 | - raise DebugGetterRetry() |
634 | diff --git a/ddebpoller.py b/ddebpoller.py |
635 | deleted file mode 100644 |
636 | index 9883782..0000000 |
637 | --- a/ddebpoller.py |
638 | +++ /dev/null |
639 | @@ -1,200 +0,0 @@ |
640 | -#!/usr/bin/python3 |
641 | - |
642 | -# Copyright (C) 2022 Canonical Ltd. |
643 | - |
644 | -# This program is free software: you can redistribute it and/or modify |
645 | -# it under the terms of the GNU General Public License as published by |
646 | -# the Free Software Foundation, either version 3 of the License, or |
647 | -# (at your option) any later version. |
648 | - |
649 | -# This program is distributed in the hope that it will be useful, |
650 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
651 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
652 | -# GNU General Public License for more details. |
653 | - |
654 | -# You should have received a copy of the GNU General Public License |
655 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
656 | - |
657 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
658 | - |
659 | -import os |
660 | -from urllib import parse |
661 | -import datetime |
662 | -from debugpoller import DebugPoller |
663 | - |
664 | - |
665 | -class DdebPoller(DebugPoller): |
666 | - """Perform the Launchpad polling and obtain a list of ddebs and source |
667 | - packages.""" |
668 | - |
669 | - def __init__( |
670 | - self, |
671 | - initial_interval=1, |
672 | - force_initial_interval=False, |
673 | - fetch_all_on_first_run=False, |
674 | - dry_run=False, |
675 | - anonymous=True, |
676 | - ): |
677 | - """Initialize the object using 'ddeb' as its name. |
678 | - |
679 | - Look at DebugPoller's docstring for an explanation about the arguments.""" |
680 | - super().__init__( |
681 | - initial_interval=initial_interval, |
682 | - force_initial_interval=force_initial_interval, |
683 | - fetch_all_on_first_run=fetch_all_on_first_run, |
684 | - dry_run=dry_run, |
685 | - anonymous=anonymous, |
686 | - ) |
687 | - |
688 | - def get_ddebs(self, archive=None, ppainfo=None): |
689 | - """Get the list of ddebs that have been published since the last |
690 | - timestamp from Launchpad. |
691 | - |
692 | - :param archive Launchpad archive: The Launchpad archive that |
693 | - should be queried for new binaries. Defaults to |
694 | - self._main_archive. |
695 | - |
696 | - :param ppainfo dict(str, str): A dictionary containing three |
697 | - keys: "ppauser", "ppaname" and "isprivateppa". It should |
698 | - only be provided when dealing with a PPA; otherwise, the |
699 | - default is None. |
700 | - |
701 | - :rtype: dict, datetime.datetime |
702 | - |
703 | - Return a dictionary containing all ddebs found, and also the |
704 | - new timestamp that should then be recorded by calling |
705 | - record_timestamp. |
706 | - |
707 | - """ |
708 | - timestamp, real_timestamp = self._get_normal_and_real_timestamps() |
709 | - |
710 | - if archive is None: |
711 | - archive = self._main_archive |
712 | - |
713 | - if ppainfo is None: |
714 | - archive_label = f"'{archive.displayname}' archive" |
715 | - else: |
716 | - archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})" |
717 | - |
718 | - self._logger.info(f"Polling ddebs created since '{real_timestamp}' from {archive_label}") |
719 | - |
720 | - result = [] |
721 | - latest_timestamp_created = timestamp |
722 | - for pkg in archive.getPublishedBinaries( |
723 | - order_by_date=True, created_since_date=real_timestamp |
724 | - ): |
725 | - if pkg.status not in ("Pending", "Published"): |
726 | - continue |
727 | - if not pkg.is_debug: |
728 | - continue |
729 | - |
730 | - ddeb_urls = pkg.binaryFileUrls() |
731 | - if len(ddeb_urls) == 0: |
732 | - # Safety check. |
733 | - continue |
734 | - |
735 | - if ( |
736 | - latest_timestamp_created is None |
737 | - or pkg.date_created > latest_timestamp_created |
738 | - ): |
739 | - latest_timestamp_created = pkg.date_created |
740 | - |
741 | - srcname = pkg.source_package_name |
742 | - binname = pkg.binary_package_name |
743 | - binver = pkg.binary_package_version |
744 | - component = pkg.component_name |
745 | - |
746 | - # We create one message (which will eventually become a |
747 | - # Celery task) per ddeb. This makes it easier later to do |
748 | - # deduplication, and also has the benefit of making the |
749 | - # downloading process more granular for multiple ddebs. |
750 | - for url in ddeb_urls: |
751 | - # Obtain the ddeb filename and its architecture. |
752 | - ddeb = os.path.basename(parse.urlparse(url).path) |
753 | - _, _, arch_and_extension = ddeb.split("_") |
754 | - arch, _ = arch_and_extension.split(".") |
755 | - |
756 | - msg = { |
757 | - "source_package": srcname, |
758 | - "binary_package": binname, |
759 | - "version": binver, |
760 | - "component": component, |
761 | - "ddeb_url": url, |
762 | - "ddeb_filename": ddeb, |
763 | - "architecture": arch, |
764 | - } |
765 | - if ppainfo is not None: |
766 | - msg.update(ppainfo) |
767 | - |
768 | - result.append(msg) |
769 | - |
770 | - return result, latest_timestamp_created |
771 | - |
772 | - def get_sources(self, archive=None, ppainfo=None): |
773 | - """Get the list of source packages that have been published since the |
774 | - last timestamp from Launchpad. |
775 | - |
776 | - :param archive Launchpad archive: The Launchpad archive that |
777 | - should be queried for new sources. Defaults to |
778 | - self._main_archive. |
779 | - |
780 | - :param ppainfo dict(str, str): A dictionary containing three |
781 | - keys: "ppauser", "ppaname" and "isprivateppa". It should |
782 | - only be provided when dealing with a PPA; otherwise, the |
783 | - default is None. |
784 | - |
785 | - :rtype: dict, datetime.datetime |
786 | - |
787 | - Return a dictionary containing all source packages found, and |
788 | - also the new timestamp that should then be recorded by calling |
789 | - record_timestamp. |
790 | - |
791 | - """ |
792 | - timestamp, real_timestamp = self._get_normal_and_real_timestamps() |
793 | - |
794 | - if archive is None: |
795 | - archive = self._main_archive |
796 | - |
797 | - if ppainfo is None: |
798 | - archive_label = f"'{archive.displayname}' archive" |
799 | - else: |
800 | - archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})" |
801 | - |
802 | - self._logger.info(f"Polling source packages created since '{real_timestamp}' from {archive_label}") |
803 | - |
804 | - result = [] |
805 | - latest_timestamp_created = timestamp |
806 | - for pkg in archive.getPublishedSources( |
807 | - order_by_date=True, created_since_date=real_timestamp |
808 | - ): |
809 | - if pkg.status not in ("Pending", "Published"): |
810 | - continue |
811 | - |
812 | - src_urls = pkg.sourceFileUrls() |
813 | - if len(src_urls) == 0: |
814 | - # Safety check. |
815 | - continue |
816 | - |
817 | - if ( |
818 | - latest_timestamp_created is None |
819 | - or pkg.date_created > latest_timestamp_created |
820 | - ): |
821 | - latest_timestamp_created = pkg.date_created |
822 | - |
823 | - srcname = pkg.source_package_name |
824 | - srcver = pkg.source_package_version |
825 | - component = pkg.component_name |
826 | - |
827 | - msg = { |
828 | - "source_package": srcname, |
829 | - "version": srcver, |
830 | - "component": component, |
831 | - "source_urls": src_urls, |
832 | - "architecture": "source", |
833 | - } |
834 | - if ppainfo is not None: |
835 | - msg.update(ppainfo) |
836 | - |
837 | - result.append(msg) |
838 | - |
839 | - return result, latest_timestamp_created |
840 | diff --git a/debian/changelog b/debian/changelog |
841 | new file mode 100644 |
842 | index 0000000..ca2645a |
843 | --- /dev/null |
844 | +++ b/debian/changelog |
845 | @@ -0,0 +1,5 @@ |
846 | +ubuntu-debuginfod (0.1.0) UNRELEASED; urgency=medium |
847 | + |
848 | + * Initial Release. |
849 | + |
850 | + -- Sergio Durigan Junior <sergio.durigan@canonical.com> Wed, 26 Apr 2023 15:20:29 -0400 |
851 | diff --git a/debian/control b/debian/control |
852 | new file mode 100644 |
853 | index 0000000..e2e8a88 |
854 | --- /dev/null |
855 | +++ b/debian/control |
856 | @@ -0,0 +1,50 @@ |
857 | +Source: ubuntu-debuginfod |
858 | +Section: devel |
859 | +Priority: optional |
860 | +Maintainer: Ubuntu Server <ubuntu-server@lists.ubuntu.com> |
861 | +Uploaders: Sergio Durigan Junior <sergio.durigan@canonical.com> |
862 | +Build-Depends: debhelper-compat (= 13), |
863 | + dh-python, |
864 | + python3-setuptools, |
865 | + pybuild-plugin-pyproject, |
866 | + python3-all, |
867 | + python3-git, |
868 | + python3-celery, |
869 | + python3-launchpadlib, |
870 | + python3-requests, |
871 | + python3-requests-oauthlib, |
872 | + python3-sdnotify, |
873 | + python3-psycopg2, |
874 | + python3-pytest <!nocheck>, |
875 | + python3-freezegun <!nocheck>, |
876 | +Standards-Version: 4.6.0 |
877 | +Homepage: https://launchpad.net/ubuntu-debuginfod |
878 | +Vcs-Browser: https://git.launchpad.net/ubuntu-debuginfod |
879 | +Vcs-Git: https://git.launchpad.net/ubuntu-debuginfod |
880 | +#Testsuite: autopkgtest-pkg-python |
881 | +Rules-Requires-Root: no |
882 | + |
883 | +Package: ubuntu-debuginfod |
884 | +Architecture: all |
885 | +Pre-Depends: adduser, ucf |
886 | +Depends: ${python3:Depends}, |
887 | + ${misc:Depends}, |
888 | + python3-ubuntu-debuginfod (= ${source:Version}), |
889 | +Description: Scripts to poll/fetch artifacts for Ubuntu's debuginfod instance |
890 | + This package offers a set of scripts and utilities that are used to |
891 | + manage Ubuntu's debuginfod instance (<https://debuginfod.ubuntu.com>). |
892 | + |
893 | +Package: python3-ubuntu-debuginfod |
894 | +Section: python |
895 | +Architecture: all |
896 | +Depends: ${python3:Depends}, |
897 | + ${misc:Depends}, |
898 | + rabbitmq-server, |
899 | + celery, |
900 | + postgresql, |
901 | +Suggests: ubuntu-debuginfod (= ${source:Version}) |
902 | +Description: Scripts to poll/fetch artifacts for Ubuntu's debuginfod instance (Python library) |
903 | + This package offers a set of scripts and utilities that are used to |
904 | + manage Ubuntu's debuginfod instance (<https://debuginfod.ubuntu.com>). |
905 | + . |
906 | + This is the Python library. |
907 | diff --git a/debian/copyright b/debian/copyright |
908 | new file mode 100644 |
909 | index 0000000..7aec2cb |
910 | --- /dev/null |
911 | +++ b/debian/copyright |
912 | @@ -0,0 +1,29 @@ |
913 | +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ |
914 | +Upstream-Name: ubuntu-debuginfod |
915 | +Upstream-Contact: Sergio Durigan Junior <sergio.durigan@canonical.com> |
916 | +Source: https://launchpad.net/ubuntu-debuginfod |
917 | + |
918 | +Files: * |
919 | +Copyright: 2022-2023 Canonical Ltd. |
920 | +License: GPL-3.0+ |
921 | + |
922 | +Files: debian/* |
923 | +Copyright: 2022-2023 Canonical Ltd. |
924 | +License: GPL-3.0+ |
925 | + |
926 | +License: GPL-3.0+ |
927 | + This program is free software: you can redistribute it and/or modify |
928 | + it under the terms of the GNU General Public License as published by |
929 | + the Free Software Foundation, either version 3 of the License, or |
930 | + (at your option) any later version. |
931 | + . |
932 | + This package is distributed in the hope that it will be useful, |
933 | + but WITHOUT ANY WARRANTY; without even the implied warranty of |
934 | + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
935 | + GNU General Public License for more details. |
936 | + . |
937 | + You should have received a copy of the GNU General Public License |
938 | + along with this program. If not, see <https://www.gnu.org/licenses/>. |
939 | + . |
940 | + On Debian/Ubuntu systems, the complete text of the GNU General |
941 | + Public License version 3 can be found in "/usr/share/common-licenses/GPL-3". |
942 | diff --git a/debian/rules b/debian/rules |
943 | new file mode 100755 |
944 | index 0000000..f31c23e |
945 | --- /dev/null |
946 | +++ b/debian/rules |
947 | @@ -0,0 +1,19 @@ |
948 | +#!/usr/bin/make -f |
949 | + |
950 | +export PYBUILD_NAME = ubuntu-debuginfod |
951 | + |
952 | +%: |
953 | + dh $@ --with python3 --buildsystem=pybuild |
954 | + |
955 | +override_dh_auto_test: |
956 | +ifeq (,$(findstring nocheck,$(DEB_BUILD_OPTIONS))) |
957 | + PYBUILD_SYSTEM=custom \ |
958 | + PYBUILD_TEST_ARGS="PYTHONPATH={build_dir} \ |
959 | + {interpreter} -m pytest -v" \ |
960 | + dh_auto_test |
961 | +endif |
962 | + |
963 | +override_dh_installsystemd: |
964 | + dh_installsystemd --name=ubuntu-debuginfod-celery |
965 | + dh_installsystemd --name=ubuntu-debuginfod-launchpad-poller --no-start --no-enable |
966 | + dh_installsystemd --name=ubuntu-debuginfod-launchpad-dispatcher --no-start --no-enable |
967 | diff --git a/debian/source/format b/debian/source/format |
968 | new file mode 100644 |
969 | index 0000000..89ae9db |
970 | --- /dev/null |
971 | +++ b/debian/source/format |
972 | @@ -0,0 +1 @@ |
973 | +3.0 (native) |
974 | diff --git a/debian/source/options b/debian/source/options |
975 | new file mode 100644 |
976 | index 0000000..cb61fa5 |
977 | --- /dev/null |
978 | +++ b/debian/source/options |
979 | @@ -0,0 +1 @@ |
980 | +extend-diff-ignore = "^[^/]*[.]egg-info/" |
981 | diff --git a/debian/ubuntu-debuginfod.install b/debian/ubuntu-debuginfod.install |
982 | new file mode 100644 |
983 | index 0000000..52b779e |
984 | --- /dev/null |
985 | +++ b/debian/ubuntu-debuginfod.install |
986 | @@ -0,0 +1,3 @@ |
987 | +services/ usr/share/ubuntu-debuginfod/ |
988 | +conf/*.default usr/share/ubuntu-debuginfod/etc-default/ |
989 | +utils/ usr/share/ubuntu-debuginfod/ |
990 | diff --git a/debian/ubuntu-debuginfod.postinst b/debian/ubuntu-debuginfod.postinst |
991 | new file mode 100644 |
992 | index 0000000..e01120c |
993 | --- /dev/null |
994 | +++ b/debian/ubuntu-debuginfod.postinst |
995 | @@ -0,0 +1,168 @@ |
996 | +#!/bin/sh |
997 | +# postinst script for ubuntu-debuginfod |
998 | +# |
999 | +# see: dh_installdeb(1) |
1000 | + |
1001 | +set -e |
1002 | + |
1003 | +# summary of how this script can be called: |
1004 | +# * <postinst> `configure' <most-recently-configured-version> |
1005 | +# * <old-postinst> `abort-upgrade' <new version> |
1006 | +# * <conflictor's-postinst> `abort-remove' `in-favour' <package> |
1007 | +# <new-version> |
1008 | +# * <postinst> `abort-remove' |
1009 | +# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' |
1010 | +# <failed-install-package> <version> `removing' |
1011 | +# <conflicting-package> <version> |
1012 | +# for details, see https://www.debian.org/doc/debian-policy/ or |
1013 | +# the debian-policy package |
1014 | + |
1015 | +# The user we will run the services as. |
1016 | +UD_USER=$(grep '^User=' /lib/systemd/system/ubuntu-debuginfod-celery.service | cut -d= -f2) |
1017 | +# The mirror directory. |
1018 | +UD_MIRROR_DIR="/srv/debug-mirror" |
1019 | +# The database name. |
1020 | +UD_DB="ubuntu-debuginfod" |
1021 | +# The location of the Launchpad credential file. |
1022 | +UD_LP_CREDENTIAL_FILE=$(grep '^Environment=LP_CREDENTIALS_FILE' /lib/systemd/system/ubuntu-debuginfod-celery.service | cut -d= -f3) |
1023 | + |
1024 | +# Install the *.default files. |
1025 | +install_etc_default_files() |
1026 | +{ |
1027 | + for file in celery launchpad-poller; do |
1028 | + ucf --three-way \ |
1029 | + "/usr/share/ubuntu-debuginfod/etc-default/ubuntu-debuginfod-${file}.default" \ |
1030 | + "/etc/default/ubuntu-debuginfod-${file}" |
1031 | + ucfr ubuntu-debuginfod \ |
1032 | + "/etc/default/ubuntu-debuginfod-${file}" |
1033 | + done |
1034 | +} |
1035 | + |
1036 | +# Create a user if it doesn't exist. |
1037 | +create_user() |
1038 | +{ |
1039 | + if ! getent passwd "${UD_USER}" > /dev/null; then |
1040 | + adduser --quiet --disabled-password \ |
1041 | + --gecos "ubuntu-debuginfod mirror user" \ |
1042 | + "${UD_USER}" |
1043 | + fi |
1044 | + |
1045 | + lpcreddir=$(dirname "${UD_LP_CREDENTIAL_FILE}") |
1046 | + if [ ! -d "${lpcreddir}" ]; then |
1047 | + mkdir -p "${lpcreddir}" |
1048 | + chown "${UD_USER}":"${UD_USER}" "${lpcreddir}" |
1049 | + fi |
1050 | +} |
1051 | + |
1052 | +# Helper function to run commands as the postgres user. |
1053 | +runuser_postgres() |
1054 | +{ |
1055 | + cd /tmp && runuser -u postgres -- "$@" && cd - |
1056 | +} |
1057 | + |
1058 | +# Create the mirror directory if it doesn't exist. |
1059 | +create_mirror_dir() |
1060 | +{ |
1061 | + if [ ! -d "${UD_MIRROR_DIR}" ]; then |
1062 | + mkdir -p "${UD_MIRROR_DIR}" |
1063 | + chown "${UD_USER}":"${UD_USER}" "${UD_MIRROR_DIR}" |
1064 | + fi |
1065 | +} |
1066 | + |
1067 | +# Create the database user if it doesn't exist. |
1068 | +create_database_user() |
1069 | +{ |
1070 | + if ! runuser_postgres psql postgres -tAc \ |
1071 | + "SELECT 1 FROM pg_roles WHERE rolname='${UD_USER}'" | \ |
1072 | + grep -qFx 1; then |
1073 | + runuser -u postgres -- createuser -d "${UD_USER}" |
1074 | + fi |
1075 | +} |
1076 | + |
1077 | +# Create the database if it doesn't exist. |
1078 | +create_database() |
1079 | +{ |
1080 | + create_database_user |
1081 | + |
1082 | + if ! runuser_postgres psql -ltq | \ |
1083 | + cut -d \| -f 1 | \ |
1084 | + grep -qFw "${UD_DB}"; then |
1085 | + runuser_postgres createdb -O "${UD_USER}" "${UD_DB}" |
1086 | + fi |
1087 | +} |
1088 | + |
1089 | +# Check that the recommended RabbitMQ configuration is present. If it |
1090 | +# isn't, display a warning. |
1091 | +check_rabbitmq_configuration() |
1092 | +{ |
1093 | + rabconf="/etc/rabbitmq/rabbitmq.conf" |
1094 | + if [ -f "${rabconf}" ] && \ |
1095 | + grep -q '^consumer_timeout' "${rabconf}"; then |
1096 | + return |
1097 | + fi |
1098 | + |
1099 | + cat << __EOF__ |
1100 | + |
1101 | + * It is suggested that RabbitMQ be configured with the following option: |
1102 | + |
1103 | + \$ cat ${rabconf} |
1104 | + consumer_timeout = 10800000 |
1105 | +__EOF__ |
1106 | +} |
1107 | + |
1108 | +# Obtain the Launchpad credentials. |
1109 | +# |
1110 | +# TODO: See if there's a way to allow the user to bail out. |
1111 | +# obtain_launchpad_credentials() |
1112 | +# { |
1113 | +# if [ ! -f "${UD_LP_CREDENTIAL_FILE}" ]; then |
1114 | +# /usr/share/ubuntu-debuginfod/utils/obtain-launchpad-credentials.py \ |
1115 | +# "${UD_LP_CREDENTIAL_FILE}" |
1116 | +# chown "${UD_USER}":"${UD_USER}" "${UD_LP_CREDENTIAL_FILE}" |
1117 | +# fi |
1118 | +# } |
1119 | + |
1120 | +# Check if the Launchpad credentials exist, and warn the user |
1121 | +# otherwise. |
1122 | +check_launchpad_credentials() |
1123 | +{ |
1124 | + if [ ! -f "${UD_LP_CREDENTIAL_FILE}" ]; then |
1125 | + cat << __EOF__ |
1126 | + |
1127 | + * The Launchpad credentials have not been obtained yet. You need to |
1128 | + obtain them in order to properly start the services. To do that, |
1129 | + please run: |
1130 | + |
1131 | + \$ runuser -u ${UD_USER} -- \\ |
1132 | + /usr/share/ubuntu-debuginfod/utils/obtain-launchpad-credentials.py \\ |
1133 | + ${UD_LP_CREDENTIAL_FILE} |
1134 | +__EOF__ |
1135 | + fi |
1136 | +} |
1137 | + |
1138 | +case "$1" in |
1139 | + configure) |
1140 | + install_etc_default_files |
1141 | + create_user |
1142 | + create_mirror_dir |
1143 | + create_database |
1144 | + check_rabbitmq_configuration |
1145 | + #obtain_launchpad_credentials |
1146 | + check_launchpad_credentials |
1147 | + ;; |
1148 | + |
1149 | + abort-upgrade|abort-remove|abort-deconfigure) |
1150 | + ;; |
1151 | + |
1152 | + *) |
1153 | + echo "postinst called with unknown argument \`$1'" >&2 |
1154 | + exit 1 |
1155 | + ;; |
1156 | +esac |
1157 | + |
1158 | +# dh_installdeb will replace this with shell code automatically |
1159 | +# generated by other debhelper scripts. |
1160 | + |
1161 | +#DEBHELPER# |
1162 | + |
1163 | +exit 0 |
1164 | diff --git a/debian/ubuntu-debuginfod.postrm b/debian/ubuntu-debuginfod.postrm |
1165 | new file mode 100644 |
1166 | index 0000000..366eee5 |
1167 | --- /dev/null |
1168 | +++ b/debian/ubuntu-debuginfod.postrm |
1169 | @@ -0,0 +1,39 @@ |
1170 | +#!/bin/sh |
1171 | + |
1172 | +set -e |
1173 | + |
1174 | +purge_etc_default_files() |
1175 | +{ |
1176 | + for file in celery launchpad-poller; do |
1177 | + if command -v ucf > /dev/null; then |
1178 | + ucf --purge \ |
1179 | + "/etc/default/ubuntu-debuginfod-${file}" |
1180 | + fi |
1181 | + |
1182 | + if command -v ucfr > /dev/null; then |
1183 | + ucfr --purge ubuntu-debuginfod \ |
1184 | + "/etc/default/ubuntu-debuginfod-${file}" |
1185 | + fi |
1186 | + done |
1187 | +} |
1188 | + |
1189 | +case "$1" in |
1190 | + purge) |
1191 | + purge_etc_default_files |
1192 | + ;; |
1193 | + |
1194 | + remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) |
1195 | + ;; |
1196 | + |
1197 | + *) |
1198 | + echo "postrm called with unknown argument \`$1'" >&2 |
1199 | + exit 1 |
1200 | + ;; |
1201 | +esac |
1202 | + |
1203 | +# dh_installdeb will replace this with shell code automatically |
1204 | +# generated by other debhelper scripts. |
1205 | + |
1206 | +#DEBHELPER# |
1207 | + |
1208 | +exit 0 |
1209 | diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service |
1210 | new file mode 120000 |
1211 | index 0000000..6537f37 |
1212 | --- /dev/null |
1213 | +++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-celery.service |
1214 | @@ -0,0 +1 @@ |
1215 | +../conf/ubuntu-debuginfod-celery.service |
1216 | \ No newline at end of file |
1217 | diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service |
1218 | new file mode 120000 |
1219 | index 0000000..9f7c034 |
1220 | --- /dev/null |
1221 | +++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.service |
1222 | @@ -0,0 +1 @@ |
1223 | +../conf/ubuntu-debuginfod-launchpad-dispatcher.service |
1224 | \ No newline at end of file |
1225 | diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer |
1226 | new file mode 120000 |
1227 | index 0000000..a921a02 |
1228 | --- /dev/null |
1229 | +++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-dispatcher.timer |
1230 | @@ -0,0 +1 @@ |
1231 | +../conf/ubuntu-debuginfod-launchpad-dispatcher.timer |
1232 | \ No newline at end of file |
1233 | diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service |
1234 | new file mode 120000 |
1235 | index 0000000..aae0131 |
1236 | --- /dev/null |
1237 | +++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.service |
1238 | @@ -0,0 +1 @@ |
1239 | +../conf/ubuntu-debuginfod-launchpad-poller.service |
1240 | \ No newline at end of file |
1241 | diff --git a/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer |
1242 | new file mode 120000 |
1243 | index 0000000..253a01c |
1244 | --- /dev/null |
1245 | +++ b/debian/ubuntu-debuginfod.ubuntu-debuginfod-launchpad-poller.timer |
1246 | @@ -0,0 +1 @@ |
1247 | +../conf/ubuntu-debuginfod-launchpad-poller.timer |
1248 | \ No newline at end of file |
1249 | diff --git a/debuggetter.py b/debuggetter.py |
1250 | deleted file mode 100644 |
1251 | index e5500a7..0000000 |
1252 | --- a/debuggetter.py |
1253 | +++ /dev/null |
1254 | @@ -1,149 +0,0 @@ |
1255 | -#!/usr/bin/python3 |
1256 | - |
1257 | -# Copyright (C) 2022 Canonical Ltd. |
1258 | - |
1259 | -# This program is free software: you can redistribute it and/or modify |
1260 | -# it under the terms of the GNU General Public License as published by |
1261 | -# the Free Software Foundation, either version 3 of the License, or |
1262 | -# (at your option) any later version. |
1263 | - |
1264 | -# This program is distributed in the hope that it will be useful, |
1265 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1266 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1267 | -# GNU General Public License for more details. |
1268 | - |
1269 | -# You should have received a copy of the GNU General Public License |
1270 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
1271 | - |
1272 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
1273 | - |
1274 | -import os |
1275 | -import requests |
1276 | -from urllib import parse |
1277 | -import logging |
1278 | -import tempfile |
1279 | -import shutil |
1280 | -import re |
1281 | - |
1282 | - |
1283 | -class DebugGetterTimeout(Exception): |
1284 | - """Default exception to raise when there's a timeout issue.""" |
1285 | - |
1286 | - |
1287 | -class DebugGetterRetry(Exception): |
1288 | - """Default exception to raise when we'd like to signal that the task |
1289 | - should be retried.""" |
1290 | - |
1291 | - |
1292 | -DEFAULT_MIRROR_DIR = "/srv/debug-mirror" |
1293 | - |
1294 | - |
1295 | -class DebugGetter: |
1296 | - """Base class for a Debug Getter.""" |
1297 | - |
1298 | - def __init__(self, subdir, mirror_dir=DEFAULT_MIRROR_DIR): |
1299 | - """Initialize the object. |
1300 | - |
1301 | - :param str mirror_dir: The directory we use to save the |
1302 | - mirrored files from Launchpad. |
1303 | - |
1304 | - :param str subdir: The subdirectory (insider mirror_dir) where |
1305 | - the module will save its files. For example, a ddeb |
1306 | - getter module should specify "ddebs" here.""" |
1307 | - self._mirror_dir = mirror_dir |
1308 | - self._subdir = subdir |
1309 | - self._logger = logging.getLogger(__name__) |
1310 | - |
1311 | - def _make_savepath(self, source_package, component): |
1312 | - """Return the full save path for a package. |
1313 | - |
1314 | - :param str source_package: The package name. |
1315 | - :param str component: The component name (main, universe, etc.).""" |
1316 | - if source_package.startswith("lib"): |
1317 | - pkgname_initials = source_package[:4] |
1318 | - else: |
1319 | - pkgname_initials = source_package[0] |
1320 | - |
1321 | - return os.path.join( |
1322 | - self._mirror_dir, |
1323 | - self._subdir, |
1324 | - component, |
1325 | - pkgname_initials, |
1326 | - source_package, |
1327 | - ) |
1328 | - |
1329 | - def _try_to_move_atomically(self, src, dst): |
1330 | - """Try to move SRC to DST atomically. |
1331 | - |
1332 | - This function assumes that SRC and DST have different filenames. |
1333 | - |
1334 | - :param str src: The source file (full path). |
1335 | - |
1336 | - :param str dst: The destination file (full path).""" |
1337 | - self._logger.debug(f"Trying to move '{src}' to '{dst}' atomically") |
1338 | - savepath = os.path.dirname(dst) |
1339 | - os.makedirs(savepath, mode=0o755, exist_ok=True) |
1340 | - newtmpfile = os.path.join(savepath, os.path.basename(src)) |
1341 | - shutil.copyfile(src, newtmpfile) |
1342 | - try: |
1343 | - shutil.move(newtmpfile, dst) |
1344 | - except shutil.SameFileError: |
1345 | - os.remove(newtmpfile) |
1346 | - self._logger.warning( |
1347 | - f"Could not create '{dst}' atomically: same file already exists" |
1348 | - ) |
1349 | - |
1350 | - def _download_from_lp(self, url, savepath, credentials=None): |
1351 | - """Download a file from Launchpad. |
1352 | - |
1353 | - This method tries to download the file in chunks into a |
1354 | - temporary file and then does an atomic move to the final |
1355 | - destination. |
1356 | - |
1357 | - :param str url: The URL that should be downloaded. |
1358 | - |
1359 | - :param str savepath: The full path (minus the filename) where |
1360 | - the file should be saved. |
1361 | - |
1362 | - :param requests_oauthlib.OAuth1 credentials: The credentials |
1363 | - to be used when downloading the artifact from Launchpad. |
1364 | - Default is None, which means anonymous.""" |
1365 | - filepath = os.path.join(savepath, os.path.basename(parse.urlparse(url).path)) |
1366 | - if os.path.exists(filepath): |
1367 | - self._logger.debug(f"'{filepath}' exists, doing nothing") |
1368 | - return |
1369 | - |
1370 | - # Launchpad only accepts authenticated requests if we use |
1371 | - # 'https://api.launchpad.net...' |
1372 | - if credentials is not None: |
1373 | - url = re.sub(r"^https://launchpad.net/", "https://api.launchpad.net/devel/", url) |
1374 | - |
1375 | - try: |
1376 | - with requests.Session() as s: |
1377 | - with s.get( |
1378 | - url, allow_redirects=True, timeout=10, stream=True, auth=credentials |
1379 | - ) as r: |
1380 | - r.raise_for_status() |
1381 | - with tempfile.NamedTemporaryFile(mode="wb") as tmpfile: |
1382 | - shutil.copyfileobj(r.raw, tmpfile) |
1383 | - tmpfile.flush() |
1384 | - self._try_to_move_atomically(tmpfile.name, filepath) |
1385 | - except ( |
1386 | - requests.ConnectionError, |
1387 | - requests.HTTPError, |
1388 | - requests.ConnectTimeout, |
1389 | - requests.ReadTimeout, |
1390 | - requests.Timeout, |
1391 | - ) as e: |
1392 | - self._logger.warning(f"Timeout while downloading '{url}': '{e}'") |
1393 | - raise DebugGetterTimeout() |
1394 | - |
1395 | - self._logger.debug(f"Saved '{filepath}'") |
1396 | - |
1397 | - @property |
1398 | - def subdir(self): |
1399 | - return self._subdir |
1400 | - |
1401 | - @subdir.setter |
1402 | - def subdir(self, subdir): |
1403 | - self._subdir = subdir |
1404 | diff --git a/poll_launchpad.py b/poll_launchpad.py |
1405 | deleted file mode 100644 |
1406 | index d8645a8..0000000 |
1407 | --- a/poll_launchpad.py |
1408 | +++ /dev/null |
1409 | @@ -1,198 +0,0 @@ |
1410 | -#!/usr/bin/python3 |
1411 | - |
1412 | -# Copyright (C) 2022 Canonical Ltd. |
1413 | - |
1414 | -# This program is free software: you can redistribute it and/or modify |
1415 | -# it under the terms of the GNU General Public License as published by |
1416 | -# the Free Software Foundation, either version 3 of the License, or |
1417 | -# (at your option) any later version. |
1418 | - |
1419 | -# This program is distributed in the hope that it will be useful, |
1420 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1421 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1422 | -# GNU General Public License for more details. |
1423 | - |
1424 | -# You should have received a copy of the GNU General Public License |
1425 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
1426 | - |
1427 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
1428 | - |
1429 | -import os |
1430 | -from ddebpoller import DdebPoller |
1431 | -from ppapoller import PPAPoller, PrivatePPAPoller |
1432 | -from debuginfod import ( |
1433 | - grab_ddebs, |
1434 | - grab_ddebs_sources, |
1435 | - grab_ppa_ddebs, |
1436 | - grab_ppa_sources, |
1437 | - grab_private_ppa_ddebs, |
1438 | - grab_private_ppa_sources, |
1439 | -) |
1440 | -from utils import DebugDB |
1441 | - |
1442 | -import psycopg2 |
1443 | - |
1444 | - |
1445 | -def _ensure_task(msg, conn, cursor): |
1446 | - """Return True if the task described by MSG should be processed. |
1447 | - |
1448 | - Also, if True, then insert the task information into the database. |
1449 | - |
1450 | - :param DebugDB conn: The database connection handler. |
1451 | - |
1452 | - :param psycopg2.cursor cursor: The cursor to the SQL database. |
1453 | - |
1454 | - :param dict msg: The message describing a task. |
1455 | - |
1456 | - :rtype: int |
1457 | - :return: The taskid if the task should be processed (or None otherwise).""" |
1458 | - if msg["architecture"] == "source": |
1459 | - pkgname = msg["source_package"] |
1460 | - else: |
1461 | - pkgname = msg["binary_package"] |
1462 | - pkgver = msg["version"] |
1463 | - architecture = msg["architecture"] |
1464 | - |
1465 | - if "ppauser" in msg.keys(): |
1466 | - sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = TRUE AND ppauser = %s AND ppaname = %s" |
1467 | - sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, ppauser, ppaname, created_date) VALUES (%s, %s, %s, TRUE, %s, %s, NOW()) RETURNING id" |
1468 | - sqlparams = (pkgname, pkgver, architecture, msg["ppauser"], msg["ppaname"]) |
1469 | - else: |
1470 | - sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = FALSE" |
1471 | - sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, created_date) VALUES (%s, %s, %s, FALSE, NOW()) RETURNING id" |
1472 | - sqlparams = (pkgname, pkgver, architecture) |
1473 | - |
1474 | - cursor.execute(sql_select, sqlparams) |
1475 | - |
1476 | - if cursor.rowcount > 0: |
1477 | - return None |
1478 | - |
1479 | - cursor.execute(sql_insert, sqlparams) |
1480 | - |
1481 | - taskid = cursor.fetchone()[0] |
1482 | - conn.commit() |
1483 | - return taskid |
1484 | - |
1485 | - |
1486 | -def _process_msg(msg, conn, cursor): |
1487 | - """Add a task to the database to download the source or binary ddebs, |
1488 | - if it isn't there already. |
1489 | - |
1490 | - :param str msg: The message to be processed. |
1491 | - |
1492 | - :param DebugDB conn: The database connection handler. |
1493 | - |
1494 | - :param psycopg2.cursor cursor: The cursor to the database. |
1495 | - |
1496 | - """ |
1497 | - taskid = _ensure_task(msg, conn, cursor) |
1498 | - if taskid is not None: |
1499 | - if msg.get("ppaname") is None: |
1500 | - ddeb_fn = grab_ddebs |
1501 | - src_fn = grab_ddebs_sources |
1502 | - elif msg["isprivateppa"] == "no": |
1503 | - ddeb_fn = grab_ppa_ddebs |
1504 | - src_fn = grab_ppa_sources |
1505 | - else: |
1506 | - ddeb_fn = grab_private_ppa_ddebs |
1507 | - src_fn = grab_private_ppa_sources |
1508 | - |
1509 | - # We append the ID to the message so that we can efficiently |
1510 | - # remove the task once it's been processed. |
1511 | - msg["taskid"] = str(taskid) |
1512 | - if msg["architecture"] == "source": |
1513 | - # Source packages have "source" as their architecture. |
1514 | - src_fn.delay(msg) |
1515 | - else: |
1516 | - ddeb_fn.delay(msg) |
1517 | - |
1518 | - |
1519 | -def _poll_launchpad_generic(poller, conn, cursor): |
1520 | - """Poll Launchpad and process the list of ddebs and source packages. |
1521 | - |
1522 | - :param DebugDB conn: The SQL connection handler. |
1523 | - |
1524 | - :param psycopg2.cursor cursor: The cursor to the SQL database.""" |
1525 | - |
1526 | - # Process ddebs. |
1527 | - messages, new_timestamp_ddebs = poller.get_ddebs() |
1528 | - for msg in messages: |
1529 | - _process_msg(msg, conn, cursor) |
1530 | - |
1531 | - # Process source packages. |
1532 | - messages, new_timestamp_src = poller.get_sources() |
1533 | - for msg in messages: |
1534 | - _process_msg(msg, conn, cursor) |
1535 | - |
1536 | - # Choose the earliest timestamp. |
1537 | - new_timestamp = min(new_timestamp_ddebs, new_timestamp_src) |
1538 | - |
1539 | - poller.record_timestamp(new_timestamp) |
1540 | - |
1541 | - |
1542 | -def _should_poll_archive(archive): |
1543 | - """Whether we should poll a certain archive. |
1544 | - |
1545 | - This function relies on the value of environment variables named |
1546 | - "POLL_{archive}". See the /etc/default/ubuntu-debuginfod-poll-lp |
1547 | - file for more details. |
1548 | - |
1549 | - :param archive string: The archive name, in uppercase. |
1550 | - |
1551 | - :rtype boolean: True if we should poll the archive, False |
1552 | - otherwise. |
1553 | - """ |
1554 | - return os.environ.get(f"POLL_{archive.upper()}") == "yes" |
1555 | - |
1556 | - |
1557 | -def poll_launchpad(conn, cursor): |
1558 | - """Poll Launchpad and process the list of ddebs and source packages |
1559 | - from the main archive. |
1560 | - |
1561 | - :param DebugDB conn: The SQL connection handler. |
1562 | - |
1563 | - :param psycopg2.cursor cursor: The cursor to the SQL database. |
1564 | - |
1565 | - """ |
1566 | - if _should_poll_archive("MAIN_ARCHIVE"): |
1567 | - _poll_launchpad_generic(DdebPoller(), conn, cursor) |
1568 | - |
1569 | - |
1570 | -def poll_launchpad_ppas(conn, cursor): |
1571 | - """Poll Launchpad and process the list of ddebs and source packages |
1572 | - from PPAs. |
1573 | - |
1574 | - :param DebugDB conn: The SQL connection handler. |
1575 | - |
1576 | - :param psycopg2.cursor cursor: The cursor to the SQL database. |
1577 | - |
1578 | - """ |
1579 | - if _should_poll_archive("PPAS"): |
1580 | - _poll_launchpad_generic(PPAPoller(), conn, cursor) |
1581 | - |
1582 | - |
1583 | -def poll_launchpad_private_ppas(conn, cursor): |
1584 | - """Poll Launchpad and process the list of ddebs and source packages |
1585 | - from private PPAs. |
1586 | - |
1587 | - :param DebugDB conn: The SQL connection handler. |
1588 | - |
1589 | - :param psycopg2.cursor cursor: The cursor to the SQL database. |
1590 | - |
1591 | - """ |
1592 | - if _should_poll_archive("PRIVATE_PPAS"): |
1593 | - _poll_launchpad_generic(PrivatePPAPoller(), conn, cursor) |
1594 | - |
1595 | - |
1596 | -def main(): |
1597 | - # Connect to the database. |
1598 | - with DebugDB() as db: |
1599 | - db.create_tables() |
1600 | - with db.cursor() as cursor: |
1601 | - poll_launchpad(db, cursor) |
1602 | - poll_launchpad_ppas(db, cursor) |
1603 | - poll_launchpad_private_ppas(db, cursor) |
1604 | - |
1605 | - |
1606 | -if __name__ == "__main__": |
1607 | - main() |
1608 | diff --git a/ppagetter.py b/ppagetter.py |
1609 | deleted file mode 100644 |
1610 | index 0f0f323..0000000 |
1611 | --- a/ppagetter.py |
1612 | +++ /dev/null |
1613 | @@ -1,227 +0,0 @@ |
1614 | -#!/usr/bin/python3 |
1615 | - |
1616 | -# Copyright (C) 2023 Canonical Ltd. |
1617 | - |
1618 | -# This program is free software: you can redistribute it and/or modify |
1619 | -# it under the terms of the GNU General Public License as published by |
1620 | -# the Free Software Foundation, either version 3 of the License, or |
1621 | -# (at your option) any later version. |
1622 | - |
1623 | -# This program is distributed in the hope that it will be useful, |
1624 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1625 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1626 | -# GNU General Public License for more details. |
1627 | - |
1628 | -# You should have received a copy of the GNU General Public License |
1629 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
1630 | - |
1631 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
1632 | - |
1633 | -import os |
1634 | -import configparser |
1635 | -from ddebgetter import ( |
1636 | - DdebGetter, |
1637 | - DdebSourceCodeGetter, |
1638 | - validate_ddeb_request, |
1639 | - validate_source_code_request, |
1640 | -) |
1641 | -from debuggetter import DebugGetterRetry, DEFAULT_MIRROR_DIR |
1642 | - |
1643 | - |
1644 | -import requests_oauthlib |
1645 | -from launchpadlib.launchpad import Credentials |
1646 | - |
1647 | - |
1648 | -def _validate_generic_ppa_request(request): |
1649 | - """Validate a request to fetch an artifact from a PPA (private or |
1650 | - not). |
1651 | - |
1652 | - :param dict[str, str] request: The dictionary containing the |
1653 | - request to be validated.""" |
1654 | - if request.get("ppauser") is None: |
1655 | - raise TypeError("No 'ppauser' in request") |
1656 | - if request.get("ppaname") is None: |
1657 | - raise TypeError("No 'ppaname' in request") |
1658 | - if request.get("isprivateppa") is None: |
1659 | - raise TypeError("No 'isprivateppa' in request") |
1660 | - |
1661 | - |
1662 | -def validate_ppa_ddeb_request(request): |
1663 | - """Validate a request to fetch a ddeb from a PPA. |
1664 | - |
1665 | - :param dict[str, str] request: The dictionary containing the |
1666 | - request to be validated.""" |
1667 | - |
1668 | - validate_ddeb_request(request) |
1669 | - _validate_generic_ppa_request(request) |
1670 | - if request["isprivateppa"] == "yes": |
1671 | - raise ValueError("Wrong request: from private PPA") |
1672 | - |
1673 | - |
1674 | -def validate_ppa_source_code_request(request): |
1675 | - """Validate a request to fetch a source package from a PPA. |
1676 | - |
1677 | - :param dict[str, str] request: The dictionary containing the |
1678 | - request to be validated.""" |
1679 | - |
1680 | - validate_source_code_request(request) |
1681 | - _validate_generic_ppa_request(request) |
1682 | - if request["isprivateppa"] == "yes": |
1683 | - raise ValueError("Wrong request: from private PPA") |
1684 | - |
1685 | - |
1686 | -def validate_private_ppa_ddeb_request(request): |
1687 | - """Validate a request to fetch a ddeb from a private PPA. |
1688 | - |
1689 | - :param dict[str, str] request: The dictionary containing the |
1690 | - request to be validated.""" |
1691 | - |
1692 | - validate_ddeb_request(request) |
1693 | - _validate_generic_ppa_request(request) |
1694 | - if request["isprivateppa"] == "no": |
1695 | - raise ValueError("Wrong request: not from private PPA") |
1696 | - |
1697 | - |
1698 | -def validate_private_ppa_source_code_request(request): |
1699 | - """Validate a request to fetch a source package from a private |
1700 | - PPA. |
1701 | - |
1702 | - :param dict[str, str] request: The dictionary containing the |
1703 | - request to be validated.""" |
1704 | - validate_source_code_request(request) |
1705 | - _validate_generic_ppa_request(request) |
1706 | - if request["isprivateppa"] == "no": |
1707 | - raise ValueError("Wrong request: not from private PPA") |
1708 | - |
1709 | - |
1710 | -class PPADdebGetter(DdebGetter): |
1711 | - """Get a ddeb from a Launchpad PPA.""" |
1712 | - |
1713 | - def __init__(self, subdir="ppas", mirror_dir=DEFAULT_MIRROR_DIR): |
1714 | - """Initialize the object. |
1715 | - |
1716 | - See DebugGetter's __init__ for an explanation of the arguments.""" |
1717 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
1718 | - |
1719 | - def process_request(self, request): |
1720 | - """Process a request, usually coming from Celery. |
1721 | - |
1722 | - :param request: The dictionary containing the information |
1723 | - necessary to fetch this ddeb. |
1724 | - :type request: dict(str : str)""" |
1725 | - validate_ppa_ddeb_request(request) |
1726 | - |
1727 | - # Adjust the subdirectory to account for the PPA name. |
1728 | - new_subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"]) |
1729 | - self.subdir = new_subdir |
1730 | - |
1731 | - self.do_process_request(request) |
1732 | - |
1733 | - |
1734 | -class PPASourceCodeGetter(DdebSourceCodeGetter): |
1735 | - """Get source code from a Launchpad PPA.""" |
1736 | - |
1737 | - def __init__(self, subdir="ppas", mirror_dir=DEFAULT_MIRROR_DIR): |
1738 | - """Initialize the object. |
1739 | - |
1740 | - See DebugGetter's __init__ for an explanation of the arguments.""" |
1741 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
1742 | - |
1743 | - def process_request(self, request): |
1744 | - """Process a request, usually coming from Celery. |
1745 | - |
1746 | - :param request: The dictionary containing the information |
1747 | - necessary to fetch this source code. |
1748 | - :type request: dict(str : str)""" |
1749 | - validate_ppa_source_code_request(request) |
1750 | - |
1751 | - # Adjust the subdirectory to account for the PPA name. |
1752 | - new_subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"]) |
1753 | - self.subdir = new_subdir |
1754 | - |
1755 | - self.do_process_request(request, fallback_to_git=False) |
1756 | - |
1757 | - |
1758 | -def obtain_lp_oauth_credentials(): |
1759 | - """Obtain the necessary OAuth credentials to perform authenticated |
1760 | - operations against Launchpad. |
1761 | - |
1762 | - :rtype requests_oauthlib.OAuth1: The OAuth1 object containing the |
1763 | - Launchpad credentials.""" |
1764 | - lpcredfile = os.environ.get("LP_CREDENTIALS_FILE") |
1765 | - if lpcredfile is None: |
1766 | - raise ValueError( |
1767 | - "Could not obtain Launchpad credentials: environment variable LP_CREDENTIALS_FILE must be set" |
1768 | - ) |
1769 | - if not os.path.isfile(lpcredfile): |
1770 | - raise FileNotFoundError( |
1771 | - f"Could not obtain Launchpad credentials: file {lpcredfile} does not exist" |
1772 | - ) |
1773 | - |
1774 | - lpcred = Credentials() |
1775 | - with open(lpcredfile, "r", encoding="UTF-8") as f: |
1776 | - lpcred.load(f) |
1777 | - |
1778 | - return requests_oauthlib.OAuth1( |
1779 | - lpcred.consumer.key, |
1780 | - client_secret=lpcred.consumer.secret, |
1781 | - resource_owner_key=lpcred.access_token.key, |
1782 | - resource_owner_secret=lpcred.access_token.secret, |
1783 | - signature_method="PLAINTEXT", |
1784 | - ) |
1785 | - |
1786 | - |
1787 | -class PrivatePPADdebGetter(PPADdebGetter): |
1788 | - """Get a ddeb from a private Launchpad PPA.""" |
1789 | - |
1790 | - def __init__(self, subdir="private-ppas", mirror_dir=DEFAULT_MIRROR_DIR): |
1791 | - """Initialize the object. |
1792 | - |
1793 | - See DebugGetter's __init__ for an explanation of the arguments.""" |
1794 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
1795 | - |
1796 | - def process_request(self, request): |
1797 | - """Process a request, usually coming from Celery. |
1798 | - |
1799 | - :param request: The dictionary containing the information |
1800 | - necessary to fetch this ddeb. |
1801 | - :type request: dict(str : str)""" |
1802 | - validate_private_ppa_ddeb_request(request) |
1803 | - |
1804 | - ppauser = request["ppauser"] |
1805 | - ppaname = request["ppaname"] |
1806 | - creds = obtain_lp_oauth_credentials() |
1807 | - |
1808 | - # Adjust the subdirectory to account for the PPA name. |
1809 | - new_subdir = os.path.join(self._subdir, ppauser, ppaname) |
1810 | - self.subdir = new_subdir |
1811 | - |
1812 | - self.do_process_request(request, credentials=creds) |
1813 | - |
1814 | - |
1815 | -class PrivatePPASourceCodeGetter(PPASourceCodeGetter): |
1816 | - """Get source code from a private Launchpad PPA.""" |
1817 | - |
1818 | - def __init__(self, subdir="private-ppas", mirror_dir=DEFAULT_MIRROR_DIR): |
1819 | - """Initialize the object. |
1820 | - |
1821 | - See DebugGetter's __init__ for an explanation of the arguments.""" |
1822 | - super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
1823 | - |
1824 | - def process_request(self, request): |
1825 | - """Process a request, usually coming from Celery. |
1826 | - |
1827 | - :param request: The dictionary containing the information |
1828 | - necessary to fetch this source code. |
1829 | - :type request: dict(str : str)""" |
1830 | - validate_private_ppa_source_code_request(request) |
1831 | - |
1832 | - ppauser = request["ppauser"] |
1833 | - ppaname = request["ppaname"] |
1834 | - creds = obtain_lp_oauth_credentials() |
1835 | - |
1836 | - # Adjust the subdirectory to account for the PPA name. |
1837 | - new_subdir = os.path.join(self._subdir, ppauser, ppaname) |
1838 | - self.subdir = new_subdir |
1839 | - |
1840 | - self.do_process_request(request, fallback_to_git=False, credentials=creds) |
1841 | diff --git a/ppapoller.py b/ppapoller.py |
1842 | deleted file mode 100644 |
1843 | index 27f844c..0000000 |
1844 | --- a/ppapoller.py |
1845 | +++ /dev/null |
1846 | @@ -1,191 +0,0 @@ |
1847 | -#!/usr/bin/python3 |
1848 | - |
1849 | -# Copyright (C) 2023 Canonical Ltd. |
1850 | - |
1851 | -# This program is free software: you can redistribute it and/or modify |
1852 | -# it under the terms of the GNU General Public License as published by |
1853 | -# the Free Software Foundation, either version 3 of the License, or |
1854 | -# (at your option) any later version. |
1855 | - |
1856 | -# This program is distributed in the hope that it will be useful, |
1857 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1858 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1859 | -# GNU General Public License for more details. |
1860 | - |
1861 | -# You should have received a copy of the GNU General Public License |
1862 | -# along with this program. If not, see <https://www.gnu.org/licenses/>. |
1863 | - |
1864 | -# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
1865 | - |
1866 | -import os |
1867 | -import lazr.restfulclient.errors |
1868 | -from ddebpoller import DdebPoller |
1869 | - |
1870 | - |
1871 | -class PPAPoller(DdebPoller): |
1872 | - """Perform the Launchpad polling and obtain a list of ddebs and source |
1873 | - packages from a PPA.""" |
1874 | - |
1875 | - def __init__( |
1876 | - self, |
1877 | - initial_interval=1, |
1878 | - force_initial_interval=False, |
1879 | - fetch_all_on_first_run=True, |
1880 | - dry_run=False, |
1881 | - anonymous=True, |
1882 | - ): |
1883 | - """Initialize the object using 'ppa' as its name. |
1884 | - |
1885 | - Look at DebugPoller's docstring for an explanation about the arguments.""" |
1886 | - super().__init__( |
1887 | - initial_interval=initial_interval, |
1888 | - force_initial_interval=force_initial_interval, |
1889 | - fetch_all_on_first_run=fetch_all_on_first_run, |
1890 | - dry_run=dry_run, |
1891 | - anonymous=anonymous, |
1892 | - ) |
1893 | - |
1894 | - def _get_list_of_ppas(self, private=False): |
1895 | - """Get the list of PPAs to be polled. |
1896 | - |
1897 | - :param private boolean: Whether the list will contain private |
1898 | - or non-private PPAs. Default is False. |
1899 | - |
1900 | - :rtype list: The list of PPAs. This list will contain one |
1901 | - dictionary per PPA, with the following keys: |
1902 | - { "ppauser" : string, |
1903 | - "ppaname" : string, |
1904 | - "isprivateppa" : string } |
1905 | - """ |
1906 | - ppafile = os.path.expanduser("~/.config/ubuntu-debuginfod/ppalist") |
1907 | - if private: |
1908 | - ppafile += "-private" |
1909 | - |
1910 | - self._logger.debug(f"Processing '{ppafile}'") |
1911 | - |
1912 | - if not os.path.isfile(ppafile): |
1913 | - self._logger.info(f"{ppafile} does not exist; doing nothing") |
1914 | - return [] |
1915 | - |
1916 | - ppas = [] |
1917 | - with open(ppafile, "r", encoding="UTF-8") as f: |
1918 | - for ppa in f.readlines(): |
1919 | - ppa = ppa.rstrip() |
1920 | - if ppa == "" or ppa.startswith("#"): |
1921 | - continue |
1922 | - ppa = ppa.split(":")[1].split("/") |
1923 | - ppa_user = ppa[0] |
1924 | - ppa_name = ppa[1] |
1925 | - self._logger.info(f"Processing ppa:{ppa_user}/{ppa_name}") |
1926 | - ppas.append( |
1927 | - { |
1928 | - "ppauser": ppa_user, |
1929 | - "ppaname": ppa_name, |
1930 | - "isprivateppa": "yes" if private else "no", |
1931 | - } |
1932 | - ) |
1933 | - return ppas |
1934 | - |
1935 | - def get_ddebs(self, private=False): |
1936 | - """Get the list of ddebs from Launchpad PPAs. |
1937 | - |
1938 | - :param private boolean: Whether the PPAs are private or not. |
1939 | - Default is False. |
1940 | - |
1941 | - :rtype dict, datetime.datetime: |
1942 | - Return a dictionary containing all source packages found, and |
1943 | - also the new timestamp that should then be recorded by calling |
1944 | - record_timestamp.""" |
1945 | - messages = [] |
1946 | - new_timestamp = self._generate_timestamp() |
1947 | - for ppa in self._get_list_of_ppas(private=private): |
1948 | - ppauser = ppa["ppauser"] |
1949 | - ppaname = ppa["ppaname"] |
1950 | - try: |
1951 | - lpuser = self._lp.people[ppauser] |
1952 | - except KeyError: |
1953 | - self._logger.error(f"Launchpad user {ppauser} does not exist") |
1954 | - continue |
1955 | - |
1956 | - try: |
1957 | - lpppa = lpuser.getPPAByName(name=ppaname) |
1958 | - except lazr.restfulclient.errors.NotFound: |
1959 | - self._logger.error( |
1960 | - f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist" |
1961 | - ) |
1962 | - continue |
1963 | - |
1964 | - self._logger.info(f"Getting list of ddebs from ppa:{ppauser}/{ppaname}") |
1965 | - msg, ts = super().get_ddebs(archive=lpppa, ppainfo=ppa) |
1966 | - messages += msg |
1967 | - |
1968 | - return messages, new_timestamp |
1969 | - |
1970 | - def get_sources(self, private=False): |
1971 | - """Get the list of source packages from Launchpad PPAs. |
1972 | - |
1973 | - :param private boolean: Whether the PPAs are private or not. |
1974 | - Default is False. |
1975 | - |
1976 | - :rtype dict, datetime.datetime: |
1977 | - Return a dictionary containing all source packages found, and |
1978 | - also the new timestamp that should then be recorded by calling |
1979 | - record_timestamp. |
1980 | - |
1981 | - """ |
1982 | - messages = [] |
1983 | - new_timestamp = self._generate_timestamp() |
1984 | - for ppa in self._get_list_of_ppas(private=private): |
1985 | - ppauser = ppa["ppauser"] |
1986 | - ppaname = ppa["ppaname"] |
1987 | - try: |
1988 | - lpuser = self._lp.people[ppauser] |
1989 | - except KeyError: |
1990 | - self._logger.error(f"Launchpad user {ppauser} does not exist") |
1991 | - continue |
1992 | - |
1993 | - try: |
1994 | - lpppa = lpuser.getPPAByName(name=ppaname) |
1995 | - except lazr.restfulclient.errors.NotFound: |
1996 | - self._logger.error( |
1997 | - f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist" |
1998 | - ) |
1999 | - continue |
2000 | - |
2001 | - self._logger.info(f"Getting list of source packages from ppa:{ppauser}/{ppaname}") |
2002 | - msg, ts = super().get_sources(archive=lpppa, ppainfo=ppa) |
2003 | - messages += msg |
2004 | - |
2005 | - return messages, new_timestamp |
2006 | - |
2007 | - |
2008 | -class PrivatePPAPoller(PPAPoller): |
2009 | - """Perform the Launchpad polling and obtain a list of ddebs and source |
2010 | - packages from a private PPA.""" |
2011 | - |
2012 | - def __init__( |
2013 | - self, |
2014 | - initial_interval=1, |
2015 | - force_initial_interval=False, |
2016 | - fetch_all_on_first_run=True, |
2017 | - dry_run=False, |
2018 | - anonymous=False, |
2019 | - ): |
2020 | - """Initialize the object using 'privateppa' as its name. |
2021 | - |
2022 | - Look at DebugPoller's docstring for an explanation about the arguments.""" |
2023 | - super().__init__( |
2024 | - initial_interval=initial_interval, |
2025 | - force_initial_interval=force_initial_interval, |
2026 | - fetch_all_on_first_run=fetch_all_on_first_run, |
2027 | - dry_run=dry_run, |
2028 | - anonymous=anonymous, |
2029 | - ) |
2030 | - |
2031 | - def get_ddebs(self): |
2032 | - """Get the list of ddebs from private Launchpad PPAs.""" |
2033 | - return super().get_ddebs(private=True) |
2034 | - |
2035 | - def get_sources(self): |
2036 | - """Get the list of source packages from private Launchpad PPAs.""" |
2037 | - return super().get_sources(private=True) |
2038 | diff --git a/pyproject.toml b/pyproject.toml |
2039 | new file mode 100644 |
2040 | index 0000000..b16fcbd |
2041 | --- /dev/null |
2042 | +++ b/pyproject.toml |
2043 | @@ -0,0 +1,40 @@ |
2044 | +[build-system] |
2045 | +requires = [ "setuptools >= 61.0.0" ] |
2046 | +build-backend = "setuptools.build_meta" |
2047 | + |
2048 | +[project] |
2049 | +name = "ubuntu_debuginfod" |
2050 | +version = "0.1.0" |
2051 | +authors = [ |
2052 | + { name="Sergio Durigan Junior", email="sergio.durigan@canonical.com" }, |
2053 | +] |
2054 | +keywords = [ "ubuntu", "debuginfod" ] |
2055 | +readme = "README.md" |
2056 | +requires-python = ">= 3.7" |
2057 | +classifiers = [ |
2058 | + "Programming Language :: Python :: 3", |
2059 | + "Development Status :: 4 - Beta", |
2060 | + "Framework :: Celery", |
2061 | + "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)", |
2062 | + "Operating System :: POSIX :: Linux", |
2063 | + "Topic :: Software Development :: Debuggers", |
2064 | +] |
2065 | +dependencies = [ |
2066 | + "GitPython", |
2067 | + "celery", |
2068 | + "launchpadlib", |
2069 | + "requests", |
2070 | + "requests-oauthlib", |
2071 | + "sdnotify", |
2072 | + "psycopg2", |
2073 | +] |
2074 | + |
2075 | +[project.optional-dependencies] |
2076 | +tests = [ |
2077 | + "pytest", |
2078 | + "freezegun", |
2079 | +] |
2080 | + |
2081 | +[project.urls] |
2082 | +"Homepage" = "https://launchpad.net/ubuntu-debuginfod" |
2083 | +"Bug Tracker" = "https://bugs.launchpad.net/ubuntu-debuginfod" |
2084 | diff --git a/services/launchpad-dispatcher.py b/services/launchpad-dispatcher.py |
2085 | new file mode 100755 |
2086 | index 0000000..81bb754 |
2087 | --- /dev/null |
2088 | +++ b/services/launchpad-dispatcher.py |
2089 | @@ -0,0 +1,451 @@ |
2090 | +#!/usr/bin/python3 |
2091 | + |
2092 | +# Copyright (C) 2022 Canonical Ltd. |
2093 | + |
2094 | +# This program is free software: you can redistribute it and/or modify |
2095 | +# it under the terms of the GNU General Public License as published by |
2096 | +# the Free Software Foundation, either version 3 of the License, or |
2097 | +# (at your option) any later version. |
2098 | + |
2099 | +# This program is distributed in the hope that it will be useful, |
2100 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2101 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2102 | +# GNU General Public License for more details. |
2103 | + |
2104 | +# You should have received a copy of the GNU General Public License |
2105 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
2106 | + |
2107 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
2108 | + |
2109 | +import os |
2110 | +import time |
2111 | +import logging |
2112 | +from urllib import parse |
2113 | +import lazr.restfulclient.errors |
2114 | +import tempfile |
2115 | +import gzip |
2116 | +import re |
2117 | +from ubuntu_debuginfod.utils.debugdb import DebugDB |
2118 | +from ubuntu_debuginfod.utils.utils import download_from_lp, obtain_lp_oauth_credentials |
2119 | +from ubuntu_debuginfod.utils.baselp import BaseLaunchpadClass |
2120 | +from ubuntu_debuginfod.debuginfod import ( |
2121 | + grab_ddebs, |
2122 | + grab_sources, |
2123 | + grab_ppa_ddebs, |
2124 | + grab_ppa_sources, |
2125 | + grab_private_ppa_ddebs, |
2126 | + grab_private_ppa_sources, |
2127 | +) |
2128 | + |
2129 | + |
2130 | +def ensure_task(msg, db, cursor): |
2131 | + """Return the task id if the task described by MSG should be |
2132 | + processed. |
2133 | + |
2134 | + Also, if true, then insert the task information into the database. |
2135 | + |
2136 | + :param DebugDB db: The database connection handler. |
2137 | + |
2138 | + :param psycopg2.cursor cursor: The cursor to the SQL database. |
2139 | + |
2140 | + :param dict[str -> str] msg: The message describing a task. |
2141 | + |
2142 | + :rtype: int |
2143 | + :return: The taskid if the task should be processed (or None otherwise).""" |
2144 | + if msg["architecture"] == "source": |
2145 | + pkgname = msg["source_package"] |
2146 | + else: |
2147 | + pkgname = msg["binary_package"] |
2148 | + pkgver = msg["version"] |
2149 | + architecture = msg["architecture"] |
2150 | + |
2151 | + if "ppauser" in msg.keys(): |
2152 | + sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = TRUE AND ppauser = %s AND ppaname = %s" |
2153 | + sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, ppauser, ppaname, created_date) VALUES (%s, %s, %s, TRUE, %s, %s, NOW()) RETURNING id" |
2154 | + sqlparams = (pkgname, pkgver, architecture, msg["ppauser"], msg["ppaname"]) |
2155 | + else: |
2156 | + sql_select = "SELECT * FROM tasks WHERE source = %s AND version = %s AND arch = %s AND isfromppa = FALSE" |
2157 | + sql_insert = "INSERT INTO tasks (source, version, arch, isfromppa, created_date) VALUES (%s, %s, %s, FALSE, NOW()) RETURNING id" |
2158 | + sqlparams = (pkgname, pkgver, architecture) |
2159 | + |
2160 | + cursor.execute(sql_select, sqlparams) |
2161 | + |
2162 | + if cursor.rowcount > 0: |
2163 | + return None |
2164 | + |
2165 | + cursor.execute(sql_insert, sqlparams) |
2166 | + |
2167 | + taskid = cursor.fetchone()[0] |
2168 | + db.commit() |
2169 | + return taskid |
2170 | + |
2171 | + |
2172 | +class LaunchpadDispatcher(BaseLaunchpadClass): |
2173 | + """Dispatch Launchpad jobs to be processed by the |
2174 | + ubuntu-debuginfod getters.""" |
2175 | + |
2176 | + SOURCE_PACKAGE_INDEXABLE_REGEX = re.compile(r"-fdebug-prefix-map=[^=]+=/usr/src/") |
2177 | + |
2178 | + def __init__(self, anonymous): |
2179 | + """Initialize class. |
2180 | + |
2181 | + :param bool anonymous: Whether we should log into Launchpad |
2182 | + anonymously or not.""" |
2183 | + self._anonymous = anonymous |
2184 | + self._logger = logging.getLogger(__name__) |
2185 | + self.launchpad_login() |
2186 | + |
2187 | + def _get_pkg(self, uri): |
2188 | + """Obtain the Launchpad entry associated with URI. |
2189 | + |
2190 | + This function handles possible exceptions that might come from |
2191 | + Launchpad, and performs retries accordingly. |
2192 | + |
2193 | + :param str uri: The URI associated with the package. |
2194 | + |
2195 | + :rtype: lazr.restfulclient.resource.Entry |
2196 | + :return: The Launchpad object representing the package.""" |
2197 | + retry = 0 |
2198 | + while True: |
2199 | + try: |
2200 | + self._logger.debug(f"Obtaining package from URI '{uri}'") |
2201 | + pkg = self._lp.load(uri) |
2202 | + except ( |
2203 | + lazr.restfulclient.errors.BadRequest, |
2204 | + lazr.restfulclient.errors.HTTPError, |
2205 | + lazr.restfulclient.errors.ServerError, |
2206 | + lazr.restfulclient.errors.UnexpectedResponseError, |
2207 | + lazr.restfulclient.errors.ResponseError, |
2208 | + IndexError, |
2209 | + ) as e: |
2210 | + self._logger.warning( |
2211 | + f"Exception when obtaining information from URI '{uri}': {e}" |
2212 | + ) |
2213 | + if retry >= 4: |
2214 | + self._logger.warning( |
2215 | + f"Maximum number of retries exceeded. Raising {e}" |
2216 | + ) |
2217 | + raise e |
2218 | + retry += 1 |
2219 | + time.sleep(2) |
2220 | + continue |
2221 | + self._logger.debug(f"Obtained package information: '{pkg}'") |
2222 | + return pkg |
2223 | + |
2224 | + def _remove_id_from_database(self, jobid, db, cursor): |
2225 | + """Helper function to remove a job ID from the database. |
2226 | + |
2227 | + This function is invoked when the job is either invalid or |
2228 | + complete. |
2229 | + |
2230 | + :param str jobid: The job ID. |
2231 | + |
2232 | + :param DebugDB db: The database handler. |
2233 | + |
2234 | + :param psycopg2.cursor cursor: The cursor to the database.""" |
2235 | + self._logger.debug(f"Removing {jobid} from dispatcher table") |
2236 | + cursor.execute("DELETE FROM jobs2dispatch WHERE id = %s", (jobid,)) |
2237 | + db.commit() |
2238 | + |
2239 | + def _dispatch_binpkg_job(self, pkg, jobid, uri, ppainfo, db, cursor): |
2240 | + """Dispatch a job for a binary package package. |
2241 | + |
2242 | + :param lazr.restfulclient.resource.Entry pkg: The binary |
2243 | + package object. |
2244 | + |
2245 | + :param str jobid: The job ID. |
2246 | + |
2247 | + :param str uri: The URI for this artifact. |
2248 | + |
2249 | + :param dict[str -> str] ppainfo: The PPA information, if |
2250 | + applicable. |
2251 | + |
2252 | + :param DebugDB db: The database handler. |
2253 | + |
2254 | + :param psycopg2.cursor cursor: The cursor to the database. |
2255 | + |
2256 | + :rtype bool: |
2257 | + :return: True if the package should be removed from the |
2258 | + database, False otherwise.""" |
2259 | + self._logger.debug( |
2260 | + f"Processing binary package {pkg}, with ID {jobid}, ppainfo = {ppainfo}" |
2261 | + ) |
2262 | + if not pkg.is_debug: |
2263 | + self._logger.debug( |
2264 | + f"Package {pkg} is not a debug package; nothing to be done" |
2265 | + ) |
2266 | + return True |
2267 | + |
2268 | + ddeb_urls = pkg.binaryFileUrls() |
2269 | + if len(ddeb_urls) == 0: |
2270 | + self._logger.debug( |
2271 | + f"Package {pkg} does not have binaryFileUrls associated with it" |
2272 | + ) |
2273 | + return True |
2274 | + |
2275 | + msg = { |
2276 | + "uri": uri, |
2277 | + "source_package": pkg.source_package_name, |
2278 | + "binary_package": pkg.binary_package_name, |
2279 | + "version": pkg.binary_package_version, |
2280 | + "component": pkg.component_name, |
2281 | + "ddeb_url": "", |
2282 | + "ddeb_filename": "", |
2283 | + "architecture": "", |
2284 | + } |
2285 | + if ppainfo is not None: |
2286 | + msg.update(ppainfo) |
2287 | + |
2288 | + for url in ddeb_urls: |
2289 | + # Obtain the ddeb filename and its architecture. |
2290 | + ddeb = os.path.basename(parse.urlparse(url).path) |
2291 | + _, _, arch_and_extension = ddeb.split("_") |
2292 | + arch, _ = arch_and_extension.split(".") |
2293 | + |
2294 | + msg["ddeb_url"] = url |
2295 | + msg["ddeb_filename"] = ddeb |
2296 | + msg["architecture"] = arch |
2297 | + |
2298 | + taskid = ensure_task(msg, db, cursor) |
2299 | + if taskid is not None: |
2300 | + msg["taskid"] = str(taskid) |
2301 | + self._logger.debug(f"Dispatching the following message: {msg}") |
2302 | + # Do the actual dispatching here. |
2303 | + if ppainfo is None: |
2304 | + grab_ddebs.delay(msg) |
2305 | + elif ppainfo["isprivateppa"] == "no": |
2306 | + grab_ppa_ddebs.delay(msg) |
2307 | + else: |
2308 | + grab_private_ppa_ddebs.delay(msg) |
2309 | + |
2310 | + return True |
2311 | + |
2312 | + def _srcpkg_is_indexable(self, binpkg): |
2313 | + """Check if the source package associated with the binary |
2314 | + package PKG is indexable, and mark it as such in the database. |
2315 | + |
2316 | + This approach has a few downsides. Namely, it relies on the |
2317 | + verboseness of build log files. If the source package |
2318 | + "prettifies" the output of "make", we're doomed. However, |
2319 | + this will have to do for now until we come up with a better |
2320 | + way to examine the ddebs. |
2321 | + |
2322 | + :param lazr.restfulclient.resource.Entry binpkg: A Launchpad |
2323 | + object to a binary package associated with the source |
2324 | + package to be examined. |
2325 | + |
2326 | + :rtype: bool |
2327 | + :return: True is the source package is indexable by |
2328 | + debuginfod, False otherwise.""" |
2329 | + if not self._anonymous: |
2330 | + # It's currently impossible for us to check build logs of |
2331 | + # anonymous PPAs, so we just assume the source is |
2332 | + # indexable. This should not be a big problem. |
2333 | + return True |
2334 | + |
2335 | + credentials = None if self._anonymous else obtain_lp_oauth_credentials() |
2336 | + log_url = self._lp.load(binpkg.build_link).build_log_url |
2337 | + if not log_url: |
2338 | + return False |
2339 | + with tempfile.TemporaryDirectory() as tempdir: |
2340 | + logfile = download_from_lp(log_url, tempdir, credentials=credentials) |
2341 | + if not logfile.endswith(".txt.gz"): |
2342 | + return False |
2343 | + try: |
2344 | + with gzip.open(logfile, "rt") as f: |
2345 | + for line in f: |
2346 | + if re.search(self.SOURCE_PACKAGE_INDEXABLE_REGEX, line): |
2347 | + return True |
2348 | + except gzip.BadGzipFile: |
2349 | + self._logger.warning(f"Could not un-gzip {logfile} (URL: {log_url})") |
2350 | + return False |
2351 | + |
2352 | + def _dispatch_srcpkg_job(self, pkg, jobid, uri, ppainfo, db, cursor): |
2353 | + """Process a job for a source package. |
2354 | + |
2355 | + :param lazr.restfulclient.resource.Entry pkg: The source |
2356 | + package object. |
2357 | + |
2358 | + :param str jobid: The job ID. |
2359 | + |
2360 | + :param str uri: The URI for this artifact. |
2361 | + |
2362 | + :param dict[str -> str] ppainfo: The PPA information, if |
2363 | + applicable. |
2364 | + |
2365 | + :param DebugDB db: The database handler. |
2366 | + |
2367 | + :param psycopg2.cursor cursor: The cursor to the database. |
2368 | + |
2369 | + :rtype bool: |
2370 | + :return: True if the package should be removed from the |
2371 | + database, False otherwise.""" |
2372 | + self._logger.debug( |
2373 | + f"Processing source package {pkg}, with ID {jobid}, ppainfo = {ppainfo}" |
2374 | + ) |
2375 | + |
2376 | + srcpkgname = pkg.source_package_name |
2377 | + if srcpkgname.startswith("linux"): |
2378 | + # We know Linux tarballs are not indexable, so let's save |
2379 | + # time. |
2380 | + return True |
2381 | + |
2382 | + src_urls = pkg.sourceFileUrls() |
2383 | + if len(src_urls) == 0: |
2384 | + self._logger.debug( |
2385 | + f"Package {pkg} does not have sourceFileUrls associated with it" |
2386 | + ) |
2387 | + return True |
2388 | + |
2389 | + # Check if the source package is actually indexable by |
2390 | + # debuginfod. |
2391 | + binpkgs = pkg.getPublishedBinaries() |
2392 | + if len(binpkgs) == 0: |
2393 | + # This means that no binary has been published yet. We |
2394 | + # return without removing the ID from the database in |
2395 | + # order to make sure this source package will be |
2396 | + # re-evaluated later. |
2397 | + return False |
2398 | + if not self._srcpkg_is_indexable(binpkgs[0]): |
2399 | + # The package is not indexable. Just remove it from the |
2400 | + # DB. |
2401 | + return True |
2402 | + |
2403 | + msg = { |
2404 | + "uri": uri, |
2405 | + "source_package": srcpkgname, |
2406 | + "version": pkg.source_package_version, |
2407 | + "component": pkg.component_name, |
2408 | + "source_urls": src_urls, |
2409 | + "architecture": "source", |
2410 | + } |
2411 | + |
2412 | + taskid = ensure_task(msg, db, cursor) |
2413 | + if taskid is not None: |
2414 | + msg["taskid"] = str(taskid) |
2415 | + self._logger.debug(f"Dispatching the following message: {msg}") |
2416 | + if ppainfo is None: |
2417 | + grab_sources.delay(msg) |
2418 | + else: |
2419 | + msg.update(ppainfo) |
2420 | + if ppainfo["isprivateppa"] == "no": |
2421 | + grab_ppa_sources.delay(msg) |
2422 | + else: |
2423 | + grab_private_ppa_sources.delay(msg) |
2424 | + |
2425 | + return True |
2426 | + |
2427 | + def _dispatch_pkg_job(self, pkg, jobid, uri, ppainfo, db, cursor): |
2428 | + """Process a job for a package. |
2429 | + |
2430 | + This is a generic function that will call the respective |
2431 | + _dispatch_{srcpkg,binpkg}_job methods when applicable. |
2432 | + |
2433 | + :param lazr.restfulclient.resource.Entry pkg: The package |
2434 | + object. |
2435 | + |
2436 | + :param str jobid: The job ID. |
2437 | + |
2438 | + :param str uri: The URI for this job. |
2439 | + |
2440 | + :param dict[str -> str] ppainfo: The PPA information, if |
2441 | + applicable. |
2442 | + |
2443 | + :param DebugDB db: The database handler. |
2444 | + |
2445 | + :param psycopg2.cursor cursor: The cursor to the database.""" |
2446 | + # We're only interested in packages that have been published |
2447 | + # or are pending publication. |
2448 | + if pkg.status in ( |
2449 | + "Published", |
2450 | + "Pending", |
2451 | + ): |
2452 | + if "/+binarypub/" in uri: |
2453 | + self._logger.debug(f"Binary package {pkg} (ID {jobid}) detected") |
2454 | + if not self._dispatch_binpkg_job(pkg, jobid, uri, ppainfo, db, cursor): |
2455 | + return |
2456 | + elif "/+sourcepub/" in uri: |
2457 | + self._logger.debug(f"Source package {pkg} (ID {jobid}) detected") |
2458 | + if not self._dispatch_srcpkg_job(pkg, jobid, uri, ppainfo, db, cursor): |
2459 | + return |
2460 | + else: |
2461 | + raise ValueError(f"Invalid URI when processing package '{pkg}': {uri}") |
2462 | + |
2463 | + self._remove_id_from_database(jobid, db, cursor) |
2464 | + |
2465 | + def dispatch_jobs(self, db, cursor): |
2466 | + """Dispatch jobs. |
2467 | + |
2468 | + This function goes over the existing jobs in the |
2469 | + "jobs2dispatch" table and filters out those jobs that are |
2470 | + interesting to debuginfod. This means that the job must: |
2471 | + |
2472 | + * Be related to debuginfo (ddebs) or source packages. |
2473 | + |
2474 | + * Be public or private, depending on which dispatcher called |
2475 | + us. |
2476 | + |
2477 | + * Be in a valid publication state (published or pending). |
2478 | + |
2479 | + :param DebugDB db: The database handler. |
2480 | + |
2481 | + :param psycopg2.cursor cursor: The cursor to the database.""" |
2482 | + cursor.execute( |
2483 | + "SELECT id, uri, isfromppa, ppauser, ppaname FROM jobs2dispatch WHERE private = %s", |
2484 | + (not self._anonymous,), |
2485 | + ) |
2486 | + results = cursor.fetchall() |
2487 | + self._logger.info(f"Starting to dispatch {len(results)} entries") |
2488 | + for (jobid, uri, isfromppa, ppauser, ppaname) in results: |
2489 | + try: |
2490 | + pkg = self._get_pkg(uri) |
2491 | + except Exception as e: |
2492 | + # Record the error and continue processing. We don't |
2493 | + # delete the entry from the table in case we decide to |
2494 | + # retry it later. |
2495 | + cursor.execute( |
2496 | + "INSERT INTO errors VALUES (%s, %s, NOW())", (uri, str(e)) |
2497 | + ) |
2498 | + continue |
2499 | + |
2500 | + ppainfo = None |
2501 | + if isfromppa: |
2502 | + ppainfo = { |
2503 | + "ppauser": ppauser, |
2504 | + "ppaname": ppaname, |
2505 | + "isprivateppa": "no" if self._anonymous else "yes", |
2506 | + } |
2507 | + self._dispatch_pkg_job(pkg, jobid, uri, ppainfo, db, cursor) |
2508 | + |
2509 | + |
2510 | +class PublicLaunchpadDispatcher(LaunchpadDispatcher): |
2511 | + """Dispatch Launchpad public jobs. |
2512 | + |
2513 | + Public here means that we don't need to log into Launchpad using |
2514 | + credentials.""" |
2515 | + |
2516 | + def __init__(self): |
2517 | + super().__init__(anonymous=True) |
2518 | + |
2519 | + |
2520 | +class PrivateLaunchpadDispatcher(LaunchpadDispatcher): |
2521 | + """Dispatch Launchpad private jobs. |
2522 | + |
2523 | + Private here means that we need to log into Launchpad using |
2524 | + credentials.""" |
2525 | + |
2526 | + def __init__(self): |
2527 | + super().__init__(anonymous=False) |
2528 | + |
2529 | + |
2530 | +def main(): |
2531 | + # Connect to the database. |
2532 | + with DebugDB() as db: |
2533 | + db.create_tables() |
2534 | + with db.cursor() as cursor: |
2535 | + PublicLaunchpadDispatcher().dispatch_jobs(db, cursor) |
2536 | + PrivateLaunchpadDispatcher().dispatch_jobs(db, cursor) |
2537 | + |
2538 | + |
2539 | +if __name__ == "__main__": |
2540 | + main() |
2541 | diff --git a/services/launchpad-poller.py b/services/launchpad-poller.py |
2542 | new file mode 100755 |
2543 | index 0000000..e38f549 |
2544 | --- /dev/null |
2545 | +++ b/services/launchpad-poller.py |
2546 | @@ -0,0 +1,156 @@ |
2547 | +#!/usr/bin/python3 |
2548 | + |
2549 | +# Copyright (C) 2022 Canonical Ltd. |
2550 | + |
2551 | +# This program is free software: you can redistribute it and/or modify |
2552 | +# it under the terms of the GNU General Public License as published by |
2553 | +# the Free Software Foundation, either version 3 of the License, or |
2554 | +# (at your option) any later version. |
2555 | + |
2556 | +# This program is distributed in the hope that it will be useful, |
2557 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2558 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2559 | +# GNU General Public License for more details. |
2560 | + |
2561 | +# You should have received a copy of the GNU General Public License |
2562 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
2563 | + |
2564 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
2565 | + |
2566 | +import os |
2567 | +from ubuntu_debuginfod.pollers.ddebpoller import DdebPoller |
2568 | +from ubuntu_debuginfod.pollers.sourcepackagepoller import SourcePackagePoller |
2569 | +from ubuntu_debuginfod.pollers.ppaddebpoller import PPADdebPoller, PrivatePPADdebPoller |
2570 | +from ubuntu_debuginfod.pollers.ppasourcepackagepoller import ( |
2571 | + PPASourcePackagePoller, |
2572 | + PrivatePPASourcePackagePoller, |
2573 | +) |
2574 | +from ubuntu_debuginfod.utils.debugdb import DebugDB |
2575 | + |
2576 | + |
2577 | +def _process_msg(msg, db, cursor): |
2578 | + """Add a task to the database to download the source or binary ddebs, |
2579 | + if it isn't there already. |
2580 | + |
2581 | + :param str msg: The message to be processed. |
2582 | + |
2583 | + :param DebugDB db: The database connection handler. |
2584 | + |
2585 | + :param psycopg2.cursor cursor: The cursor to the database. |
2586 | + |
2587 | + """ |
2588 | + if "ppauser" in msg.keys(): |
2589 | + sqlinsert = "INSERT INTO jobs2dispatch (uri, private, isfromppa, ppauser, ppaname, date) VALUES (%s, %s, %s, %s, %s, NOW())" |
2590 | + sqlvalues = ( |
2591 | + msg["uri"], |
2592 | + msg["isprivateppa"] == "yes", |
2593 | + True, |
2594 | + msg["ppauser"], |
2595 | + msg["ppaname"], |
2596 | + ) |
2597 | + else: |
2598 | + sqlinsert = "INSERT INTO jobs2dispatch (uri, private, isfromppa, date) VALUES (%s, %s, %s, NOW())" |
2599 | + sqlvalues = (msg["uri"], False, False) |
2600 | + |
2601 | + cursor.execute(sqlinsert, sqlvalues) |
2602 | + db.commit() |
2603 | + |
2604 | + |
2605 | +def _poll_launchpad_generic(poller, db, cursor): |
2606 | + """Poll Launchpad and process the list of ddebs and source packages. |
2607 | + |
2608 | + :param DebugDB db: The SQL connection handler. |
2609 | + |
2610 | + :param psycopg2.cursor cursor: The cursor to the SQL database.""" |
2611 | + |
2612 | + messages, timestamp = poller.poll_artifacts() |
2613 | + for msg in messages: |
2614 | + _process_msg(msg, db, cursor) |
2615 | + |
2616 | + if timestamp is not None: |
2617 | + poller.record_timestamp(timestamp) |
2618 | + # # Process ddebs. |
2619 | + # messages, new_timestamp_ddebs = poller.get_ddebs() |
2620 | + # for msg in messages: |
2621 | + # _process_msg(msg, db, cursor) |
2622 | + |
2623 | + # # Process source packages. |
2624 | + # messages, new_timestamp_src = poller.get_sources() |
2625 | + # for msg in messages: |
2626 | + # _process_msg(msg, db, cursor) |
2627 | + |
2628 | + # # Choose the earliest timestamp. |
2629 | + # new_timestamp = min(new_timestamp_ddebs, new_timestamp_src) |
2630 | + |
2631 | + # poller.record_timestamp(new_timestamp) |
2632 | + |
2633 | + |
2634 | +def _should_poll_archive(archive): |
2635 | + """Whether we should poll a certain archive. |
2636 | + |
2637 | + This function relies on the value of environment variables named |
2638 | + "POLL_{archive}". See the /etc/default/ubuntu-debuginfod-poll-lp |
2639 | + file for more details. |
2640 | + |
2641 | + :param archive string: The archive name, in uppercase. |
2642 | + |
2643 | + :rtype: boolean |
2644 | + |
2645 | + :return: True if we should poll the archive, False otherwise.""" |
2646 | + return os.environ.get(f"POLL_{archive.upper()}") == "yes" |
2647 | + |
2648 | + |
2649 | +def poll_launchpad(db, cursor): |
2650 | + """Poll Launchpad and process the list of ddebs and source packages |
2651 | + from the main archive. |
2652 | + |
2653 | + :param DebugDB db: The SQL connection handler. |
2654 | + |
2655 | + :param psycopg2.cursor cursor: The cursor to the SQL database. |
2656 | + |
2657 | + """ |
2658 | + if _should_poll_archive("MAIN_ARCHIVE"): |
2659 | + _poll_launchpad_generic(DdebPoller(), db, cursor) |
2660 | + _poll_launchpad_generic(SourcePackagePoller(), db, cursor) |
2661 | + |
2662 | + |
2663 | +def poll_launchpad_ppas(db, cursor): |
2664 | + """Poll Launchpad and process the list of ddebs and source packages |
2665 | + from PPAs. |
2666 | + |
2667 | + :param DebugDB db: The SQL connection handler. |
2668 | + |
2669 | + :param psycopg2.cursor cursor: The cursor to the SQL database. |
2670 | + |
2671 | + """ |
2672 | + if _should_poll_archive("PPAS"): |
2673 | + _poll_launchpad_generic(PPADdebPoller(), db, cursor) |
2674 | + _poll_launchpad_generic(PPASourcePackagePoller(), db, cursor) |
2675 | + |
2676 | + |
2677 | +def poll_launchpad_private_ppas(db, cursor): |
2678 | + """Poll Launchpad and process the list of ddebs and source packages |
2679 | + from private PPAs. |
2680 | + |
2681 | + :param DebugDB db: The SQL connection handler. |
2682 | + |
2683 | + :param psycopg2.cursor cursor: The cursor to the SQL database. |
2684 | + |
2685 | + """ |
2686 | + if _should_poll_archive("PRIVATE_PPAS"): |
2687 | + _poll_launchpad_generic(PrivatePPADdebPoller(), db, cursor) |
2688 | + _poll_launchpad_generic(PrivatePPASourcePackagePoller(), db, cursor) |
2689 | + |
2690 | + |
2691 | +def main(): |
2692 | + # Connect to the database. |
2693 | + with DebugDB() as db: |
2694 | + db.create_tables() |
2695 | + with db.cursor() as cursor: |
2696 | + poll_launchpad(db, cursor) |
2697 | + poll_launchpad_ppas(db, cursor) |
2698 | + poll_launchpad_private_ppas(db, cursor) |
2699 | + |
2700 | + |
2701 | +if __name__ == "__main__": |
2702 | + main() |
2703 | diff --git a/setup.cfg b/setup.cfg |
2704 | new file mode 100644 |
2705 | index 0000000..2f0184a |
2706 | --- /dev/null |
2707 | +++ b/setup.cfg |
2708 | @@ -0,0 +1,14 @@ |
2709 | +[options] |
2710 | +packages = find: |
2711 | +include_package_data = True |
2712 | + |
2713 | +[flake8] |
2714 | +# Ignore line-too-long |
2715 | +extend-ignore = E501 |
2716 | + |
2717 | +[mypy] |
2718 | +# Ignore warning about missing packages when assessing imports. |
2719 | +ignore_missing_imports = True |
2720 | + |
2721 | +[pylint.'MESSAGES CONTROL'] |
2722 | +disable = logging-fstring-interpolation,line-too-long,missing-module-docstring,too-many-arguments,import-error,invalid-name,too-few-public-methods,duplicate-code |
2723 | diff --git a/tests/test_baselp.py b/tests/test_baselp.py |
2724 | new file mode 100644 |
2725 | index 0000000..ba27670 |
2726 | --- /dev/null |
2727 | +++ b/tests/test_baselp.py |
2728 | @@ -0,0 +1,58 @@ |
2729 | +#!/usr/bin/python3 |
2730 | + |
2731 | +# Copyright (C) 2023 Canonical Ltd. |
2732 | + |
2733 | +# This program is free software: you can redistribute it and/or modify |
2734 | +# it under the terms of the GNU General Public License as published by |
2735 | +# the Free Software Foundation, either version 3 of the License, or |
2736 | +# (at your option) any later version. |
2737 | + |
2738 | +# This program is distributed in the hope that it will be useful, |
2739 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2740 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2741 | +# GNU General Public License for more details. |
2742 | + |
2743 | +# You should have received a copy of the GNU General Public License |
2744 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
2745 | + |
2746 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
2747 | + |
2748 | +from unittest.mock import patch |
2749 | +import unittest |
2750 | +from ubuntu_debuginfod.utils import baselp |
2751 | + |
2752 | + |
2753 | +class FakeArchive: |
2754 | + """A fake Launchpad archive object.""" |
2755 | + |
2756 | + def __init__(self): |
2757 | + self.main_archive = [] |
2758 | + |
2759 | + |
2760 | +class FakeLaunchpad: |
2761 | + """A fake Launchpad object.""" |
2762 | + |
2763 | + def __init__(self): |
2764 | + self.distributions = {"ubuntu": FakeArchive()} |
2765 | + |
2766 | + |
2767 | +@patch( |
2768 | + "launchpadlib.launchpad.Launchpad.login_anonymously", return_value=FakeLaunchpad() |
2769 | +) |
2770 | +@patch("launchpadlib.launchpad.Launchpad.login_with", return_value=FakeLaunchpad()) |
2771 | +class TestBaseLP(baselp.BaseLaunchpadClass): |
2772 | + def test_anonymous_login(self, mock_login_with, mock_login_anon): |
2773 | + self._anonymous = True |
2774 | + self.launchpad_login() |
2775 | + assert mock_login_anon.called is True |
2776 | + assert mock_login_with.called is False |
2777 | + |
2778 | + def test_non_anonymous_login(self, mock_login_with, mock_login_anon): |
2779 | + self._anonymous = False |
2780 | + self.launchpad_login() |
2781 | + assert mock_login_anon.called is False |
2782 | + assert mock_login_with.called is True |
2783 | + |
2784 | + |
2785 | +if __name__ == "__main__": |
2786 | + unittest.main() |
2787 | diff --git a/tests/test_common_lp_getter.py b/tests/test_common_lp_getter.py |
2788 | new file mode 100644 |
2789 | index 0000000..5663dc0 |
2790 | --- /dev/null |
2791 | +++ b/tests/test_common_lp_getter.py |
2792 | @@ -0,0 +1,91 @@ |
2793 | +#!/usr/bin/python3 |
2794 | + |
2795 | +# Copyright (C) 2023 Canonical Ltd. |
2796 | + |
2797 | +# This program is free software: you can redistribute it and/or modify |
2798 | +# it under the terms of the GNU General Public License as published by |
2799 | +# the Free Software Foundation, either version 3 of the License, or |
2800 | +# (at your option) any later version. |
2801 | + |
2802 | +# This program is distributed in the hope that it will be useful, |
2803 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2804 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2805 | +# GNU General Public License for more details. |
2806 | + |
2807 | +# You should have received a copy of the GNU General Public License |
2808 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
2809 | + |
2810 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
2811 | + |
2812 | +import os |
2813 | +import re |
2814 | +import unittest |
2815 | +import pytest |
2816 | +from ubuntu_debuginfod.getters import common_lp_getter |
2817 | + |
2818 | + |
2819 | +class TestCommonLaunchpadGetter: |
2820 | + |
2821 | + VALID_GENERIC_REQ = { |
2822 | + "architecture": "amd64", |
2823 | + "uri": "URL", |
2824 | + } |
2825 | + |
2826 | + def test_make_savepath_no_lib(self): |
2827 | + subdir = "test" |
2828 | + d = common_lp_getter.CommonLaunchpadGetter(subdir=subdir) |
2829 | + source_package = "foo" |
2830 | + component = "bar" |
2831 | + expected_path = os.path.join( |
2832 | + d.DEFAULT_MIRROR_DIR, subdir, component, source_package[0], source_package |
2833 | + ) |
2834 | + assert d.make_savepath(source_package, component) == expected_path |
2835 | + |
2836 | + def test_make_savepath_lib(self): |
2837 | + subdir = "test" |
2838 | + d = common_lp_getter.CommonLaunchpadGetter(subdir=subdir) |
2839 | + source_package = "libfoo" |
2840 | + component = "bar" |
2841 | + expected_path = os.path.join( |
2842 | + d.DEFAULT_MIRROR_DIR, subdir, component, source_package[:4], source_package |
2843 | + ) |
2844 | + assert d.make_savepath(source_package, component) == expected_path |
2845 | + |
2846 | + @pytest.mark.parametrize( |
2847 | + "req, exception, ex_message", |
2848 | + [ |
2849 | + (None, TypeError, "Invalid request"), |
2850 | + ({"foo": "bar"}, ValueError, "No 'architecture' in request"), |
2851 | + ( |
2852 | + {"architecture": None}, |
2853 | + ValueError, |
2854 | + "Invalid 'architecture' (None) in request", |
2855 | + ), |
2856 | + ({"architecture": "amd64"}, ValueError, "No 'uri' in request"), |
2857 | + ( |
2858 | + {"architecture": "amd64", "uri": None}, |
2859 | + ValueError, |
2860 | + "Invalid 'uri' (None) in request", |
2861 | + ), |
2862 | + ], |
2863 | + ) |
2864 | + def test_validate_request_invalid_request(self, req, exception, ex_message): |
2865 | + d = common_lp_getter.CommonLaunchpadGetter(subdir="test") |
2866 | + with pytest.raises(exception, match=re.escape(ex_message)): |
2867 | + d.validate_request(req) |
2868 | + |
2869 | + def test_validate_request_valid_request(self): |
2870 | + d = common_lp_getter.CommonLaunchpadGetter(subdir="test") |
2871 | + assert d.validate_request(self.VALID_GENERIC_REQ) is None |
2872 | + |
2873 | + def test_download_artifact_raises_notimplemented(self): |
2874 | + d = common_lp_getter.CommonLaunchpadGetter(subdir="test") |
2875 | + with pytest.raises( |
2876 | + NotImplementedError, |
2877 | + match=re.escape("Wrong call to 'download_artifact' from base class"), |
2878 | + ): |
2879 | + d.download_artifact() |
2880 | + |
2881 | + |
2882 | +if __name__ == "__main__": |
2883 | + unittest.main() |
2884 | diff --git a/tests/test_common_ppa_getter.py b/tests/test_common_ppa_getter.py |
2885 | new file mode 100644 |
2886 | index 0000000..8eb736a |
2887 | --- /dev/null |
2888 | +++ b/tests/test_common_ppa_getter.py |
2889 | @@ -0,0 +1,112 @@ |
2890 | +#!/usr/bin/python3 |
2891 | + |
2892 | +# Copyright (C) 2023 Canonical Ltd. |
2893 | + |
2894 | +# This program is free software: you can redistribute it and/or modify |
2895 | +# it under the terms of the GNU General Public License as published by |
2896 | +# the Free Software Foundation, either version 3 of the License, or |
2897 | +# (at your option) any later version. |
2898 | + |
2899 | +# This program is distributed in the hope that it will be useful, |
2900 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2901 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2902 | +# GNU General Public License for more details. |
2903 | + |
2904 | +# You should have received a copy of the GNU General Public License |
2905 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
2906 | + |
2907 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
2908 | + |
2909 | +import re |
2910 | +import pytest |
2911 | +import unittest |
2912 | +from ubuntu_debuginfod.getters import common_ppa_getter |
2913 | + |
2914 | + |
2915 | +class MockValidateRequest: |
2916 | + def validate_request(self, request): |
2917 | + pass |
2918 | + |
2919 | + |
2920 | +class MockCommonPPAGetter(common_ppa_getter.CommonPPAGetter, MockValidateRequest): |
2921 | + def __init__(self): |
2922 | + super().__init__() |
2923 | + |
2924 | + |
2925 | +class TestCommonPPAGetter: |
2926 | + @pytest.mark.parametrize( |
2927 | + "req, exception, ex_message", |
2928 | + [ |
2929 | + ( |
2930 | + {"architecture": "amd64", "uri": "URL", "ddeb_url": "URL"}, |
2931 | + TypeError, |
2932 | + "No 'ppauser' in request", |
2933 | + ), |
2934 | + ( |
2935 | + { |
2936 | + "architecture": "amd64", |
2937 | + "ddeb_url": "URL", |
2938 | + "uri": "URL", |
2939 | + "ppauser": "foo", |
2940 | + }, |
2941 | + TypeError, |
2942 | + "No 'ppaname' in request", |
2943 | + ), |
2944 | + ( |
2945 | + { |
2946 | + "architecture": "amd64", |
2947 | + "uri": "URL", |
2948 | + "ddeb_url": "URL", |
2949 | + "ppauser": "foo", |
2950 | + "ppaname": "bar", |
2951 | + }, |
2952 | + TypeError, |
2953 | + "No 'isprivateppa' in request", |
2954 | + ), |
2955 | + ( |
2956 | + { |
2957 | + "architecture": "amd64", |
2958 | + "uri": "URL", |
2959 | + "ddeb_url": "URL", |
2960 | + "ppauser": None, |
2961 | + "ppaname": "bar", |
2962 | + "isprivateppa": "no", |
2963 | + }, |
2964 | + ValueError, |
2965 | + "Invalid 'ppauser' (None) in request", |
2966 | + ), |
2967 | + ( |
2968 | + { |
2969 | + "architecture": "amd64", |
2970 | + "uri": "URL", |
2971 | + "ddeb_url": "URL", |
2972 | + "ppauser": "foo", |
2973 | + "ppaname": None, |
2974 | + "isprivateppa": "no", |
2975 | + }, |
2976 | + ValueError, |
2977 | + "Invalid 'ppaname' (None) in request", |
2978 | + ), |
2979 | + ( |
2980 | + { |
2981 | + "architecture": "amd64", |
2982 | + "uri": "URL", |
2983 | + "ddeb_url": "URL", |
2984 | + "ppauser": "foo", |
2985 | + "ppaname": "bar", |
2986 | + "isprivateppa": None, |
2987 | + }, |
2988 | + ValueError, |
2989 | + "Invalid 'isprivateppa' (None) in request", |
2990 | + ), |
2991 | + ], |
2992 | + ) |
2993 | + def test_validate_generic_ppa_request_invalid_request( |
2994 | + self, req, exception, ex_message |
2995 | + ): |
2996 | + with pytest.raises(exception, match=re.escape(ex_message)): |
2997 | + MockCommonPPAGetter().validate_request(req) |
2998 | + |
2999 | + |
3000 | +if __name__ == "__main__": |
3001 | + unittest.main() |
3002 | diff --git a/tests/test_ddebgetter.py b/tests/test_ddebgetter.py |
3003 | new file mode 100644 |
3004 | index 0000000..3a67894 |
3005 | --- /dev/null |
3006 | +++ b/tests/test_ddebgetter.py |
3007 | @@ -0,0 +1,64 @@ |
3008 | +#!/usr/bin/python3 |
3009 | + |
3010 | +# Copyright (C) 2023 Canonical Ltd. |
3011 | + |
3012 | +# This program is free software: you can redistribute it and/or modify |
3013 | +# it under the terms of the GNU General Public License as published by |
3014 | +# the Free Software Foundation, either version 3 of the License, or |
3015 | +# (at your option) any later version. |
3016 | + |
3017 | +# This program is distributed in the hope that it will be useful, |
3018 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3019 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3020 | +# GNU General Public License for more details. |
3021 | + |
3022 | +# You should have received a copy of the GNU General Public License |
3023 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3024 | + |
3025 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3026 | + |
3027 | +import re |
3028 | +import pytest |
3029 | +import unittest |
3030 | +from ubuntu_debuginfod.getters import ddebgetter |
3031 | + |
3032 | + |
3033 | +class TestDdebGetter: |
3034 | + |
3035 | + VALID_DDEB_REQ = { |
3036 | + "architecture": "amd64", |
3037 | + "ddeb_url": "URL", |
3038 | + "uri": "URL", |
3039 | + } |
3040 | + |
3041 | + def test_validate_ddeb_request_valid_request(self): |
3042 | + assert ddebgetter.DdebGetter().validate_request(self.VALID_DDEB_REQ) is None |
3043 | + |
3044 | + @pytest.mark.parametrize( |
3045 | + "req, exception, ex_message", |
3046 | + [ |
3047 | + (None, TypeError, "Invalid request"), |
3048 | + ( |
3049 | + {"architecture": "amd64", "uri": "URL", "foo": "bar"}, |
3050 | + TypeError, |
3051 | + "No 'ddeb_url' in request", |
3052 | + ), |
3053 | + ( |
3054 | + {"architecture": "amd64", "uri": "URL", "ddeb_url": None}, |
3055 | + ValueError, |
3056 | + "Invalid 'ddeb_url' (None) in request", |
3057 | + ), |
3058 | + ( |
3059 | + {"architecture": "source", "uri": "URL", "ddeb_url": "URL"}, |
3060 | + ValueError, |
3061 | + "Wrong request: source fetch", |
3062 | + ), |
3063 | + ], |
3064 | + ) |
3065 | + def test_validate_ddeb_request_invalid_request(self, req, exception, ex_message): |
3066 | + with pytest.raises(exception, match=re.escape(ex_message)): |
3067 | + ddebgetter.DdebGetter().validate_request(req) |
3068 | + |
3069 | + |
3070 | +if __name__ == "__main__": |
3071 | + unittest.main() |
3072 | diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py |
3073 | new file mode 100644 |
3074 | index 0000000..0a64163 |
3075 | --- /dev/null |
3076 | +++ b/tests/test_exceptions.py |
3077 | @@ -0,0 +1,33 @@ |
3078 | +#!/usr/bin/python3 |
3079 | + |
3080 | +# Copyright (C) 2023 Canonical Ltd. |
3081 | + |
3082 | +# This program is free software: you can redistribute it and/or modify |
3083 | +# it under the terms of the GNU General Public License as published by |
3084 | +# the Free Software Foundation, either version 3 of the License, or |
3085 | +# (at your option) any later version. |
3086 | + |
3087 | +# This program is distributed in the hope that it will be useful, |
3088 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3089 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3090 | +# GNU General Public License for more details. |
3091 | + |
3092 | +# You should have received a copy of the GNU General Public License |
3093 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3094 | + |
3095 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3096 | + |
3097 | +import unittest |
3098 | +from ubuntu_debuginfod.utils import exceptions |
3099 | + |
3100 | + |
3101 | +class TestExceptions: |
3102 | + def test_timeout_exception_inherits_ok(self): |
3103 | + assert issubclass(exceptions.UbuntuDebuginfodTimeout, Exception) |
3104 | + |
3105 | + def test_retry_exception_inherits_ok(self): |
3106 | + assert issubclass(exceptions.UbuntuDebuginfodRetry, Exception) |
3107 | + |
3108 | + |
3109 | +if __name__ == "__main__": |
3110 | + unittest.main() |
3111 | diff --git a/tests/test_getter.py b/tests/test_getter.py |
3112 | new file mode 100644 |
3113 | index 0000000..933370c |
3114 | --- /dev/null |
3115 | +++ b/tests/test_getter.py |
3116 | @@ -0,0 +1,38 @@ |
3117 | +#!/usr/bin/python3 |
3118 | + |
3119 | +# Copyright (C) 2023 Canonical Ltd. |
3120 | + |
3121 | +# This program is free software: you can redistribute it and/or modify |
3122 | +# it under the terms of the GNU General Public License as published by |
3123 | +# the Free Software Foundation, either version 3 of the License, or |
3124 | +# (at your option) any later version. |
3125 | + |
3126 | +# This program is distributed in the hope that it will be useful, |
3127 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3128 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3129 | +# GNU General Public License for more details. |
3130 | + |
3131 | +# You should have received a copy of the GNU General Public License |
3132 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3133 | + |
3134 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3135 | + |
3136 | +import unittest |
3137 | +from ubuntu_debuginfod import getter |
3138 | + |
3139 | + |
3140 | +class TestGetter: |
3141 | + def test_getter_get_subdir(self): |
3142 | + subdir = "test" |
3143 | + d = getter.Getter(subdir=subdir) |
3144 | + assert d.subdir == subdir |
3145 | + |
3146 | + def test_getter_set_subdir(self): |
3147 | + d = getter.Getter(subdir="test") |
3148 | + newsubdir = "newtest" |
3149 | + d.subdir = newsubdir |
3150 | + assert d.subdir == newsubdir |
3151 | + |
3152 | + |
3153 | +if __name__ == "__main__": |
3154 | + unittest.main() |
3155 | diff --git a/tests/test_ppaddebgetter.py b/tests/test_ppaddebgetter.py |
3156 | new file mode 100644 |
3157 | index 0000000..3ca3dd1 |
3158 | --- /dev/null |
3159 | +++ b/tests/test_ppaddebgetter.py |
3160 | @@ -0,0 +1,94 @@ |
3161 | +#!/usr/bin/python3 |
3162 | + |
3163 | +# Copyright (C) 2023 Canonical Ltd. |
3164 | + |
3165 | +# This program is free software: you can redistribute it and/or modify |
3166 | +# it under the terms of the GNU General Public License as published by |
3167 | +# the Free Software Foundation, either version 3 of the License, or |
3168 | +# (at your option) any later version. |
3169 | + |
3170 | +# This program is distributed in the hope that it will be useful, |
3171 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3172 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3173 | +# GNU General Public License for more details. |
3174 | + |
3175 | +# You should have received a copy of the GNU General Public License |
3176 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3177 | + |
3178 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3179 | + |
3180 | +import re |
3181 | +import pytest |
3182 | +import unittest |
3183 | +from ubuntu_debuginfod.getters import ppaddebgetter |
3184 | + |
3185 | + |
3186 | +class TestPPADdebGetter: |
3187 | + |
3188 | + VALID_PUBLIC_PPA_DDEB_REQ = { |
3189 | + "architecture": "amd64", |
3190 | + "ddeb_url": "URL", |
3191 | + "ppauser": "foo", |
3192 | + "ppaname": "bar", |
3193 | + "isprivateppa": "no", |
3194 | + "uri": "URL", |
3195 | + } |
3196 | + |
3197 | + VALID_PRIVATE_PPA_DDEB_REQ = { |
3198 | + "architecture": "amd64", |
3199 | + "ddeb_url": "URL", |
3200 | + "ppauser": "foo", |
3201 | + "ppaname": "bar", |
3202 | + "isprivateppa": "yes", |
3203 | + "uri": "URL", |
3204 | + } |
3205 | + |
3206 | + def test_validate_ppa_generic_valid_request(self): |
3207 | + assert ( |
3208 | + ppaddebgetter.PPADdebGetter().validate_request( |
3209 | + self.VALID_PUBLIC_PPA_DDEB_REQ |
3210 | + ) |
3211 | + is None |
3212 | + ) |
3213 | + assert ( |
3214 | + ppaddebgetter.PrivatePPADdebGetter().validate_request( |
3215 | + self.VALID_PRIVATE_PPA_DDEB_REQ |
3216 | + ) |
3217 | + is None |
3218 | + ) |
3219 | + |
3220 | + def test_validate_ppa_ddeb_request_valid_request(self): |
3221 | + assert ( |
3222 | + ppaddebgetter.PPADdebGetter().validate_request( |
3223 | + self.VALID_PUBLIC_PPA_DDEB_REQ |
3224 | + ) |
3225 | + is None |
3226 | + ) |
3227 | + |
3228 | + def test_validate_ppa_ddeb_request_invalid_request(self): |
3229 | + with pytest.raises( |
3230 | + ValueError, match=re.escape("Wrong request: from private PPA") |
3231 | + ): |
3232 | + ppaddebgetter.PPADdebGetter().validate_request( |
3233 | + self.VALID_PRIVATE_PPA_DDEB_REQ |
3234 | + ) |
3235 | + |
3236 | + def test_validate_private_ppa_ddeb_request_valid_request(self): |
3237 | + assert ( |
3238 | + ppaddebgetter.PrivatePPADdebGetter().validate_request( |
3239 | + self.VALID_PRIVATE_PPA_DDEB_REQ |
3240 | + ) |
3241 | + is None |
3242 | + ) |
3243 | + |
3244 | + def test_validate_private_ddeb_request_invalid_request(self): |
3245 | + with pytest.raises( |
3246 | + ValueError, match=re.escape("Wrong request: not from private PPA") |
3247 | + ): |
3248 | + ppaddebgetter.PrivatePPADdebGetter().validate_request( |
3249 | + self.VALID_PUBLIC_PPA_DDEB_REQ |
3250 | + ) |
3251 | + |
3252 | + |
3253 | +if __name__ == "__main__": |
3254 | + unittest.main() |
3255 | diff --git a/tests/test_ppasourcepackagegetter.py b/tests/test_ppasourcepackagegetter.py |
3256 | new file mode 100644 |
3257 | index 0000000..aec5b95 |
3258 | --- /dev/null |
3259 | +++ b/tests/test_ppasourcepackagegetter.py |
3260 | @@ -0,0 +1,94 @@ |
3261 | +#!/usr/bin/python3 |
3262 | + |
3263 | +# Copyright (C) 2023 Canonical Ltd. |
3264 | + |
3265 | +# This program is free software: you can redistribute it and/or modify |
3266 | +# it under the terms of the GNU General Public License as published by |
3267 | +# the Free Software Foundation, either version 3 of the License, or |
3268 | +# (at your option) any later version. |
3269 | + |
3270 | +# This program is distributed in the hope that it will be useful, |
3271 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3272 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3273 | +# GNU General Public License for more details. |
3274 | + |
3275 | +# You should have received a copy of the GNU General Public License |
3276 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3277 | + |
3278 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3279 | + |
3280 | +import re |
3281 | +import pytest |
3282 | +import unittest |
3283 | +from ubuntu_debuginfod.getters import ppasourcepackagegetter |
3284 | + |
3285 | + |
3286 | +class TestPPASourcePackageGetter: |
3287 | + |
3288 | + VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ = { |
3289 | + "architecture": "source", |
3290 | + "source_urls": ["URL1", "URL2"], |
3291 | + "ppauser": "foo", |
3292 | + "ppaname": "bar", |
3293 | + "isprivateppa": "no", |
3294 | + "uri": "URL", |
3295 | + } |
3296 | + |
3297 | + VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ = { |
3298 | + "architecture": "source", |
3299 | + "source_urls": ["URL1", "URL2"], |
3300 | + "ppauser": "foo", |
3301 | + "ppaname": "bar", |
3302 | + "isprivateppa": "yes", |
3303 | + "uri": "URL", |
3304 | + } |
3305 | + |
3306 | + def test_validate_request_valid_request(self): |
3307 | + assert ( |
3308 | + ppasourcepackagegetter.PPASourcePackageGetter().validate_request( |
3309 | + self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ |
3310 | + ) |
3311 | + is None |
3312 | + ) |
3313 | + assert ( |
3314 | + ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request( |
3315 | + self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ |
3316 | + ) |
3317 | + is None |
3318 | + ) |
3319 | + |
3320 | + def test_validate_ppa_source_code_request_valid_request(self): |
3321 | + assert ( |
3322 | + ppasourcepackagegetter.PPASourcePackageGetter().validate_request( |
3323 | + self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ |
3324 | + ) |
3325 | + is None |
3326 | + ) |
3327 | + |
3328 | + def test_validate_ppa_source_code_request_invalid_request(self): |
3329 | + with pytest.raises( |
3330 | + ValueError, match=re.escape("Wrong request: from private PPA") |
3331 | + ): |
3332 | + ppasourcepackagegetter.PPASourcePackageGetter().validate_request( |
3333 | + self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ |
3334 | + ) |
3335 | + |
3336 | + def test_validate_private_ppa_source_code_request_valid_request(self): |
3337 | + assert ( |
3338 | + ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request( |
3339 | + self.VALID_PRIVATE_PPA_SOURCE_PACKAGE_REQ |
3340 | + ) |
3341 | + is None |
3342 | + ) |
3343 | + |
3344 | + def test_validate_private_ppa_source_code_request_invalid_request(self): |
3345 | + with pytest.raises( |
3346 | + ValueError, match=re.escape("Wrong request: not from private PPA") |
3347 | + ): |
3348 | + ppasourcepackagegetter.PrivatePPASourcePackageGetter().validate_request( |
3349 | + self.VALID_PUBLIC_PPA_SOURCE_PACKAGE_REQ |
3350 | + ) |
3351 | + |
3352 | + |
3353 | +if __name__ == "__main__": |
3354 | + unittest.main() |
3355 | diff --git a/tests/test_sourcepackagegetter.py b/tests/test_sourcepackagegetter.py |
3356 | new file mode 100644 |
3357 | index 0000000..f82c4f7 |
3358 | --- /dev/null |
3359 | +++ b/tests/test_sourcepackagegetter.py |
3360 | @@ -0,0 +1,69 @@ |
3361 | +#!/usr/bin/python3 |
3362 | + |
3363 | +# Copyright (C) 2023 Canonical Ltd. |
3364 | + |
3365 | +# This program is free software: you can redistribute it and/or modify |
3366 | +# it under the terms of the GNU General Public License as published by |
3367 | +# the Free Software Foundation, either version 3 of the License, or |
3368 | +# (at your option) any later version. |
3369 | + |
3370 | +# This program is distributed in the hope that it will be useful, |
3371 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3372 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3373 | +# GNU General Public License for more details. |
3374 | + |
3375 | +# You should have received a copy of the GNU General Public License |
3376 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3377 | + |
3378 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3379 | + |
3380 | +import re |
3381 | +import pytest |
3382 | +import unittest |
3383 | +from ubuntu_debuginfod.getters import sourcepackagegetter |
3384 | + |
3385 | + |
3386 | +class TestSourcePackageGetter: |
3387 | + |
3388 | + VALID_SOURCE_PACKAGE_REQ = { |
3389 | + "architecture": "source", |
3390 | + "source_urls": [], |
3391 | + "uri": "URL", |
3392 | + } |
3393 | + |
3394 | + def test_validate_request_valid_request(self): |
3395 | + assert ( |
3396 | + sourcepackagegetter.SourcePackageGetter().validate_request( |
3397 | + self.VALID_SOURCE_PACKAGE_REQ |
3398 | + ) |
3399 | + is None |
3400 | + ) |
3401 | + |
3402 | + @pytest.mark.parametrize( |
3403 | + "req, exception, ex_message", |
3404 | + [ |
3405 | + (None, TypeError, "Invalid request"), |
3406 | + ( |
3407 | + {"architecture": "source", "uri": "URL", "foo": "bar"}, |
3408 | + TypeError, |
3409 | + "No 'source_urls' in request", |
3410 | + ), |
3411 | + ( |
3412 | + {"architecture": "source", "uri": "URL", "source_urls": None}, |
3413 | + ValueError, |
3414 | + "Invalid 'source_urls' (None) in request", |
3415 | + ), |
3416 | + ( |
3417 | + {"architecture": "amd64", "uri": "URL", "source_urls": "URL"}, |
3418 | + ValueError, |
3419 | + "Wrong request: ddeb fetch", |
3420 | + ), |
3421 | + ], |
3422 | + ) |
3423 | + def test_validate_request_invalid_request(self, req, exception, ex_message): |
3424 | + with pytest.raises(exception, match=re.escape(ex_message)): |
3425 | + sourcepackagegetter.SourcePackageGetter().validate_request(req) |
3426 | + |
3427 | + |
3428 | +if __name__ == "__main__": |
3429 | + unittest.main() |
3430 | diff --git a/tests/test_utils.py b/tests/test_utils.py |
3431 | new file mode 100644 |
3432 | index 0000000..7cac497 |
3433 | --- /dev/null |
3434 | +++ b/tests/test_utils.py |
3435 | @@ -0,0 +1,114 @@ |
3436 | +#!/usr/bin/python3 |
3437 | + |
3438 | +# Copyright (C) 2023 Canonical Ltd. |
3439 | + |
3440 | +# This program is free software: you can redistribute it and/or modify |
3441 | +# it under the terms of the GNU General Public License as published by |
3442 | +# the Free Software Foundation, either version 3 of the License, or |
3443 | +# (at your option) any later version. |
3444 | + |
3445 | +# This program is distributed in the hope that it will be useful, |
3446 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3447 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3448 | +# GNU General Public License for more details. |
3449 | + |
3450 | +# You should have received a copy of the GNU General Public License |
3451 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3452 | + |
3453 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3454 | + |
3455 | +import os |
3456 | +import re |
3457 | +import unittest |
3458 | +import datetime |
3459 | +import tempfile |
3460 | +import filecmp |
3461 | +import pytest |
3462 | +from freezegun import freeze_time |
3463 | +from ubuntu_debuginfod.utils import utils |
3464 | + |
3465 | + |
3466 | +class TestUtils: |
3467 | + def test_generate_timestamp_without_interval(self, *args): |
3468 | + curdate = datetime.datetime.now(datetime.timezone.utc) |
3469 | + with freeze_time(curdate): |
3470 | + ret = utils.generate_timestamp() |
3471 | + assert isinstance(ret, datetime.datetime) |
3472 | + assert ret == curdate |
3473 | + |
3474 | + def test_generate_timestamp_with_interval(self, *args): |
3475 | + curdate = datetime.datetime.now(datetime.timezone.utc) |
3476 | + interval = 8 |
3477 | + with freeze_time(curdate): |
3478 | + ret = utils.generate_timestamp(interval=interval) |
3479 | + assert isinstance(ret, datetime.datetime) |
3480 | + assert ret == curdate - datetime.timedelta(hours=interval) |
3481 | + |
3482 | + @pytest.mark.skip( |
3483 | + reason="currently broken when dealing with files in the same directory" |
3484 | + ) |
3485 | + def test_move_atomically_same_dir(self): |
3486 | + with tempfile.TemporaryDirectory() as tmp: |
3487 | + file1 = os.path.join(tmp, "file1") |
3488 | + file2 = os.path.join(tmp, "file2") |
3489 | + |
3490 | + with open(file1, "w") as f: |
3491 | + f.write("This is a test") |
3492 | + f.flush() |
3493 | + |
3494 | + utils.move_atomically(file1, file2) |
3495 | + filecmp.clear_cache() |
3496 | + assert filecmp.cmp(file1, file2, shallow=False) |
3497 | + |
3498 | + def test_move_atomically_different_dirs(self): |
3499 | + with tempfile.TemporaryDirectory() as tmp1: |
3500 | + file1 = os.path.join(tmp1, "file1") |
3501 | + with open(file1, "w") as f: |
3502 | + f.write("This is a test") |
3503 | + |
3504 | + with tempfile.TemporaryDirectory() as tmp2: |
3505 | + file2 = os.path.join(tmp2, "file2") |
3506 | + |
3507 | + utils.move_atomically(file1, file2) |
3508 | + filecmp.clear_cache() |
3509 | + assert filecmp.cmp(file1, file2, shallow=False) |
3510 | + |
3511 | + @pytest.mark.parametrize( |
3512 | + "version,expected_result", |
3513 | + [ |
3514 | + ("1.0-1", "1.0-1"), |
3515 | + ("1.0-1~ppa1", "1.0-1_ppa1"), |
3516 | + ("2:1.0-1", "2%1.0-1"), |
3517 | + ("1.0..1", "1.0.#.1"), |
3518 | + ("1.0..1.", "1.0.#.1.#"), |
3519 | + ("1.0-1.lock", "1.0-1.#lock"), |
3520 | + ("3:1.0-1..~ppa1.lock", "3%1.0-1.#._ppa1.#lock"), |
3521 | + ], |
3522 | + ) |
3523 | + def test_git_dep14_tag(self, version, expected_result): |
3524 | + assert utils.git_dep14_tag(version) == expected_result |
3525 | + |
3526 | + def test_obtain_lp_oauth_credentials_no_LP_CREDENTIALS_FILE_set(self): |
3527 | + if "LP_CREDENTIALS_FILE" in os.environ: |
3528 | + del os.environ["LP_CREDENTIALS_FILE"] |
3529 | + with pytest.raises( |
3530 | + ValueError, |
3531 | + match=re.escape( |
3532 | + "Could not obtain Launchpad credentials: environment variable LP_CREDENTIALS_FILE must be set" |
3533 | + ), |
3534 | + ): |
3535 | + utils.obtain_lp_oauth_credentials() |
3536 | + |
3537 | + def test_obtain_lp_oauth_credentials_LP_CREDENTIALS_FILE_invalid_file(self): |
3538 | + os.environ["LP_CREDENTIALS_FILE"] = "/path/to/nonexistent/file" |
3539 | + with pytest.raises( |
3540 | + FileNotFoundError, |
3541 | + match=re.escape( |
3542 | + "Could not obtain Launchpad credentials: file /path/to/nonexistent/file does not exist" |
3543 | + ), |
3544 | + ): |
3545 | + utils.obtain_lp_oauth_credentials() |
3546 | + |
3547 | + |
3548 | +if __name__ == "__main__": |
3549 | + unittest.main() |
3550 | diff --git a/ubuntu_debuginfod/__init__.py b/ubuntu_debuginfod/__init__.py |
3551 | new file mode 100644 |
3552 | index 0000000..c63b042 |
3553 | --- /dev/null |
3554 | +++ b/ubuntu_debuginfod/__init__.py |
3555 | @@ -0,0 +1,10 @@ |
3556 | +"""Init.""" |
3557 | + |
3558 | +import logging |
3559 | + |
3560 | +logging.basicConfig( |
3561 | + level=logging.INFO, |
3562 | + format=( |
3563 | + "%(levelname) -10s %(asctime)s %(name) -20s %(funcName) " "-25s : %(message)s" |
3564 | + ), |
3565 | +) |
3566 | diff --git a/debuginfod.py b/ubuntu_debuginfod/debuginfod.py |
3567 | similarity index 61% |
3568 | rename from debuginfod.py |
3569 | rename to ubuntu_debuginfod/debuginfod.py |
3570 | index ef39a95..54d73c9 100644 |
3571 | --- a/debuginfod.py |
3572 | +++ b/ubuntu_debuginfod/debuginfod.py |
3573 | @@ -24,18 +24,24 @@ from celery.signals import ( |
3574 | task_postrun, |
3575 | celeryd_init, |
3576 | ) |
3577 | +import sdnotify |
3578 | |
3579 | -from debuggetter import DebugGetterTimeout, DebugGetterRetry |
3580 | -from ddebgetter import DdebGetter, DdebSourceCodeGetter |
3581 | -from ppagetter import ( |
3582 | +from ubuntu_debuginfod.utils.exceptions import ( |
3583 | + UbuntuDebuginfodRetry, |
3584 | + UbuntuDebuginfodTimeout, |
3585 | +) |
3586 | +from ubuntu_debuginfod.getters.ddebgetter import DdebGetter |
3587 | +from ubuntu_debuginfod.getters.sourcepackagegetter import SourcePackageGetter |
3588 | +from ubuntu_debuginfod.getters.ppaddebgetter import ( |
3589 | PPADdebGetter, |
3590 | - PPASourceCodeGetter, |
3591 | PrivatePPADdebGetter, |
3592 | - PrivatePPASourceCodeGetter, |
3593 | ) |
3594 | -from utils import DebugDB |
3595 | - |
3596 | -import sdnotify |
3597 | +from ubuntu_debuginfod.getters.ppasourcepackagegetter import ( |
3598 | + PPASourcePackageGetter, |
3599 | + PrivatePPASourcePackageGetter, |
3600 | +) |
3601 | +from ubuntu_debuginfod.utils.debugdb import DebugDB |
3602 | +from ubuntu_debuginfod.utils.utils import record_error_into_db |
3603 | |
3604 | app = Celery("debuginfod", broker="pyamqp://guest@localhost//") |
3605 | |
3606 | @@ -55,29 +61,14 @@ app.conf.update( |
3607 | |
3608 | sdnotifier = sdnotify.SystemdNotifier() |
3609 | |
3610 | - |
3611 | -def _record_error_into_db(exc, msg): |
3612 | - """Record an error into the database. |
3613 | - |
3614 | - :param Exception exc: The Exception that was triggered. |
3615 | - |
3616 | - :param dict msg: The message that triggere the error.""" |
3617 | - with DebugDB() as db: |
3618 | - with db.cursor() as cur: |
3619 | - cur.execute( |
3620 | - "INSERT INTO errors (task, exception, date) VALUES (%s, %s, NOW())", |
3621 | - ( |
3622 | - str(msg), |
3623 | - str(exc), |
3624 | - ), |
3625 | - ) |
3626 | +# 1 day |
3627 | +DEFAULT_RETRY_DELAY = 24 * 60 * 60 |
3628 | |
3629 | |
3630 | @app.task( |
3631 | name="grab_ddebs", |
3632 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3633 | - # 1 day |
3634 | - default_retry_delay=24 * 60 * 60, |
3635 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3636 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3637 | # Launchpad can be problematic, so we retry every day for a |
3638 | # week. |
3639 | retry_kwargs={"max_retries": 7}, |
3640 | @@ -91,41 +82,40 @@ def grab_ddebs(msg): |
3641 | g = DdebGetter() |
3642 | g.process_request(msg) |
3643 | except Exception as e: |
3644 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3645 | - e, DebugGetterTimeout |
3646 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3647 | + e, UbuntuDebuginfodTimeout |
3648 | ): |
3649 | # This is some other kind of error that we need to deal |
3650 | # with. Mark it as such. |
3651 | - _record_error_into_db(e, msg) |
3652 | + record_error_into_db(e, msg) |
3653 | # We still need to raise the exception here. Celery will |
3654 | # reschedule the task if applicable. |
3655 | raise e |
3656 | |
3657 | |
3658 | @app.task( |
3659 | - name="grab_ddebs_sources", |
3660 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3661 | - # 1 day |
3662 | - default_retry_delay=24 * 60 * 60, |
3663 | + name="grab_sources", |
3664 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3665 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3666 | # Launchpad can be problematic, so we retry every day for a |
3667 | # week. |
3668 | retry_kwargs={"max_retries": 7}, |
3669 | ) |
3670 | -def grab_ddebs_sources(msg): |
3671 | - """Dispatch the DdebSourceCodeGetter task. |
3672 | +def grab_sources(msg): |
3673 | + """Dispatch the SourceCodeGetter task. |
3674 | |
3675 | :param dict[str, str] msg: The dictionary containing the message |
3676 | that will be processed by the getter.""" |
3677 | try: |
3678 | - g = DdebSourceCodeGetter() |
3679 | + g = SourcePackageGetter() |
3680 | g.process_request(msg) |
3681 | except Exception as e: |
3682 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3683 | - e, DebugGetterTimeout |
3684 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3685 | + e, UbuntuDebuginfodTimeout |
3686 | ): |
3687 | # This is some other kind of error that we need to deal |
3688 | # with. Mark it as such. |
3689 | - _record_error_into_db(e, msg) |
3690 | + record_error_into_db(e, msg) |
3691 | # We still need to raise the exception here. Celery will |
3692 | # reschedule the task if applicable. |
3693 | raise e |
3694 | @@ -133,9 +123,8 @@ def grab_ddebs_sources(msg): |
3695 | |
3696 | @app.task( |
3697 | name="grab_ppa_ddebs", |
3698 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3699 | - # 1 day |
3700 | - default_retry_delay=24 * 60 * 60, |
3701 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3702 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3703 | # Launchpad can be problematic, so we retry every day for a |
3704 | # week. |
3705 | retry_kwargs={"max_retries": 7}, |
3706 | @@ -149,12 +138,12 @@ def grab_ppa_ddebs(msg): |
3707 | g = PPADdebGetter() |
3708 | g.process_request(msg) |
3709 | except Exception as e: |
3710 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3711 | - e, DebugGetterTimeout |
3712 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3713 | + e, UbuntuDebuginfodTimeout |
3714 | ): |
3715 | # This is some other kind of error that we need to deal |
3716 | # with. Mark it as such. |
3717 | - _record_error_into_db(e, msg) |
3718 | + record_error_into_db(e, msg) |
3719 | # We still need to raise the exception here. Celery will |
3720 | # reschedule the task if applicable. |
3721 | raise e |
3722 | @@ -162,28 +151,27 @@ def grab_ppa_ddebs(msg): |
3723 | |
3724 | @app.task( |
3725 | name="grab_ppa_sources", |
3726 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3727 | - # 1 day |
3728 | - default_retry_delay=24 * 60 * 60, |
3729 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3730 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3731 | # Launchpad can be problematic, so we retry every day for a |
3732 | # week. |
3733 | retry_kwargs={"max_retries": 7}, |
3734 | ) |
3735 | def grab_ppa_sources(msg): |
3736 | - """Dispatch the PPASourceCodeGetter task. |
3737 | + """Dispatch the PPASourcePackageGetter task. |
3738 | |
3739 | :param dict[str, str] msg: The dictionary containing the message |
3740 | that will be processed by the getter.""" |
3741 | try: |
3742 | - g = PPASourceCodeGetter() |
3743 | + g = PPASourcePackageGetter() |
3744 | g.process_request(msg) |
3745 | except Exception as e: |
3746 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3747 | - e, DebugGetterTimeout |
3748 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3749 | + e, UbuntuDebuginfodTimeout |
3750 | ): |
3751 | # This is some other kind of error that we need to deal |
3752 | # with. Mark it as such. |
3753 | - _record_error_into_db(e, msg) |
3754 | + record_error_into_db(e, msg) |
3755 | # We still need to raise the exception here. Celery will |
3756 | # reschedule the task if applicable. |
3757 | raise e |
3758 | @@ -191,9 +179,8 @@ def grab_ppa_sources(msg): |
3759 | |
3760 | @app.task( |
3761 | name="grab_private_ppa_ddebs", |
3762 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3763 | - # 1 day |
3764 | - default_retry_delay=24 * 60 * 60, |
3765 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3766 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3767 | # Launchpad can be problematic, so we retry every day for a |
3768 | # week. |
3769 | retry_kwargs={"max_retries": 7}, |
3770 | @@ -207,12 +194,12 @@ def grab_private_ppa_ddebs(msg): |
3771 | g = PrivatePPADdebGetter() |
3772 | g.process_request(msg) |
3773 | except Exception as e: |
3774 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3775 | - e, DebugGetterTimeout |
3776 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3777 | + e, UbuntuDebuginfodTimeout |
3778 | ): |
3779 | # This is some other kind of error that we need to deal |
3780 | # with. Mark it as such. |
3781 | - _record_error_into_db(e, msg) |
3782 | + record_error_into_db(e, msg) |
3783 | # We still need to raise the exception here. Celery will |
3784 | # reschedule the task if applicable. |
3785 | raise e |
3786 | @@ -220,28 +207,27 @@ def grab_private_ppa_ddebs(msg): |
3787 | |
3788 | @app.task( |
3789 | name="grab_private_ppa_sources", |
3790 | - autoretry_for=(DebugGetterTimeout, DebugGetterRetry), |
3791 | - # 1 day |
3792 | - default_retry_delay=24 * 60 * 60, |
3793 | + autoretry_for=(UbuntuDebuginfodTimeout, UbuntuDebuginfodRetry), |
3794 | + default_retry_delay=DEFAULT_RETRY_DELAY, |
3795 | # Launchpad can be problematic, so we retry every day for a |
3796 | # week. |
3797 | retry_kwargs={"max_retries": 7}, |
3798 | ) |
3799 | def grab_private_ppa_sources(msg): |
3800 | - """Dispatch the PrivatePPASourceCodeGetter task. |
3801 | + """Dispatch the PrivatePPASourcePackageGetter task. |
3802 | |
3803 | :param dict[str, str] msg: The dictionary containing the message |
3804 | that will be processed by the getter.""" |
3805 | try: |
3806 | - g = PrivatePPASourceCodeGetter() |
3807 | + g = PrivatePPASourcePackageGetter() |
3808 | g.process_request(msg) |
3809 | except Exception as e: |
3810 | - if not isinstance(e, DebugGetterRetry) and not isinstance( |
3811 | - e, DebugGetterTimeout |
3812 | + if not isinstance(e, UbuntuDebuginfodRetry) and not isinstance( |
3813 | + e, UbuntuDebuginfodTimeout |
3814 | ): |
3815 | # This is some other kind of error that we need to deal |
3816 | # with. Mark it as such. |
3817 | - _record_error_into_db(e, msg) |
3818 | + record_error_into_db(e, msg) |
3819 | # We still need to raise the exception here. Celery will |
3820 | # reschedule the task if applicable. |
3821 | raise e |
3822 | @@ -259,13 +245,42 @@ def notify_worker_shutting_down(**kwargs): |
3823 | sdnotifier.notify("STOPPING=1") |
3824 | |
3825 | |
3826 | +def _get_msg_field(msg, field): |
3827 | + """Return a specific field from msg. |
3828 | + |
3829 | + :param dict[str -> str] msg: The dictionary received by this |
3830 | + Celery job. |
3831 | + |
3832 | + :param str field: The field name that we want. |
3833 | + |
3834 | + :rtype str or list: |
3835 | + :return: The value for field. |
3836 | + |
3837 | + :raises: RuntimeError if field does not exist or if it doesn't |
3838 | + have a non-None value.""" |
3839 | + if field not in msg.keys(): |
3840 | + e = RuntimeError(f"{field} could not be found in {msg}") |
3841 | + record_error_into_db(e, "Failed during _get_msg_field") |
3842 | + raise e |
3843 | + val = msg[field] |
3844 | + if val is None: |
3845 | + e = RuntimeError(f"{field} is None") |
3846 | + record_error_into_db(e, "Failed during _get_msg_field") |
3847 | + raise e |
3848 | + return val |
3849 | + |
3850 | + |
3851 | @task_postrun.connect |
3852 | -def task_postrun(sender, args, state, **kwargs): |
3853 | +def debuginfod_task_postrun(sender, args, state, **kwargs): |
3854 | """Process a task that's just finished. |
3855 | |
3856 | We keep the state of each task in a SQL database, and this |
3857 | function is responsible for cleaning up the state for tasks that have |
3858 | - succeeded.""" |
3859 | + succeeded. |
3860 | + |
3861 | + Moreover, it also records information about the downloaded |
3862 | + artifact so that we can keep track of it later (i.e., when we need |
3863 | + to clean things up).""" |
3864 | if state not in ("SUCCESS",): |
3865 | return |
3866 | |
3867 | @@ -274,13 +289,9 @@ def task_postrun(sender, args, state, **kwargs): |
3868 | msg = args[0] |
3869 | if not msg: |
3870 | e = RuntimeError("Could not obtain message to be processed.") |
3871 | - _record_error_into_db(e, "Failed during task_postrun") |
3872 | - raise e |
3873 | - if not msg.get("taskid"): |
3874 | - e = RuntimeError("Could not obtain taskid.") |
3875 | - _record_error_into_db(e, "Failed during task_postrun") |
3876 | + record_error_into_db(e, "Failed during task_postrun") |
3877 | raise e |
3878 | - taskid = msg["taskid"] |
3879 | + taskid = _get_msg_field(msg, "taskid") |
3880 | cur.execute("DELETE FROM tasks WHERE id = %s", (taskid,)) |
3881 | |
3882 | |
3883 | diff --git a/ubuntu_debuginfod/getter.py b/ubuntu_debuginfod/getter.py |
3884 | new file mode 100644 |
3885 | index 0000000..7ced991 |
3886 | --- /dev/null |
3887 | +++ b/ubuntu_debuginfod/getter.py |
3888 | @@ -0,0 +1,57 @@ |
3889 | +#!/usr/bin/python3 |
3890 | + |
3891 | +# Copyright (C) 2022 Canonical Ltd. |
3892 | + |
3893 | +# This program is free software: you can redistribute it and/or modify |
3894 | +# it under the terms of the GNU General Public License as published by |
3895 | +# the Free Software Foundation, either version 3 of the License, or |
3896 | +# (at your option) any later version. |
3897 | + |
3898 | +# This program is distributed in the hope that it will be useful, |
3899 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3900 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3901 | +# GNU General Public License for more details. |
3902 | + |
3903 | +# You should have received a copy of the GNU General Public License |
3904 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3905 | + |
3906 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3907 | + |
3908 | +import logging |
3909 | + |
3910 | + |
3911 | +class Getter: |
3912 | + """Base class for a Debug Getter.""" |
3913 | + |
3914 | + # The default directory where the debug artifacts will be saved. |
3915 | + DEFAULT_MIRROR_DIR = "/srv/debug-mirror" |
3916 | + |
3917 | + def __init__(self, subdir, mirror_dir=None, anonymous=True): |
3918 | + """Initialize the object. |
3919 | + |
3920 | + :param str mirror_dir: The directory we use to save the |
3921 | + mirrored files from Launchpad. |
3922 | + |
3923 | + :param str subdir: The subdirectory (insider mirror_dir) where |
3924 | + the module will save its files. For example, a ddeb |
3925 | + getter module should specify "ddebs"here. If None, use |
3926 | + DEFAULT_MIRROR_DIR. |
3927 | + |
3928 | + :param bool anonymous: Whether this getter handles anonymous |
3929 | + (public) artifacts. Default to True.""" |
3930 | + self._mirror_dir = self.DEFAULT_MIRROR_DIR if mirror_dir is None else mirror_dir |
3931 | + self._subdir = subdir |
3932 | + self._anonymous = anonymous |
3933 | + self._logger = logging.getLogger(__name__) |
3934 | + self._logger.debug( |
3935 | + f"Initializing Getter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}" |
3936 | + ) |
3937 | + |
3938 | + @property |
3939 | + def subdir(self): |
3940 | + return self._subdir |
3941 | + |
3942 | + @subdir.setter |
3943 | + def subdir(self, subdir): |
3944 | + self._logger.debug(f"Setting {type(self).__name__} subdir to '{subdir}'") |
3945 | + self._subdir = subdir |
3946 | diff --git a/ubuntu_debuginfod/getters/__init__.py b/ubuntu_debuginfod/getters/__init__.py |
3947 | new file mode 100644 |
3948 | index 0000000..14e8999 |
3949 | --- /dev/null |
3950 | +++ b/ubuntu_debuginfod/getters/__init__.py |
3951 | @@ -0,0 +1 @@ |
3952 | +"""Init.""" |
3953 | diff --git a/ubuntu_debuginfod/getters/common_lp_getter.py b/ubuntu_debuginfod/getters/common_lp_getter.py |
3954 | new file mode 100644 |
3955 | index 0000000..4852678 |
3956 | --- /dev/null |
3957 | +++ b/ubuntu_debuginfod/getters/common_lp_getter.py |
3958 | @@ -0,0 +1,104 @@ |
3959 | +#!/usr/bin/python3 |
3960 | + |
3961 | +# Copyright (C) 2022-2023 Canonical Ltd. |
3962 | + |
3963 | +# This program is free software: you can redistribute it and/or modify |
3964 | +# it under the terms of the GNU General Public License as published by |
3965 | +# the Free Software Foundation, either version 3 of the License, or |
3966 | +# (at your option) any later version. |
3967 | + |
3968 | +# This program is distributed in the hope that it will be useful, |
3969 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3970 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3971 | +# GNU General Public License for more details. |
3972 | + |
3973 | +# You should have received a copy of the GNU General Public License |
3974 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
3975 | + |
3976 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
3977 | + |
3978 | +import os |
3979 | +from ubuntu_debuginfod.getter import Getter |
3980 | +from ubuntu_debuginfod.utils.debugdb import DebugDB |
3981 | +from ubuntu_debuginfod.utils.utils import obtain_lp_oauth_credentials |
3982 | + |
3983 | + |
3984 | +class CommonLaunchpadGetter(Getter): |
3985 | + """Common getter class for dealing with Launchpad.""" |
3986 | + |
3987 | + def __init__(self, subdir, mirror_dir=None, anonymous=True): |
3988 | + """Initialize the object. |
3989 | + |
3990 | + See Getter's __init__ for an explanation of the arguments.""" |
3991 | + super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=anonymous) |
3992 | + self._logger.debug( |
3993 | + f"Initializing CommonLaunchpadGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}" |
3994 | + ) |
3995 | + |
3996 | + def make_savepath(self, source_package, component): |
3997 | + """Return the full save path for a package. |
3998 | + |
3999 | + :param str source_package: The package name. |
4000 | + |
4001 | + :param str component: The component name (main, universe, |
4002 | + etc.).""" |
4003 | + if source_package.startswith("lib"): |
4004 | + pkgname_initials = source_package[:4] |
4005 | + else: |
4006 | + pkgname_initials = source_package[0] |
4007 | + |
4008 | + return os.path.join( |
4009 | + self._mirror_dir, |
4010 | + self._subdir, |
4011 | + component, |
4012 | + pkgname_initials, |
4013 | + source_package, |
4014 | + ) |
4015 | + |
4016 | + def process_request(self, request): |
4017 | + """Process a request. |
4018 | + |
4019 | + :param dict[str, str] request: The dictionary containing the |
4020 | + information necessary to fetch this ddeb. |
4021 | + |
4022 | + :param requests_oauthlib.OAuth1 credentials: The credentials |
4023 | + to be used when downloading the artifact from Launchpad. |
4024 | + Default is None, which means anonymous.""" |
4025 | + self._logger.debug( |
4026 | + f"Processing request from {type(self).__name__} to download artifact: {request}" |
4027 | + ) |
4028 | + credentials = None if self._anonymous else obtain_lp_oauth_credentials() |
4029 | + fpath = self.download_artifact( |
4030 | + request, |
4031 | + credentials=credentials, |
4032 | + ) |
4033 | + if fpath is not None: |
4034 | + with DebugDB() as db: |
4035 | + with db.cursor() as cur: |
4036 | + cur.execute( |
4037 | + "INSERT INTO artifacts (fullpath, uri, private, date) VALUES (%s, %s, %s, NOW())", |
4038 | + (fpath, request["uri"], not self._anonymous), |
4039 | + ) |
4040 | + |
4041 | + def validate_request(self, request): |
4042 | + """Validate a request coming from Celery. |
4043 | + |
4044 | + :param dict[str, str] request: The dictionary containing the |
4045 | + request to be validated.""" |
4046 | + if request is None: |
4047 | + raise TypeError("Invalid request (None)") |
4048 | + |
4049 | + if "architecture" not in request.keys(): |
4050 | + raise ValueError("No 'architecture' in request") |
4051 | + if request.get("architecture") is None: |
4052 | + raise ValueError("Invalid 'architecture' (None) in request") |
4053 | + |
4054 | + if "uri" not in request.keys(): |
4055 | + raise ValueError("No 'uri' in request") |
4056 | + if request.get("uri") is None: |
4057 | + raise ValueError("Invalid 'uri' (None) in request") |
4058 | + |
4059 | + def download_artifact(self, *args): |
4060 | + """Interface for download_artifact. Subclasses of |
4061 | + CommonLaunchpadGetter are expected to implement it.""" |
4062 | + raise NotImplementedError("Wrong call to 'download_artifact' from base class") |
4063 | diff --git a/ubuntu_debuginfod/getters/common_ppa_getter.py b/ubuntu_debuginfod/getters/common_ppa_getter.py |
4064 | new file mode 100644 |
4065 | index 0000000..290bbe7 |
4066 | --- /dev/null |
4067 | +++ b/ubuntu_debuginfod/getters/common_ppa_getter.py |
4068 | @@ -0,0 +1,65 @@ |
4069 | +#!/usr/bin/python3 |
4070 | + |
4071 | +# Copyright (C) 2023 Canonical Ltd. |
4072 | + |
4073 | +# This program is free software: you can redistribute it and/or modify |
4074 | +# it under the terms of the GNU General Public License as published by |
4075 | +# the Free Software Foundation, either version 3 of the License, or |
4076 | +# (at your option) any later version. |
4077 | + |
4078 | +# This program is distributed in the hope that it will be useful, |
4079 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4080 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4081 | +# GNU General Public License for more details. |
4082 | + |
4083 | +# You should have received a copy of the GNU General Public License |
4084 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4085 | + |
4086 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4087 | + |
4088 | +import os |
4089 | +import logging |
4090 | + |
4091 | + |
4092 | +class CommonPPAGetter: |
4093 | + """A common class for PPA getters.""" |
4094 | + |
4095 | + def __init__(self, *args, **kwargs): |
4096 | + """Initialize the object. |
4097 | + |
4098 | + We're not interested in doing anything here, since this is a |
4099 | + base class that should be used in multiple-inheritance |
4100 | + scenarios.""" |
4101 | + super().__init__(*args, **kwargs) |
4102 | + self._logger = logging.getLogger(__name__) |
4103 | + self._logger.debug("Initializing CommonPPAGetter") |
4104 | + |
4105 | + def validate_request(self, request): |
4106 | + """Validate a request to fetch an artifact from a PPA (private or |
4107 | + not). |
4108 | + |
4109 | + :param dict[str, str] request: The dictionary containing the |
4110 | + request to be validated.""" |
4111 | + super().validate_request(request) |
4112 | + keys = request.keys() |
4113 | + if "ppauser" not in keys: |
4114 | + raise TypeError("No 'ppauser' in request") |
4115 | + if "ppaname" not in keys: |
4116 | + raise TypeError("No 'ppaname' in request") |
4117 | + if "isprivateppa" not in keys: |
4118 | + raise TypeError("No 'isprivateppa' in request") |
4119 | + |
4120 | + if request.get("ppauser") is None: |
4121 | + raise ValueError("Invalid 'ppauser' (None) in request") |
4122 | + if request.get("ppaname") is None: |
4123 | + raise ValueError("Invalid 'ppaname' (None) in request") |
4124 | + if request.get("isprivateppa") is None: |
4125 | + raise ValueError("Invalid 'isprivateppa' (None) in request") |
4126 | + |
4127 | + def process_request(self, request, credentials=None): |
4128 | + """Process a request, usually coming from Celery. |
4129 | + |
4130 | + :param dict[str -> str] request: The dictionary containing the |
4131 | + information necessary to fetch this ddeb.""" |
4132 | + self.subdir = os.path.join(self._subdir, request["ppauser"], request["ppaname"]) |
4133 | + super().process_request(request, credentials=credentials) |
4134 | diff --git a/ubuntu_debuginfod/getters/ddebgetter.py b/ubuntu_debuginfod/getters/ddebgetter.py |
4135 | new file mode 100644 |
4136 | index 0000000..633ed93 |
4137 | --- /dev/null |
4138 | +++ b/ubuntu_debuginfod/getters/ddebgetter.py |
4139 | @@ -0,0 +1,70 @@ |
4140 | +#!/usr/bin/python3 |
4141 | + |
4142 | +# Copyright (C) 2022 Canonical Ltd. |
4143 | + |
4144 | +# This program is free software: you can redistribute it and/or modify |
4145 | +# it under the terms of the GNU General Public License as published by |
4146 | +# the Free Software Foundation, either version 3 of the License, or |
4147 | +# (at your option) any later version. |
4148 | + |
4149 | +# This program is distributed in the hope that it will be useful, |
4150 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4151 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4152 | +# GNU General Public License for more details. |
4153 | + |
4154 | +# You should have received a copy of the GNU General Public License |
4155 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4156 | + |
4157 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4158 | + |
4159 | +from ubuntu_debuginfod.getters.common_lp_getter import CommonLaunchpadGetter |
4160 | +from ubuntu_debuginfod.utils.utils import download_from_lp |
4161 | + |
4162 | + |
4163 | +class DdebGetter(CommonLaunchpadGetter): |
4164 | + """Get (fetch) a ddeb.""" |
4165 | + |
4166 | + def __init__(self, subdir="ddebs", mirror_dir=None, anonymous=True): |
4167 | + """Initialize the object. |
4168 | + |
4169 | + See Getter's __init__ for an explanation of the arguments.""" |
4170 | + super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=anonymous) |
4171 | + self._logger.debug( |
4172 | + f"Initializing DdebGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}" |
4173 | + ) |
4174 | + |
4175 | + def download_artifact(self, request, credentials=None): |
4176 | + """Download a ddeb associated with a package. |
4177 | + |
4178 | + :param dict[str, str] request: The dictionary containing the |
4179 | + information necessary to fetch this ddeb. |
4180 | + |
4181 | + :param requests_oauthlib.OAuth1 credentials: The credentials |
4182 | + to be used when downloading the artifact from Launchpad. |
4183 | + Default is None, which means anonymous. |
4184 | + |
4185 | + :rtype: str |
4186 | + :return: The full path for the file that was saved, or None if |
4187 | + the file already exists.""" |
4188 | + self.validate_request(request) |
4189 | + source_package = request["source_package"] |
4190 | + component = request["component"] |
4191 | + ddeb_url = request["ddeb_url"] |
4192 | + savepath = self.make_savepath(source_package, component) |
4193 | + self._logger.debug(f"Downloading '{ddeb_url}' into '{savepath}'") |
4194 | + if credentials is not None: |
4195 | + self._logger.debug("Downloading with credentials") |
4196 | + return download_from_lp(ddeb_url, savepath, credentials=credentials) |
4197 | + |
4198 | + def validate_request(self, request): |
4199 | + """Validate a request to fetch a ddeb from the main archive. |
4200 | + |
4201 | + :param dict[str, str] request: The dictionary containing the |
4202 | + request to be validated.""" |
4203 | + super().validate_request(request) |
4204 | + if "ddeb_url" not in request.keys(): |
4205 | + raise TypeError("No 'ddeb_url' in request") |
4206 | + if request.get("ddeb_url") is None: |
4207 | + raise ValueError("Invalid 'ddeb_url' (None) in request") |
4208 | + if request["architecture"] == "source": |
4209 | + raise ValueError("Wrong request: source fetch") |
4210 | diff --git a/ubuntu_debuginfod/getters/ppaddebgetter.py b/ubuntu_debuginfod/getters/ppaddebgetter.py |
4211 | new file mode 100644 |
4212 | index 0000000..4d86232 |
4213 | --- /dev/null |
4214 | +++ b/ubuntu_debuginfod/getters/ppaddebgetter.py |
4215 | @@ -0,0 +1,65 @@ |
4216 | +#!/usr/bin/python3 |
4217 | + |
4218 | +# Copyright (C) 2023 Canonical Ltd. |
4219 | + |
4220 | +# This program is free software: you can redistribute it and/or modify |
4221 | +# it under the terms of the GNU General Public License as published by |
4222 | +# the Free Software Foundation, either version 3 of the License, or |
4223 | +# (at your option) any later version. |
4224 | + |
4225 | +# This program is distributed in the hope that it will be useful, |
4226 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4227 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4228 | +# GNU General Public License for more details. |
4229 | + |
4230 | +# You should have received a copy of the GNU General Public License |
4231 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4232 | + |
4233 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4234 | + |
4235 | +from ubuntu_debuginfod.getters.ddebgetter import DdebGetter |
4236 | +from ubuntu_debuginfod.getters.common_ppa_getter import CommonPPAGetter |
4237 | + |
4238 | + |
4239 | +class PPADdebGetter(CommonPPAGetter, DdebGetter): |
4240 | + """Get a ddeb from a Launchpad PPA.""" |
4241 | + |
4242 | + def __init__(self, subdir="ppas", mirror_dir=None): |
4243 | + """Initialize the object. |
4244 | + |
4245 | + See Getter's __init__ for an explanation of the arguments.""" |
4246 | + super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=True) |
4247 | + self._logger.debug( |
4248 | + f"Initializing PPADdebGetter with subdir={subdir}, mirror_dir={mirror_dir}" |
4249 | + ) |
4250 | + |
4251 | + def validate_request(self, request): |
4252 | + """Validate a request to fetch a ddeb from a PPA. |
4253 | + |
4254 | + :param dict[str, str] request: The dictionary containing the |
4255 | + request to be validated.""" |
4256 | + super().validate_request(request) |
4257 | + if request["isprivateppa"] == "yes": |
4258 | + raise ValueError("Wrong request: from private PPA") |
4259 | + |
4260 | + |
4261 | +class PrivatePPADdebGetter(CommonPPAGetter, DdebGetter): |
4262 | + """Get a ddeb from a private Launchpad PPA.""" |
4263 | + |
4264 | + def __init__(self, subdir="private-ppas", mirror_dir=None): |
4265 | + """Initialize the object. |
4266 | + |
4267 | + See Getter's __init__ for an explanation of the arguments.""" |
4268 | + super().__init__(subdir=subdir, mirror_dir=mirror_dir, anonymous=False) |
4269 | + self._logger.debug( |
4270 | + f"Initializing PrivatePPADdebGetter with subdir={subdir}, mirror_dir={mirror_dir}" |
4271 | + ) |
4272 | + |
4273 | + def validate_request(self, request): |
4274 | + """Validate a request to fetch a ddeb from a private PPA. |
4275 | + |
4276 | + :param dict[str, str] request: The dictionary containing the |
4277 | + request to be validated.""" |
4278 | + super().validate_request(request) |
4279 | + if request["isprivateppa"] == "no": |
4280 | + raise ValueError("Wrong request: not from private PPA") |
4281 | diff --git a/ubuntu_debuginfod/getters/ppasourcepackagegetter.py b/ubuntu_debuginfod/getters/ppasourcepackagegetter.py |
4282 | new file mode 100644 |
4283 | index 0000000..ae488a4 |
4284 | --- /dev/null |
4285 | +++ b/ubuntu_debuginfod/getters/ppasourcepackagegetter.py |
4286 | @@ -0,0 +1,70 @@ |
4287 | +#!/usr/bin/python3 |
4288 | + |
4289 | +# Copyright (C) 2023 Canonical Ltd. |
4290 | + |
4291 | +# This program is free software: you can redistribute it and/or modify |
4292 | +# it under the terms of the GNU General Public License as published by |
4293 | +# the Free Software Foundation, either version 3 of the License, or |
4294 | +# (at your option) any later version. |
4295 | + |
4296 | +# This program is distributed in the hope that it will be useful, |
4297 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4298 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4299 | +# GNU General Public License for more details. |
4300 | + |
4301 | +# You should have received a copy of the GNU General Public License |
4302 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4303 | + |
4304 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4305 | + |
4306 | +from ubuntu_debuginfod.getters.sourcepackagegetter import SourcePackageGetter |
4307 | +from ubuntu_debuginfod.getters.common_ppa_getter import CommonPPAGetter |
4308 | + |
4309 | + |
4310 | +class PPASourcePackageGetter(CommonPPAGetter, SourcePackageGetter): |
4311 | + """Get a source package from a Launchpad PPA.""" |
4312 | + |
4313 | + def __init__(self, subdir="ppas", mirror_dir=None): |
4314 | + """Initialize the object. |
4315 | + |
4316 | + See Getter's __init__ for an explanation of the arguments.""" |
4317 | + super().__init__( |
4318 | + subdir=subdir, mirror_dir=mirror_dir, anonymous=True, fallback_to_git=False |
4319 | + ) |
4320 | + self._logger.debug( |
4321 | + f"Initializing PPASourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}" |
4322 | + ) |
4323 | + |
4324 | + def validate_request(self, request): |
4325 | + """Validate a request to fetch a source package from a PPA. |
4326 | + |
4327 | + :param dict[str, str] request: The dictionary containing the |
4328 | + request to be validated.""" |
4329 | + super().validate_request(request) |
4330 | + if request["isprivateppa"] == "yes": |
4331 | + raise ValueError("Wrong request: from private PPA") |
4332 | + |
4333 | + |
4334 | +class PrivatePPASourcePackageGetter(CommonPPAGetter, SourcePackageGetter): |
4335 | + """Get a source package from a private Launchpad PPA.""" |
4336 | + |
4337 | + def __init__(self, subdir="private-ppas", mirror_dir=None): |
4338 | + """Initialize the object. |
4339 | + |
4340 | + See Getter's __init__ for an explanation of the arguments.""" |
4341 | + super().__init__( |
4342 | + subdir=subdir, mirror_dir=mirror_dir, anonymous=False, fallback_to_git=False |
4343 | + ) |
4344 | + self._logger.debug( |
4345 | + f"Initializing PrivatePPASourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}" |
4346 | + ) |
4347 | + |
4348 | + def validate_request(self, request): |
4349 | + """Validate a request to fetch a source package from a private |
4350 | + PPA. |
4351 | + |
4352 | + :param dict[str, str] request: The dictionary containing the |
4353 | + request to be validated.""" |
4354 | + super().validate_request(request) |
4355 | + if request["isprivateppa"] == "no": |
4356 | + raise ValueError("Wrong request: not from private PPA") |
4357 | diff --git a/ubuntu_debuginfod/getters/sourcepackagegetter.py b/ubuntu_debuginfod/getters/sourcepackagegetter.py |
4358 | new file mode 100644 |
4359 | index 0000000..2287620 |
4360 | --- /dev/null |
4361 | +++ b/ubuntu_debuginfod/getters/sourcepackagegetter.py |
4362 | @@ -0,0 +1,255 @@ |
4363 | +#!/usr/bin/python3 |
4364 | + |
4365 | +# Copyright (C) 2022-2023 Canonical Ltd. |
4366 | + |
4367 | +# This program is free software: you can redistribute it and/or modify |
4368 | +# it under the terms of the GNU General Public License as published by |
4369 | +# the Free Software Foundation, either version 3 of the License, or |
4370 | +# (at your option) any later version. |
4371 | + |
4372 | +# This program is distributed in the hope that it will be useful, |
4373 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4374 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4375 | +# GNU General Public License for more details. |
4376 | + |
4377 | +# You should have received a copy of the GNU General Public License |
4378 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4379 | + |
4380 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4381 | + |
4382 | +import os |
4383 | +import lzma |
4384 | +import tarfile |
4385 | +import subprocess |
4386 | +import tempfile |
4387 | +from git import Git, Repo |
4388 | +from git.exc import GitCommandError |
4389 | +from ubuntu_debuginfod.getters.common_lp_getter import CommonLaunchpadGetter |
4390 | +from ubuntu_debuginfod.utils.utils import ( |
4391 | + git_dep14_tag, |
4392 | + download_from_lp, |
4393 | + move_atomically, |
4394 | +) |
4395 | +from ubuntu_debuginfod.utils.exceptions import UbuntuDebuginfodRetry |
4396 | + |
4397 | + |
4398 | +def adjust_tar_filepath(tarinfo): |
4399 | + """Adjust the filepath for a TarInfo file. |
4400 | + |
4401 | + This function is needed because TarFile.add strips the leading |
4402 | + slash from the filenames, so we have to workaround it by |
4403 | + re-adding the slash ourselves. |
4404 | + |
4405 | + This function is intended to be used as a callback provided to |
4406 | + TarFile.add. |
4407 | + |
4408 | + :param TarInfo tarinfo: The tarinfo.""" |
4409 | + tarinfo.name = os.path.join("/", tarinfo.name) |
4410 | + return tarinfo |
4411 | + |
4412 | + |
4413 | +class SourcePackageGetter(CommonLaunchpadGetter): |
4414 | + """Get (fetch) the source package artifact.""" |
4415 | + |
4416 | + def __init__( |
4417 | + self, subdir="ddebs", mirror_dir=None, anonymous=True, fallback_to_git=True |
4418 | + ): |
4419 | + """Initialize the object. |
4420 | + |
4421 | + :param boolean fallback_to_git: Whether we should try to fetch |
4422 | + the source package using git if the regular approach (via |
4423 | + dget) fails. Default to True. |
4424 | + |
4425 | + See Getter's __init__ for an explanation of the other arguments.""" |
4426 | + super().__init__(subdir=subdir, mirror_dir=mirror_dir) |
4427 | + self._fallback_to_git = fallback_to_git |
4428 | + self._logger.debug( |
4429 | + f"Initializing SourcePackageGetter with subdir={subdir}, mirror_dir={mirror_dir}, anonymous={anonymous}, fallback_to_git={fallback_to_git}" |
4430 | + ) |
4431 | + |
4432 | + def _download_source_package_from_git(self, source_package, version, filepath): |
4433 | + """Download the source package using Launchpad's git repository. |
4434 | + |
4435 | + :param str source_package: Source package name. |
4436 | + |
4437 | + :param str version: Source package version. |
4438 | + |
4439 | + :param str filepath: The full pathname where the resulting |
4440 | + source package tarball should be saved. |
4441 | + |
4442 | + :rtype: bool |
4443 | + :return: This method returns True when the operation succeeds, or False |
4444 | + *iff* the "git clone" command fails to run. Otherwise, this |
4445 | + function will throw an exception.""" |
4446 | + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir: |
4447 | + g = Git() |
4448 | + git_dir = os.path.join(source_dir, f"{source_package}") |
4449 | + tag = "applied/" + git_dep14_tag(f"{version}") |
4450 | + self._logger.debug( |
4451 | + f"Cloning '{source_package}' git repo into '{git_dir}' (tag: '{tag}')" |
4452 | + ) |
4453 | + try: |
4454 | + g.clone( |
4455 | + f"https://git.launchpad.net/ubuntu/+source/{source_package}", |
4456 | + git_dir, |
4457 | + depth=1, |
4458 | + branch=tag, |
4459 | + ) |
4460 | + except GitCommandError as e: |
4461 | + # Couldn't perform the download. Let's signal and |
4462 | + # bail out. |
4463 | + self._logger.warning( |
4464 | + f"Could not clone git repo for '{source_package}': {e}" |
4465 | + ) |
4466 | + return False |
4467 | + repo = Repo(git_dir) |
4468 | + prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/") |
4469 | + self._logger.debug( |
4470 | + f"Archiving git repo for '{source_package}-{version}' as '{filepath}'" |
4471 | + ) |
4472 | + with tempfile.NamedTemporaryFile() as tmpfile: |
4473 | + with lzma.open(tmpfile.name, "w") as xzfile: |
4474 | + repo.archive(xzfile, prefix=prefix_path, format="tar") |
4475 | + move_atomically(tmpfile.name, filepath) |
4476 | + |
4477 | + return True |
4478 | + |
4479 | + def _download_source_package_from_dsc( |
4480 | + self, source_package, version, filepath, source_urls, credentials=None |
4481 | + ): |
4482 | + """Download the source package using the .dsc file. |
4483 | + |
4484 | + :param str source_package: Source package name. |
4485 | + |
4486 | + :param str version: Source package version. |
4487 | + |
4488 | + :param str filepath: The full pathname where the resulting |
4489 | + source package tarball should be saved. |
4490 | + |
4491 | + :param list source_urls: List of URLs used to fetch the source |
4492 | + package. This is usually the list returned by the |
4493 | + sourceFileUrls() Launchpad API call. |
4494 | + |
4495 | + :param requests_oauthlib.OAuth1 credentials: The credentials |
4496 | + to be used when downloading the artifact from Launchpad. |
4497 | + Default is None, which means anonymous. |
4498 | + |
4499 | + :rtype: bool |
4500 | + :return: This method returns True when the operation succeeds, or False |
4501 | + *iff* the "dpkg-source -x"command fails to run. |
4502 | + Otherwise, this function will throw an exception.""" |
4503 | + with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir: |
4504 | + for url in source_urls: |
4505 | + download_from_lp(url, source_dir, credentials=credentials) |
4506 | + |
4507 | + dscfile = None |
4508 | + for f in os.listdir(source_dir): |
4509 | + newf = os.path.join(source_dir, f) |
4510 | + if os.path.isfile(newf) and f.endswith(".dsc"): |
4511 | + dscfile = newf |
4512 | + break |
4513 | + |
4514 | + if dscfile is None: |
4515 | + self._logger.warning( |
4516 | + "Could not find .dsc file, even though it should exist." |
4517 | + ) |
4518 | + return False |
4519 | + |
4520 | + outdir = os.path.join(source_dir, "outdir") |
4521 | + self._logger.debug(f"Will call 'dpkg-source -x {dscfile} {outdir}'") |
4522 | + try: |
4523 | + subprocess.run( |
4524 | + ["/usr/bin/dpkg-source", "-x", dscfile, outdir], |
4525 | + cwd=source_dir, |
4526 | + check=True, |
4527 | + ) |
4528 | + except subprocess.CalledProcessError: |
4529 | + self._logger.warning("Call to 'dpkg-source -x' failed.") |
4530 | + return False |
4531 | + |
4532 | + if not os.path.isdir(outdir): |
4533 | + self._logger.warning( |
4534 | + f"'{outdir}' has not been created by 'dpkg-source -x'." |
4535 | + ) |
4536 | + return False |
4537 | + |
4538 | + prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/") |
4539 | + |
4540 | + with tempfile.NamedTemporaryFile() as tmpfile: |
4541 | + with tarfile.open(tmpfile.name, "w:xz") as tfile: |
4542 | + tfile.add(outdir, arcname=prefix_path, filter=adjust_tar_filepath) |
4543 | + move_atomically(tmpfile.name, filepath) |
4544 | + |
4545 | + return True |
4546 | + |
4547 | + def download_artifact( |
4548 | + self, |
4549 | + request, |
4550 | + credentials=None, |
4551 | + ): |
4552 | + """Download the source package. |
4553 | + |
4554 | + :param str source_package: Source package name. |
4555 | + |
4556 | + :param str version: Source package version. |
4557 | + |
4558 | + :param str component: Source package component. |
4559 | + |
4560 | + :param list source_urls: List of source file URLS. |
4561 | + |
4562 | + :param boolean fallback_to_git: Whether we should try to fetch |
4563 | + the source package using git if the regular approach (via |
4564 | + dget) fails. Default to True. |
4565 | + |
4566 | + :param requests_oauthlib.OAuth1 credentials: The credentials |
4567 | + to be used when downloading the artifact from Launchpad. |
4568 | + Default is None, which means anonymous. |
4569 | + |
4570 | + :rtype: str |
4571 | + :return: The full path for the file that was saved, or None if |
4572 | + the file already exists.""" |
4573 | + source_package = request["source_package"] |
4574 | + version = request["version"] |
4575 | + component = request["component"] |
4576 | + source_urls = request["source_urls"] |
4577 | + |
4578 | + savepath = self.make_savepath(source_package, component) |
4579 | + txzfilepath = os.path.join(savepath, f"{source_package}-{version}.tar.xz") |
4580 | + if os.path.exists(txzfilepath): |
4581 | + self._logger.debug(f"'{txzfilepath}' already exists; doing nothing.") |
4582 | + return None |
4583 | + |
4584 | + if self._download_source_package_from_dsc( |
4585 | + source_package, version, txzfilepath, source_urls, credentials=credentials |
4586 | + ): |
4587 | + self._logger.info( |
4588 | + f"Downloaded source package from dsc for '{source_package}-{version}' as '{txzfilepath}'" |
4589 | + ) |
4590 | + return txzfilepath |
4591 | + |
4592 | + if self._fallback_to_git and self._download_source_package_from_git( |
4593 | + source_package, version, txzfilepath |
4594 | + ): |
4595 | + self._logger.info( |
4596 | + f"Downloaded source package from git for '{source_package}-{version}' as '{txzfilepath}'" |
4597 | + ) |
4598 | + return txzfilepath |
4599 | + |
4600 | + # In the (likely?) event that there is a problem with |
4601 | + # Launchpad, let's raise an exception signalling that we'd |
4602 | + # like to retry the task. |
4603 | + raise UbuntuDebuginfodRetry() |
4604 | + |
4605 | + def validate_request(self, request): |
4606 | + """Validate a request to fetch a source package from the main |
4607 | + archive. |
4608 | + |
4609 | + :param dict[str, str] request: The dictionary containing the |
4610 | + request to be validated.""" |
4611 | + super().validate_request(request) |
4612 | + if "source_urls" not in request.keys(): |
4613 | + raise TypeError("No 'source_urls' in request") |
4614 | + if request.get("source_urls") is None: |
4615 | + raise ValueError("Invalid 'source_urls' (None) in request") |
4616 | + if request["architecture"] != "source": |
4617 | + raise ValueError("Wrong request: ddeb fetch") |
4618 | diff --git a/debugpoller.py b/ubuntu_debuginfod/poller.py |
4619 | similarity index 66% |
4620 | rename from debugpoller.py |
4621 | rename to ubuntu_debuginfod/poller.py |
4622 | index 4e4a657..d6f32d0 100644 |
4623 | --- a/debugpoller.py |
4624 | +++ b/ubuntu_debuginfod/poller.py |
4625 | @@ -18,14 +18,13 @@ |
4626 | # Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4627 | |
4628 | import os |
4629 | -import sys |
4630 | -from launchpadlib.launchpad import Launchpad |
4631 | import datetime |
4632 | import logging |
4633 | +from ubuntu_debuginfod.utils.utils import generate_timestamp |
4634 | |
4635 | |
4636 | -class DebugPoller: |
4637 | - """The DebugPoller class implements the basics of a poller for the |
4638 | +class Poller: |
4639 | + """The Poller class implements the basics of a poller for the |
4640 | debuginfod service.""" |
4641 | |
4642 | # The timestamp file we will save our timestamp into. |
4643 | @@ -43,7 +42,7 @@ class DebugPoller: |
4644 | dry_run=False, |
4645 | anonymous=True, |
4646 | ): |
4647 | - """Initalize the DebugPoller object. |
4648 | + """Initalize the Poller object. |
4649 | |
4650 | :param int initial_interval: The initial interval (in hours), |
4651 | i.e., if no timestamp file has been found then we use this |
4652 | @@ -65,60 +64,27 @@ class DebugPoller: |
4653 | the timestamp in the file when the operation finishes. |
4654 | Default is False. |
4655 | |
4656 | - :param bool anonymous: Whether we should login anonymously. |
4657 | - Default is True. |
4658 | - |
4659 | - """ |
4660 | - if anonymous: |
4661 | - self._lp = Launchpad.login_anonymously( |
4662 | - "ubuntu-debuginfod poller", "production", version="devel" |
4663 | - ) |
4664 | - else: |
4665 | - self._lp = Launchpad.login_with( |
4666 | - "ubuntu-debuginfod poller", "production", version="devel" |
4667 | - ) |
4668 | - |
4669 | - self._main_archive = self._lp.distributions["ubuntu"].main_archive |
4670 | - |
4671 | - self._logger = logging.getLogger(__name__) |
4672 | - logging.basicConfig( |
4673 | - level=logging.INFO, |
4674 | - format=( |
4675 | - "%(levelname) -10s %(asctime)s %(name) -20s %(funcName) " |
4676 | - "-25s : %(message)s" |
4677 | - ), |
4678 | - ) |
4679 | - |
4680 | + :param bool anonymous: Whether we should poll anonymously. |
4681 | + Default is True.""" |
4682 | self._initial_interval = initial_interval |
4683 | self._force_initial_interval = force_initial_interval |
4684 | self._fetch_all_on_first_run = fetch_all_on_first_run |
4685 | self._dry_run = dry_run |
4686 | + self._anonymous = anonymous |
4687 | + self._logger = logging.getLogger(__name__) |
4688 | + self._logger.debug( |
4689 | + f"Initializing Poller with initial_interval={initial_interval}, force_initial_interval={force_initial_interval}, fetch_all_on_first_run={fetch_all_on_first_run}, dry_run={dry_run}, anonymous={anonymous}" |
4690 | + ) |
4691 | |
4692 | @property |
4693 | def timestamp_filename(self): |
4694 | """Get the filename that contains the timestamp for this module. |
4695 | |
4696 | - :rtype str: The timestamp filename.""" |
4697 | - module_name = type(self).__name__.replace('Poller', '').lower() |
4698 | + :rtype: str |
4699 | + :return: The timestamp filename.""" |
4700 | + module_name = type(self).__name__.replace("Poller", "").lower() |
4701 | return f"{self.TIMESTAMP_FILE}-{module_name}" |
4702 | |
4703 | - def _generate_timestamp(self, interval=None): |
4704 | - """Generate a timestamp that can be used when querying Launchpad |
4705 | - (via getPublished{Sources,Binaries}). |
4706 | - |
4707 | - :param interval: Specify how long ago (in hours) the timestamp must |
4708 | - refer to. If not specified, the timestamp is generated for |
4709 | - the current time. |
4710 | - :type interval: int or None""" |
4711 | - d = datetime.datetime.now(datetime.timezone.utc) |
4712 | - |
4713 | - if interval is not None: |
4714 | - self._logger.debug(f"Generating timestamp with interval {interval}") |
4715 | - d = d - datetime.timedelta(hours=interval) |
4716 | - |
4717 | - self._logger.debug(f"Generated timestamp '{d}'") |
4718 | - return d |
4719 | - |
4720 | def _get_timestamp(self): |
4721 | """Get the timestamp from the timestamp file, or generate a |
4722 | new one. |
4723 | @@ -138,15 +104,17 @@ class DebugPoller: |
4724 | self._logger.debug( |
4725 | f"Or force_initial_interval = {self._force_initial_interval}" |
4726 | ) |
4727 | - return self._generate_timestamp(interval=self._initial_interval) |
4728 | + return generate_timestamp(interval=self._initial_interval) |
4729 | |
4730 | - with open(tfile, "r", encoding="UTF-8") as f: |
4731 | - ts = int(f.readline().rstrip()) |
4732 | - d = datetime.datetime.fromtimestamp(ts, tz=datetime.timezone.utc) |
4733 | + with open(tfile, "r", encoding="UTF-8") as file: |
4734 | + ts_int = int(file.readline().rstrip()) |
4735 | + last_timestamp = datetime.datetime.fromtimestamp( |
4736 | + ts_int, tz=datetime.timezone.utc |
4737 | + ) |
4738 | |
4739 | - if not isinstance(d, datetime.datetime): |
4740 | - raise RuntimeError(f"Invalid timestamp {d}") |
4741 | - return d |
4742 | + if not isinstance(last_timestamp, datetime.datetime): |
4743 | + raise RuntimeError(f"Invalid timestamp {last_timestamp}") |
4744 | + return last_timestamp |
4745 | |
4746 | def _get_normal_and_real_timestamps(self): |
4747 | """Get the previous timestamp and adjust it to account for |
4748 | @@ -175,5 +143,8 @@ class DebugPoller: |
4749 | os.makedirs(dirname, mode=0o755, exist_ok=True) |
4750 | with open(tfile, "w", encoding="UTF-8") as f: |
4751 | epoch = datetime.datetime.fromtimestamp(0, tz=datetime.timezone.utc) |
4752 | - ts = int((timestamp - epoch).total_seconds()) |
4753 | - f.write("%d" % ts) |
4754 | + ts_int = int((timestamp - epoch).total_seconds()) |
4755 | + self._logger.debug( |
4756 | + f"Recording timestamp '{ts_int}' for {type(self).__name__}" |
4757 | + ) |
4758 | + f.write(f"{ts_int}") |
4759 | diff --git a/ubuntu_debuginfod/pollers/__init__.py b/ubuntu_debuginfod/pollers/__init__.py |
4760 | new file mode 100644 |
4761 | index 0000000..14e8999 |
4762 | --- /dev/null |
4763 | +++ b/ubuntu_debuginfod/pollers/__init__.py |
4764 | @@ -0,0 +1 @@ |
4765 | +"""Init.""" |
4766 | diff --git a/ubuntu_debuginfod/pollers/common_lp_poller.py b/ubuntu_debuginfod/pollers/common_lp_poller.py |
4767 | new file mode 100644 |
4768 | index 0000000..93a958e |
4769 | --- /dev/null |
4770 | +++ b/ubuntu_debuginfod/pollers/common_lp_poller.py |
4771 | @@ -0,0 +1,86 @@ |
4772 | +#!/usr/bin/python3 |
4773 | + |
4774 | +# Copyright (C) 2022 Canonical Ltd. |
4775 | + |
4776 | +# This program is free software: you can redistribute it and/or modify |
4777 | +# it under the terms of the GNU General Public License as published by |
4778 | +# the Free Software Foundation, either version 3 of the License, or |
4779 | +# (at your option) any later version. |
4780 | + |
4781 | +# This program is distributed in the hope that it will be useful, |
4782 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4783 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4784 | +# GNU General Public License for more details. |
4785 | + |
4786 | +# You should have received a copy of the GNU General Public License |
4787 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4788 | + |
4789 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4790 | + |
4791 | +from ubuntu_debuginfod.utils.utils import lp_pub_iterator |
4792 | +from ubuntu_debuginfod.poller import Poller |
4793 | +from ubuntu_debuginfod.utils.baselp import BaseLaunchpadClass |
4794 | + |
4795 | + |
4796 | +class CommonLaunchpadPoller(Poller, BaseLaunchpadClass): |
4797 | + """Common poller class when dealing with Launchpad archives.""" |
4798 | + |
4799 | + def __init__(self, *args, **kwargs): |
4800 | + """Initialize the object. |
4801 | + |
4802 | + Look at BaseLaunchpadClass's docstring for an explanation |
4803 | + about the arguments.""" |
4804 | + super().__init__(*args, **kwargs) |
4805 | + self.launchpad_login() |
4806 | + self._logger.debug("Initializing CommonLaunchpadPoller") |
4807 | + |
4808 | + def poll_artifacts(self, archive, ppainfo, poller_fn): |
4809 | + """Poll artifacts from an archive. |
4810 | + |
4811 | + :param lazr.restfulclient.resource.Entry archive: The Launchpad archive. |
4812 | + |
4813 | + :param dict[str -> str] ppainfo: The dictionary containing |
4814 | + information about the PPA, if applicable. Otherwise, |
4815 | + None. |
4816 | + |
4817 | + :param function poller_fn: The polling function to be called. |
4818 | + |
4819 | + :rtype: dict[str -> str], datetime.datetime |
4820 | + :return: A dictionary containing "uri" as key, listing all |
4821 | + packages' self_link found. The most recent package |
4822 | + timestamp processed.""" |
4823 | + timestamp, real_timestamp = self._get_normal_and_real_timestamps() |
4824 | + |
4825 | + if ppainfo is None: |
4826 | + archive_label = f"'{archive.displayname}' archive" |
4827 | + else: |
4828 | + archive_label = f"ppa:{ppainfo['ppauser']}/{ppainfo['ppaname']} (private: {ppainfo['isprivateppa']})" |
4829 | + |
4830 | + self._logger.info( |
4831 | + f"Polling artifacts ({type(self).__name__}) created since '{real_timestamp}' from {archive_label}" |
4832 | + ) |
4833 | + |
4834 | + result = [] |
4835 | + latest_timestamp_created = timestamp |
4836 | + # Create an iterator for the Launchpad publications. |
4837 | + pkgiter = lp_pub_iterator( |
4838 | + poller_fn(order_by_date=True, created_since_date=real_timestamp) |
4839 | + ) |
4840 | + |
4841 | + for pkg in pkgiter: |
4842 | + msg = { |
4843 | + "uri": pkg.self_link, |
4844 | + } |
4845 | + |
4846 | + if ( |
4847 | + latest_timestamp_created is None |
4848 | + or pkg.date_created > latest_timestamp_created |
4849 | + ): |
4850 | + latest_timestamp_created = pkg.date_created |
4851 | + |
4852 | + if ppainfo is not None: |
4853 | + msg.update(ppainfo) |
4854 | + |
4855 | + result.append(msg) |
4856 | + |
4857 | + return result, latest_timestamp_created |
4858 | diff --git a/ubuntu_debuginfod/pollers/common_ppa_poller.py b/ubuntu_debuginfod/pollers/common_ppa_poller.py |
4859 | new file mode 100644 |
4860 | index 0000000..7f1c2ca |
4861 | --- /dev/null |
4862 | +++ b/ubuntu_debuginfod/pollers/common_ppa_poller.py |
4863 | @@ -0,0 +1,121 @@ |
4864 | +#!/usr/bin/python3 |
4865 | + |
4866 | +# Copyright (C) 2023 Canonical Ltd. |
4867 | + |
4868 | +# This program is free software: you can redistribute it and/or modify |
4869 | +# it under the terms of the GNU General Public License as published by |
4870 | +# the Free Software Foundation, either version 3 of the License, or |
4871 | +# (at your option) any later version. |
4872 | + |
4873 | +# This program is distributed in the hope that it will be useful, |
4874 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4875 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4876 | +# GNU General Public License for more details. |
4877 | + |
4878 | +# You should have received a copy of the GNU General Public License |
4879 | +# along with this program. If not, see <https://www.gnu.org/licenses/>. |
4880 | + |
4881 | +# Authors: Sergio Durigan Junior <sergio.durigan@canonical.com> |
4882 | + |
4883 | +import os |
4884 | +import lazr.restfulclient.errors |
4885 | +from ubuntu_debuginfod.utils.utils import generate_timestamp |
4886 | + |
4887 | + |
4888 | +class CommonPPAPoller: |
4889 | + """A common poller for PPAs.""" |
4890 | + |
4891 | + PPALIST_PATH = os.path.expanduser("~/.config/ubuntu-debuginfod/ppalist") |
4892 | + PRIVATE_PPALIST_PATH = os.path.expanduser( |
4893 | + "~/.config/ubuntu-debuginfod/ppalist-private" |
4894 | + ) |
4895 | + |
4896 | + def __init__(self, *args, **kwargs): |
4897 | + """Initialize the object. |
4898 | + |
4899 | + This class is meant to be use in a multiple-inheritance scenario.""" |
4900 | + super().__init__(*args, **kwargs) |
4901 | + self._logger.debug("Initializing CommonPPAPoller") |
4902 | + |
4903 | + def get_list_of_ppas(self): |
4904 | + """Get the list of PPAs to be polled. |
4905 | + |
4906 | + :rtype: list |
4907 | + :return: The list of PPAs. This list will contain one |
4908 | + dictionary per PPA, with the following keys: |
4909 | + |
4910 | + { "ppauser" : string, |
4911 | + "ppaname" : string, |
4912 | + "isprivateppa" : string }""" |
4913 | + ppafile = self.PPALIST_PATH if self._anonymous else self.PRIVATE_PPALIST_PATH |
4914 | + |
4915 | + self._logger.debug(f"Processing '{ppafile}'") |
4916 | + |
4917 | + if not os.path.isfile(ppafile): |
4918 | + self._logger.info(f"{ppafile} does not exist; doing nothing") |
4919 | + return [] |
4920 | + |
4921 | + ppas = [] |
4922 | + with open(ppafile, "r", encoding="UTF-8") as f: |
4923 | + for ppa in f.readlines(): |
4924 | + ppa = ppa.rstrip() |
4925 | + if ppa == "" or ppa.startswith("#"): |
4926 | + continue |
4927 | + ppa = ppa.split(":")[1].split("/") |
4928 | + ppa_user = ppa[0] |
4929 | + ppa_name = ppa[1] |
4930 | + self._logger.info(f"Processing ppa:{ppa_user}/{ppa_name}") |
4931 | + ppas.append( |
4932 | + { |
4933 | + "ppauser": ppa_user, |
4934 | + "ppaname": ppa_name, |
4935 | + "isprivateppa": "no" if self._anonymous else "yes", |
4936 | + } |
4937 | + ) |
4938 | + return ppas |
4939 | + |
4940 | + def poll_artifacts(self): |
4941 | + """Get the list of artifacts from Launchpad PPAs. |
4942 | + |
4943 | + :param poller_fn function: The poller function that should be |
4944 | + used when polling for artifacts. |
4945 | + |
4946 | + :param lp_people launchpadlib.launchpad.PersonSet: The pointer |
4947 | + to a Launchpad "people"set. This is usually |
4948 | + "self._lp.people". |
4949 | + |
4950 | + :rtype: dict, datetime.datetime |
4951 | + :return: Return a dictionary containing all artifacts found, |
4952 | + and also the new timestamp that should then be recorded by |
4953 | + calling record_timestamp. Timestamp can be None, in which |
4954 | + case nothing should be recorded.""" |
4955 | + ppalist = self.get_list_of_ppas() |
4956 | + if len(ppalist) == 0: |
4957 | + return [], None |
4958 | + |
4959 | + messages = [] |
4960 | + new_timestamp = generate_timestamp() |
4961 | + for ppainfo in ppalist: |
4962 | + ppauser = ppainfo["ppauser"] |
4963 | + ppaname = ppainfo["ppaname"] |
4964 | + try: |
4965 | + lpuser = self._lp.people[ppauser] |
4966 | + except KeyError: |
4967 | + self._logger.error(f"Launchpad user {ppauser} does not exist") |
4968 | + continue |
4969 | + |
4970 | + try: |
4971 | + lpppa = lpuser.getPPAByName(name=ppaname) |
4972 | + except lazr.restfulclient.errors.NotFound: |
4973 | + self._logger.error( |
4974 | + f"Launchpad PPA ppa:{ppauser}/{ppaname} does not exist" |
4975 | + ) |
4976 | + continue |
4977 | + |
4978 | + self._logger.info( |
4979 | + f"Getting list of artifacts ({type(self).__name__}) from ppa:{ppauser}/{ppaname} (private: {not self._anonymous})" |
4980 | + ) |
4981 | + msg, _ = super().poll_artifacts(archive=lpppa, ppainfo=ppainfo) |
4982 | + messages += msg |
4983 | + |
4984 | + return messages, new_timestamp |
4985 | diff --git a/ubuntu_debuginfod/pollers/ddebpoller.py b/ubuntu_debuginfod/pollers/ddebpoller.py |
4986 | new file mode 100644 |
4987 | index 0000000..ea96030 |
4988 | --- /dev/null |
4989 | +++ b/ubuntu_debuginfod/pollers/ddebpoller.py |
4990 | @@ -0,0 +1,69 @@ |
4991 | +#!/usr/bin/python3 |
4992 | + |
4993 | +# Copyright (C) 2022 Canonical Ltd. |
4994 | + |
4995 | +# This program is free software: you can redistribute it and/or modify |
4996 | +# it under the terms of the GNU General Public License as published by |
4997 | +# the Free Software Foundation, either version 3 of the License, or |
4998 | +# (at your option) any later version. |
4999 | + |
5000 | +# This program is distributed in the hope that it will be useful, |
Hi Sergio,
As we discussed before, I will review this one :)