Merge into master : overhaul : lp:ubuntu-debuginfod : Git : Code : ubuntu-debuginfod

Status:	Superseded
Proposed branch:	ubuntu-debuginfod:overhaul
Merge into:	ubuntu-debuginfod:master
Diff against target:	0 lines
Related bugs:	Link a bug report

Reviewer	Date Requested	Status
Athos Ribeiro	2023-04-25	Pending
Canonical Server Reporter	2023-04-25	Pending
Review via email: mp+441905@code.launchpad.net

Description of the change

This MP is the big overhaul/rewrite of ubuntu-debuginfod that we talked about.

There are many new things being done here, but most of the diff is basically code reorganization to make better use of Python's capabilities.

I'll try to make a list of important things to consider when reviewing this code:

1) I'm make heavy use of base classes for pollers and getters. Using pollers and as example:

- You will see that there is the most basic class, Poller, which implements only the necessary/common bits for a generic poller to operate.

- On top of it, there is a CommonLaunchpadPoller class, which extends Poller and implements generic methods for both a ddeb and a source package poller to work. The class also inherits from BaseLaunchpadClass, which follows the mixin design pattern and is useful because it provides a "launchpad_login" methods to classes that need this feature.

- On top of CommonLaunchpadPoller you will find the DdebPoller and SourcePackagePoller classes, which finally implement the necessary method (in this case, poll_artifacts) that will do the polling of their respective artifacts.

- PPAs are kind of a special beast, so they have their own base class (CommonPPAPoller) which is actually crafted to be used specifically in multiple-inheritance scenarios. This base class is used alongside both DdebPoller and SourcePackagePoller to implement the PPA methods to fetch ddebs and source packages (for public and private PPAs).

The design of getters follows exactly the same rationale.

2) You will find that I broke down the "poll_lp.py" service in two:

- The first one, "launchpad-poller.py", only performs the polling of artifacts from LP and adds them to a local database named "jobs2dispatch". It's important to say that this poller does *not* do any complex filtering to determine if the artifact is actually interesting for debuginfod or not; it simply dumps the contents of a getPublished{Sources,Binaries} call into the local table. This was required because the service was hitting some strange "Error 400" problems with LP, and part of the reason seemed to be the fact that "poll_lp.py" was performing too many calls to the API.

- There is a second service now, called "launchpad-dispatcher.py". This service is responsible for iterating over the entries of the "jobs2dispatch" table and determine if they are interesting to us. If positive, the service dispatches the job to Celery.

Currently, the dispatcher is a single service that runs sequentially over the table. Until now I haven't found any issues with this approach, but eventually I intend to transform "launchpad-dispatcher.py" into a Celery service as well, so that the processing & dispatching operations can be done in parallel.

3) This is now a Python package. You should be able to use virtualenv and pip to install it locally. I also revamped the README file and included instructions on how to set things up locally.

4) I disabled a bunch of pylint warnings that were, IMO, false positives in my case. pylint is happy with the code, as well as flake8 and mypy. But I'm open to suggestions/comments, of course.

5) The package now has a testsuite. It is aimed at unittesting so far, but it did help me catch a few silly mistakes when I was developing things. I consider the testsuite to be a WIP; I still want to add more tests and, eventually, work towards having a proper CI/CD setup (thanks for the idea, Athos!).

I think that's it. I'm not in a big hurry to get this approved/deployed, so I'm fine to wait if you would like to talk in person about the MP during the upcoming sprint.