Merge ubuntu-debuginfod:getter into ubuntu-debuginfod:master

[Sergio Durigan Junior]

This MP implements the "getter" classes, along with a Celery application that makes use of those classes and transforms them into tasks.

The idea here is to follow what I did for the poller: have a basic class that implements primitives, and extend it in order to handle a specific scenario. In our case, there are currently two objects that will need to be fetched: a list of ddebs, and the source code for the package (with patches applied). The former is a simple thing to do, but the latter is more involved.

Before we even start downloading the source code, we need to determine whether it is suitable to be consumed by debuginfod. We do that by inspecting the DWARF of each ddeb we've downloaded and looking for a specific pathname that should be embedded into the DW_AT_comp_dir tag. If we find it, we can proceed with downloading the source code. And then, a bit more processing needs to be done...

There are two basic ways of grabbing the source code: I can either download the source artifacts from LP and use dpkg-source to extract them (a la "dget"), or I can use git-ubuntu's "applied" tags. You will notice that both methods have been implemented, and will be tried in order.

The getters are idempotent, which is almost a requirement due to the retry mechanism that Celery offers. Please let me know if you need more details about the Celery parts and I'll be happy to provide them.

You can run this application by invoking:

celery -A debuginfod worker -c 4 --loglevel=INFO

I will update the README file soon to cover that.

[Athos Ribeiro (athos-ribeiro)]

Thanks, Sergio!

I added a few comments below and inline:

- The README file needs update to the dependencies now you also require requests here.

- You could use the logger lazy string generation by using %s instead of f strings.

review: Needs Information

[Lena Voytek (lvoytek)]

Added some additional comments alongside Athos'

Thanks!

review: Needs Information

[Sergio Durigan Junior (sergiodj)]

Thanks, Athos and Lena.

I think I've addressed everything, except for Athos' comment about using lazy strings instead of f-strings. Please let me know what you think.

[Lena Voytek (lvoytek)]

All changes brought up look good now, thanks!

review: Approve

[Bryce Harrington (bryce)]

I also wondered about suggesting using %s style strings for logger in an earlier review. Logger also supports f-string style for lazy substitution, there's a config variable you set when you're creating the logger object.

However, I didn't mention it because I don't have much experience with logger myself (I should start using it more), and couldn't express the advantage of doing it that way vs. passing static strings. I think it's fine the way Sergio's coded it, but am curious if there's a strong rationale for using logger's string formatting instead?

[Athos Ribeiro (athos-ribeiro)]

Thanks Sergio.

@Bryce, for the logger lazy evaluation, the advantage is that python will only generate the strings (i.e., perform the substitutions) in case the string is actually going to be logged (instead of always generating the string).

review: Approve

[Sergio Durigan Junior (sergiodj)]

Thanks for the review, Athos, and thanks for the further clarification, Bryce.

I merged this branch after replacing [] by .get, as suggested by Athos.

[Bryce Harrington (bryce)]

Lena and Athos already are on top of the review, so leaving approval to them, but here's a few more review comments. Mostly style suggestions and nitpicks, nothing that I can see that should block landing.

[Sergio Durigan Junior (sergiodj)]

Thanks for the review, Bryce.

As you might have noticed, I merged the branch already, but your comments are valuable as ever and I intend to address them in a subsequent commit.

diff --git a/README.md b/README.md
index 995ba36..8fe7f00 100644
--- a/README.md
+++ b/README.md
@@ -13,6 +13,8 @@ python3-debian
 python3-git
 python3-celery
 python3-launchpadlib
+python3-requests
+python3-sdnotify
 ```
 
 * Applications
diff --git a/ddebgetter.py b/ddebgetter.py
new file mode 100644
index 0000000..62eebc6
--- /dev/null
+++ b/ddebgetter.py
@@ -0,0 +1,383 @@
+#!/usr/bin/python3
+
+# Copyright (C) 2022 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Authors: Sergio Durigan Junior <[e-mail]>
+
+import os
+from urllib import parse
+import lzma
+import tarfile
+import subprocess
+
+from debian import debfile
+import tempfile
+from elftools.common.utils import bytelist2string
+from elftools.common.exceptions import ELFError, ELFParseError, DWARFError
+from elftools.elf.elffile import ELFFile
+
+from git import Git, Repo
+from git.exc import GitCommandError
+
+from debuggetter import DebugGetter, DebugGetterRetry, DEFAULT_MIRROR_DIR
+
+
+class DdebGetter(DebugGetter):
+    """Get (fetch) a ddeb."""
+
+    def __init__(self, mirror_dir=DEFAULT_MIRROR_DIR):
+        """Initialize the object.
+
+        See DebugGetter's __init__ for an explanation of the arguments."""
+        super().__init__(subdir="ddebs", mirror_dir=mirror_dir)
+
+    def process_request(self, request):
+        """Process a request, usually coming from Celery.
+
+        :param request: The dictionary containing the information
+            necessary to fetch this ddeb.
+        :type request: dict(str : str)
+        """
+        if not request or not request.get("ddebs"):
+            return
+
+        self._logger.debug(f"Processing request to download ddebs: {request}")
+        self._download_ddebs(
+            request["source_package"],
+            request["component"],
+            request["ddebs"],
+        )
+
+    def _download_ddebs(self, source_package, component, urls):
+        """Download the ddebs associated with a package/version.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str component: Source package component.
+
+        :param list urls: List of ddeb URLs."""
+        savepath = self._make_savepath(source_package, component)
+        os.makedirs(savepath, mode=0o755, exist_ok=True)
+        for url in urls:
+            self._logger.debug(f"Downloading '{url}' into '{savepath}'")
+            self._download_from_lp(url, savepath)
+
+
+# The function below was taken from git-ubuntu.  We need to make
+# sure we perform the same version -> tag transformation as it
+# does.
+def _git_dep14_tag(version):
+    """Munge a version string according to http://dep.debian.net/deps/dep14/
+
+    :param str version: The version to be adjusted."""
+    version = version.replace("~", "_")
+    version = version.replace(":", "%")
+    version = version.replace("..", ".#.")
+    if version.endswith("."):
+        version = version + "#"
+    if version.endswith(".lock"):
+        pre, _, _ = version.partition(".lock")
+        version = pre + ".#lock"
+    return version
+
+
+class DdebSourceCodeGetter(DebugGetter):
+    """Get (fetch) the source code associated with a ddeb."""
+
+    def __init__(self, mirror_dir="/srv/debug-mirror"):
+        super().__init__(mirror_dir=mirror_dir, subdir="ddebs")
+
+    def process_request(self, request):
+        """Process a request, usually coming from Celery.
+
+        :param request: The dictionary containing the information
+            necessary to fetch this source code.
+        :type request: dict(str : str)
+        """
+        if not request or not request.get("ddebs"):
+            return
+
+        self._logger.debug(
+            f"Processing request to download source code (for ddebs): {request}"
+        )
+        self._process_source(
+            request["source_package"],
+            request["version"],
+            request["component"],
+            request["ddebs"],
+            request["sources"],
+        )
+
+    def _get_comp_dirs(self, debug_file, filename):
+        """Get the list of available DW_AT_comp_dir declarations from the
+        DWARF file.
+
+        :param TextIOWrapper debug_file: The .debug file.
+        """
+        elf_file = ELFFile(debug_file)
+
+        if not elf_file.has_dwarf_info():
+            self._logger.debug(f"'{filename}' doesn't have DWARF")
+            return []
+
+        result = []
+        dwarf_file = elf_file.get_dwarf_info()
+        for cu in dwarf_file.iter_CUs():
+            attr = cu.get_top_DIE().attributes
+            if "DW_AT_comp_dir" in attr.keys():
+                value = attr["DW_AT_comp_dir"].value
+                if isinstance(value, bytes):
+                    result.append(bytelist2string(value).decode("UTF-8"))
+        self._logger.debug(
+            f"Found the following DW_AT_comp_dir for '{filename}':\n{result}"
+        )
+        return result
+
+    def _should_fetch_source_for_debugfile(self, source_package, version, filepath):
+        """Whether we should fetch the source code for the specified source
+        package/deb.
+
+        We return True if the debug file associated with the source
+        package has any DW_AT_comp_dir entity whose path points to
+        "/usr/src/source_package-version/".
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str filepath: Full path to the .ddeb file."""
+        deb_file = debfile.DebFile(filepath)
+        # The path we expect to find in any of the DW_AT_comp_dirs.
+        expected_path = f"/usr/src/{source_package}-{version}/"
+        for deb_internal_file in list(deb_file.data):
+            if not deb_internal_file.endswith(".debug"):
+                continue
+            with tempfile.TemporaryFile() as debug_file:
+                debug_file.write(deb_file.data.get_content(deb_internal_file))
+                try:
+                    comp_dirs = self._get_comp_dirs(debug_file, deb_internal_file)
+                except (ELFError, ELFParseError, DWARFError) as e:
+                    self._logger.warning(
+                        f"Exception while looking for DW_AT_comp_dirs: {e}"
+                    )
+                    # There's no point in retrying anything here, so
+                    # we just generate a warning and keep going.
+                    continue
+
+                for comp_dir in comp_dirs:
+                    if comp_dir.startswith(expected_path):
+                        self._logger.debug(f"Found a good DW_AT_comp_dir: {comp_dir}")
+                        return True
+        return False
+
+    def _should_fetch_source(self, source_package, version, component, filenames):
+        """Return True if we should fetch the source code for a specific
+        package, False otherwise.
+
+        Whether or not we should fetch the source code is determined
+        by the presence of a specific path prefix (/usr/src/...) in
+        the DW_AT_comp_dir declarations of a DWARF file.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str component: Source package component (main,
+            universe, etc.)
+
+        :param list urls: is the list of URLs used to fetch the respective debug
+            information for the package.
+        """
+        savepath = self._make_savepath(source_package, component)
+        for fname in filenames:
+            filepath = os.path.join(savepath, fname)
+            if not os.path.isfile(filepath):
+                continue
+            if self._should_fetch_source_for_debugfile(
+                source_package, version, filepath
+            ):
+                return True
+        return False
+
+    def _download_source_code_from_git(self, source_package, version, filepath):
+        """Download the source code using Launchpad's git repository.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str filepath: The full pathname where the resulting
+            source code tarball should be saved.
+        """
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
+            g = Git()
+            git_dir = os.path.join(source_dir, f"{source_package}")
+            tag = "applied/" + _git_dep14_tag(f"{version}")
+            self._logger.debug(
+                f"Cloning '{source_package}' git repo into '{git_dir}' (tag: '{tag}')"
+            )
+            try:
+                g.clone(
+                    f"https://git.launchpad.net/ubuntu/+source/{source_package}",
+                    git_dir,
+                    depth=1,
+                    branch=tag,
+                )
+            except GitCommandError as e:
+                # Couldn't perform the download.  Let's signal and
+                # bail out.
+                self._logger.warning(
+                    f"Could not clone git repo for '{source_package}': {e}"
+                )
+                return False
+            repo = Repo(git_dir)
+            prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
+            with lzma.open(filepath, "w") as xzfile:
+                self._logger.debug(
+                    f"Archiving git repo for '{source_package}-{version}' as '{filepath}'"
+                )
+                repo.archive(xzfile, prefix=prefix_path, format="tar")
+
+        return True
+
+    def _adjust_tar_filepath(self, tarinfo):
+        """Adjust the filepath for a TarInfo file.
+
+        This function is needed because TarFile.add strips the leading
+        slash from the filenames, so we have to workaround it by
+        re-adding the slash ourselves.
+
+        This function is intended to be used as a callback provided to
+        TarFile.add.
+
+        :param TarInfo tarinfo: The tarinfo.
+        """
+        tarinfo.name = f"/{tarinfo.name}"
+        return tarinfo
+
+    def _download_source_code_from_dsc(
+        self, source_package, version, filepath, source_urls
+    ):
+
+        """Download the source code using the .dsc file.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str filepath: The full pathname where the resulting
+            source code tarball should be saved.
+
+        :param list source_urls: List of URLs used to fetch the source
+            package.  This is usually the list returned by the
+            sourceFileUrls() Launchpad API call.
+        """
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as source_dir:
+            for url in source_urls:
+                self._download_from_lp(url, source_dir)
+
+            dscfile = os.path.join(source_dir, f"{source_package}_{version}.dsc")
+            if not os.path.isfile(dscfile):
+                self._logger.warning(f"'{dscfile}' doesn't exist, but it should.")
+                return False
+
+            outdir = os.path.join(source_dir, "outdir")
+            try:
+                subprocess.run(
+                    ["/usr/bin/dpkg-source", "-x", dscfile, outdir],
+                    cwd=source_dir,
+                    check=True,
+                )
+            except subprocess.CalledProcessError:
+                self._logger.warning(f"Call to 'dpkg-source -x' failed.")
+                return False
+
+            if not os.path.isdir(outdir):
+                self._logger.warning(
+                    f"'{outdir}' has not been created by 'dpkg-source -x'."
+                )
+                return False
+
+            prefix_path = os.path.join("/usr/src/", f"{source_package}-{version}/")
+
+            with tarfile.open(filepath, "w:xz") as tfile:
+                tfile.add(outdir, arcname=prefix_path, filter=self._adjust_tar_filepath)
+
+            return True
+
+    def _download_source_code(self, source_package, version, component, source_urls):
+        """Download the source code for a package.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str component: Source package component.
+
+        :param list source_urls: List of source file URLS."""
+        savepath = self._make_savepath(source_package, component)
+        txzfilepath = os.path.join(savepath, f"{source_package}-{version}.tar.xz")
+        if os.path.exists(txzfilepath):
+            self._logger.debug(f"'{txzfilepath}' already exists; doing nothing.")
+            return
+
+        if self._download_source_code_from_dsc(
+            source_package, version, txzfilepath, source_urls
+        ):
+            self._logger.info(
+                f"Downloaded source code from dsc for '{source_package}-{version}' as '{txzfilepath}'"
+            )
+            return
+
+        if self._download_source_code_from_git(source_package, version, txzfilepath):
+            self._logger.info(
+                f"Downloaded source code from git for '{source_package}-{version}' as '{txzfilepath}'"
+            )
+            return
+
+        # In the (likely?) event that there is a problem with
+        # Launchpad, let's raise an exception signalling that we'd
+        # like to retry the task.
+        raise DebugGetterRetry()
+
+    def _process_source(
+        self, source_package, version, component, ddeb_urls, source_urls
+    ):
+        """Process the request to fetch the source code.
+
+        We only download the source code if we know it will be
+        properly indexed by debuginfod.
+
+        :param str source_package: Source package name.
+
+        :param str version: Source package version.
+
+        :param str component: Source package component.
+
+        :param list urls: List of ddeb URLs."""
+        # Convert URLs into filenames.
+        filenames = [
+            os.path.basename(parse.urlparse(fname).path) for fname in ddeb_urls
+        ]
+        if not self._should_fetch_source(source_package, version, component, filenames):
+            self._logger.info(
+                f"Should not fetch source code for '{source_package}-{version}'"
+            )
+            return
+
+        self._download_source_code(source_package, version, component, source_urls)
diff --git a/ddebpoller.py b/ddebpoller.py
index a19d92c..24ba613 100644
--- a/ddebpoller.py
+++ b/ddebpoller.py
@@ -65,6 +65,7 @@ class DdebPoller(DebugPoller):
                     "component": pkg.component_name,
                     "distro_series": distro_series,
                     "ddebs": ddeb_urls,
+                    "sources": pkg.sourceFileUrls(),
                 }
                 self._logger.debug(
                     f"For source package '{pkgname}-{pkgver}', found {ddebs_len}:\n{ddeb_urls}"
diff --git a/debuggetter.py b/debuggetter.py
new file mode 100644
index 0000000..9bb6102
--- /dev/null
+++ b/debuggetter.py
@@ -0,0 +1,92 @@
+#!/usr/bin/python3
+
+# Copyright (C) 2022 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Authors: Sergio Durigan Junior <[e-mail]>
+
+import os
+import requests
+from urllib import parse
+import logging
+
+
+class DebugGetterTimeout(Exception):
+    """Default exception to raise when there's a timeout issue."""
+
+
+class DebugGetterRetry(Exception):
+    """Default exception to raise when we'd like to signal that the task
+    should be retried."""
+
+
+DEFAULT_MIRROR_DIR = "/srv/debug-mirror"
+
+class DebugGetter:
+    """Base class for a Debug Getter."""
+
+    def __init__(self, subdir, mirror_dir=DEFAULT_MIRROR_DIR):
+        """Initialize the object.
+
+        :param str mirror_dir: The directory we use to save the
+            mirrored files from Launchpad.
+
+        :param str subdir: The subdirectory (insider mirror_dir) where
+            the module will save its files.  For example, a ddeb
+            getter module should specify "ddebs" here.
+        """
+        self._mirror_dir = mirror_dir
+        self._subdir = subdir
+        self._logger = logging.getLogger(__name__)
+
+    def _make_savepath(self, source_package, component):
+        """Return the full save path for a package.
+
+        :param str source_package: The package name.
+        :param str component: The component name (main, universe, etc.).
+        """
+        return os.path.join(
+            self._mirror_dir,
+            self._subdir,
+            component,
+            source_package[:1],
+            source_package,
+        )
+
+    def _download_from_lp(self, url, savepath):
+        filepath = os.path.join(savepath, os.path.basename(parse.urlparse(url).path))
+        if os.path.exists(filepath):
+            self._logger.debug(f"'{filepath}' exists, doing nothing")
+            return
+
+        try:
+            with requests.Session() as s:
+                with s.get(url, allow_redirects=True, timeout=10, stream=True) as r:
+                    r.raise_for_status()
+                    with open(filepath, "wb") as f:
+                        # 10 MB for chunk_size should be enough...
+                        for chunk in r.iter_content(chunk_size=10 * 1024 * 1024):
+                            f.write(chunk)
+        except (
+            requests.ConnectionError,
+            requests.HTTPError,
+            requests.ConnectTimeout,
+            requests.ReadTimeout,
+            requests.Timeout,
+        ) as e:
+            self._logger.warning(f"Timeout while downloading '{url}': '{e}'")
+            raise DebugGetterTimeout()
+
+        self._logger.debug(f"Saved '{filepath}'")
diff --git a/debuginfod.py b/debuginfod.py
new file mode 100644
index 0000000..d64efd4
--- /dev/null
+++ b/debuginfod.py
@@ -0,0 +1,80 @@
+#!/usr/bin/python3
+
+# Copyright (C) 2022 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+# Authors: Sergio Durigan Junior <[e-mail]>
+
+from celery import Celery
+
+from debuggetter import DebugGetterTimeout, DebugGetterRetry
+
+from ddebgetter import DdebGetter, DdebSourceCodeGetter
+
+app = Celery("debuginfod", broker="pyamqp://guest@localhost//")
+
+app.conf.update(
+    task_serializer="json",
+    accept_content=["json"],
+    result_serializer="json",
+    timezone="America/Toronto",
+    enable_utc=True,
+    task_acks_late=True,
+    # Only one task per worker.
+    worker_prefetch_multiplier=1,
+    worker_log_format="[%(asctime)s: %(levelname)s/%(processName)s] %(funcName)s: %(message)s",
+)
+
+
[e-mail](
+    name="grab_ddebs",
+    autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
+    # 1 day
+    default_retry_delay=24 * 60 * 60,
+    # Launchpad can be problematic, so we retry every day for a
+    # week.
+    retry_kwargs={"max_retries": 7},
+)
+def grab_ddebs(msg):
+    """Dispatch the DdebGetter task.
+
+    :param dict(str -> str) msg: The dictionary containing the message
+        that will be processed by the getter.
+    """
+    g = DdebGetter()
+    g.process_request(msg)
+
+
[e-mail](
+    name="grab_ddebs_sources",
+    autoretry_for=(DebugGetterTimeout, DebugGetterRetry),
+    # 1 day
+    default_retry_delay=24 * 60 * 60,
+    # Launchpad can be problematic, so we retry every day for a
+    # week.
+    retry_kwargs={"max_retries": 7},
+)
+def grab_ddebs_sources(msg):
+    """Dispatch the DdebSourceCodeGetter task.
+
+    :param dict(str -> str) msg: The dictionary containing the message
+        that will be processed by the getter.
+    """
+    g = DdebSourceCodeGetter()
+    g.process_request(msg)
+
+
+if __name__ == "__main__":
+    app.start()