Merge ~litios/ubuntu-cve-tracker:components-for-esm into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

Right now, umt tooling will report that "main" is the component for all packages in ESM releases. This recently caused an issue while building a package because of dependency resolution.

This MR tries to approach this issue by looking at the original release to find out which the real component is.

[David Fernandez Gonzalez (litios)]

$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import source_map;import cve_lib; srcmap = source_map.load()
>>> cve_lib.is_supported(srcmap, "ffmpeg", "esm-apps/bionic")
True
>>> cve_lib.is_supported(srcmap, "ffmpeg", "bionic")
False
>>> print(srcmap['app-engine/focal']['python2.7'])
{'pocket': '', 'section': 'universe'}
>>> print(srcmap['bitglass-django/xenial']['u-msgpack-python'])
{'pocket': '', 'section': 'main'}

$ umt search python2.7
Running search command.

Ubuntu packages:

trusty/esm: 2.7.6-8ubuntu0.6+esm13, Pocket: release, Component: main
esm-infra/xenial: 2.7.12-1ubuntu0~16.04.18+esm3, Pocket: release, Component: main
esm-apps/focal: 2.7.18-1~20.04.3+esm1, Pocket: release, Component: universe
esm-apps/jammy: 2.7.18-13ubuntu1.1+esm2, Pocket: release, Component: universe
trusty: 2.7.6-8ubuntu0.5, Pocket: updates, Component: main
xenial: 2.7.12-1ubuntu0~16.04.18, Pocket: updates, Component: main
bionic: 2.7.17-1~18.04ubuntu1.10, Pocket: updates, Component: main
focal: 2.7.18-1~20.04.3, Pocket: updates, Component: universe
jammy: 2.7.18-13ubuntu1.1, Pocket: updates, Component: universe
kinetic: 2.7.18-13ubuntu2, Pocket: release, Component: universe
lunar: 2.7.18-13ubuntu2, Pocket: release, Component: universe

Other packages:

buster: 2.7.16-2+deb10u1, Pocket: release, Component: main
stable: 2.7.18-8, Pocket: release, Component: main
stretch: 2.7.13-2+deb9u6, Pocket: updates, Component: main

[Eduardo Barretto (ebarretto)]

lgtm, thanks!

review: Approve

[Alex Murray (alexmurray)]

LGTM - thanks - just one minor change to please include a comment / docstring to explain what get_active_releases_with_esm() does compared to get_active_esm_releases()

review: Approve

[David Fernandez Gonzalez (litios)]

Docstrings were added to clarify that.
Thanks, Eduardo and Alex, for reviewing!

diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py
index 6968bd4..e10fb9d 100755
--- a/scripts/cve_lib.py
+++ b/scripts/cve_lib.py
@@ -2366,20 +2366,50 @@ def is_active_release(rel):
 
 # takes a standard release name
 # XXX should perhaps adjust that
-def is_active_esm_release(rel):
-    if not is_active_release(rel):
-        esm_rel = get_esm_name(rel)
+def is_active_esm_release(rel, component='main'):
+    if not is_active_release(rel) or \
+        component == 'universe' or component == 'multiverse':
+        esm_rel = get_esm_name(rel, component)
         if esm_rel:
             return esm_rel not in eol_releases
     return False
 
+def get_active_releases_with_esm():
+    """Return Ubuntu releases with, at least, one active ESM release."""
+    active_esm_releases = []
+    all_esm_releases = set(esm_releases + esm_apps_releases + esm_infra_releases + ros_esm_releases)
 
-def get_esm_name(rel, component=None):
+    # Get ESM active releases that are EOL
+    for esm_rel in all_esm_releases:
+        if is_active_esm_release(esm_rel):
+            active_esm_releases.append(esm_rel)
+
+    # Get active releases that also have ESM (apps)
+    for esm_rel in all_esm_releases:
+        if is_active_release(esm_rel):
+            active_esm_releases.append(esm_rel)
+
+    return active_esm_releases
+
+def get_active_esm_releases():
+    """Return all active ESM releases."""
+    active_esm_releases = []
+    for rel in get_active_releases_with_esm():
+        for component in components:
+            if is_active_esm_release(rel, component):
+                active_esm_releases.append(get_esm_name(rel, component))
+
+    return set(active_esm_releases)
+
+# Defaults to main for historical reasons
+def get_esm_name(rel, component='main'):
     if rel in esm_releases:
         return rel + '/esm'
-    elif rel in esm_apps_releases and component:
+    elif rel in esm_apps_releases and \
+        (component == 'universe' or component == 'multiverse'):
         return 'esm-apps/' + rel
-    elif rel in esm_infra_releases:
+    elif rel in esm_infra_releases and \
+        (component == 'main' or component == 'restricted'):
         return 'esm-infra/' + rel
     elif rel in ros_esm_releases:
         return 'ros-esm/' + rel
@@ -2402,6 +2432,12 @@ def is_supported(map, pkg, rel, cvedata=None):
        ('universe-binary' in cvedata['tags'][pkg] or
         'not-ue' in cvedata['tags'][pkg]):
         return False
+
+    # If it's inside a subproject, it's supported
+    if (rel in external_releases or rel in get_active_esm_releases()) and rel in map \
+        and pkg in map[rel]:
+        return True
+
     # Look for a supported component
     if rel in map and pkg in map[rel] and \
        (map[rel][pkg]['section'] == 'main' or
diff --git a/scripts/source_map.py b/scripts/source_map.py
index 91f5dbe..e4d221e 100755
--- a/scripts/source_map.py
+++ b/scripts/source_map.py
@@ -462,6 +462,10 @@ def load_subprojects_lists(releases=None):
     if releases is None:
         releases = cve_lib.all_releases
 
+    all_sources_esm = {}
+    for item in _find_sources(releases=cve_lib.get_active_releases_with_esm(), skip_eol_releases=False):
+        load_sources_collection(item, all_sources_esm)
+
     for rel in releases:
         _, _, _, details = cve_lib.get_subproject_details(rel)
         if details is None:
@@ -501,7 +505,16 @@ def load_subprojects_lists(releases=None):
                 if pkg not in map[rel]:
                     map[rel][pkg] = dict()
                     map[rel][pkg]['pocket'] = ''
-                    map[rel][pkg]['section'] = 'main'
+
+                    orig_rel = cve_lib.get_orig_rel_name(rel)
+                    if not orig_rel in all_sources_esm or pkg not in all_sources_esm[orig_rel]:
+                        # Not an ESM subproject
+                        _, orig_rel = cve_lib.product_series(rel)
+
+                    if orig_rel in all_sources_esm and pkg in all_sources_esm[orig_rel]:
+                        map[rel][pkg]['section'] = all_sources_esm[orig_rel][pkg]['section']
+                    else:
+                        map[rel][pkg]['section'] = 'main'
 
                     if '|' in pkg:
                         main_package_name = pkg.split('|')[0]