Merge ~litios/ubuntu-cve-tracker:aliases-v2 into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

This MR is to use aliases the other way around as it is right now:
* supported.txt will contain the name of the package in the subproject
* aliases.yaml will contain the aliases for the package name in Ubuntu

Example:

$ cat subprojects/soss/conda-focal/supported.txt| grep cryptography
cryptography
$ cat subprojects/soss/conda-focal/aliases.yaml
---
cryptography: [python-cryptography]

When doing triage, we want to make the process as easy as possible; therefore, the tool will expect that the user always provides the Ubuntu name of the package. It will warn the user if using the subproject name instead of the Ubuntu one (if it exists), as this will break any other aliases in other subprojects.

In case the package doesn't exist in Ubuntu, the tool will warn the user that it exists in a subproject and the person doing triage can provide the subproject name(s) so the CVE gets created.

Example output with check-cves and check-syntax: https://pastebin.canonical.com/p/PyzT5tvjmP/

[David Fernandez Gonzalez (litios)]

It seems like the preview diff is broken, all the changes to CVEs are already merged in master.

[Alex Murray (alexmurray)]

A few questions inline.

review: Needs Fixing

[David Fernandez Gonzalez (litios)]

Thanks for the input Alex!

I was relying too much on the user to do the manual work of figuring out the right alias when triaging.

I added some new commits to make it more transparent.

About the multi-versioned packages Steve mentioned, the last commit should also fix the issue.

[David Fernandez Gonzalez (litios)]

I added a final commit to deal with aliases in SOSS.

SOSS package names have the format PACKAGE|VERSION, and that's what the contents of supported.txt are.

Right now, the format of aliases.yaml would be something like this to match it:

"cryptography|1.0.0": ["python-cryptography"],
"cryptography|2.0.0": ["python-cryptography"],
...

but really, whatever is after the | doesn't seem useful for aliases, as the package is always the same. So the proposed format from @ccdm94 was:

"cryptography": ["python-cryptography"],

and that should cover all of the "cryptography|VERSION" listed in `supported.txt`.

Testing: https://pastebin.canonical.com/p/fXYYYN9jmV/

[Alex Murray (alexmurray)]

LGTM - but one minor query about whether we even need check-syntax-fixup anymore and hence could these changes to it be removed and the file deleted instead?

review: Approve

[Alex Murray (alexmurray)]

Actually as pointed out by mdeslaur, check-syntax-fixup is still useful for a bunch of use-cases so please ignore those comments.

[David Fernandez Gonzalez (litios)]

Thanks Alex!

diff --git a/scripts/active_edit b/scripts/active_edit
index 9d0df27..ec36cfe 100755
--- a/scripts/active_edit
+++ b/scripts/active_edit
@@ -43,6 +43,8 @@ for release in cve_lib.all_releases:
         source_releases.append(release)
 source = source_map.load(releases=source_releases, skip_eol_releases=False)
 
+added_rel_pkg = []
+
 def pkg_in_rel(pkg,rel):
     if rel in ['upstream']:
         return True
@@ -88,10 +90,12 @@ def _add_pkg(p, fp, fixed, parent, embargoed):
         if len(fixed[p]) > 2:
             fixed_in_release_version = fixed[p][2]
     for rel in ['upstream'] + list(source.keys()):
+        aliases = source_map.get_aliases_of_ubuntu_package(source, p, rel)
+
         # determine default state but override this if pkg_db has a
         # better one
         state = "needs-triage"
-        if not pkg_in_rel(p, rel):
+        if not pkg_in_rel(p, rel) and not aliases:
             # package doesn't exist in this release - see if it wants a
             # DNE entry
             if release_wants_dne(rel):
@@ -128,9 +132,13 @@ def _add_pkg(p, fp, fixed, parent, embargoed):
             print('%s_%s: %s' % (rel, p, state), file=fp)
         # TODO: figure out how to store subproject details for embargoed CVEs
         elif not embargoed:
-            # add this to subprojects for rel
-            with open(os.path.join(cve_lib.get_external_subproject_cve_dir(rel), cve), "a") as f:
-                print('%s_%s: %s' % (rel, p, state), file=f)
+            pkgs_to_add = aliases if aliases else [p]
+            for pkg in pkgs_to_add:
+                rel_pkgname = rel + '/' + pkg
+                if not rel_pkgname in added_rel_pkg:
+                    with open(os.path.join(cve_lib.get_external_subproject_cve_dir(rel), cve), "a") as f:
+                        print('%s_%s: %s' % (rel, pkg, state), file=f)
+                    added_rel_pkg.append(rel_pkgname)
 
 def add_pkg(p, fp, fixed, parent, embargoed):
     # skip packages which don't actually exist in any release
diff --git a/scripts/check-cves b/scripts/check-cves
index 6eb9f58..2bc15d2 100755
--- a/scripts/check-cves
+++ b/scripts/check-cves
@@ -884,9 +884,26 @@ class CVEHandler(xml.sax.handler.ContentHandler):
         external_subprojects = {}
         for subproject in cve_lib.external_releases:
             if subproject in source:
+                aliases = source_map.get_all_aliases(source, subproject)
                 for hint in software_hints_from_cve_description:
-                    if hint in source[subproject] and hint not in common_words:
+                    if hint in common_words:
+                        continue
+
+                    if hint in source[subproject]:
+                        if 'aliases' in source[subproject][hint]:
+                            external_subprojects.setdefault(subproject, set()).add(','.join(source[subproject][hint]['aliases']))
+                        else:
+                            external_subprojects.setdefault(subproject, set()).add(hint)
+                    elif hint in aliases:
                         external_subprojects.setdefault(subproject, set()).add(hint)
+                    else:
+                        pkgs = source_map.get_packages_from_generic_name(source, hint, subproject)
+                        for pkg in pkgs:
+                            if 'aliases' in source[subproject][pkg]:
+                                external_subprojects.setdefault(subproject, set()).add(','.join(source[subproject][pkg]['aliases']))
+                            else:
+                                external_subprojects.setdefault(subproject, set()).add(pkg)
+
         return external_subprojects
 
     def display_cve(self, cve, file=sys.stdout, line_prefix=None, wrap_desc=False):
diff --git a/scripts/check-syntax b/scripts/check-syntax
index def63e3..4fb0cf8 100755
--- a/scripts/check-syntax
+++ b/scripts/check-syntax
@@ -429,35 +429,35 @@ for cve in args:
 
         # Check aliases
         for rel in all_required_releases:
-            if rel in source and pkg in source[rel] and \
-                'aliases' in source[rel][pkg]:
-                aliases_releases.add(rel)
-
-                # This package-release uses aliases,
-                # it shouldn't be listed
-                if rel in listed_releases and pkg \
-                    not in source[rel][pkg]['aliases']:
-                    filename = srcmap["pkgs"][pkg][nearby_rel][0]
-                    linenum = srcmap["pkgs"][pkg][nearby_rel][1]
-                    print(
-                        "%s: %d: package '%s' not in '%s'"
-                        % (filename, linenum, pkg, rel),
-                        file=sys.stderr,
-                    )
-
-                failed = 0
-                for alias in source[rel][pkg]['aliases']:
-                    if alias not in data["pkgs"].keys():
+            if rel in source:
+                aliases = source_map.get_aliases_of_ubuntu_package(source, pkg, rel)
+                if aliases:
+                    if rel in listed_releases and pkg \
+                        not in aliases:
                         filename = srcmap["pkgs"][pkg][nearby_rel][0]
                         linenum = srcmap["pkgs"][pkg][nearby_rel][1]
                         print(
-                            "%s: %d: %s missing release '%s'"
-                            # put the error on a line near where this entry should go
-                            % (filename, linenum, alias, rel),
+                            "%s: %d: package '%s' not in '%s'"
+                            % (filename, linenum, pkg, rel),
                             file=sys.stderr,
                         )
-                        cve_okay = False
-                        failed += 1
+
+                    aliases_releases.add(rel)
+
+                    # This package-release uses aliases,
+                    # it shouldn't be listed
+
+                    for alias in aliases:
+                        if alias not in data["pkgs"].keys():
+                            filename = srcmap["pkgs"][pkg][nearby_rel][0]
+                            linenum = srcmap["pkgs"][pkg][nearby_rel][1]
+                            print(
+                                "%s: %d: %s missing release '%s'"
+                                # put the error on a line near where this entry should go
+                                % (filename, linenum, alias, rel),
+                                file=sys.stderr,
+                            )
+                            cve_okay = False
 
         missing_releases = all_required_releases - listed_releases - aliases_releases
         nearby_rel = list(listed_releases - missing_releases)[0]
diff --git a/scripts/check-syntax-fixup b/scripts/check-syntax-fixup
index c7c77c2..697368c 100755
--- a/scripts/check-syntax-fixup
+++ b/scripts/check-syntax-fixup
@@ -59,11 +59,31 @@ def delete_from_file(filename: str, linenum: int, dryrun=False, verbose=False):
     else:
         print("%s: %d: would delete this line" % (os.path.relpath(filename), linenum))
 
+
+def identify_subproject_line_number(filename: str, pkg: str, rel: str):
+    # file may not already exist
+    contents = []
+    try:
+        with open(filename, "r") as f:
+            contents = f.readlines()
+    except FileNotFoundError:
+        pass
+
+    for linenum,line in enumerate(contents):
+        line_rel, line_pkg = line.split(":")[0].split('_')
+        if pkg == line_pkg and rel == line_rel:
+            return linenum + 1
+
+    return -1
+
 def get_pkg_rel_from_msg(msg):
     parts = msg.split(" ")
     if 'DOES exist' in msg or 'not in' in msg:
         pkg = parts[1].replace("'", "")
         rel = parts[-1].replace("'", "")
+    elif 'unknown package' in msg:
+        pkg = parts[2].replace("'", "")
+        rel = parts[-1].replace("'", "")
     else:
         pkg = parts[0]
         rel = parts[-1].replace("'", "")
@@ -153,8 +173,11 @@ for line in args.infile:
             cve = os.path.join(
                 cve_lib.get_external_subproject_cve_dir(rel), os.path.basename(cve)
             )
-            # linenum is only relevant to the original cve file
-            linenum = 1
+            linenum = identify_subproject_line_number(cve, pkg, rel)
+            if linenum == -1:
+                if 'DOES exist' in msg:
+                    continue
+                linenum = 1 # We are inserting
 
         # Remove the 'DNE' line before adding the new one
         if 'DOES exist' in msg:
@@ -171,8 +194,10 @@ for line in args.infile:
             cve = os.path.join(
                 cve_lib.get_external_subproject_cve_dir(rel), os.path.basename(cve)
             )
-            # linenum is only relevant to the original cve file
-            linenum = 1
+            linenum = identify_subproject_line_number(cve, pkg, rel)
+            if linenum == -1:
+                print(line, file=sys.stderr)
+                continue
 
         # delete this line since
         delete_from_file(cve, linenum, args.dry_run, args.verbose)
diff --git a/scripts/source_map.py b/scripts/source_map.py
index 311c1f6..616c5ec 100755
--- a/scripts/source_map.py
+++ b/scripts/source_map.py
@@ -423,6 +423,29 @@ def get_built_using_header():
     header += "\n" + "-" * 78
     return header
 
+def get_all_aliases(sources, rel):
+    aliases = []
+    for pkg in sources[rel]:
+        if 'aliases' in sources[rel][pkg]:
+            for alias in sources[rel][pkg]['aliases']:
+                aliases.append(alias)
+    return aliases
+
+def get_aliases_of_ubuntu_package(sources, pkg_name, rel_name):
+    aliases = []
+    for rel in sources:
+        for pkg in sources[rel]:
+            if rel == rel_name and 'aliases' in sources[rel][pkg]:
+                if pkg_name in sources[rel][pkg]['aliases']:
+                    aliases.append(pkg)
+    return aliases
+
+def get_packages_from_generic_name(sources, generic_name, rel):
+    pkgs = []
+    for pkg in sources[rel]:
+        if 'generic_name' in sources[rel][pkg] and sources[rel][pkg]['generic_name'] == generic_name:
+            pkgs.append(pkg)
+    return pkgs
 
 def load_subprojects_lists(releases=None):
     map = dict()
@@ -471,17 +494,21 @@ def load_subprojects_lists(releases=None):
                     map[rel][pkg]['pocket'] = ''
                     map[rel][pkg]['section'] = 'main'
 
+                    if '|' in pkg:
+                        main_package_name = pkg.split('|')[0]
+                        map[rel][pkg]['generic_name'] = main_package_name
+
             if 'aliases' in details:
                 with open(details['aliases'], 'r') as file:
                     aliases = yaml.safe_load(file)
 
                     for pkg in aliases:
-                        if pkg in map[rel]:
-                            map[rel][pkg]['aliases'] = aliases[pkg]
-                        #else:
-                        #    print("WARN: pkg %s found in aliases but not in supported list for %s. Skipping" % (pkg, rel))
-            else:
-                pass
+                        for src_pkg in map[rel]:
+                            if pkg == src_pkg or \
+                                ('generic_name' in map[rel][src_pkg] and pkg == map[rel][src_pkg]['generic_name']):
+                                map[rel][src_pkg]['aliases'] = aliases[pkg]
+                            #else:
+                            #    print("WARN: pkg %s found in aliases but not in supported list for %s. Skipping" % (pkg, rel))
 
     return map