Merge ~litios/ubuntu-cve-tracker:add-static-built-using into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

The tool to report the packages used during the building uses Built-Using to gather them. This is not the only case for other formats like Golang.

This is a proposition to extend this to tag 'Static-Built-Using` for the tool itself and also allow other custom tags in the future like the ones in https://wiki.debian.org/Static-Built-Using for source_map.

As this is not the only tool using the functions from source_map I've tried to minimize changes to prevent other scripts from breaking and make it as generic as possible.

On a side note, the script process-cves, on line 1092, use these functions to tell the user about packages that may need to be rebuilt. I would also like to propose adding the static check there too. The worst-case scenario would be nothing extra to report.

[Eduardo Barretto (ebarretto)]

overall lgtm, I made some small comments and I would wait for Steve's review.

[Shengjing Zhu (zhsj)]

There will be some period that Built-Using and Static-Built-Using are both used on different packages. If we only have one tag (default_built_using_tag), then we only get a partial results. Would it be better to have a results for all tags? And maybe include X-Cargo-Built-Using as well.

[Alex Murray (alexmurray)]

I agree - I think the tool should gather and report the results of all known 'Built-Using' tags rather than relying on the user to invoke it will all the various possible combinations of different tags.

review: Needs Fixing

[David Fernandez Gonzalez (litios)]

I definitely agree with that. I fixed it to include the Cargo tag and also make it gather and print all the tags.

Example with Static-Built-Using: https://pastebin.canonical.com/p/JWrpFc3zPG/
Example with X-Cargo-Built-Using: https://pastebin.canonical.com/p/hFkyhmyW6Z/

[Eduardo Barretto (ebarretto)]

lgtm, thanks!

review: Approve

diff --git a/scripts/source_map.py b/scripts/source_map.py
index e53a71e..ef849d5 100755
--- a/scripts/source_map.py
+++ b/scripts/source_map.py
@@ -19,6 +19,7 @@ import sys
 import cve_lib
 import yaml
 
+built_using_tags = ["Built-Using", "Static-Built-Using", "X-Cargo-Built-Using"]
 apt_pkg.init_system()
 
 
@@ -338,8 +339,9 @@ def load_packages_collection(item, map):
             else:
                 map[release][pkg]['source'] = parser.section['Package']
 
-            if 'Built-Using' in parser.section:
-                map[release][pkg]['built-using'] = parser.section['Built-Using'].split(', ')
+            for tag in built_using_tags:
+                if tag in parser.section:
+                    map[release][pkg][tag] = parser.section[tag].split(', ')
 
             map[release][pkg]['architecture'] = parser.section['Architecture']
 
@@ -354,38 +356,38 @@ def load_built_using_collection(pmap, releases=None, component=None):
             continue
 
         for pkg in pmap[rel]:
-            if 'built-using' in pmap[rel][pkg]:
-                # Built-Using for a binary in the Packages file lists the
-                # originating source package of the embedded binary
-                section = pmap[rel][pkg]['section']
-                if component is not None and section != component:
-                    continue
+            for tag in built_using_tags:
+                if tag in pmap[rel][pkg]:
+                    section = pmap[rel][pkg]['section']
+                    if component is not None and section != component:
+                        continue
 
-                pocket = rel
-                if pmap[rel][pkg]['pocket'] != '':
-                    pocket += "-%s" % pmap[rel][pkg]['pocket']
-
-                for (s, c, v) in map(lambda x: x.split(' ', 3),
-                                     pmap[rel][pkg]['built-using']):
-                    v = v.rstrip(')')
-                    if s not in built_using:
-                        built_using[s] = dict()
-                    if v not in built_using[s]:
-                        built_using[s][v] = dict()
-                    if section not in built_using[s][v]:
-                        built_using[s][v][section] = dict()
-                    if pocket not in built_using[s][v][section]:
-                        built_using[s][v][section][pocket] = []
-                    if pkg not in built_using[s][v][section][pocket]:
-                        built_using[s][v][section][pocket].append(
-                            (pkg, pmap[rel][pkg]['version']))
+                    pocket = rel
+                    if pmap[rel][pkg]['pocket'] != '':
+                        pocket += "-%s" % pmap[rel][pkg]['pocket']
+
+                    for (s, c, v) in map(lambda x: x.split(' ', 3),
+                                        pmap[rel][pkg][tag]):
+                        v = v.rstrip(')')
+                        if s not in built_using:
+                            built_using[s] = dict()
+                        if v not in built_using[s]:
+                            built_using[s][v] = dict()
+                        if section not in built_using[s][v]:
+                            built_using[s][v][section] = dict()
+                        if pocket not in built_using[s][v][section]:
+                            built_using[s][v][section][pocket] = []
+                        if pkg not in built_using[s][v][section][pocket]:
+                            built_using[s][v][section][pocket].append(
+                                (pkg, pmap[rel][pkg]['version'], tag))
 
     return built_using
 
 
-built_using_source_format = '%-35s'
-built_using_pocket_format = '%-15s'
+built_using_source_format = '%-55s'
+built_using_pocket_format = '%-20s'
 built_using_component_format = '%-11s'
+built_using_tag_format = '%-24s'
 built_using_usedby_format = '%-35s'
 
 
@@ -407,13 +409,15 @@ def get_built_using(built_using_map, src):
                         continue
                 elif src_version != version:
                     continue
+
             for section in sorted(built_using_map[src][version]):
                 for pocket in sorted(built_using_map[src][version][section]):
-                    for s, v in sorted(
+                    for s, v, t in sorted(
                             built_using_map[src][version][section][pocket]):
                         out += built_using_source_format % ("%s (%s) " % (src, version))
                         out += built_using_pocket_format % pocket
                         out += built_using_component_format % section
+                        out += built_using_tag_format % t
                         out += built_using_usedby_format % s
                         out += '\n'
 
@@ -424,8 +428,9 @@ def get_built_using_header():
     header = built_using_source_format % "Source (version)"
     header += built_using_pocket_format % "Pocket"
     header += built_using_component_format % "Component"
+    header += built_using_tag_format % 'Tag'
     header += built_using_usedby_format % "Used by"
-    header += "\n" + "-" * 78
+    header += "\n" + "-" * 120
     return header
 
 def get_all_aliases(sources, rel):