Merge ~ebarretto/ubuntu-cve-tracker:report-released-packages into ubuntu-cve-tracker:master

[Eduardo Barretto]

NOTE: This PR SHOULD NOT be merged until we get to work on the metric scripts that use this report.

This PR is intended to prepare for upcoming releases and enhance the report-released-packages script. The PR contain 3 commits:

1. Add in_universe function to cve_lib:
   Even though we have is_universe, it doesn't actually check if srcpkg or binary in universe. So, in order to not break other scripts, adding a new function to check if a srcpkg or binary is in universe was needed.

2. Adapt report-released-packages to better track archive updates:
   With 16.04 esm near launch, released-esm and experimental directory will go away, so we needed a better way to track archive updates.

3. Alter report-released-packages to show number of updates per package and number of cves:
   The current report-released-packages show the number of updates, but that is actually the number of CVEs fixed. So I've extended to also get the number of updates per package.

[Emilia Torino (emitorino)]

Changes look good but is difficult to review the code without understanding how is exercised (e.g. I am not familiar with the content of map[rel][pkg]['section'] to understand if that's correct) or the impact it could have on whoever is consuming them (e.g. changing packages[pkg] = 1 to packages[pkg] = {}).

review: Approve

[Steve Beattie (sbeattie)]

Hi Eduardo,

I don't know much about this script, does it need to run on people.c.c or elsewhere?

[Eduardo Barretto (ebarretto)]

> Hi Eduardo,
>
> I don't know much about this script, does it need to run on people.c.c or
> elsewhere?

Yes, I believe this runs in keule in a cron job and is what the kpi scripts uses to get some of the metrics on esm.

[Steve Beattie (sbeattie)]

On Wed, Apr 21, 2021 at 02:46:21PM -0000, Eduardo Barretto wrote:
> 1. Add in_universe function to cve_lib:
> Even though we have is_universe, it doesn't actually check if srcpkg or binary in universe. So, in order to not break other scripts, adding a new function to check if a srcpkg or binary is in universe was needed.

I'm not sure what you mean here. is_universe() ends up calling
is_supported(), which at the end (after doing a bunch of work around
overlays) does:

    # Allow for a tagged override to declare a pkg (from the perspective of
    # a given CVE item) to be unsupported.
    if cvedata and pkg in cvedata['tags'] and \
       ('universe-binary' in cvedata['tags'][pkg] or
        'not-ue' in cvedata['tags'][pkg]):
        return False
    # Look for component or if we have a partially supported release, and if
    # partially supported only support those packages that are supported
    if pkg in map[rel] and \
       (map[rel][pkg]['section'] == 'main' or
        map[rel][pkg]['section'] == 'restricted') and \
       (len(supported_pkgs[rel]) == 0 or pkg in supported_pkgs[rel]):
        return True
    return False

which looks for the 'universe-binary' tag[0] in any of the attached cves.
Else, it looks to see whether the source packages is in 'main'
or 'restricted' (the two supported components) for that release.
Alternatively, it looks in the esm-infra lists for that package (I
believe) to see if it's supported there. For non-ESM-infra releases,
things that aren't in 'main' or 'restricted' will be in either
'universe' or 'multiverse' which is really what our colloquial usage
of the term 'universe' covers.

[0] or 'not-ue', which looks to have only affected 2017 kernels, 'ue' here
    I believe means 'ubuntu-engineering' and that those kernels were
    supported by a team outside of ubuntu engineering.

--
Steve Beattie
<email address hidden>

[Eduardo Barretto (ebarretto)]

> On Wed, Apr 21, 2021 at 02:46:21PM -0000, Eduardo Barretto wrote:
> > 1. Add in_universe function to cve_lib:
> > Even though we have is_universe, it doesn't actually check if srcpkg or
> binary in universe. So, in order to not break other scripts, adding a new
> function to check if a srcpkg or binary is in universe was needed.
>
> I'm not sure what you mean here. is_universe() ends up calling
> is_supported(), which at the end (after doing a bunch of work around
> overlays) does:
>
> # Allow for a tagged override to declare a pkg (from the perspective of
> # a given CVE item) to be unsupported.
> if cvedata and pkg in cvedata['tags'] and \
> ('universe-binary' in cvedata['tags'][pkg] or
> 'not-ue' in cvedata['tags'][pkg]):
> return False
> # Look for component or if we have a partially supported release, and if
> # partially supported only support those packages that are supported
> if pkg in map[rel] and \
> (map[rel][pkg]['section'] == 'main' or
> map[rel][pkg]['section'] == 'restricted') and \
> (len(supported_pkgs[rel]) == 0 or pkg in supported_pkgs[rel]):
> return True
> return False
>
> which looks for the 'universe-binary' tag[0] in any of the attached cves.
> Else, it looks to see whether the source packages is in 'main'
> or 'restricted' (the two supported components) for that release.
> Alternatively, it looks in the esm-infra lists for that package (I
> believe) to see if it's supported there. For non-ESM-infra releases,

The current problem is exactly on the esm-infra lists of packages, with 16.04 ESM
coming up, we don't have a list of packages anymore as the scope changed:
https://wiki.ubuntu.com/SecurityTeam/ESM/16.04#Maintained_Packages

That also means that source_map right now won't load esm-infra/xenial (e.g.
check source_map.load() and source_map.load_ppa(), it is based in the esm-infra
list of packages). Also, source_map won't load xenial, as it will be listed in
cve_lib.eol_releases.

In order to make is_supported work with upcoming esm, we would need to address
those changes.

Also, for 14.04 ESM we do have a list of universe packages, pretty much like the
esm-infra list, so is_supported is returning True for those universe packages
which would then make is_universe return False. This needs to be investigated
further, as I believe some tools expect to see those 14.04 universe packages as
"supported".

That's why I created this new in_universe function.

We can certainly have a meeting and try to come up with a better solution.

diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py
index 5a2b740..ace68c1 100755
--- a/scripts/cve_lib.py
+++ b/scripts/cve_lib.py
@@ -1518,6 +1518,15 @@ def any_universe(map, pkg, releases, cvedata):
             return True
     return False
 
+def in_universe(map, pkg, rel, cve, cvedata):
+    if pkg in map[rel] and map[rel][pkg]['section'] == 'universe':
+        return True
+    else:
+        if not cvedata:
+            cvedata = load_cve(find_cve(cve))
+        if pkg in cvedata['tags'] and 'universe-binary' in cvedata['tags'][pkg]:
+            return True
+    return False
 
 def load_debian_dsas(filename, verbose=True):
     dsa = None
diff --git a/scripts/report-released-packages b/scripts/report-released-packages
index db0f802..82d0dc0 100755
--- a/scripts/report-released-packages
+++ b/scripts/report-released-packages
@@ -7,14 +7,9 @@
 # Public License, Version 2 or later. See http://www.gnu.org/copyleft/gpl.html
 # for details.
 
-import datetime
-import os
-import re
 import sys
 import optparse
 import cve_lib
-import time
-import json
 
 import source_map
 
@@ -22,15 +17,15 @@ packages = dict()
 assignees = dict()
 pkg_assignees = dict()
 popularity = dict()
-pockets = {'extended': dict(),
+pockets = {'esm': dict(),
            'ppa': dict(),
            'archive': dict()}
 
 parser = optparse.OptionParser()
 parser.add_option("-S", "--skip-devel", help="Show only those CVEs *not* in the current devel release", action="store_true")
 parser.add_option("-D", "--only-devel", help="Show only those CVEs in the current devel release", action="store_true")
-parser.add_option("-e", "--only-extended", help="Show only packages with extended release status", action="store_true")
-parser.add_option("-x", "--not-extended", help="Show only those packages without extended release status", action="store_true")
+parser.add_option("-e", "--only-esm", help="Show only packages updated by the esm team", action="store_true")
+parser.add_option("-x", "--not-esm", help="Show only those packages not updated by the esm team", action="store_true")
 parser.add_option("-p", "--packages", help="Report only on the given packages", action="append", type="string")
 parser.add_option("-P", "--pkgfamily", help="Perform pkg family renamings (use argument multiple times for 'linux' and/or 'xen')", action="append", default=[])
 parser.add_option("-X", "--exclude", help="Ignore specified packages", action="append", type="string")
@@ -38,7 +33,7 @@ parser.add_option("-r", "--release", help="Report only for the given releases", 
 parser.add_option("-d", "--debug", help="Report debug information while loading", action="store_true")
 (opt, args) = parser.parse_args()
 
-map = source_map.load()
+srcmap = source_map.load(subprojects=cve_lib.find_subprojects())
 releases = cve_lib.all_releases
 for eol in cve_lib.eol_releases:
     if eol in releases:
@@ -65,40 +60,42 @@ for cve in sorted(cves):
             continue
 
         found = False
-        extended = False
+        esm = False
         ppa = False
         archive = False
+        version = 0
         for r in releases:
             status = table[cve]['pkgs'][pkg]
             if status.has_key(r):
                 if status[r][0].startswith('released'):
                     #print >>sys.stderr, "%s for %s in %s (%s)" % (cve, pkg, r, table[cve][pkg][r])
                     found = True
+                    version = status[r][1]
+                    rel = r
 
-                    # if has some extended release status or has esm in version
-                    # then is considered extended
-                    if status[r][0].startswith('released-'):
-                        extended = True
+                    if '/' in r and 'trusty' not in r:
+                        rel = r.split('/')[1]
                     if 'esm' in status[r][1]:
-                        extended = True
+                        esm = True
                         ppa = True
-                    else:
+                    elif cve_lib.in_universe(srcmap, pkg, rel, cve, None):
+                        esm = True
                         archive = True
 
         if not found:
             continue
 
-        if opt.only_extended and not extended:
+        if opt.only_esm and not esm:
             continue
 
-        if opt.not_extended and extended:
+        if opt.not_esm and esm:
             continue
 
-        if extended:
-            if pockets['extended'].has_key(pkg):
-                pockets['extended'][pkg] += 1
+        if esm:
+            if pockets['esm'].has_key(pkg):
+                pockets['esm'][pkg] += 1
             else:
-                pockets['extended'][pkg] = 1
+                pockets['esm'][pkg] = 1
         if ppa:
             if pockets['ppa'].has_key(pkg):
                 pockets['ppa'][pkg] += 1
@@ -112,16 +109,20 @@ for cve in sorted(cves):
 
 
         if packages.has_key(pkg):
-            packages[pkg] += 1
+            packages[pkg]['cves'] += 1
+            if version not in packages[pkg]['version']:
+                packages[pkg]['version'].append(version)
         else:
-            packages[pkg] = 1
+            packages[pkg] = {}
+            packages[pkg]['cves'] = 1
+            packages[pkg]['version'] = [version]
 
 
-sys.stdout.write("Updates\tPackage\n")
+sys.stdout.write("Updates\tCVEs Fixed\tPackage\n")
 sys.stdout.write("---------------------------------------------------------------------\n")
 
 for pkg in sorted(packages.keys()):
-    sys.stdout.write("%s\t%s" % (str(packages[pkg]), pkg))
+    sys.stdout.write("%s\t%s\t\t%s" % (str(len(packages[pkg]['version'])), str(packages[pkg]['cves']), pkg))
 
     extra_info = []
     for p in pockets.keys():