Merge ~emitorino/ubuntu-cve-tracker:add_support_for_non_ubuntu_releases into ubuntu-cve-tracker:master

[Emilia Torino]

- scripts/generate-oval: ignore non-Ubuntu projects during OVAL generation

Running: $ mkdir /tmp/oval && ./scripts/generate-oval --output-dir /tmp/oval/ active/CVE-2021-3121
Traceback (most recent call last):
  File "./scripts/generate-oval", line 736, in <module>
    main()
  File "./scripts/generate-oval", line 212, in main
    cve_data = parse_cve_file(filepath, cache)
  File "./scripts/generate-oval", line 446, in parse_cve_file
    parse_package_status(key, package, value, filepath, cache)
  File "./scripts/generate-oval", line 253, in parse_package_status
    status['bin-pkgs'] = cache.get_binarypkgs(package, get_orig_rel_name(release))
  File "./scripts/generate-oval", line 639, in get_binarypkgs
    source = self._lookup_latest_source_package(pname, release)
  File "./scripts/generate-oval", line 595, in _lookup_latest_source_package
    self.releases[release] = self.ubuntu.getSeries(name_or_version=release)
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 605, in __call__
    response, content = self.root._browser._request(
  File "/usr/lib/python3/dist-packages/lazr/restfulclient/_browser.py", line 438, in _request
    raise error
lazr.restfulclient.errors.NotFound: HTTP Error 404: Not Found
Response headers:
---
content-length: 42
content-security-policy: frame-ancestors 'self';
content-type: text/plain
date: Mon, 03 May 2021 17:43:01 GMT
server: zope.server.http (HTTP)
status: 404
strict-transport-security: max-age=15552000
vary: Accept
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-launchpad-revision: 02afa4875ac52c169f5cddf0d1bcdd6e149a3754
x-lazr-notifications: []
x-powered-by: Zope (www.zope.org), Python (www.python.org)
x-xss-protection: 1; mode=block
---
Response body:
---
b"No such distribution series: 'git/github'."
---

After this change:
$ mkdir /tmp/oval && ./scripts/generate-oval --output-dir /tmp/oval/ active/CVE-2021-3121
Progress: [####################] 100% (1 of 1 CVEs processed)
$ ls -la /tmp/oval/
total 108
drwxrwxr-x 2 emitorino emitorino 4096 may 3 14:41 .
drwxrwxrwt 36 root root 57344 may 3 14:41 ..
-rw-rw-r-- 1 emitorino emitorino 6165 may 3 14:41 com.ubuntu.bionic.cve.oval.xml
-rw-rw-r-- 1 emitorino emitorino 6117 may 3 14:41 com.ubuntu.focal.cve.oval.xml
-rw-rw-r-- 1 emitorino emitorino 6145 may 3 14:41 com.ubuntu.groovy.cve.oval.xml
-rw-rw-r-- 1 emitorino emitorino 6173 may 3 14:41 com.ubuntu.hirsute.cve.oval.xml
-rw-rw-r-- 1 emitorino emitorino 3591 may 3 14:41 com.ubuntu.trusty.cve.oval.xml
-rw-rw-r-- 1 emitorino emitorino 6270 may 3 14:41 com.ubuntu.xenial.cve.oval.xml

[Eduardo Barretto (ebarretto)]

LGTM!

review: Approve

[Steve Beattie (sbeattie)]

So even with this fix, the oval generator is still failing with the following error from launchpad:

Response body:
---
b"No such distribution series: 'esm-infra/xenial'."
---

I also think that it's probably better to be explicit what we're trying to generate for, rather than enumerating all the things we shouldn't be generating.

a996724... by Emilia Torino on 2021-05-04

Fixing boolean condition while checking for esm string patterns

Signed-off-by: Maria Emilia Torino <email address hidden>

[Emilia Torino (emitorino)]

> So even with this fix, the oval generator is still failing with the following
> error from launchpad:
>
> Response body:
> ---
> b"No such distribution series: 'esm-infra/xenial'."
Eduardo's commit https://git.launchpad.net/ubuntu-cve-tracker/commit/scripts/cve_lib.py?id=96875b1ef8673c6f441852948c4067b7119b1413 fixed this issue, but iiuc introduced another one since this same script was generating outputs like:

WARNING: Unknown package field "trusty/esm" in trusty/esm_shadow in "./active/CVE-2005-4890"
WARNING: Unknown package field "esm-infra/xenial" in esm-infra/xenial_shadow in "./active/CVE-2005-4890"

Its my understanding that the issue is present in the boolean condition: https://git.launchpad.net/ubuntu-cve-tracker/tree/scripts/cve_lib.py#n1525 should be 'and' instead of 'or' as this last commit proposes: https://git.launchpad.net/~emitorino/ubuntu-cve-tracker/commit/?id=a996724046c295a0dd60fb6efd0a06418c1ea373

> ---
>
> I also think that it's probably better to be explicit what we're trying to
> generate for, rather than enumerating all the things we shouldn't be
> generating.

I agree, but on the other side we are adding several changes to $UCT and I am not sure about the effects a refactoring in this lib could generate. Maybe we can include it in a follow up MP?

[Steve Beattie (sbeattie)]

> > I also think that it's probably better to be explicit what we're trying to
> > generate for, rather than enumerating all the things we shouldn't be
> > generating.
>
> I agree, but on the other side we are adding several changes to $UCT and I am
> not sure about the effects a refactoring in this lib could generate. Maybe we
> can include it in a follow up MP?

Yes, that's fine.

Thanks for digging in to this and coming up with fixes!

Merged.

review: Approve

diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py
index 04ff225..0292ef8 100755
--- a/scripts/cve_lib.py
+++ b/scripts/cve_lib.py
@@ -1522,7 +1522,7 @@ def get_esm_name(rel, component=None):
 
 # get the original name of an esm release
 def get_orig_rel_name(rel):
-    if not rel.endswith('/esm') or not rel.startswith('esm-'):
+    if not rel.endswith('/esm') and not rel.startswith('esm-'):
         return rel
     if rel.startswith('esm-'):
         return rel.split('/')[1]
diff --git a/scripts/generate-oval b/scripts/generate-oval
index 0b055e3..e03dec5 100755
--- a/scripts/generate-oval
+++ b/scripts/generate-oval
@@ -89,6 +89,11 @@ ignored_releases = [
     'yakkety', 'zesty', 'artful', 'cosmic', 'disco', 'eoan'
 ]
 
+# For now, ignore non-Ubuntu projects
+ignored_projects = [
+    'git/github'
+]
+
 all_releases = list(supported_releases.keys()) + ignored_releases
 
 ignored_package_fields = [
@@ -425,7 +430,7 @@ def parse_cve_file(filepath, cache):
             if ignore_source_package(package):
                 continue
 
-            if key in ignored_package_fields or key in ignored_releases:
+            if key in ignored_package_fields or key in ignored_releases or key in ignored_projects:
                 continue
 
             if (package not in packages):