Merge ~alexmurray/ubuntu-cve-tracker:check-cves-ignore-based-on-urlref-regex into ubuntu-cve-tracker:master

[Alex Murray]

scripts/check-cves: add support for ignoring CVEs based on ref URLs

Wordpress Plugin CVEs seem to be reported by the same CNA so support matching
their URL prefix to ignore these common CVEs.

I have tested this during CVE triage today and it seems to work well and is a
great time saver.

Signed-off-by: Alex Murray <email address hidden>

[Alex Murray (alexmurray)]

Anyone want to take a look at this? The change is small so should be easy to review - I've been using it during CVE triage this week and it has been really useful FWIW.

[Marc Deslauriers (mdeslaur)]

LGTM, ack, thanks!

review: Approve

[Seth Arnold (seth-arnold)]

Does this 'feel' slow while using it? it looks a bit like it recompiles every regex on every cve, and I'm wondering if it would make sense to explicitly compile these first, or write one complex regex that matches all of these and compile that one, to skip repeated executions. (At least, I didn't quickly spot a python version of https://docs.rs/regex/latest/regex/struct.RegexSet.html that could do this easily for you.)

[Alex Murray (alexmurray)]

@seth-arnold - I didn't notice any performance impact but I agree that we should ideally be pre-compiling these. Also they should probably be stored in a separate "configuration" file to allow easily adding new ones during the CVE triage process.

diff --git a/scripts/check-cves b/scripts/check-cves
index c49011c..2cc026d 100755
--- a/scripts/check-cves
+++ b/scripts/check-cves
@@ -118,6 +118,14 @@ IGNORE_STRINGS = [
     "Oracle Enterprise Manager", "F5 BIG-IP", "Adobe Acrobat and Reader"
 ]
 
+IGNORE_URLS = {
+    "https://patchstack.com/database/vulnerability/.*/wordpress.*": "WordPress Plugin",
+    "https://www.wordfence.com/threat-intel/vulnerabilities/id/.*?source=cve": "WordPress Plugin",
+    "https://wpscan.com/vulnerability/.*": "WordPress Plugin",
+    "https://www.zyxel.com/global/en/support/security-advisories/.*": "Zyxel",
+    "https://www.manageengine.com/.*": "Zoho ManageEngine",
+    "https://www.qnap.com/en/security-advisory/.*": "QNAP",
+}
 
 def merge_list(list1, list2, intersection=None):
     """Write the union of list and list2 into list. If intersection is not
@@ -980,10 +988,19 @@ class CVEHandler(xml.sax.handler.ContentHandler):
             words = self.get_software_hints_from_cve_description(self.cve_data[cve]['desc'])
             if action == 'skip':
                 # try and hint if any of the references use a URL that is within
-                # a known packages Homepage from HOMEPAGES_MAP
+                # a known packages Homepage from HOMEPAGES_MAP or within
+                # IGNORE_URLS
                 if 'refs' in self.cve_data[cve]:
                     for ref in self.cve_data[cve]['refs']:
                         url = ref[2]
+                        for url_re in IGNORE_URLS:
+                            if re.search(url_re, url):
+                                print(f"Detected URL: {url} matches {url_re}")
+                                action = 'ignore'
+                                reason = IGNORE_URLS[url_re]
+                                break
+                        if action == 'ignore':
+                            break
                         for srcpkg in HOMEPAGES_MAP:
                             for homepage in HOMEPAGES_MAP[srcpkg]:
                                 # TODO: do smarter matching than this
@@ -997,7 +1014,7 @@ class CVEHandler(xml.sax.handler.ContentHandler):
                 # otherwise it will likely confuse the human doing CVE triage to
                 # suggest a package name that doesn't actually exist
                 hints = words & allsrcs
-                if len(hints) > 0:
+                if len(hints) > 0 and action != "ignore":
                     packages = []
                     for hint in hints:
                         # use preferred name of package instead of the one that matched