Merge ~rodrigo-zaiden/ubuntu-cve-tracker:sis_generate_usn_ignore_cves_changelog into ubuntu-cve-tracker:master

[Rodrigo Figueiredo Zaiden]

scripts/sis-generate-usn: check ignored cves from argument when parsing changelog

 when generating USN, we first parse CVEs from changelog before updating
 the CVE set (removing the CVE(s)) with the ignored CVEs from the argument
 '--ignore-cves'. So, if there is a CVE that will fail when parsing the
 changelog, we cannot use the '--ignore-cves' argument because it fails
 before reaching the CVE set update for ignored cves from command.

 verifying if the parsed CVE is listed to be ignored gives us the chance
 to skip that check and won't add the CVE to the USN.

Scenario:
CVE-2023-4563 is a duplicated CVE, it was identified as duplicate after
the kernel team prepared the kernels and listed it as fixed in the changelog.
When identified, it was moved to the ignored/ folder.
The CVE that it was duplicate of (CVE-2023-4244) is also listed in the
changelog.

It can also happen if a CVE has a typo when being mentioned in the changelog.

How to reproduce it:
$ cd $UCT
$./scripts/prepare-kernel-usn.py -d -n -p Proposed jammy jammy/linux-nvidia-6.2: 6.2.0-1011.11 --ignore CVE-2023-4563
...
ERROR: CVE-2023-4563 does not exist in UCT in either retired
Traceback (most recent call last):
  File "/home/rodrigo/git-pulls/ubuntu-cve-tracker/./scripts/prepare-kernel-usn.py", line 334, in <module>
    subprocess.check_call(cmd, stdout=f, universal_newlines=True)
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/rodrigo/git-pulls/ubuntu-cve-tracker/scripts/sis-generate-usn', '--kernel-mode', '--no-new-warn', '--ignore-cves', 'CVE-2023-4563', '--cves', 'CVE-2022-45886,CVE-2023-4273,CVE-2023-21264,CVE-2023-4155,CVE-2023-38427,CVE-2023-42752,CVE-2023-42755,CVE-2022-48425,CVE-2023-4244,CVE-2023-4569,CVE-2023-4128,CVE-2023-38431,CVE-2022-45919,CVE-2023-34319,CVE-2023-5197,CVE-2023-4622,CVE-2022-45887,CVE-2023-4132,CVE-2023-3772,CVE-2023-20588,CVE-2023-3212,CVE-2023-4921,CVE-2023-40283,CVE-2023-4881,CVE-2023-1206,CVE-2023-31083,CVE-2023-42753,CVE-2023-4623,CVE-2023-2156,CVE-2023-4194,CVE-2023-42756,CVE-2023-20569', '--filter-bins', '^linux-image-(\\d|generic|virtual|lowlatency|power|server|omap|raspi|riscv|snapdragon|highbank|allwinner|aws|bluefield|gcp|gke|ibm|intel|nvidia|oem|oracle|azure|joule|kvm|euclid|dell|starfive|xilinx)', 'N-1', '--binaries-json', 'jammy-binaries.json', 'linux-meta-nvidia-6.2_6.2.0.1011.13_source.changes', 'linux-signed-nvidia-6.2_6.2.0-1011.11_source.changes', 'linux-signed-nvidia-6.2_6.2.0-1011.11_arm64.changes', 'linux-nvidia-6.2_6.2.0-1011.11_amd64.changes', 'linux-nvidia-6.2_6.2.0-1011.11_arm64.changes', 'linux-nvidia-6.2_6.2.0-1011.11_source.changes', 'linux-meta-nvidia-6.2_6.2.0.1011.13_arm64.changes', 'linux-meta-nvidia-6.2_6.2.0.1011.13_amd64.changes', 'linux-signed-nvidia-6.2_6.2.0-1011.11_amd64.changes']' returned non-zero exit status 1.

Notes:
When we first executed prepare-kernel-usn script, we identified that it was ok
to ignore that CVE. But even when we want to ignore it, it keeps failing
because the script first check if there is an error with the CVE and stop.
just later it would update the CVE set to remove the ignored CVEs.

It can also happen when there is a typo when listing CVEs in the changelog
(It happens with the above execution with CVE-2023-42572, that is a typo
for CVE-2023-42752)

currently, the trace is:
    sis-generate-usn
    |_ parse_changes()
    | |_ parse_CVEs()
    | |_ check_cve_priority() //fails if there is an error
    X
    |_ if opt.ignore_cves:
        CVEs.difference_update(set(opt.ignore_cves.split(',')))
        //won't have the chance to use the ignored CVEs

this commit will:
    sis-generate-usn
    |_ parse_changes()
    | |_ parse_CVEs() //skip the following check if CVE is in ignored list
    | |_ check_cve_priority() //fails if there is an error
    |
    |_ if opt.ignore_cves:
        CVEs.difference_update(set(opt.ignore_cves.split(',')))
        //will update the CVE set with ignored CVEs with success

[Alex Murray (alexmurray)]

LGTM!

review: Approve

[Steve Beattie (sbeattie)]

Thank you for the excellent detailed explanation of what's going wrong, it makes reviewing the change so much easier. Merged, thanks!

review: Approve

diff --git a/scripts/sis-generate-usn b/scripts/sis-generate-usn
index 10a57d2..45f9180 100755
--- a/scripts/sis-generate-usn
+++ b/scripts/sis-generate-usn
@@ -214,6 +214,9 @@ def parse_CVEs(text, cvelines):
     cvere = re.compile("((?:CVE|cve)-\d\d\d\d-\d{4,7})")
     for cve in cvere.finditer(text):
         cve_number = cve.group().upper()
+        if opt.ignore_cves and cve_number in opt.ignore_cves:
+            print("WARN: found '%s' in changelog but ignored in command line" % cve_number, file=sys.stderr)
+            continue
         check_cve_priority(cve_number)
         result.add(cve_number)