Merge ~rodrigo-zaiden/ubuntu-cve-tracker:kernel_usn_option into ubuntu-cve-tracker:master

[Rodrigo Figueiredo Zaiden]

prepare-kernel-usn.py and sis-generate-usn: add new argument

 with the optional argument --filter-cves it is possible to filter out
 CVEs that are marked as released but where found in the changelog.
 It is mainly needed in the kernel USNs context as kernel changelogs
 may include previous version changelogs.

When dealing with kernel releases, it is common to have changelogs that will include previous versions changelogs in the current.
When running `prepare-kernel-usn.py` script against a version like this, it will throw an error saying that there are CVEs in the changelog that were not passed in command line, and for this cases, we have to re-run the script ignoring those CVEs.

example:
focal/linux-aws : 5.4.0-1100.108 - https://launchpad.net/ubuntu/+source/linux-aws/5.4.0-1100.108
"linux-aws (5.4.0-1099.107)" and olders can be found there.

 $ cd $UCT
 $ ./scripts/prepare-kernel-usn.py -n -p Proposed focal focal/linux-aws: 5.4.0-1100.108
 ...
 ValueError: CVEs found in changelog but not command line: CVE-2022-3567 CVE-2022-3524 CVE-2022-3643 CVE-2022-2196 CVE-2022-3621 CVE-2022-41218 CVE-2022-3564 CVE-2023-0461 CVE-2022-4382 CVE-2022-3061 CVE-2023-0266 CVE-2022-3594 CVE-2022-42703 CVE-2022-43945 CVE-2022-2663 CVE-2022-4139 CVE-2023-23559 CVE-2021-3669 CVE-2022-3545 CVE-2022-3566 CVE-2022-3565 CVE-2022-47520 CVE-2022-42896 CVE-2022-45934
 ...

Those CVEs are already released in older kernels.
with the change proposed:

 $ cd $UCT
 $ ./scripts/prepare-kernel-usn.py -n -p Proposed focal focal/linux-aws: 5.4.0-1100.108 --filter-cves
 ...
 INFO: CVEs '{'CVE-2022-41218', 'CVE-2022-42703', 'CVE-2022-3524', 'CVE-2022-3621', 'CVE-2022-3567', 'CVE-2022-4139', 'CVE-2022-3565', 'CVE-2022-3564', 'CVE-2022-42896', 'CVE-2022-3566', 'CVE-2022-45934', 'CVE-2023-0266', 'CVE-2022-3061', 'CVE-2022-4382', 'CVE-2022-47520', 'CVE-2022-2663', 'CVE-2022-3643', 'CVE-2023-23559', 'CVE-2023-0461', 'CVE-2022-2196', 'CVE-2022-3545', 'CVE-2022-43945', 'CVE-2022-3594', 'CVE-2021-3669'}' in changelog are already released, ignoring
 ...

[Seth Arnold (seth-arnold)]

The name feels a bit generic for what it does; if there's a good reason to go with this, that's fine, but I'd like to suggest something more like:

--ignore-released-cves
--filter-released-cves
--skip-released-cves
--ignore-released-cves-in-changelog
--filter-released-cves-in-changelog
--skip-released-cves-in-changelog

or something else :) -- something that's more specific about what exact operation is going to be done, and the cves that it is going to be done to.

Thanks

[Rodrigo Figueiredo Zaiden (rodrigo-zaiden)]

Hi Seth,

> The name feels a bit generic for what it does; if there's a good reason to go
> with this, that's fine, but I'd like to suggest something more like:
>
> --ignore-released-cves
> --filter-released-cves
> --skip-released-cves
> --ignore-released-cves-in-changelog
> --filter-released-cves-in-changelog
> --skip-released-cves-in-changelog
>
> or something else :) -- something that's more specific about what exact
> operation is going to be done, and the cves that it is going to be done to.

Thanks for the input, better be clear.
I got one of your ideas (--ignore-released-cves-in-changelog) and added in a new commit alongside with a fix for a condition that I just realized it was missing.

>
> Thanks

Thank you very much

[Rodrigo Figueiredo Zaiden (rodrigo-zaiden)]

Just used this feature in the last cycle's releases: 2023.03.20, and it proved to be useful, for some of the linux-azure and linux-aws kernels, old changelogs were included in the current release changelog and while using the feature I was able to quickly skip the already released CVEs and didn't get issues.

As it is using a new argument, --ignore-released-cves-in-changelog, the impacts for other users are close to nothing as the default is not to use the argument.

Does anyone opposes to merging this to master ?

Thanks!

[Steve Beattie (sbeattie)]

Sorry for the delay, this looks good to me and I had tested the script with the changed argument.

review: Approve

diff --git a/scripts/prepare-kernel-usn.py b/scripts/prepare-kernel-usn.py
index ee26610..ac92da5 100755
--- a/scripts/prepare-kernel-usn.py
+++ b/scripts/prepare-kernel-usn.py
@@ -174,6 +174,7 @@ parser.add_argument('-P', '--ppa', action='store', default='ubuntu', help='Use k
 parser.add_argument('-F', '--force', action='store_true', default=False, help='Override sanity checks and continue anyway')
 parser.add_argument('--use-changes', action='store', default=None, help='Use previously downloaded changes files from prior run (sets --keep-changes as well')
 parser.add_argument('--keep-changes', action='store_true', default=False, help='Keep changes files downloaded by sis-changes')
+parser.add_argument("--ignore-released-cves-in-changelog", action='store_true', help="Filter out CVEs already marked as released")
 parser.add_argument('release', action='store', nargs=1,  help='Primary release name (e.g. xenial)')
 parser.add_argument('kernel', action=KernelVersionAction, nargs='+',  help='Kernel source package name and versions; e.g. "linux 4.4.0-42.62. Source package can be a release/name pair"')
 args = parser.parse_args()
@@ -311,6 +312,9 @@ try:
     else:
         cmd += ['--cves', ','.join(cves)]
 
+    if args.ignore_released_cves_in_changelog:
+        cmd += ['--ignore-released-cves-in-changelog']
+
     if args.embargoed:
         cmd += ['--embargoed']
 
diff --git a/scripts/sis-generate-usn b/scripts/sis-generate-usn
index de70fba..2089776 100755
--- a/scripts/sis-generate-usn
+++ b/scripts/sis-generate-usn
@@ -33,6 +33,7 @@ opter.add_option("--filter-bins", metavar="REGEX", help="Only include binary pac
 opter.add_option("--cves", metavar="CVES", help="Comma separated list of CVEs to use instead of the normal *_source.changes autodetection (must be a superset).", default=None)
 opter.add_option("--add-cves", metavar="CVES", help="Comma separated list of CVEs to use in addition to the normal *_source.changes autodetection.", default=None)
 opter.add_option("--ignore-cves", metavar="CVES", help="Comma separated list of CVEs to ignore when doing CVE autodetection.", default=None)
+opter.add_option("--ignore-released-cves-in-changelog", help="Filter out CVEs already marked as released", action='store_true')
 opter.add_option("--embargoed", help="Include embargoed directory when looking for CVE descriptions", action='store_true')
 opter.add_option("--include-eol", help="Include EoL releases", action='store_true')
 opter.add_option("--binaries-json", help="Path to JSON mapping of binary packages to versions (can repeat, default: binaries.json)", action='append', default=[])
@@ -551,6 +552,21 @@ for json_file in opt.binaries_json:
 if opt.ignore_cves:
     CVEs.difference_update(set(opt.ignore_cves.split(',')))
 
+# filter out CVEs from the changelog that are marked as released
+# usually needed in kernel USNs that may contain previous changelogs
+# appended to the current.
+if opt.ignore_released_cves_in_changelog:
+    released = set()
+    for cve_num in CVEs:
+        cve = cve_lib.load_cve(cve_lib.find_cve(cve_num))
+        for release in info.keys():
+            for pkg in info[release].keys():
+                if pkg in cve["pkgs"].keys():
+                    if cve["pkgs"][pkg][release][0] == "released":
+                        released.add(cve_num)
+    print("INFO: CVEs '%s' in changelog(s) are already released, ignoring" % (released), file=sys.stderr)
+    CVEs.difference_update(released)
+
 if opt.cves:
     superset = set(opt.cves.split(','))
     if not superset.issuperset(CVEs):