Merge ~cache-use-only/ubuntu-cve-tracker:kcve-merge into ubuntu-cve-tracker:master

[Yuxuan Luo]

scripts: add kcve-merge

Adds a Python3 script to facilitate merging designated kernel CVE
changes from autotriage branch to upstream security branch.

Signed-off-by: Yuxuan Luo <email address hidden>

Examples:

```bash
$ ./kcve-merge --cve CVE-2023-4273 CVE-2023-2235 --to 'pending.*'
$ ./kcve-merge --kernel jammy_linux-azure devel_linux-aws --cve CVE-2023-4273 CVE-2023-2235 --to 'pending.*'
$ ./kcve-merge --cve-list ./cve_list.txt
$ ./kcve-merge --cve-list ./cve_list.txt --from 'ignored'
```

[Seth Arnold (seth-arnold)]

Thanks for putting this together, this looks like a nice way to work with smaller pieces of the kernel triage process. I've got a few comments inline; my thoughts aren't necessarily something you need to do, but I do hope it's useful.

[Yuxuan Luo (cache-use-only)]

> Thanks for putting this together, this looks like a nice way to work with
> smaller pieces of the kernel triage process. I've got a few comments inline;
> my thoughts aren't necessarily something you need to do, but I do hope it's
> useful.
Thank you for your intuitive and comprehensive review! I have force pushed a newer version with additional notes in it, hope it helps with future maintenance.

[Seth Arnold (seth-arnold)]

Thanks so much for the updates, I'm finding this newer version easier to read and understand.

Now that I understand it better, I've got some more thoughts :) as before, they're not strictly things you have to do, but I do hope that they lead to a better end result.

Thanks

[Yuxuan Luo (cache-use-only)]

> Thanks so much for the updates, I'm finding this newer version easier to read
> and understand.
>
> Now that I understand it better, I've got some more thoughts :) as before,
> they're not strictly things you have to do, but I do hope that they lead to a
> better end result.
>
> Thanks

I took your advice and re-structured the script so that it should be much easier to read. As always, thanks for sharing your thoughts!

[Seth Arnold (seth-arnold)]

Hi Yuxuan, this looks great! Thanks for iterating on it with me.

review: Approve

[Eduardo Barretto (ebarretto)]

Marked this as merged in b1ce336199e0992a8a4e2488b30e12b4cb8caef4

diff --git a/scripts/kcve-merge b/scripts/kcve-merge
 new file mode 100755
index 0000000..9897b1d
--- /dev/null
+++ b/scripts/kcve-merge
@@ -0,0 +1,307 @@
+#!/usr/bin/env python3
+#
+# Merge autotriage branch to master
+#
+
+import os
+import re
+import sys
+import pathlib
+import argparse
+import fileinput
+import subprocess
+
+AUTOBRANCH = 'autotriage'
+SECBRANCH = 'master'
+
+def warn(prompt: str):
+    print("\033[93m" + prompt + "\033[0m")
+
+
+def error(prompt: str):
+    print("\033[91m" + prompt + "\033[0m")
+
+
+def info(prompt: str):
+    print("\033[94m" + prompt + "\033[0m")
+
+
+# Import cve_lib in UCT
+uct_path = os.environ.get('UCT')
+cwd = os.getcwd()
+try:
+    assert (uct_path is not None)
+    os.chdir(uct_path)
+    sys.path.append(uct_path + '/scripts')
+    import cve_lib
+except AssertionError:
+    error('Please configure $UCT in your shell source file')
+    error('Example: export UCT=/home/ubuntu/Documents/ubuntu-cve-tracker')
+    error('Current $UCT: ' + '{uct_path}')
+    sys.exit(1)
+except FileNotFoundError:
+    error('Could not change directory to: ' + uct_path)
+    sys.exit(1)
+except:
+    error('Error importing cve_lib')
+    sys.exit(1)
+
+
+def sanitize_cves(arg_cve: list[str], arg_cve_list: str) -> list[str]:
+    """
+    Sanitize and build a list of unique CVE IDs based on the given arguments
+
+    Parameters
+    ----------
+    - arg_cve: list[str]
+        - A list of CVEs
+        - Example: ['CVE-2023-1111', 'CVE-2022-2222']
+    - arg_cve_list: str
+        - Path to the file containing a list of CVE, one per line
+        - Example: ../cve_list.txt
+
+    Return
+    ------
+    - cve_path_list: list[str]
+        - A list of unique CVE IDs
+        - Example: ['CVE-2023-1111', 'CVE-2022-1234']
+    """
+
+    cve_set = set()
+
+    # If no --cve nor --cve-list is provided, selecting all kernel CVEs by default
+    if arg_cve is None and arg_cve_list is None:
+        cmd = ["git", "--no-pager", "diff", "--name-only", SECBRANCH+".."+AUTOBRANCH]
+
+        dump = subprocess.run(cmd, capture_output=True).stdout.decode()
+        for line in dump.splitlines():
+            # Skip non-CVE files
+            if "/CVE" not in line:
+                continue
+
+            cve_number = line.split(sep='/')[1]
+            cve_set.add(cve_number)
+        return list(cve_set)
+
+    if arg_cve is not None:
+        for cve_num in arg_cve:
+            cve_set.add(cve_num)
+
+    if arg_cve_list is not None:
+        file_path = arg_cve_list
+
+        # The cwd has been set to UCT for the sake of cve_lib;
+        # therefore, manually concat relative path to the calling
+        # work directory
+        if not os.path.isabs(arg_cve_list):
+            file_path = os.path.join(cwd, pathlib.Path(arg_cve_list))
+
+        try:
+            with open(file_path) as cve_file:
+                for line in cve_file.read().splitlines():
+                    cve_set.add(cve_lib.find_cve(line))
+        except FileNotFoundError as e:
+            warn(e.strerror + ": " + file_path)
+        except Exception as e:
+            error("Unknown error when parsing " + file_path + ": " + e)
+
+    return list(cve_set)
+
+
+def build_path_list(cve_list: list[str]) -> list[str]:
+    """Build a list of absolute path to CVEs with the given CVEs
+
+    Parameters
+    ----------
+    - cve_list: list[str]
+        - A list of CVEs
+        - Example: ['CVE-2023-1111', 'CVE-2022-2222']
+
+    Return
+    ------
+    - cve_path_list: list[str]
+        - A list of absolute path to CVEs
+        - Example: ['/home/ubuntu-cve-tracker/active/CVE-2023-1111']
+    """
+    cve_path_list = []
+
+    for cve_num in cve_list:
+        try:
+            cve_path_list.append(cve_lib.find_cve(cve_num))
+        except ValueError:
+            warn("Cannot find given CVE number: " + cve_num)
+            continue
+
+    return cve_path_list
+
+
+def build_status_dict(cve_list: list[str]) -> {}:
+    """Build a dictionary of diffs
+
+    Parameter
+    ---------
+    - cve_list: list[str]
+        - A list of CVEs
+
+    Returns
+    -------
+    - A Python dictionary of status line changes
+        - {
+            CVE_path:{
+                kernel: status
+            }
+          }
+    - Example:
+    {
+        '/home/ubuntu/ubuntu-cve-tracker/active/CVE-2022-1111': {
+            'devel_linux': 'ignored (ESM criteria, was needed)',
+            'jammy_linux-aws' : 'pending',
+        },
+        '/home/ubuntu/ubuntu-cve-tracker/active/CVE-2022-1112': {
+            'focal_linux': 'needed',
+        }
+    }
+    """
+    status_line_re = re.compile(r"_linux.*: ")
+    to_status = {}
+
+    for cve_path in cve_list:
+        to_status[cve_path] = {}
+        cmd = ["git", "--no-pager", "diff", "--no-color", SECBRANCH+".."+AUTOBRANCH, "--", cve_path]
+
+        try:
+            dump = subprocess.run(cmd, capture_output=True).stdout.decode()
+            for line in dump.splitlines():
+                if status_line_re.search(line):
+                    splitted = line[1::].split(sep=':')
+                    if line[0] == '+' and line[1] != '+':
+                        release = splitted[0]
+                        status = splitted[1][1::]  # get rid of the leading space
+                        to_status[cve_path][release] = status
+        except OSError:
+            error("Error executing: " + ' '.join(cmd))
+        except:
+            error("Unknown error occurs when building status dictionary")
+
+    return to_status
+
+def main():
+    parser = argparse.ArgumentParser(
+        prog='kcve-merge',
+        description='Merge selective status change from autotriage branch to master branch',
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        '--kernel',
+        action='extend',
+        help='Specify the kernel of CVEs to merge: RELEASE_SOURCE',
+        nargs='+',
+    )
+    parser.add_argument(
+        '--cve',
+        action='extend',
+        help='Specify the CVE to merge its status change: CVE-2023-3611',
+        nargs='+',
+    )
+    parser.add_argument(
+        '--from',
+        help='Filter the from status using regex',
+    )
+    parser.add_argument(
+        '--to',
+        help='Filter the to status using regex',
+    )
+    parser.add_argument(
+        '--cve-list',
+        help='Provide a CVE list: ./cve_priority.txt',
+        type=str
+    )
+    args = parser.parse_args()
+    arg_kernel: list[str] = args.kernel
+    arg_cve: list[str] = args.cve
+    arg_from = vars(args).get("from")
+    arg_to = args.to
+    arg_cve_list = args.cve_list
+
+    cve_list = []
+    cve_path_list = []
+    to_status = {}
+    status_line_re = re.compile(r"_linux.*: ")
+    if arg_from is not None:
+        try:
+            from_re = re.compile(arg_from)
+        except:
+            error("Error compiling from-status regex: " + arg_from)
+            exit(1)
+    if arg_to is not None:
+        try:
+            to_re = re.compile(arg_to)
+        except:
+            error("Error compiling to-status regex: " + arg_to)
+            exit(1)
+
+    curr_branch = subprocess.run("git rev-parse --abbrev-ref HEAD", shell=True, capture_output=True).stdout.decode().strip()
+    if curr_branch != SECBRANCH:
+        error("Please checkout UCT to '" + SECBRANCH + "' branch")
+        error("currently it is on: " + curr_branch)
+        exit(-1)
+
+    cve_list = sanitize_cves(arg_cve, arg_cve_list)
+
+    # Print prompt
+    info("Merging:")
+    # print first 5 CVEs
+    for cve_num in cve_list[:5]:
+        print("  " + cve_num)
+    if len(cve_list) - 5 > 0:
+        print("  ...and " + str(len(cve_list) - 5) + " more")
+    if arg_kernel is None:
+        info("With all kernels")
+    else:
+        info("with:")
+        for kernel in arg_kernel:
+            print("  " + kernel)
+    if arg_from is not None:
+        info("which from-status matches: " + arg_from)
+    if arg_to is not None:
+        info("which to-status matches: " + arg_to)
+
+    cve_path_list = build_path_list(cve_list)
+    to_status = build_status_dict(cve_path_list)
+
+    # Apply changes
+    for cve_path in cve_path_list:
+        with fileinput.input(files=cve_path, inplace=True) as f:
+            for line in f:
+                if status_line_re.search(line):
+                    kernel = line.split(sep=':')[0]
+                    status = line.split(sep=' ')[1]
+
+                    if (arg_kernel is None or kernel in arg_kernel) and kernel in to_status[cve_path].keys():
+                        if arg_from is None or from_re.search(status):
+                            if arg_to is None or to_re.search(to_status[cve_path][kernel]):
+                                print(kernel + ": " + to_status[cve_path][kernel], end='\n')
+                                continue
+                print(line, end='')
+
+    # Prepare for git push
+    git_status = subprocess.run(["git", "status", "--short"], capture_output=True).stdout.decode()
+    if len(git_status) == 0:
+        warn('No changes being made')
+    else:
+        for cve_num in cve_path_list:
+            try:
+                cve_lib.git_add(cve_num)
+            except ValueError as e:
+                error(e)
+        try:
+            if len(cve_path_list) == 1:
+                cve_lib.git_commit("kernel/" + cve_path_list[0].split(sep="/")[-1] + ": autotriage")
+            else:
+                cve_lib.git_commit("kernel: autotriage")
+        except ValueError as e:
+            error(e.args[0])
+
+
+if __name__ == '__main__':
+    sys.exit(main())