Merge ~cache-use-only/ubuntu-cve-tracker:kcve-merge into ubuntu-cve-tracker:master

Proposed by Yuxuan Luo
Status: Needs review
Proposed branch: ~cache-use-only/ubuntu-cve-tracker:kcve-merge
Merge into: ubuntu-cve-tracker:master
Diff against target: 313 lines (+307/-0)
1 file modified
scripts/kcve-merge (+307/-0)
Reviewer Review Type Date Requested Status
Seth Arnold Approve
Review via email: mp+450275@code.launchpad.net

Commit message

scripts: add kcve-merge

Adds a Python3 script to facilitate merging designated kernel CVE
changes from autotriage branch to upstream security branch.

Signed-off-by: Yuxuan Luo <email address hidden>

Description of the change

Examples:

```bash
$ ./kcve-merge --cve CVE-2023-4273 CVE-2023-2235 --to 'pending.*'
$ ./kcve-merge --kernel jammy_linux-azure devel_linux-aws --cve CVE-2023-4273 CVE-2023-2235 --to 'pending.*'
$ ./kcve-merge --cve-list ./cve_list.txt
$ ./kcve-merge --cve-list ./cve_list.txt --from 'ignored'
```

To post a comment you must log in.
Revision history for this message
Seth Arnold (seth-arnold) wrote :

Thanks for putting this together, this looks like a nice way to work with smaller pieces of the kernel triage process. I've got a few comments inline; my thoughts aren't necessarily something you need to do, but I do hope it's useful.

Revision history for this message
Yuxuan Luo (cache-use-only) wrote :

> Thanks for putting this together, this looks like a nice way to work with
> smaller pieces of the kernel triage process. I've got a few comments inline;
> my thoughts aren't necessarily something you need to do, but I do hope it's
> useful.
Thank you for your intuitive and comprehensive review! I have force pushed a newer version with additional notes in it, hope it helps with future maintenance.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Thanks so much for the updates, I'm finding this newer version easier to read and understand.

Now that I understand it better, I've got some more thoughts :) as before, they're not strictly things you have to do, but I do hope that they lead to a better end result.

Thanks

Revision history for this message
Yuxuan Luo (cache-use-only) wrote :

> Thanks so much for the updates, I'm finding this newer version easier to read
> and understand.
>
> Now that I understand it better, I've got some more thoughts :) as before,
> they're not strictly things you have to do, but I do hope that they lead to a
> better end result.
>
> Thanks

I took your advice and re-structured the script so that it should be much easier to read. As always, thanks for sharing your thoughts!

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Hi Yuxuan, this looks great! Thanks for iterating on it with me.

review: Approve

Unmerged commits

e9b1692... by Yuxuan Luo

scripts: add kcve-merge

Adds a Python3 script to facilitate merging designated kernel CVE
changes from autotriage branch to upstream security branch.

Signed-off-by: Yuxuan Luo <email address hidden>

Failed
[SUCCEEDED] unit-tests:0 (build)
[FAILED] check-cves:0 (build)
12 of 2 results

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/scripts/kcve-merge b/scripts/kcve-merge
2new file mode 100755
3index 0000000..9897b1d
4--- /dev/null
5+++ b/scripts/kcve-merge
6@@ -0,0 +1,307 @@
7+#!/usr/bin/env python3
8+#
9+# Merge autotriage branch to master
10+#
11+
12+import os
13+import re
14+import sys
15+import pathlib
16+import argparse
17+import fileinput
18+import subprocess
19+
20+AUTOBRANCH = 'autotriage'
21+SECBRANCH = 'master'
22+
23+def warn(prompt: str):
24+ print("\033[93m" + prompt + "\033[0m")
25+
26+
27+def error(prompt: str):
28+ print("\033[91m" + prompt + "\033[0m")
29+
30+
31+def info(prompt: str):
32+ print("\033[94m" + prompt + "\033[0m")
33+
34+
35+# Import cve_lib in UCT
36+uct_path = os.environ.get('UCT')
37+cwd = os.getcwd()
38+try:
39+ assert (uct_path is not None)
40+ os.chdir(uct_path)
41+ sys.path.append(uct_path + '/scripts')
42+ import cve_lib
43+except AssertionError:
44+ error('Please configure $UCT in your shell source file')
45+ error('Example: export UCT=/home/ubuntu/Documents/ubuntu-cve-tracker')
46+ error('Current $UCT: ' + '{uct_path}')
47+ sys.exit(1)
48+except FileNotFoundError:
49+ error('Could not change directory to: ' + uct_path)
50+ sys.exit(1)
51+except:
52+ error('Error importing cve_lib')
53+ sys.exit(1)
54+
55+
56+def sanitize_cves(arg_cve: list[str], arg_cve_list: str) -> list[str]:
57+ """
58+ Sanitize and build a list of unique CVE IDs based on the given arguments
59+
60+ Parameters
61+ ----------
62+ - arg_cve: list[str]
63+ - A list of CVEs
64+ - Example: ['CVE-2023-1111', 'CVE-2022-2222']
65+ - arg_cve_list: str
66+ - Path to the file containing a list of CVE, one per line
67+ - Example: ../cve_list.txt
68+
69+ Return
70+ ------
71+ - cve_path_list: list[str]
72+ - A list of unique CVE IDs
73+ - Example: ['CVE-2023-1111', 'CVE-2022-1234']
74+ """
75+
76+ cve_set = set()
77+
78+ # If no --cve nor --cve-list is provided, selecting all kernel CVEs by default
79+ if arg_cve is None and arg_cve_list is None:
80+ cmd = ["git", "--no-pager", "diff", "--name-only", SECBRANCH+".."+AUTOBRANCH]
81+
82+ dump = subprocess.run(cmd, capture_output=True).stdout.decode()
83+ for line in dump.splitlines():
84+ # Skip non-CVE files
85+ if "/CVE" not in line:
86+ continue
87+
88+ cve_number = line.split(sep='/')[1]
89+ cve_set.add(cve_number)
90+ return list(cve_set)
91+
92+ if arg_cve is not None:
93+ for cve_num in arg_cve:
94+ cve_set.add(cve_num)
95+
96+ if arg_cve_list is not None:
97+ file_path = arg_cve_list
98+
99+ # The cwd has been set to UCT for the sake of cve_lib;
100+ # therefore, manually concat relative path to the calling
101+ # work directory
102+ if not os.path.isabs(arg_cve_list):
103+ file_path = os.path.join(cwd, pathlib.Path(arg_cve_list))
104+
105+ try:
106+ with open(file_path) as cve_file:
107+ for line in cve_file.read().splitlines():
108+ cve_set.add(cve_lib.find_cve(line))
109+ except FileNotFoundError as e:
110+ warn(e.strerror + ": " + file_path)
111+ except Exception as e:
112+ error("Unknown error when parsing " + file_path + ": " + e)
113+
114+ return list(cve_set)
115+
116+
117+def build_path_list(cve_list: list[str]) -> list[str]:
118+ """Build a list of absolute path to CVEs with the given CVEs
119+
120+ Parameters
121+ ----------
122+ - cve_list: list[str]
123+ - A list of CVEs
124+ - Example: ['CVE-2023-1111', 'CVE-2022-2222']
125+
126+ Return
127+ ------
128+ - cve_path_list: list[str]
129+ - A list of absolute path to CVEs
130+ - Example: ['/home/ubuntu-cve-tracker/active/CVE-2023-1111']
131+ """
132+ cve_path_list = []
133+
134+ for cve_num in cve_list:
135+ try:
136+ cve_path_list.append(cve_lib.find_cve(cve_num))
137+ except ValueError:
138+ warn("Cannot find given CVE number: " + cve_num)
139+ continue
140+
141+ return cve_path_list
142+
143+
144+def build_status_dict(cve_list: list[str]) -> {}:
145+ """Build a dictionary of diffs
146+
147+ Parameter
148+ ---------
149+ - cve_list: list[str]
150+ - A list of CVEs
151+
152+ Returns
153+ -------
154+ - A Python dictionary of status line changes
155+ - {
156+ CVE_path:{
157+ kernel: status
158+ }
159+ }
160+ - Example:
161+ {
162+ '/home/ubuntu/ubuntu-cve-tracker/active/CVE-2022-1111': {
163+ 'devel_linux': 'ignored (ESM criteria, was needed)',
164+ 'jammy_linux-aws' : 'pending',
165+ },
166+ '/home/ubuntu/ubuntu-cve-tracker/active/CVE-2022-1112': {
167+ 'focal_linux': 'needed',
168+ }
169+ }
170+ """
171+ status_line_re = re.compile(r"_linux.*: ")
172+ to_status = {}
173+
174+ for cve_path in cve_list:
175+ to_status[cve_path] = {}
176+ cmd = ["git", "--no-pager", "diff", "--no-color", SECBRANCH+".."+AUTOBRANCH, "--", cve_path]
177+
178+ try:
179+ dump = subprocess.run(cmd, capture_output=True).stdout.decode()
180+ for line in dump.splitlines():
181+ if status_line_re.search(line):
182+ splitted = line[1::].split(sep=':')
183+ if line[0] == '+' and line[1] != '+':
184+ release = splitted[0]
185+ status = splitted[1][1::] # get rid of the leading space
186+ to_status[cve_path][release] = status
187+ except OSError:
188+ error("Error executing: " + ' '.join(cmd))
189+ except:
190+ error("Unknown error occurs when building status dictionary")
191+
192+ return to_status
193+
194+def main():
195+ parser = argparse.ArgumentParser(
196+ prog='kcve-merge',
197+ description='Merge selective status change from autotriage branch to master branch',
198+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
199+ )
200+ parser.add_argument(
201+ '--kernel',
202+ action='extend',
203+ help='Specify the kernel of CVEs to merge: RELEASE_SOURCE',
204+ nargs='+',
205+ )
206+ parser.add_argument(
207+ '--cve',
208+ action='extend',
209+ help='Specify the CVE to merge its status change: CVE-2023-3611',
210+ nargs='+',
211+ )
212+ parser.add_argument(
213+ '--from',
214+ help='Filter the from status using regex',
215+ )
216+ parser.add_argument(
217+ '--to',
218+ help='Filter the to status using regex',
219+ )
220+ parser.add_argument(
221+ '--cve-list',
222+ help='Provide a CVE list: ./cve_priority.txt',
223+ type=str
224+ )
225+ args = parser.parse_args()
226+ arg_kernel: list[str] = args.kernel
227+ arg_cve: list[str] = args.cve
228+ arg_from = vars(args).get("from")
229+ arg_to = args.to
230+ arg_cve_list = args.cve_list
231+
232+ cve_list = []
233+ cve_path_list = []
234+ to_status = {}
235+ status_line_re = re.compile(r"_linux.*: ")
236+ if arg_from is not None:
237+ try:
238+ from_re = re.compile(arg_from)
239+ except:
240+ error("Error compiling from-status regex: " + arg_from)
241+ exit(1)
242+ if arg_to is not None:
243+ try:
244+ to_re = re.compile(arg_to)
245+ except:
246+ error("Error compiling to-status regex: " + arg_to)
247+ exit(1)
248+
249+ curr_branch = subprocess.run("git rev-parse --abbrev-ref HEAD", shell=True, capture_output=True).stdout.decode().strip()
250+ if curr_branch != SECBRANCH:
251+ error("Please checkout UCT to '" + SECBRANCH + "' branch")
252+ error("currently it is on: " + curr_branch)
253+ exit(-1)
254+
255+ cve_list = sanitize_cves(arg_cve, arg_cve_list)
256+
257+ # Print prompt
258+ info("Merging:")
259+ # print first 5 CVEs
260+ for cve_num in cve_list[:5]:
261+ print(" " + cve_num)
262+ if len(cve_list) - 5 > 0:
263+ print(" ...and " + str(len(cve_list) - 5) + " more")
264+ if arg_kernel is None:
265+ info("With all kernels")
266+ else:
267+ info("with:")
268+ for kernel in arg_kernel:
269+ print(" " + kernel)
270+ if arg_from is not None:
271+ info("which from-status matches: " + arg_from)
272+ if arg_to is not None:
273+ info("which to-status matches: " + arg_to)
274+
275+ cve_path_list = build_path_list(cve_list)
276+ to_status = build_status_dict(cve_path_list)
277+
278+ # Apply changes
279+ for cve_path in cve_path_list:
280+ with fileinput.input(files=cve_path, inplace=True) as f:
281+ for line in f:
282+ if status_line_re.search(line):
283+ kernel = line.split(sep=':')[0]
284+ status = line.split(sep=' ')[1]
285+
286+ if (arg_kernel is None or kernel in arg_kernel) and kernel in to_status[cve_path].keys():
287+ if arg_from is None or from_re.search(status):
288+ if arg_to is None or to_re.search(to_status[cve_path][kernel]):
289+ print(kernel + ": " + to_status[cve_path][kernel], end='\n')
290+ continue
291+ print(line, end='')
292+
293+ # Prepare for git push
294+ git_status = subprocess.run(["git", "status", "--short"], capture_output=True).stdout.decode()
295+ if len(git_status) == 0:
296+ warn('No changes being made')
297+ else:
298+ for cve_num in cve_path_list:
299+ try:
300+ cve_lib.git_add(cve_num)
301+ except ValueError as e:
302+ error(e)
303+ try:
304+ if len(cve_path_list) == 1:
305+ cve_lib.git_commit("kernel/" + cve_path_list[0].split(sep="/")[-1] + ": autotriage")
306+ else:
307+ cve_lib.git_commit("kernel: autotriage")
308+ except ValueError as e:
309+ error(e.args[0])
310+
311+
312+if __name__ == '__main__':
313+ sys.exit(main())

Subscribers

People subscribed via source and target branches

to status/vote changes: