Merge ~litios/ubuntu-cve-tracker:sync-from-usns-extract-cves into ubuntu-cve-tracker:master
- Git
- lp:~litios/ubuntu-cve-tracker
- sync-from-usns-extract-cves
- Merge into master
Status: | Merged |
---|---|
Merged at revision: | 57fd8ff4e536e0686a44c9668b9091925badd85c |
Proposed branch: | ~litios/ubuntu-cve-tracker:sync-from-usns-extract-cves |
Merge into: | ubuntu-cve-tracker:master |
Diff against target: |
775 lines (+463/-242) 3 files modified
.launchpad.yaml (+1/-1) scripts/sync-from-usns.py (+241/-241) scripts/test_sync_from_usns.py (+221/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Spyros Seimenis | Approve | ||
Review via email: mp+443786@code.launchpad.net |
Commit message
Description of the change
USN-3777-1 was triggering an error in sync-from-usns due to the last paragraph.
If the package is a type of kernel, the function extract_
This PR addresses this issue, allowing for the CVE list to be specified in the middle of the description too, as in USN-3777-1.
As of today, we are manually ignoring these issues by specifying to skip parsing them in meta_lists/
David Fernandez Gonzalez (litios) wrote : | # |
Hey Spyros, thanks for all the input. I did some changes:
* Refactor the code to work with regex.
* Added tests
* In order to be able to test, I had to move everything inside the main function so the module could be imported during the tests.
Let me know if it looks good!
Spyros Seimenis (sespiros) wrote : | # |
Hi David, thanks for the changes and the tests! I think overall it reads much better now. A couple of final comments inline.
- (nitpick) you could tighten the CVE regex to CVE-\d{4}-\d{4,7}
- (nitpick) cve_list maybe would read better than cves_chunk as a variable name
Also it seems that the CI tests are failing, not sure if it's because of this or transient error, you can run these locally with
sudo snap install --classic lpci
cd $UCT
lpci
David Fernandez Gonzalez (litios) wrote : | # |
Thanks for the input, it was useful! I added your suggestions to a new commit.
Regarding CI/CD, it is failing but that's because of the check-cves job, the unit-tests is passing which is the one running the test from this PR, so it should be good but let me know if you think there is something else wrong:
============ 869 passed, 1 skipped, 32 warnings in 60.82s (0:01:00) ============
David Fernandez Gonzalez (litios) : | # |
Spyros Seimenis (sespiros) wrote : | # |
So these comments may have been a bit unrelated to the PR but I just found the way we are trying to catch user error confusing in that code. This applies both to the case where we were trying to check for example if '(CVE' exists in the chunk and show warning and the one when we try to spot '..' at the end and fix it rather than warn.
David Fernandez Gonzalez (litios) wrote : | # |
So I agree with the '(CVE' part and I'm currently changing this, I'm testing locally before pushing but I think the '..' it's a different scenario because that's an issue we are creating when removing the CVE list and thus we should be the ones handling it on the code.
Spyros Seimenis (sespiros) : | # |
Preview Diff
1 | diff --git a/.launchpad.yaml b/.launchpad.yaml | |||
2 | index a0a847c..a089c1d 100644 | |||
3 | --- a/.launchpad.yaml | |||
4 | +++ b/.launchpad.yaml | |||
5 | @@ -36,7 +36,7 @@ jobs: | |||
6 | 36 | rm -f embargoed | 36 | rm -f embargoed |
7 | 37 | mkdir embargoed | 37 | mkdir embargoed |
8 | 38 | echo "Running unit tests..." | 38 | echo "Running unit tests..." |
10 | 39 | pytest-3 ./scripts/test_cve_lib.py ./scripts/test_kernel_lib.py ./scripts/test_usn_lib.py ./scripts/test_source_map.py ./scripts/test_publish-cves-to-website-api.py ./test/test_oval_lib_unit.py | 39 | pytest-3 ./scripts/test_cve_lib.py ./scripts/test_kernel_lib.py ./scripts/test_usn_lib.py ./scripts/test_source_map.py ./scripts/test_publish-cves-to-website-api.py ./scripts/test_sync_from_usns.py ./test/test_oval_lib_unit.py |
11 | 40 | # ideally we would also run ./scripts/check-cves --test here as well as it | 40 | # ideally we would also run ./scripts/check-cves --test here as well as it |
12 | 41 | # is more of a unit test but it requires packages-mirror so run it as part | 41 | # is more of a unit test but it requires packages-mirror so run it as part |
13 | 42 | # of check-syntax below | 42 | # of check-syntax below |
14 | diff --git a/scripts/sync-from-usns.py b/scripts/sync-from-usns.py | |||
15 | index ee138b5..b05463e 100755 | |||
16 | --- a/scripts/sync-from-usns.py | |||
17 | +++ b/scripts/sync-from-usns.py | |||
18 | @@ -27,46 +27,7 @@ import usn_lib | |||
19 | 27 | 27 | ||
20 | 28 | from source_map import version_compare, load | 28 | from source_map import version_compare, load |
21 | 29 | 29 | ||
62 | 30 | cves = dict() | 30 | def extract_cve_descriptions(usn, usnnum, verbose): |
23 | 31 | |||
24 | 32 | config = cve_lib.read_config() | ||
25 | 33 | |||
26 | 34 | parser = argparse.ArgumentParser(description="Sync cve status from USN database") | ||
27 | 35 | parser.add_argument("--usn", help="Limit report/update to a single USN", metavar="USN", default=None) | ||
28 | 36 | parser.add_argument("-u", "--update", help="Update CVEs with released package versions", action='store_true') | ||
29 | 37 | parser.add_argument("-U", "--use-usn", help="Use the version in the USN if it differs from the version in UCT", action='store_true') | ||
30 | 38 | parser.add_argument("-v", "--verbose", help="Report logic while processing USNs", action='store_true') | ||
31 | 39 | parser.add_argument("-d", "--debug", help="Report additional debugging while processing USNs", action='store_true') | ||
32 | 40 | parser.add_argument("-r", "--retired", help="Process retired CVEs in addition to active ones", action='store_true') | ||
33 | 41 | parser.add_argument('-g', "--git-stage", help="When updating, stage for commit by adding to git's index (requires --update)", action='store_true') | ||
34 | 42 | parser.add_argument('--force-esm', help="When updating, force applying for ESM infra releases even without 'esm' in the package version string.", action='store_true') | ||
35 | 43 | parser.add_argument("database", nargs="?", help="Use alternate USN database (default: %(default)s)", action='store', default=config['usn_db_copy']) | ||
36 | 44 | args = parser.parse_args() | ||
37 | 45 | |||
38 | 46 | if args.git_stage: | ||
39 | 47 | if not args.update: | ||
40 | 48 | print('--git-stage option requires --update as well, exiting', file=sys.stderr) | ||
41 | 49 | exit(1) | ||
42 | 50 | if not cve_lib.git_is_tree_clean(debug=True): | ||
43 | 51 | print('Please commit or stash your existing changes to UCT first. Aborting.', | ||
44 | 52 | file=sys.stderr) | ||
45 | 53 | exit(1) | ||
46 | 54 | |||
47 | 55 | if args.force_esm and not args.usn: | ||
48 | 56 | print('--force-esm option requires a specific usn to operate on, exiting.', file=sys.stderr) | ||
49 | 57 | exit(1) | ||
50 | 58 | |||
51 | 59 | if args.debug: | ||
52 | 60 | print("Loading %s ..." % (args.database), file=sys.stderr) | ||
53 | 61 | reverted = usn_lib.get_reverted() | ||
54 | 62 | ignored_description = usn_lib.get_ignored_description() | ||
55 | 63 | db = usn_lib.load_database(args.database) | ||
56 | 64 | usnlist = [args.usn] | ||
57 | 65 | if not args.usn: | ||
58 | 66 | usnlist = db | ||
59 | 67 | |||
60 | 68 | |||
61 | 69 | def extract_cve_descriptions(usn, usnnum): | ||
63 | 70 | descriptions = dict() | 31 | descriptions = dict() |
64 | 71 | cves = set() | 32 | cves = set() |
65 | 72 | for cve in usn.get('cves', []): | 33 | for cve in usn.get('cves', []): |
66 | @@ -91,242 +52,281 @@ def extract_cve_descriptions(usn, usnnum): | |||
67 | 91 | 52 | ||
68 | 92 | # Drop un-parened USN qualifiers | 53 | # Drop un-parened USN qualifiers |
69 | 93 | affected = re.compile(' (Only )?Ubuntu [^ ]+( LTS)?(, (and )?Ubuntu [^ ]+( LTS)?)? (was|were) (not )?affected\.') | 54 | affected = re.compile(' (Only )?Ubuntu [^ ]+( LTS)?(, (and )?Ubuntu [^ ]+( LTS)?)? (was|were) (not )?affected\.') |
71 | 94 | 55 | cve_list_regex = re.compile(r' ?\((CVE-\d{4}-\d{4,7},? ?)+\)') | |
72 | 56 | cve_regex = re.compile(r'CVE-\d{4}-\d{4,7}') | ||
73 | 95 | if len(chunks) == 1: | 57 | if len(chunks) == 1: |
74 | 96 | # This description applies to all the CVEs | 58 | # This description applies to all the CVEs |
75 | 97 | for cve in cves: | 59 | for cve in cves: |
76 | 98 | descriptions[cve] = textwrap.fill(description, 75) | 60 | descriptions[cve] = textwrap.fill(description, 75) |
77 | 99 | else: | 61 | else: |
79 | 100 | # Extract trailing (CVE-YYYY-NNNN...) | 62 | # Extracting (CVE-YYYY-NNNN, CVE-...) |
80 | 101 | for chunk in chunks: | 63 | for chunk in chunks: |
81 | 102 | chunk = affected.sub('', chunk) | 64 | chunk = affected.sub('', chunk) |
85 | 103 | if ' (CVE' not in chunk: | 65 | cve_list = cve_list_regex.search(chunk) |
86 | 104 | if args.verbose: | 66 | description = cve_list_regex.sub('', chunk) |
87 | 105 | print("USN %s: CVE not mentioned in chunk: '%s' (ignored)" % (usnnum, chunk), file=sys.stderr) | 67 | |
88 | 68 | if not cve_list: | ||
89 | 69 | if verbose: | ||
90 | 70 | print("USN %s: CVE list is missing: '%s'" % (usnnum, chunk), file=sys.stderr) | ||
91 | 106 | continue | 71 | continue |
108 | 107 | parts = chunk.split(' (CVE-') | 72 | |
109 | 108 | cvelist = 'CVE-%s' % parts.pop() | 73 | # In case the CVE list is in the middle of the description, |
110 | 109 | # Keep only the non-parathesis part | 74 | # so we can preserve the dot. |
111 | 110 | chunk = parts[0] | 75 | if description[-2:] == '..': |
112 | 111 | # Fixup ")." into ")" | 76 | description = description[:-1] |
113 | 112 | if cvelist.endswith(').'): | 77 | |
114 | 113 | cvelist = cvelist[:-2] + ')' | 78 | cves = cve_regex.findall(cve_list.group()) |
115 | 114 | # Validate closing paren | 79 | for cve in cves: |
116 | 115 | if not cvelist.endswith(")"): | 80 | descriptions[cve] = textwrap.fill(description, 75) |
101 | 116 | bad_cve = cvelist.split(')')[0] | ||
102 | 117 | if usnnum not in ignored_description or bad_cve not in ignored_description[usnnum]: | ||
103 | 118 | raise ValueError("USN %s: CVE list does not end with ')': '%s'" % (usnnum, cvelist)) | ||
104 | 119 | cvelist = cvelist[:-1] | ||
105 | 120 | cvelist = cvelist.split(", ") | ||
106 | 121 | for cve in cvelist: | ||
107 | 122 | descriptions[cve] = textwrap.fill(chunk, 75) | ||
117 | 123 | 81 | ||
118 | 124 | return descriptions | 82 | return descriptions |
119 | 125 | 83 | ||
143 | 126 | srcmap = {} | 84 | def parse_args(): |
144 | 127 | for usn in usnlist: | 85 | parser = argparse.ArgumentParser(description="Sync cve status from USN database") |
145 | 128 | ubuntu_descriptions = dict() | 86 | parser.add_argument("--usn", help="Limit report/update to a single USN", metavar="USN", default=None) |
146 | 129 | if args.debug: | 87 | parser.add_argument("-u", "--update", help="Update CVEs with released package versions", action='store_true') |
147 | 130 | print('Checking %s' % (usn), file=sys.stderr) | 88 | parser.add_argument("-U", "--use-usn", help="Use the version in the USN if it differs from the version in UCT", action='store_true') |
148 | 131 | if 'cves' not in db[usn]: | 89 | parser.add_argument("-v", "--verbose", help="Report logic while processing USNs", action='store_true') |
149 | 132 | continue | 90 | parser.add_argument("-d", "--debug", help="Report additional debugging while processing USNs", action='store_true') |
150 | 133 | 91 | parser.add_argument("-r", "--retired", help="Process retired CVEs in addition to active ones", action='store_true') | |
151 | 134 | # Should we update Ubuntu-Description? (only post USN 800 let's say) | 92 | parser.add_argument('-g', "--git-stage", help="When updating, stage for commit by adding to git's index (requires --update)", action='store_true') |
152 | 135 | # Ignored non "-1" USNs for sanity... | 93 | parser.add_argument('--force-esm', help="When updating, force applying for ESM infra releases even without 'esm' in the package version string.", action='store_true') |
153 | 136 | usn_parts = [int(x) for x in usn.split('-')] | 94 | parser.add_argument("database", nargs="?", help="Use alternate USN database (default: %(default)s)", action='store', default=config['usn_db_copy']) |
154 | 137 | if usn_parts[0] > 800 and usn_parts[1] == 1: | 95 | args = parser.parse_args() |
155 | 138 | update_descriptions = False | 96 | return args |
156 | 139 | for rel in db[usn]['releases']: | 97 | |
157 | 140 | # FIXME: known stable kernel release list should be specified somewhere | 98 | if __name__ == '__main__': |
158 | 141 | # else. | 99 | config = cve_lib.read_config() |
159 | 142 | if len(set(db[usn]['releases'][rel].get('sources', [])).intersection(set(cve_lib.kernel_srcs))) > 0: | 100 | |
160 | 143 | update_descriptions = True | 101 | args = parse_args() |
161 | 144 | if args.debug: | 102 | if args.git_stage: |
162 | 145 | print('Extracting Ubuntu-Description from %s' % (usn), file=sys.stderr) | 103 | if not args.update: |
163 | 146 | break | 104 | print('--git-stage option requires --update as well, exiting', file=sys.stderr) |
164 | 147 | if update_descriptions and usn not in ignored_description: | 105 | exit(1) |
165 | 148 | ubuntu_descriptions = extract_cve_descriptions(db[usn], usn) | 106 | if not cve_lib.git_is_tree_clean(debug=True): |
166 | 107 | print('Please commit or stash your existing changes to UCT first. Aborting.', | ||
167 | 108 | file=sys.stderr) | ||
168 | 109 | exit(1) | ||
169 | 110 | |||
170 | 111 | if args.force_esm and not args.usn: | ||
171 | 112 | print('--force-esm option requires a specific usn to operate on, exiting.', file=sys.stderr) | ||
172 | 113 | exit(1) | ||
173 | 149 | 114 | ||
175 | 150 | for cve in db[usn]['cves']: | 115 | if args.debug: |
176 | 116 | print("Loading %s ..." % (args.database), file=sys.stderr) | ||
177 | 117 | |||
178 | 118 | cves = dict() | ||
179 | 119 | reverted = usn_lib.get_reverted() | ||
180 | 120 | ignored_description = usn_lib.get_ignored_description() | ||
181 | 121 | db = usn_lib.load_database(args.database) | ||
182 | 122 | usnlist = [args.usn] | ||
183 | 123 | if not args.usn: | ||
184 | 124 | usnlist = db | ||
185 | 125 | |||
186 | 126 | srcmap = {} | ||
187 | 127 | for usn in usnlist: | ||
188 | 128 | ubuntu_descriptions = dict() | ||
189 | 151 | if args.debug: | 129 | if args.debug: |
194 | 152 | print('Want %s' % (cve), file=sys.stderr) | 130 | print('Checking %s' % (usn), file=sys.stderr) |
195 | 153 | if not cve.startswith('CVE-'): | 131 | if 'cves' not in db[usn]: |
192 | 154 | if args.debug: | ||
193 | 155 | print("Skipping (does not start with 'CVE-')", file=sys.stderr) | ||
196 | 156 | continue | 132 | continue |
199 | 157 | # Skip checking CVEs that were reverted for a given USN | 133 | |
200 | 158 | if usn in reverted and cve in reverted[usn]: | 134 | # Should we update Ubuntu-Description? (only post USN 800 let's say) |
201 | 135 | # Ignored non "-1" USNs for sanity... | ||
202 | 136 | usn_parts = [int(x) for x in usn.split('-')] | ||
203 | 137 | if usn_parts[0] > 800 and usn_parts[1] == 1: | ||
204 | 138 | update_descriptions = False | ||
205 | 139 | for rel in db[usn]['releases']: | ||
206 | 140 | # FIXME: known stable kernel release list should be specified somewhere | ||
207 | 141 | # else. | ||
208 | 142 | if len(set(db[usn]['releases'][rel].get('sources', [])).intersection(set(cve_lib.kernel_srcs))) > 0: | ||
209 | 143 | update_descriptions = True | ||
210 | 144 | if args.debug: | ||
211 | 145 | print('Extracting Ubuntu-Description from %s' % (usn), file=sys.stderr) | ||
212 | 146 | break | ||
213 | 147 | if update_descriptions and usn not in ignored_description: | ||
214 | 148 | ubuntu_descriptions = extract_cve_descriptions(db[usn], usn, args.verbose) | ||
215 | 149 | |||
216 | 150 | for cve in db[usn]['cves']: | ||
217 | 159 | if args.debug: | 151 | if args.debug: |
227 | 160 | print("Skipping (was reverted)", file=sys.stderr) | 152 | print('Want %s' % (cve), file=sys.stderr) |
228 | 161 | continue | 153 | if not cve.startswith('CVE-'): |
220 | 162 | filename = '%s/%s' % (cve_lib.active_dir, cve) | ||
221 | 163 | if os.path.exists('%s/%s' % (cve_lib.retired_dir, cve)): | ||
222 | 164 | if args.retired: | ||
223 | 165 | # include retired CVEs (may create false warnings) | ||
224 | 166 | filename = '%s/%s' % (cve_lib.retired_dir, cve) | ||
225 | 167 | else: | ||
226 | 168 | # Skip retired CVEs | ||
229 | 169 | if args.debug: | 154 | if args.debug: |
231 | 170 | print("Skipping (already retired)", file=sys.stderr) | 155 | print("Skipping (does not start with 'CVE-')", file=sys.stderr) |
232 | 171 | continue | 156 | continue |
245 | 172 | if os.path.exists('%s/%s' % (cve_lib.ignored_dir, cve)): | 157 | # Skip checking CVEs that were reverted for a given USN |
246 | 173 | # Skip ignored CVEs, may have been REJECTED after USN publication | 158 | if usn in reverted and cve in reverted[usn]: |
247 | 174 | if args.debug: | 159 | if args.debug: |
248 | 175 | print("Skipping (already ignored)", file=sys.stderr) | 160 | print("Skipping (was reverted)", file=sys.stderr) |
237 | 176 | continue | ||
238 | 177 | if os.path.exists(filename): | ||
239 | 178 | if args.verbose: | ||
240 | 179 | print('USN %s refers to %s' % (usn, cve)) | ||
241 | 180 | try: | ||
242 | 181 | data = cve_lib.load_cve(filename) | ||
243 | 182 | except ValueError as e: | ||
244 | 183 | print(e, file=sys.stderr) | ||
249 | 184 | continue | 161 | continue |
257 | 185 | cves.setdefault(cve, data) | 162 | filename = '%s/%s' % (cve_lib.active_dir, cve) |
258 | 186 | 163 | if os.path.exists('%s/%s' % (cve_lib.retired_dir, cve)): | |
259 | 187 | # update Ubuntu-Description | 164 | if args.retired: |
260 | 188 | if cve in ubuntu_descriptions: | 165 | # include retired CVEs (may create false warnings) |
261 | 189 | if usn in ignored_description and cve in ignored_description[usn]: | 166 | filename = '%s/%s' % (cve_lib.retired_dir, cve) |
255 | 190 | if args.debug: | ||
256 | 191 | print("Skipping update of description due to ignore list", file=sys.stderr) | ||
262 | 192 | else: | 167 | else: |
266 | 193 | desc = ubuntu_descriptions[cve] | 168 | # Skip retired CVEs |
267 | 194 | if data.get('Ubuntu-Description', None) != '\n' + desc: | 169 | if args.debug: |
268 | 195 | print("USN %s has updated Ubuntu-Description for %s:\n %s" % (usn, cve, "\n ".join(desc.strip().splitlines())), file=sys.stderr) | 170 | print("Skipping (already retired)", file=sys.stderr) |
269 | 171 | continue | ||
270 | 172 | if os.path.exists('%s/%s' % (cve_lib.ignored_dir, cve)): | ||
271 | 173 | # Skip ignored CVEs, may have been REJECTED after USN publication | ||
272 | 174 | if args.debug: | ||
273 | 175 | print("Skipping (already ignored)", file=sys.stderr) | ||
274 | 176 | continue | ||
275 | 177 | if os.path.exists(filename): | ||
276 | 178 | if args.verbose: | ||
277 | 179 | print('USN %s refers to %s' % (usn, cve)) | ||
278 | 180 | try: | ||
279 | 181 | data = cve_lib.load_cve(filename) | ||
280 | 182 | except ValueError as e: | ||
281 | 183 | print(e, file=sys.stderr) | ||
282 | 184 | continue | ||
283 | 185 | cves.setdefault(cve, data) | ||
284 | 186 | |||
285 | 187 | # update Ubuntu-Description | ||
286 | 188 | if cve in ubuntu_descriptions: | ||
287 | 189 | if usn in ignored_description and cve in ignored_description[usn]: | ||
288 | 196 | if args.debug: | 190 | if args.debug: |
290 | 197 | print("[%s]\n[%s]" % (data.get('Ubuntu-Description', ''), '\n' + desc), file=sys.stderr) | 191 | print("Skipping update of description due to ignore list", file=sys.stderr) |
291 | 192 | else: | ||
292 | 193 | desc = ubuntu_descriptions[cve] | ||
293 | 194 | if data.get('Ubuntu-Description', None) != '\n' + desc: | ||
294 | 195 | print("USN %s has updated Ubuntu-Description for %s:\n %s" % (usn, cve, "\n ".join(desc.strip().splitlines())), file=sys.stderr) | ||
295 | 196 | if args.debug: | ||
296 | 197 | print("[%s]\n[%s]" % (data.get('Ubuntu-Description', ''), '\n' + desc), file=sys.stderr) | ||
297 | 198 | if args.update: | ||
298 | 199 | cve_lib.update_multiline_field(filename, 'Ubuntu-Description', desc) | ||
299 | 200 | if args.git_stage: | ||
300 | 201 | cve_lib.git_add(filename) | ||
301 | 202 | |||
302 | 203 | # update References | ||
303 | 204 | if 'References' in data: | ||
304 | 205 | usn_ref = "https://ubuntu.com/security/notices/USN-" + usn | ||
305 | 206 | found = False | ||
306 | 207 | if usn_ref in data['References']: | ||
307 | 208 | found = True | ||
308 | 209 | if not found: | ||
309 | 210 | print("%s references %s" % (usn_ref, cve), file=sys.stderr) | ||
310 | 198 | if args.update: | 211 | if args.update: |
312 | 199 | cve_lib.update_multiline_field(filename, 'Ubuntu-Description', desc) | 212 | cve_lib.add_reference(filename, usn_ref) |
313 | 200 | if args.git_stage: | 213 | if args.git_stage: |
314 | 201 | cve_lib.git_add(filename) | 214 | cve_lib.git_add(filename) |
315 | 202 | 215 | ||
324 | 203 | # update References | 216 | # Record what the PublicDate field was when we published, in case |
325 | 204 | if 'References' in data: | 217 | # NVD moves it around. |
326 | 205 | usn_ref = "https://ubuntu.com/security/notices/USN-" + usn | 218 | if 'PublicDateAtUSN' not in data: |
327 | 206 | found = False | 219 | if data['PublicDate'].strip() == "": |
328 | 207 | if usn_ref in data['References']: | 220 | print("Yikes, empty PublicDate for %s" % (cve), file=sys.stderr) |
329 | 208 | found = True | 221 | sys.exit(1) |
322 | 209 | if not found: | ||
323 | 210 | print("%s references %s" % (usn_ref, cve), file=sys.stderr) | ||
330 | 211 | if args.update: | 222 | if args.update: |
332 | 212 | cve_lib.add_reference(filename, usn_ref) | 223 | cve_lib.prepend_field(filename, 'PublicDateAtUSN', data['PublicDate']) |
333 | 213 | if args.git_stage: | 224 | if args.git_stage: |
334 | 214 | cve_lib.git_add(filename) | 225 | cve_lib.git_add(filename) |
335 | 215 | 226 | ||
365 | 216 | # Record what the PublicDate field was when we published, in case | 227 | for rel in db[usn]['releases']: |
366 | 217 | # NVD moves it around. | 228 | if 'sources' not in db[usn]['releases'][rel]: |
367 | 218 | if 'PublicDateAtUSN' not in data: | 229 | if args.debug: |
368 | 219 | if data['PublicDate'].strip() == "": | 230 | print(" strange: %s listed, but without any changed sources -- skipping release" % (rel)) |
369 | 220 | print("Yikes, empty PublicDate for %s" % (cve), file=sys.stderr) | 231 | continue |
370 | 221 | sys.exit(1) | 232 | cve_rel = rel |
371 | 222 | if args.update: | 233 | if not cve_lib.is_active_release(rel) and cve_lib.is_active_esm_release(rel): |
372 | 223 | cve_lib.prepend_field(filename, 'PublicDateAtUSN', data['PublicDate']) | 234 | cve_rel = cve_lib.get_esm_name(rel) |
373 | 224 | if args.git_stage: | 235 | for src in db[usn]['releases'][rel]['sources']: |
374 | 225 | cve_lib.git_add(filename) | 236 | version = db[usn]['releases'][rel]['sources'][src]['version'] |
375 | 226 | 237 | esm_version_match = re.search("[\+~]esm\d+", version) | |
376 | 227 | for rel in db[usn]['releases']: | 238 | if esm_version_match: |
377 | 228 | if 'sources' not in db[usn]['releases'][rel]: | 239 | if cve_lib.is_active_release(rel): |
349 | 229 | if args.debug: | ||
350 | 230 | print(" strange: %s listed, but without any changed sources -- skipping release" % (rel)) | ||
351 | 231 | continue | ||
352 | 232 | cve_rel = rel | ||
353 | 233 | if not cve_lib.is_active_release(rel) and cve_lib.is_active_esm_release(rel): | ||
354 | 234 | cve_rel = cve_lib.get_esm_name(rel) | ||
355 | 235 | for src in db[usn]['releases'][rel]['sources']: | ||
356 | 236 | version = db[usn]['releases'][rel]['sources'][src]['version'] | ||
357 | 237 | esm_version_match = re.search("[\+~]esm\d+", version) | ||
358 | 238 | if esm_version_match: | ||
359 | 239 | if cve_lib.is_active_release(rel): | ||
360 | 240 | cve_rel = cve_lib.get_esm_name(rel, 'universe') | ||
361 | 241 | else: | ||
362 | 242 | if not rel in srcmap: | ||
363 | 243 | srcmap[rel] = load(releases=[rel], skip_eol_releases=False)[rel] | ||
364 | 244 | if cve_lib.is_universe(srcmap, src, rel, None): | ||
378 | 245 | cve_rel = cve_lib.get_esm_name(rel, 'universe') | 240 | cve_rel = cve_lib.get_esm_name(rel, 'universe') |
379 | 246 | else: | 241 | else: |
395 | 247 | cve_rel = cve_lib.get_esm_name(rel) | 242 | if not rel in srcmap: |
396 | 248 | # If the version doesn't match 'esm' and this is | 243 | srcmap[rel] = load(releases=[rel], skip_eol_releases=False)[rel] |
397 | 249 | # for an esm release, then skip (because that would | 244 | if cve_lib.is_universe(srcmap, src, rel, None): |
398 | 250 | # match for all the USNs that were published prior | 245 | cve_rel = cve_lib.get_esm_name(rel, 'universe') |
399 | 251 | # to a release going into ESM infra status), | 246 | else: |
400 | 252 | # *unless* the --force-esm argument has been passed | 247 | cve_rel = cve_lib.get_esm_name(rel) |
401 | 253 | # for updates prepared by other teams that do not | 248 | # If the version doesn't match 'esm' and this is |
402 | 254 | # use the "esm" in the version string convention, | 249 | # for an esm release, then skip (because that would |
403 | 255 | # like the kernel team. | 250 | # match for all the USNs that were published prior |
404 | 256 | elif not args.force_esm and not esm_version_match and 'esm' in cve_rel: | 251 | # to a release going into ESM infra status), |
405 | 257 | continue | 252 | # *unless* the --force-esm argument has been passed |
406 | 258 | 253 | # for updates prepared by other teams that do not | |
407 | 259 | if src not in cves[cve]['pkgs'] or cve_rel not in cves[cve]['pkgs'][src]: | 254 | # use the "esm" in the version string convention, |
408 | 260 | # HACK: ignore abandoned linux topic branches | 255 | # like the kernel team. |
409 | 261 | if src in ['linux-ti-omap', 'linux-qcm-msm']: | 256 | elif not args.force_esm and not esm_version_match and 'esm' in cve_rel: |
410 | 262 | continue | 257 | continue |
414 | 263 | # HACK: ignore firefox-* packages since we track | 258 | |
415 | 264 | # xulrunner. These existed only from hardy-karmic. | 259 | if src not in cves[cve]['pkgs'] or cve_rel not in cves[cve]['pkgs'][src]: |
416 | 265 | if src in ['firefox-3.0', 'firefox-3.1', 'firefox-3.5']: | 260 | # HACK: ignore abandoned linux topic branches |
417 | 261 | if src in ['linux-ti-omap', 'linux-qcm-msm']: | ||
418 | 262 | continue | ||
419 | 263 | # HACK: ignore firefox-* packages since we track | ||
420 | 264 | # xulrunner. These existed only from hardy-karmic. | ||
421 | 265 | if src in ['firefox-3.0', 'firefox-3.1', 'firefox-3.5']: | ||
422 | 266 | continue | ||
423 | 267 | # skip eol releases | ||
424 | 268 | if not cve_lib.is_active_release(rel) and not cve_lib.is_active_esm_release(rel): | ||
425 | 269 | continue | ||
426 | 270 | print("USN-%s touches %s in %s with %s (but is not listed in %s)" % (usn, src, cve_rel, cve, filename), file=sys.stderr) | ||
427 | 266 | continue | 271 | continue |
430 | 267 | # skip eol releases | 272 | state, notes = cves[cve]['pkgs'][src][cve_rel] |
431 | 268 | if not cve_lib.is_active_release(rel) and not cve_lib.is_active_esm_release(rel): | 273 | |
432 | 274 | # A CVE is tied to a USN, which means sometimes the CVE | ||
433 | 275 | # doesn't affect all releases of package, so skip | ||
434 | 276 | # not-affected without comment | ||
435 | 277 | if state == 'not-affected': | ||
436 | 278 | if args.verbose: | ||
437 | 279 | print(" %s/%s marked 'not-affected' -- ignoring" % (src, cve_rel)) | ||
438 | 269 | continue | 280 | continue |
439 | 270 | print("USN-%s touches %s in %s with %s (but is not listed in %s)" % (usn, src, cve_rel, cve, filename), file=sys.stderr) | ||
440 | 271 | continue | ||
441 | 272 | state, notes = cves[cve]['pkgs'][src][cve_rel] | ||
442 | 273 | |||
443 | 274 | # A CVE is tied to a USN, which means sometimes the CVE | ||
444 | 275 | # doesn't affect all releases of package, so skip | ||
445 | 276 | # not-affected without comment | ||
446 | 277 | if state == 'not-affected': | ||
447 | 278 | if args.verbose: | ||
448 | 279 | print(" %s/%s marked 'not-affected' -- ignoring" % (src, cve_rel)) | ||
449 | 280 | continue | ||
450 | 281 | |||
451 | 282 | if state == 'DNE' and cve_lib.is_active_esm_release(rel): | ||
452 | 283 | if args.verbose: | ||
453 | 284 | print(" %s/%s marked 'DNE' -- ignoring" % (src, cve_rel)) | ||
454 | 285 | continue | ||
455 | 286 | # if state == 'pending' and notes == db[usn]['releases'][rel]['sources'][src]['version']: | ||
456 | 287 | # # Found aligned pending/released pair | ||
457 | 288 | # pass | ||
458 | 289 | 281 | ||
462 | 290 | if state not in ['needed', 'deferred', 'pending', 'released', 'active', 'needs-triage', 'ignored']: | 282 | if state == 'DNE' and cve_lib.is_active_esm_release(rel): |
463 | 291 | print("USN-%s fixed %s in %s %s/%s (but is marked %s)!?" % (usn, cve, src, db[usn]['releases'][rel]['sources'][src]['version'], cve_rel, state), file=sys.stderr) | 283 | if args.verbose: |
464 | 292 | continue | 284 | print(" %s/%s marked 'DNE' -- ignoring" % (src, cve_rel)) |
465 | 285 | continue | ||
466 | 286 | # if state == 'pending' and notes == db[usn]['releases'][rel]['sources'][src]['version']: | ||
467 | 287 | # # Found aligned pending/released pair | ||
468 | 288 | # pass | ||
469 | 293 | 289 | ||
484 | 294 | if state != 'released': | 290 | if state not in ['needed', 'deferred', 'pending', 'released', 'active', 'needs-triage', 'ignored']: |
485 | 295 | # CVE db is the "master" for when a CVE was fixed, | 291 | print("USN-%s fixed %s in %s %s/%s (but is marked %s)!?" % (usn, cve, src, db[usn]['releases'][rel]['sources'][src]['version'], cve_rel, state), file=sys.stderr) |
472 | 296 | # so only fill in the version from the USN if the | ||
473 | 297 | # fixed version is not already known to the CVE db. | ||
474 | 298 | detail = "" | ||
475 | 299 | version = notes | ||
476 | 300 | usn_ver = db[usn]['releases'][rel]['sources'][src]['version'] | ||
477 | 301 | if version == "": | ||
478 | 302 | version = usn_ver | ||
479 | 303 | elif version != usn_ver: | ||
480 | 304 | detail = " (USN: %s ) " % (usn_ver) | ||
481 | 305 | print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, cve_rel, state), file=sys.stderr) | ||
482 | 306 | if version_compare(version, usn_ver) > 0 and not (state == 'deferred' or state == 'ignored' or args.use_usn): | ||
483 | 307 | print("ERROR: Version in CVE (%s) is higher than USN version! Skipping" % cve, file=sys.stderr) | ||
486 | 308 | continue | 292 | continue |
487 | 309 | if args.use_usn or state == 'deferred' or state == 'ignored': | ||
488 | 310 | version = usn_ver | ||
489 | 311 | if args.update: | ||
490 | 312 | cve_lib.update_state(filename, src, cve_rel, 'released', version) | ||
491 | 313 | 293 | ||
493 | 314 | if esm_version_match: | 294 | if state != 'released': |
494 | 295 | # CVE db is the "master" for when a CVE was fixed, | ||
495 | 296 | # so only fill in the version from the USN if the | ||
496 | 297 | # fixed version is not already known to the CVE db. | ||
497 | 298 | detail = "" | ||
498 | 299 | version = notes | ||
499 | 300 | usn_ver = db[usn]['releases'][rel]['sources'][src]['version'] | ||
500 | 301 | if version == "": | ||
501 | 302 | version = usn_ver | ||
502 | 303 | elif version != usn_ver: | ||
503 | 304 | detail = " (USN: %s ) " % (usn_ver) | ||
504 | 305 | print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, cve_rel, state), file=sys.stderr) | ||
505 | 306 | if version_compare(version, usn_ver) > 0 and not (state == 'deferred' or state == 'ignored' or args.use_usn): | ||
506 | 307 | print("ERROR: Version in CVE (%s) is higher than USN version! Skipping" % cve, file=sys.stderr) | ||
507 | 315 | continue | 308 | continue |
525 | 316 | 309 | if args.use_usn or state == 'deferred' or state == 'ignored': | |
526 | 317 | if not cve_rel in srcmap: | 310 | version = usn_ver |
527 | 318 | srcmap[cve_rel] = load(releases=[cve_rel], skip_eol_releases=False)[cve_rel] | 311 | if args.update: |
528 | 319 | 312 | cve_lib.update_state(filename, src, cve_rel, 'released', version) | |
529 | 320 | esm_rel = cve_lib.get_esm_name(cve_rel, 'universe' if cve_lib.is_universe(srcmap, src, cve_rel, None) else 'main') | 313 | |
530 | 321 | if esm_rel and esm_rel in cves[cve]['pkgs'][src]: | 314 | if esm_version_match: |
531 | 322 | status_esm = cves[cve]['pkgs'][src][esm_rel][0] | 315 | continue |
532 | 323 | if status_esm != 'released' and status_esm != 'not-affected' and status_esm != 'ignored': | 316 | |
533 | 324 | print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, esm_rel, status_esm), file=sys.stderr) | 317 | if not cve_rel in srcmap: |
534 | 325 | cve_lib.update_state(filename, src, esm_rel, 'not-affected', version) | 318 | srcmap[cve_rel] = load(releases=[cve_rel], skip_eol_releases=False)[cve_rel] |
535 | 326 | 319 | ||
536 | 327 | if args.git_stage: | 320 | esm_rel = cve_lib.get_esm_name(cve_rel, 'universe' if cve_lib.is_universe(srcmap, src, cve_rel, None) else 'main') |
537 | 328 | cve_lib.git_add(filename) | 321 | if esm_rel and esm_rel in cves[cve]['pkgs'][src]: |
538 | 329 | elif args.debug: | 322 | status_esm = cves[cve]['pkgs'][src][esm_rel][0] |
539 | 330 | print(" %s/%s marked 'released' -- ignoring" % (src, cve_rel)) | 323 | if status_esm != 'released' and status_esm != 'not-affected' and status_esm != 'ignored': |
540 | 331 | else: | 324 | print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, esm_rel, status_esm), file=sys.stderr) |
541 | 332 | print("USN-%s fixed %s but it is neither active nor retired" % (usn, cve), file=sys.stderr) | 325 | cve_lib.update_state(filename, src, esm_rel, 'not-affected', version) |
542 | 326 | |||
543 | 327 | if args.git_stage: | ||
544 | 328 | cve_lib.git_add(filename) | ||
545 | 329 | elif args.debug: | ||
546 | 330 | print(" %s/%s marked 'released' -- ignoring" % (src, cve_rel)) | ||
547 | 331 | else: | ||
548 | 332 | print("USN-%s fixed %s but it is neither active nor retired" % (usn, cve), file=sys.stderr) | ||
549 | diff --git a/scripts/test_sync_from_usns.py b/scripts/test_sync_from_usns.py | |||
550 | 333 | new file mode 100644 | 333 | new file mode 100644 |
551 | index 0000000..c97a8ac | |||
552 | --- /dev/null | |||
553 | +++ b/scripts/test_sync_from_usns.py | |||
554 | @@ -0,0 +1,221 @@ | |||
555 | 1 | import pytest | ||
556 | 2 | import mock | ||
557 | 3 | import importlib | ||
558 | 4 | import usn_lib | ||
559 | 5 | |||
560 | 6 | descriptions = [ | ||
561 | 7 | { | ||
562 | 8 | 'description': | ||
563 | 9 | ''' | ||
564 | 10 | Andy Lutomirski and Mika Penttilä discovered that the KVM implementation | ||
565 | 11 | in the Linux kernel did not properly check privilege levels when emulating | ||
566 | 12 | some instructions. An unprivileged attacker in a guest VM could use this to | ||
567 | 13 | escalate privileges within the guest. (CVE-2018-10853, CVE-2023-1010) | ||
568 | 14 | |||
569 | 15 | It was discovered that a use-after-free vulnerability existed in the IRDA | ||
570 | 16 | implementation in the Linux kernel. A local attacker could use this to | ||
571 | 17 | cause a denial of service (system crash) or possibly execute arbitrary | ||
572 | 18 | code. (CVE-2018-6555) | ||
573 | 19 | ''', | ||
574 | 20 | 'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2023-1010'], | ||
575 | 21 | 'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM ' | ||
576 | 22 | 'implementation in\n' | ||
577 | 23 | 'the Linux kernel did not properly check privilege levels ' | ||
578 | 24 | 'when emulating\n' | ||
579 | 25 | 'some instructions. An unprivileged attacker in a guest VM ' | ||
580 | 26 | 'could use this to\n' | ||
581 | 27 | 'escalate privileges within the guest.', | ||
582 | 28 | 'CVE-2018-6555': 'It was discovered that a use-after-free vulnerability ' | ||
583 | 29 | 'existed in the IRDA\n' | ||
584 | 30 | 'implementation in the Linux kernel. A local attacker could ' | ||
585 | 31 | 'use this to\n' | ||
586 | 32 | 'cause a denial of service (system crash) or possibly ' | ||
587 | 33 | 'execute arbitrary\n' | ||
588 | 34 | 'code.', | ||
589 | 35 | 'CVE-2023-1010': 'Andy Lutomirski and Mika Penttilä discovered that the KVM ' | ||
590 | 36 | 'implementation in\n' | ||
591 | 37 | 'the Linux kernel did not properly check privilege levels ' | ||
592 | 38 | 'when emulating\n' | ||
593 | 39 | 'some instructions. An unprivileged attacker in a guest VM ' | ||
594 | 40 | 'could use this to\n' | ||
595 | 41 | 'escalate privileges within the guest.'} | ||
596 | 42 | }, | ||
597 | 43 | { | ||
598 | 44 | 'description':""" | ||
599 | 45 | Andy Lutomirski and Mika Penttilä discovered that the KVM implementation | ||
600 | 46 | in the Linux kernel did not properly check privilege levels when emulating | ||
601 | 47 | some instructions. An unprivileged attacker in a guest VM could use this to | ||
602 | 48 | escalate privileges within the guest. (CVE-2018-10853) | ||
603 | 49 | |||
604 | 50 | USN 3652-1 added a mitigation for Speculative Store Bypass | ||
605 | 51 | a.k.a. Spectre Variant 4 (CVE-2018-3639). This update provides the | ||
606 | 52 | corresponding mitigation for ARM64 processors. Please note that for | ||
607 | 53 | this mitigation to be effective, an updated firmware for the processor | ||
608 | 54 | may be required. | ||
609 | 55 | """, | ||
610 | 56 | 'cves': ['CVE-2018-10853', 'CVE-2018-3639'], | ||
611 | 57 | 'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM ' | ||
612 | 58 | 'implementation in\n' | ||
613 | 59 | 'the Linux kernel did not properly check privilege levels ' | ||
614 | 60 | 'when emulating\n' | ||
615 | 61 | 'some instructions. An unprivileged attacker in a guest VM ' | ||
616 | 62 | 'could use this to\n' | ||
617 | 63 | 'escalate privileges within the guest.', | ||
618 | 64 | 'CVE-2018-3639': 'USN 3652-1 added a mitigation for Speculative Store Bypass ' | ||
619 | 65 | 'a.k.a. Spectre\n' | ||
620 | 66 | 'Variant 4. This update provides the corresponding ' | ||
621 | 67 | 'mitigation for ARM64\n' | ||
622 | 68 | 'processors. Please note that for this mitigation to be ' | ||
623 | 69 | 'effective, an\n' | ||
624 | 70 | 'updated firmware for the processor may be required.'} | ||
625 | 71 | }, | ||
626 | 72 | { | ||
627 | 73 | 'description':""" | ||
628 | 74 | Jann Horn discovered that microprocessors utilizing speculative | ||
629 | 75 | execution and branch prediction may allow unauthorized memory | ||
630 | 76 | reads via sidechannel attacks. This flaw is known as Spectre. A | ||
631 | 77 | local attacker could use this to expose sensitive information, | ||
632 | 78 | including kernel memory. This update provides mitigations for the | ||
633 | 79 | i386 (CVE-2017-9999 only), amd64, ppc64el, and s390x architectures. | ||
634 | 80 | (CVE-2017-5715, CVE-2017-5753) | ||
635 | 81 | |||
636 | 82 | USN-3522-1 mitigated CVE-2017-5754 (Meltdown) for the amd64 | ||
637 | 83 | architecture in Ubuntu 16.04 LTS. This update provides the | ||
638 | 84 | corresponding mitigations for the ppc64el architecture. Original | ||
639 | 85 | advisory details: | ||
640 | 86 | |||
641 | 87 | Jann Horn discovered that microprocessors utilizing speculative | ||
642 | 88 | execution and indirect branch prediction may allow unauthorized memory | ||
643 | 89 | reads via sidechannel attacks. This flaw is known as Meltdown. A local | ||
644 | 90 | attacker could use this to expose sensitive information, including | ||
645 | 91 | kernel memory. (CVE-2017-5754) | ||
646 | 92 | """, | ||
647 | 93 | 'cves': ['CVE-2017-5715', 'CVE-2017-5753', 'CVE-2017-5754'], | ||
648 | 94 | 'result': {'CVE-2017-5715': 'Jann Horn discovered that microprocessors utilizing ' | ||
649 | 95 | 'speculative execution\n' | ||
650 | 96 | 'and branch prediction may allow unauthorized memory reads ' | ||
651 | 97 | 'via sidechannel\n' | ||
652 | 98 | 'attacks. This flaw is known as Spectre. A local attacker ' | ||
653 | 99 | 'could use this to\n' | ||
654 | 100 | 'expose sensitive information, including kernel memory. This ' | ||
655 | 101 | 'update provides\n' | ||
656 | 102 | 'mitigations for the i386 (CVE-2017-9999 only), amd64, ' | ||
657 | 103 | 'ppc64el, and s390x\n' | ||
658 | 104 | 'architectures.', | ||
659 | 105 | 'CVE-2017-5753': 'Jann Horn discovered that microprocessors utilizing ' | ||
660 | 106 | 'speculative execution\n' | ||
661 | 107 | 'and branch prediction may allow unauthorized memory reads ' | ||
662 | 108 | 'via sidechannel\n' | ||
663 | 109 | 'attacks. This flaw is known as Spectre. A local attacker ' | ||
664 | 110 | 'could use this to\n' | ||
665 | 111 | 'expose sensitive information, including kernel memory. This ' | ||
666 | 112 | 'update provides\n' | ||
667 | 113 | 'mitigations for the i386 (CVE-2017-9999 only), amd64, ' | ||
668 | 114 | 'ppc64el, and s390x\n' | ||
669 | 115 | 'architectures.', | ||
670 | 116 | 'CVE-2017-5754': 'Jann Horn discovered that microprocessors utilizing ' | ||
671 | 117 | 'speculative execution\n' | ||
672 | 118 | 'and indirect branch prediction may allow unauthorized ' | ||
673 | 119 | 'memory reads via\n' | ||
674 | 120 | 'sidechannel attacks. This flaw is known as Meltdown. A ' | ||
675 | 121 | 'local attacker could\n' | ||
676 | 122 | 'use this to expose sensitive information, including kernel ' | ||
677 | 123 | 'memory.'} | ||
678 | 124 | }, | ||
679 | 125 | { | ||
680 | 126 | 'description':""" | ||
681 | 127 | Andy Lutomirski and Mika Penttilä discovered that the KVM implementation | ||
682 | 128 | in the Linux kernel did not properly check privilege levels when emulating | ||
683 | 129 | some instructions. An unprivileged attacker in a guest VM could use this to | ||
684 | 130 | escalate privileges within the guest. (CVE-2018-10853, CVE-2023-1010) | ||
685 | 131 | |||
686 | 132 | It was discovered that a use-after-free vulnerability existed in the IRDA | ||
687 | 133 | implementation in the Linux kernel. A local attacker could use this to | ||
688 | 134 | cause a denial of service (system crash) or possibly execute arbitrary | ||
689 | 135 | code. (CVE-2018-6555). | ||
690 | 136 | """, | ||
691 | 137 | 'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2023-1010'], | ||
692 | 138 | 'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM ' | ||
693 | 139 | 'implementation in\n' | ||
694 | 140 | 'the Linux kernel did not properly check privilege levels ' | ||
695 | 141 | 'when emulating\n' | ||
696 | 142 | 'some instructions. An unprivileged attacker in a guest VM ' | ||
697 | 143 | 'could use this to\n' | ||
698 | 144 | 'escalate privileges within the guest.', | ||
699 | 145 | 'CVE-2018-6555': 'It was discovered that a use-after-free vulnerability ' | ||
700 | 146 | 'existed in the IRDA\n' | ||
701 | 147 | 'implementation in the Linux kernel. A local attacker could ' | ||
702 | 148 | 'use this to\n' | ||
703 | 149 | 'cause a denial of service (system crash) or possibly ' | ||
704 | 150 | 'execute arbitrary\n' | ||
705 | 151 | 'code.', | ||
706 | 152 | 'CVE-2023-1010': 'Andy Lutomirski and Mika Penttilä discovered that the KVM ' | ||
707 | 153 | 'implementation in\n' | ||
708 | 154 | 'the Linux kernel did not properly check privilege levels ' | ||
709 | 155 | 'when emulating\n' | ||
710 | 156 | 'some instructions. An unprivileged attacker in a guest VM ' | ||
711 | 157 | 'could use this to\n' | ||
712 | 158 | 'escalate privileges within the guest.'} | ||
713 | 159 | } | ||
714 | 160 | ] | ||
715 | 161 | |||
716 | 162 | broken_descriptions = [ | ||
717 | 163 | { | ||
718 | 164 | 'description': | ||
719 | 165 | """ | ||
720 | 166 | Andy Lutomirski and Mika Penttilä discovered that the KVM implementation | ||
721 | 167 | in the Linux kernel did not properly check privilege levels when emulating | ||
722 | 168 | some instructions. An unprivileged attacker in a guest VM could use this to | ||
723 | 169 | escalate privileges within the guest. (CVE-2018-10853) | ||
724 | 170 | |||
725 | 171 | It was discovered that a use-after-free vulnerability existed in the IRDA | ||
726 | 172 | implementation in the Linux kernel. A local attacker could use this to | ||
727 | 173 | cause a denial of service (system crash) or possibly execute arbitrary | ||
728 | 174 | code. (CVE-2018-6555, CVE-2018-6556 | ||
729 | 175 | """, | ||
730 | 176 | 'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2018-6556'], | ||
731 | 177 | 'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM implementation in\nthe Linux kernel did not properly check privilege levels when emulating\nsome instructions. An unprivileged attacker in a guest VM could use this to\nescalate privileges within the guest.'}, | ||
732 | 178 | 'error': "USN 1-1: CVE list is missing: 'It was discovered that a use-after-free vulnerability existed in the IRDA implementation in the Linux kernel. A local attacker could use this to cause a denial of service (system crash) or possibly execute arbitrary code. (CVE-2018-6555, CVE-2018-6556'\n" | ||
733 | 179 | }, | ||
734 | 180 | { | ||
735 | 181 | 'description': | ||
736 | 182 | """ | ||
737 | 183 | Andy Lutomirski and Mika Penttilä discovered that the KVM implementation | ||
738 | 184 | in the Linux kernel did not properly check privilege levels when emulating | ||
739 | 185 | some instructions. An unprivileged attacker in a guest VM could use this to | ||
740 | 186 | escalate privileges within the guest. (CVE-2018-10853) | ||
741 | 187 | |||
742 | 188 | USN 3652-1 added a mitigation for Speculative Store Bypass | ||
743 | 189 | a.k.a. Spectre Variant 4 (CVE-2018-3639. This update provides the | ||
744 | 190 | corresponding mitigation for ARM64 processors. Please note that for | ||
745 | 191 | this mitigation to be effective, an updated firmware for the processor | ||
746 | 192 | may be required. | ||
747 | 193 | """, | ||
748 | 194 | 'cves': ['CVE-2018-10853', 'CVE-2018-3639'], | ||
749 | 195 | 'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM implementation in\nthe Linux kernel did not properly check privilege levels when emulating\nsome instructions. An unprivileged attacker in a guest VM could use this to\nescalate privileges within the guest.'}, | ||
750 | 196 | 'error': "USN 1-1: CVE list is missing: 'USN 3652-1 added a mitigation for Speculative Store Bypass a.k.a. Spectre Variant 4 (CVE-2018-3639. This update provides the corresponding mitigation for ARM64 processors. Please note that for this mitigation to be effective, an updated firmware for the processor may be required.'\n" | ||
751 | 197 | } | ||
752 | 198 | ] | ||
753 | 199 | |||
754 | 200 | class TestDescriptions: | ||
755 | 201 | @pytest.mark.parametrize("usn", descriptions) | ||
756 | 202 | @mock.patch("usn_lib.load_database") | ||
757 | 203 | def test_descriptions(self, _load_database_mock, usn): | ||
758 | 204 | _load_database_mock.return_value = [] | ||
759 | 205 | sync_from_usns = importlib.import_module("sync-from-usns") | ||
760 | 206 | fake_usn = '1-1' | ||
761 | 207 | db = {fake_usn: {'description': usn['description'], 'cves': usn['cves']}} | ||
762 | 208 | result = sync_from_usns.extract_cve_descriptions(db[fake_usn], fake_usn, False) | ||
763 | 209 | assert result == usn['result'] | ||
764 | 210 | |||
765 | 211 | @pytest.mark.parametrize("usn", broken_descriptions) | ||
766 | 212 | @mock.patch("usn_lib.load_database") | ||
767 | 213 | def test_descriptions_exception(self, _load_database_mock, usn, capsys): | ||
768 | 214 | _load_database_mock.return_value = [] | ||
769 | 215 | sync_from_usns = importlib.import_module("sync-from-usns") | ||
770 | 216 | fake_usn = '1-1' | ||
771 | 217 | db = {fake_usn: {'description': usn['description'], 'cves': usn['cves']}} | ||
772 | 218 | result = sync_from_usns.extract_cve_descriptions(db[fake_usn], fake_usn, True) | ||
773 | 219 | captured = capsys.readouterr() | ||
774 | 220 | assert result == usn['result'] | ||
775 | 221 | assert captured.err == usn['error'] |
Some comments (+ inline comments).
Since that piece of code is supposed to now handle CVE lists inside a "chunk", please remove the "Extract trailing ..." comment or replace with "Extract CVE list from chunk".
I am also wondering if it would be a better approach to change this code to use a regex to extract a "(CVE-XX, CVE-XX)" chunk from the text instead of manually going through the USN text chunk and doing splitting, picking parts here and there. I think it would improve readability + maintainability.
Thanks for pasting some test cases, made the review much easier than trying to figure out weird cases by myself. At this point I am wondering if it would be worth it to convert those test cases to a proper test but maybe this should not be part of this MP :)