Merge ~alexmurray/ubuntu-cve-tracker:noboilerplates-3 into ubuntu-cve-tracker:master
- Git
- lp:~alexmurray/ubuntu-cve-tracker
- noboilerplates-3
- Merge into master
Status: | Merged |
---|---|
Merged at revision: | 7120fd2d1eeb59d24bedb5dae0e1ab13d85b1b45 |
Proposed branch: | ~alexmurray/ubuntu-cve-tracker:noboilerplates-3 |
Merge into: | ubuntu-cve-tracker:master |
Diff against target: |
1534 lines (+505/-582) 19 files modified
README (+4/-5) README.mozilla (+3/-2) README.usn (+1/-1) README.webkit (+3/-3) scripts/active_edit (+123/-269) scripts/add-derived-kernel (+33/-11) scripts/add_meta_info.py (+4/-2) scripts/boilerplate-to-json.py (+151/-0) scripts/check-cves (+110/-220) scripts/check-syntax (+10/-17) scripts/check-syntax-fixup (+0/-5) scripts/cve-mode.el (+1/-3) scripts/cve.vim (+0/-1) scripts/cve_lib.py (+54/-24) scripts/dup-status-for-pkg (+1/-1) scripts/kernel-triage-missing-break-fix (+1/-1) scripts/release-cycle-devel-opens (+1/-1) scripts/release-cycle-released (+1/-1) scripts/sync-from-eol.py (+4/-15) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Alex Murray | Needs Fixing | ||
Seth Arnold | Approve | ||
Eduardo Barretto | Approve | ||
Review via email: mp+427753@code.launchpad.net |
Commit message
Description of the change
This series of commits is designed to allow us to replace the boilerplate files and the package_
To test this branch you will need to generate this package-db.json as:
./scripts/
Then you should be able to do CVE triage and run active_edit etc etc as normal.
The main benefit this change gives is a better way to represent this data *plus* it ensures that when say updating a CVE with a boilerplate target (e.g. active_edit -C CVE-NNNN-XXXX -p openjdk) - that the packages captured by that boilerplate (ie openjdk-10 etc) all get added to the CVE file as expected *along with all the various subprojects*. This should mean we don't have to use check-syntax-fixup so much.
Steve Beattie (sbeattie) wrote : | # |
Steve Beattie (sbeattie) wrote : | # |
Again, have not read in detail...
On Wed, Aug 03, 2022 at 07:05:15AM -0000, Alex Murray wrote:
> diff --git a/scripts/
> index d219586..f4c96c9 100755
> --- a/scripts/
> +++ b/scripts/
> @@ -105,6 +105,6 @@ echo "Updating status of released CVEs (ok if empty and prints help)..."
> xargs --no-run-if-empty ./scripts/
>
> printf "\n\nPlease update cve_lib.py, kernel_lib.py, sis-generate-usn, cve-alert.sh, and prepare-
> -echo "Please add package description information if necessary to meta_lists/
> +echo "Please add package description information if necessary to meta_lists/
> echo "If this kernel has not been published yet, please add it to the unpublished_kernels list in check-syntax."
> echo "Also, before publishing a USN against the new kernel, ensure packages_mirror has been run"
Plesae note that one of the things the `add-derived-
for the new kernel being added for tracking in UCT is add it to the
boilerplate (I subsequently adjust the addition's location in the
boilerplate to try to keep them relatively organized). Is there a way
add-derived-kernel could be updated to do this under the new structure?
(Yes, there is post-hoc manual editing to be done after
add-derived-kernel does its thing, but my intent is to reduce that as
much as possible, to avoid errors.)
--
Steve Beattie
<email address hidden>
Alex Murray (alexmurray) wrote : | # |
Thanks for the comments @sbeattie - see https:/
Alex Murray (alexmurray) wrote : | # |
So this merge request has had no real review for almost a whole month... I would really like to get it merged so will likely just merge it next week unless anyone wants to give it a look over... TIA 🙏
Eduardo Barretto (ebarretto) wrote : | # |
Small fix in indentation, but the code looks good to me.
I believe you will send another PR, later when things are working in this new structure, to do the cleanup in cve_lib (load_boilerpla
Alex Murray (alexmurray) wrote : | # |
Thanks @ebarretto - I have fixed the indentation in https:/
Yes, once this is merged in then I plan to actually remove the boilerplate files *and* generate the new package-db.json and commit this to UCT (plus remove the now obsolete package_
Eduardo Barretto (ebarretto) wrote : | # |
lgtm, thanks!
Seth Arnold (seth-arnold) wrote : | # |
Sorry for not getting to this sooner :( I like quite a lot of it. I'd love to be rid of the boilerplates.
I've got a few comments inline.
Thanks
Alex Murray (alexmurray) wrote : | # |
Thanks Seth - fixed the typo and moved the parse_embedded_
Regarding wordwrap() etc - that is simply stolen from check-cves and moved into cve_lib now so that we can reuse it - so it will have the same functionality as it did before - I have not modified it.
Alex Murray (alexmurray) wrote : | # |
I just noticed that I missed a few other scripts that still refer to the boilerplate files so they will need to be fixed up before this gets merged:
scripts/
scripts/
scripts/
scripts/
Basically `git grep boilerplate` should return nothing when this MR is actually ready.
Seth Arnold (seth-arnold) wrote : | # |
scripts/
active/
scripts/
scripts/
active/
to:
active/CVE-*
scripts/
Getting rid of the boilerplates looks like it'll simplify a lot :)
Preview Diff
1 | diff --git a/README b/README |
2 | index 433dde7..8817769 100644 |
3 | --- a/README |
4 | +++ b/README |
5 | @@ -597,7 +597,7 @@ Stable Release Actions |
6 | ---------------------- |
7 | When a stable release is published, the active CVEs need to be adjusted to |
8 | reflect the new stable release. e.g. when trusty was published: |
9 | - perl -pi -e 's/^((#?)devel_(.*))/$2trusty_$3\n$1/g' active/{CVE-,00boilerplate}* |
10 | + perl -pi -e 's/^((#?)devel_(.*))/$2trusty_$3\n$1/g' active/CVE-* |
11 | The script tools will need to be adjusted as well. There is usually some |
12 | lag time between the new devel archive opening and the stable release |
13 | getting published. This means that "devel" will disappear from ubuntu-table |
14 | @@ -611,7 +611,7 @@ briefly: |
15 | |
16 | scripts/cve_lib.py should have an empty 'devel_release'. |
17 | |
18 | -Move all active CVEs and boilerplates from "devel" to release state: |
19 | +Move all active CVEs from "devel" to release state: |
20 | ./scripts/release-cycle-released $RELEASE |
21 | |
22 | |
23 | @@ -619,7 +619,7 @@ Development Release Actions |
24 | --------------------------- |
25 | Fill in releases and devel_release in ubuntu-cve-tools/scripts/cve_lib.py |
26 | |
27 | -Move all active CVEs and boilerplates from latest release to devel state: |
28 | +Move all active CVEs from latest release to devel state: |
29 | ./scripts/release-cycle-devel-opens $LATEST_STABLE_RELEASE |
30 | |
31 | Add release to non-ports and ports section of |
32 | @@ -635,7 +635,6 @@ Here is how: |
33 | |
34 | 2. update the CVEs: |
35 | $ sed -i 's/^<release>_\(.*\): \(needed\|needs\-triage\)/<release>_\1: \2 (reached end-of-life)/g' ./active/CVE-* |
36 | -$ sed -i '/^<release>_\(.*\): /d' ./active/00boilerplate* |
37 | |
38 | 3. retire the CVEs (see 'Retiring items', above) |
39 | |
40 | @@ -678,7 +677,7 @@ New Backport Kernel |
41 | When a new backport kernel is added, update scripts/cve_lib.py's |
42 | kernel_srcs and description_overrides. |
43 | |
44 | -Then update the 00boilerplate.linux with its entry, add that entry to each |
45 | +Then update the meta_lists/package-db.json with its entry, add that entry to each |
46 | CVE with an entry for the LTS the kernel was added to, and update all the |
47 | statuses for the newly added kernel, based off the version it will be |
48 | branched from. For example, to add a new kernel backported from Vivid to |
49 | diff --git a/README.mozilla b/README.mozilla |
50 | index 5e06594..7c0f1f2 100644 |
51 | --- a/README.mozilla |
52 | +++ b/README.mozilla |
53 | @@ -45,7 +45,7 @@ CVE Triage |
54 | ---------- |
55 | CVEs in Firefox are tracked in the xulrunner source packages for builds that |
56 | use the system xulrunner, and firefox source packages for those that use a |
57 | -static build. active/00boilerplate.firefox is used to capture the source |
58 | +static build. meta_lists/package-db.json is used to capture the source |
59 | package relationships when triaging CVEs for firefox (ie, you only need to |
60 | specify 'firefox' as the source package). |
61 | |
62 | @@ -65,6 +65,7 @@ xulrunner-1.9.2: system xul for reverese dependencies that process web content. |
63 | firefox: Ubuntu 8.04 LTS and higher (static build of 3.6.x or higher) |
64 | |
65 | Additionally, the following share a common codebase and are affected by the |
66 | -same CVE often enough that they warrant being part of the firefox boilerplate: |
67 | +same CVE often enough that they warrant being part of the firefox pkgs set in |
68 | +meta_lists/package-db.json: |
69 | seamonkey |
70 | thunkerbird |
71 | diff --git a/README.usn b/README.usn |
72 | index ac8156f..e5c3f58 100644 |
73 | --- a/README.usn |
74 | +++ b/README.usn |
75 | @@ -30,7 +30,7 @@ versions. |
76 | Make sure the source package description was properly set for every affected |
77 | release (i.e. the value for the usn.py --source-description argument in the |
78 | generated usn shell script is set). If it was not automatically populated, make |
79 | -sure the source package is present in $UCT/meta_list/package_info_overrides.json |
80 | +sure the source package is present in $UCT/meta_list/package-db.json |
81 | and add it if it is not. |
82 | |
83 | ** CVEs references |
84 | diff --git a/README.webkit b/README.webkit |
85 | index 187d619..a52bf88 100644 |
86 | --- a/README.webkit |
87 | +++ b/README.webkit |
88 | @@ -75,7 +75,7 @@ chromium-browser |
89 | |
90 | This package contains a fork of the WebKit code base (aka 'Blink'). It is |
91 | maintained separately by Google and is tracked in the 'chromium-browser' |
92 | -boilerplate. |
93 | +entry in meta_lists/package-db.json. |
94 | |
95 | oxide |
96 | ----- |
97 | @@ -83,11 +83,11 @@ oxide |
98 | Oxide is bindings for the chromium content api and therefore contains a fork |
99 | of the webkit code base (aka, 'Blink'). The chromium content api is maintained |
100 | separately by Google and Oxide is maintained by Canonical. Oxide is tracked in |
101 | -the 'chromium-browser' boilerplate. |
102 | +the 'chromium-browser' entry in meta_lists/package-db.json. |
103 | |
104 | CVE Triage |
105 | ---------- |
106 | -active/00boilerplate.webkit is used to capture the source package relationships |
107 | +meta_lists/package-db.json is used to capture the source package relationships |
108 | when triaging CVEs for webkit (ie, you only need to specify 'webkit' as the |
109 | source package). |
110 | |
111 | diff --git a/scripts/active_edit b/scripts/active_edit |
112 | index a85f279..482446c 100755 |
113 | --- a/scripts/active_edit |
114 | +++ b/scripts/active_edit |
115 | @@ -10,9 +10,9 @@ |
116 | |
117 | import optparse |
118 | import os |
119 | +import pathlib |
120 | import re |
121 | import sys |
122 | -import time |
123 | |
124 | import cve_lib |
125 | import source_map |
126 | @@ -21,7 +21,6 @@ releases = ['upstream'] + cve_lib.all_releases |
127 | |
128 | max_file_size = 10 * 1024 * 1024 # 10MB |
129 | cvedir = cve_lib.active_dir |
130 | -boilerplates = cvedir |
131 | |
132 | parser = optparse.OptionParser() |
133 | parser.add_option("-p", "--package", dest="pkgs", help="Package name and optional version where package is fixed (with optional Ubuntu release and version in that release)", metavar="NAME[,VERSION[,RELEASE,RELEASE_VERSION]]", action="append") |
134 | @@ -30,7 +29,8 @@ parser.add_option("-r", "--reference-url", dest="ref_urls", help="URL references |
135 | parser.add_option("-c", "--cve", dest="cve", help="CVE entry", metavar="CVE-YYYY-NNNN") |
136 | parser.add_option("-e", "--embargoed", dest="embargoed", help="This is an embargoed entry", action="store_true") |
137 | parser.add_option("-y", "--yes", dest="autoconfirm", help="Do not ask for confirmation", action="store_true") |
138 | -parser.add_option("-P", "--public", help="Record date the CVE went public", metavar="YYYY-MM-DD") |
139 | +parser.add_option("-P", "--public", dest="public_date", help="Record date the CVE went public", metavar="YYYY-MM-DD") |
140 | +parser.add_option("--priority", help="Record a priority for the CVE", default=None) |
141 | parser.add_option("-C", "--cvss", help="CVSS3.1 rating", metavar="CVSS:3.1/AV:_/AC:_/PR:_/UI:_/S:_/C:_/I:_/A:_") |
142 | (options, args) = parser.parse_args() |
143 | |
144 | @@ -70,6 +70,7 @@ def create_or_update_external_subproject_cves(cve, pkgname): |
145 | continue |
146 | affected_releases.append(release) |
147 | |
148 | + ans = "y" |
149 | if not options.autoconfirm: |
150 | print("\n") |
151 | for release in affected_releases: |
152 | @@ -101,245 +102,130 @@ def release_wants_dne(release): |
153 | _, product, _, _ = cve_lib.get_subproject_details(release) |
154 | return product != None and product == cve_lib.PRODUCT_UBUNTU |
155 | |
156 | -def update_cve(cve, pkgname, fixed_in=None, fixed_in_release=None, fixed_in_release_version=None): |
157 | - '''Update an existing CVE file''' |
158 | - with open(os.path.join(cvedir, cve), "r") as f: |
159 | - lines = f.read(max_file_size).split('\n') |
160 | - |
161 | - skipped = [] |
162 | - added_lines = "" |
163 | - |
164 | - tmp_releases = get_releases(pkgname) |
165 | - |
166 | - # If we are using 00boilerplate.<pkgname> and the package is DNE on all |
167 | - # current releases, then don't add the the stanza for this release. This |
168 | - # allows us to use generic boilerplate names like 00boilerplate.gnutls or |
169 | - # 00boilerplate.openjdk without adding useless extra stanzas. |
170 | - # TODO: this still doesn't handle adding the contents of the boilerplate |
171 | - # to an existing CVE (ie, ./scripts/sctive_edit -p openjdk -c CVE-YYYY-NNNN |
172 | - # where CVE-YYYY-NNNN already exists) |
173 | - if os.path.exists(os.path.join(boilerplates, '00boilerplate.%s' % pkgname)): |
174 | - pkg_exists_somewhere = False |
175 | - for r in tmp_releases: |
176 | - if r == 'upstream' or (r in cve_lib.eol_releases \ |
177 | - and not cve_lib.is_active_esm_release(r)): |
178 | - continue |
179 | - if pkg_in_rel(pkgname,r): |
180 | - pkg_exists_somewhere = True |
181 | - break |
182 | - if not pkg_exists_somewhere: |
183 | - print("skipping '" + pkgname + "' (DNE on all current releases)\n", file=sys.stderr) |
184 | - return |
185 | - |
186 | - for line in lines: |
187 | - for r in tmp_releases: |
188 | - if r == cve_lib.devel_release or r == '': |
189 | - r = 'devel' |
190 | - if not re.match(r'^' + r + ".*:", line): |
191 | - continue |
192 | - tmp = line.split(':') |
193 | - match = "%s_%s" % (r, pkgname) |
194 | - if match == tmp[0]: |
195 | - skipped.append(r) |
196 | - |
197 | - if len(skipped) == 0: |
198 | - added_lines += '\nPatches_' + pkgname + ':\n' |
199 | - |
200 | - higher_not_affected = False |
201 | - for release in tmp_releases: |
202 | - # don't add any external releases |
203 | - if release in cve_lib.external_releases: |
204 | - continue |
205 | - |
206 | - r = release |
207 | - if r == cve_lib.devel_release or r == '': |
208 | - r = 'devel' |
209 | - |
210 | - if r in skipped: |
211 | - print("skipping '" + pkgname + "' for " + r + " (already included)\n", file=sys.stderr) |
212 | - else: |
213 | - # skip eol_releases releases without esm support |
214 | - if r in cve_lib.eol_releases \ |
215 | - and (not cve_lib.is_active_esm_release(r) or r == 'precise'): |
216 | - continue |
217 | - state = "needs-triage" |
218 | - if not pkg_in_rel(pkgname, release): |
219 | - # package doesn't exist in this release - see if it wants a |
220 | - # DNE entry |
221 | - if release_wants_dne(release): |
222 | - state = "DNE" |
223 | - else: |
224 | - continue |
225 | - elif cve_lib.is_active_esm_release(r): |
226 | - state = "ignored (out of standard support)" |
227 | - elif r == 'upstream' and fixed_in is not None: |
228 | - state = "released (%s)" % fixed_in |
229 | - elif fixed_in_release_version and r == fixed_in_release: |
230 | - state = "not-affected (%s)" % fixed_in_release_version |
231 | - higher_not_affected = True |
232 | - elif higher_not_affected: |
233 | - state = "not-affected" |
234 | - added_lines += '%s_%s: %s\n' % (r, pkgname, state) |
235 | - |
236 | - if len(releases) == len(skipped): |
237 | - print("\nNothing to add!\n") |
238 | - return |
239 | - |
240 | - if not options.autoconfirm: |
241 | - print("\n" + added_lines) |
242 | - print("\nAppend the above to " + os.path.join(cvedir, cve) + " (y|N)? ") |
243 | - ans = sys.stdin.readline().lower() |
244 | - print("\n") |
245 | - else: |
246 | - ans = "y" |
247 | |
248 | - if ans.startswith("y"): |
249 | - file = open(os.path.join(cvedir, cve), "a") |
250 | - file.write(added_lines) |
251 | - file.close() |
252 | +def create_or_update_cve(cve, packages, priority=None, bug_urls=None, ref_urls=None, public_date=None, desc=None, cvss=None): |
253 | + |
254 | + pkgs = [] |
255 | + fixed = {} |
256 | + # parse optional fixed_in release and version from package name |
257 | + for p in packages: |
258 | + tmp_p = p.split(',') |
259 | + pkg = tmp_p[0] |
260 | + pkgs.append(pkg) |
261 | + fixed[pkg] = tmp_p[1:] |
262 | + |
263 | + update = False |
264 | + try: |
265 | + dst = cve_lib.find_cve(cve) |
266 | + update = True |
267 | + except ValueError: |
268 | + dst = os.path.join(pathlib.Path().parent.resolve(), "active", cve) |
269 | + |
270 | + # collect notes from pkg_db and add any extra pkgs from pkg_db as well |
271 | + notes = [] |
272 | + for p in pkgs: |
273 | + if p in pkg_db: |
274 | + notes = notes + pkg_db[p]["notes"] |
275 | + pkgs = pkgs + list(pkg_db[p]["pkgs"].keys()) |
276 | + |
277 | + # normalise the list of packages |
278 | + pkgs = sorted(list(set(pkgs))) |
279 | + |
280 | + # remove any packages which don't actually exist in any release |
281 | + for p in pkgs: |
282 | + keep = False |
283 | + for r in source.keys(): |
284 | + keep |= p in source[r] |
285 | + if not keep: |
286 | + pkgs.remove(p) |
287 | + |
288 | + if update: |
289 | + mode = "a" |
290 | else: |
291 | - print("Aborted\n") |
292 | - |
293 | -def create_cve(cve, pkgname, fixed_in=None, fixed_in_release=None, fixed_in_release_version=None): |
294 | - '''Create a new CVE file''' |
295 | - src = os.path.join(boilerplates, '00boilerplate') |
296 | - if os.path.exists(src + "." + pkgname): |
297 | - src = src + "." + pkgname |
298 | - boiler = open(src, "r") |
299 | - lines = boiler.read(max_file_size).splitlines() |
300 | - boiler.close() |
301 | - |
302 | - cand_pat = re.compile(r'^Candidate:') |
303 | - ref_pat = re.compile(r'^References:') |
304 | - bugs_pat = re.compile(r'^Bugs:') |
305 | - cvss_pat = re.compile(r'^CVSS:') |
306 | - pkg_pat = re.compile(r'^#?[a-z/\-]+_(PKG|%s):' % pkgname) |
307 | - patch_pat = re.compile(r'^#?Patches_(PKG|%s):' % pkgname) |
308 | - |
309 | - tmp_releases = get_releases(pkgname) |
310 | - added = set() |
311 | - |
312 | - contents = "" |
313 | - higher_not_affected = False |
314 | - for line in lines: |
315 | - if (cand_pat.search(line)): |
316 | - contents += line + os.path.basename(cve) + '\n' |
317 | - elif line.startswith('PublicDate:'): |
318 | - if options.embargoed: |
319 | - if options.public: |
320 | - # use public date as CRD |
321 | - contents += "CRD: %s\n" % options.public |
322 | - else: |
323 | - contents += "CRD: <TBD>\n" |
324 | - if options.public: |
325 | - contents += "PublicDate: %s\n" % options.public |
326 | - else: |
327 | - # default to today-- this will be refreshed by check-cves |
328 | - contents += "PublicDate: %s\n" % time.strftime("%Y-%m-%d", time.gmtime()) |
329 | - elif (cvss_pat.search(line)): |
330 | - if options.cvss: |
331 | - contents += 'CVSS: ' + options.cvss + '\n' |
332 | - else: |
333 | - contents += 'CVSS:\n' |
334 | - elif (patch_pat.search(line)): |
335 | - contents += '\nPatches_' + pkgname + ':\n' |
336 | - elif (pkg_pat.search(line)): |
337 | - for release in tmp_releases: |
338 | - if release == cve_lib.devel_release or release == '': |
339 | - release = 'devel' |
340 | - |
341 | - # skip eol_releases releases without esm support |
342 | - if release in cve_lib.eol_releases \ |
343 | - and not cve_lib.is_active_esm_release(release): |
344 | + mode = "w" |
345 | + with open(dst, mode, encoding="utf-8") as fp: |
346 | + if not update: |
347 | + print('Candidate: %s' % (cve), file=fp) |
348 | + print('PublicDate: %s' % (public_date if public_date else "unknown"), file=fp) |
349 | + print('References:\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=%s' % (cve), file=fp) |
350 | + for url in (ref_urls if ref_urls else []): |
351 | + print(" %s" % url, file=fp) |
352 | + print('Description:', file=fp) |
353 | + for desc_line in (cve_lib.wrap_text(desc).split('\n') if desc else []): |
354 | + print(" %s" % (desc_line), file=fp) |
355 | + print('Ubuntu-Description:', file=fp) |
356 | + print('Notes:', file=fp) |
357 | + for note in notes: |
358 | + for note_line in cve_lib.wrap_text(note[1], 75 - len(note[0]) - 2).split('\n'): |
359 | + print(" %s> %s" % (note[0], note_line), file=fp) |
360 | + print('Mitigation:', file=fp) |
361 | + print('Bugs:', file=fp) |
362 | + for url in (bug_urls if bug_urls else []): |
363 | + print(" %s" % url, file=fp) |
364 | + print('Priority: %s' % (priority if priority else "untriaged"), file=fp) |
365 | + print('Discovered-by:', file=fp) |
366 | + print('Assigned-to:', file=fp) |
367 | + print('CVSS:', file=fp) |
368 | + for entry in (cvss if cvss else []): |
369 | + src, cvss = entry |
370 | + print(' %s: %s' % (src, cvss), file=fp) |
371 | + |
372 | + # add package info from pkg_db |
373 | + for p in pkgs: |
374 | + print('', file=fp) |
375 | + print('Patches_%s:' % p, file=fp) |
376 | + # find which releases p exists in |
377 | + higher_not_affected = False |
378 | + fixed_in = None |
379 | + fixed_in_release = None |
380 | + fixed_in_release_version = None |
381 | + if p in fixed and len(fixed[p]) > 0: |
382 | + fixed_in = fixed[p][0] |
383 | + if len(fixed[p]) > 1: |
384 | + fixed_in_release = fixed[p][1] |
385 | + if len(fixed[p]) > 2: |
386 | + fixed_in_release_version = fixed[p][2] |
387 | + for rel in ['upstream'] + list(source.keys()): |
388 | + # determine default state but override this if pkg_db has a |
389 | + # better one |
390 | + state = "needs-triage" |
391 | + if not pkg_in_rel(p, rel): |
392 | + # package doesn't exist in this release - see if it wants a |
393 | + # DNE entry |
394 | + if release_wants_dne(rel): |
395 | + state = "DNE" |
396 | + else: |
397 | + continue |
398 | + if rel == cve_lib.devel_release: |
399 | + # devel is present in source.keys() so use that instead |
400 | + # of the codename |
401 | continue |
402 | + if cve_lib.is_active_esm_release(rel): |
403 | + state = "ignored (out of standard support)" |
404 | + elif rel == 'upstream' and fixed_in is not None: |
405 | + state = "released (%s)" % fixed_in |
406 | + elif fixed_in_release_version and rel == fixed_in_release: |
407 | + state = "not-affected (%s)" % fixed_in_release_version |
408 | + higher_not_affected = True |
409 | + elif higher_not_affected: |
410 | + state = "not-affected" |
411 | + |
412 | + # use pkg_db state if one exists |
413 | + if p in pkg_db and p in pkg_db[p]["pkgs"] and rel in pkg_db[p]["pkgs"][p]: |
414 | + state_tuple = pkg_db[p]["pkgs"][p][rel] |
415 | + state = state_tuple[0] |
416 | + if len(state_tuple[1]) > 0: |
417 | + state = state + " (%s)" % state_tuple[1] |
418 | + if rel not in cve_lib.external_releases: |
419 | + print('%s_%s: %s' % (rel, p, state), file=fp) |
420 | + else: |
421 | + # add this to subprojects for rel |
422 | + with open(os.path.join(cve_lib.get_external_subproject_cve_dir(rel), cve), "a") as f: |
423 | + print('%s_%s: %s' % (rel, p, state), file=f) |
424 | |
425 | - rel_pat = re.compile('#?' + release + '_') |
426 | - |
427 | - if (rel_pat.search(line)): |
428 | - state = "needs-triage" |
429 | - if not pkg_in_rel(pkgname, release): |
430 | - # package doesn't exist in this release - see if it wants a |
431 | - # DNE entry |
432 | - if release_wants_dne(release): |
433 | - state = "DNE" |
434 | - else: |
435 | - continue |
436 | - elif cve_lib.is_active_esm_release(release): |
437 | - state = "ignored (out of standard support)" |
438 | - elif release == 'upstream' and fixed_in is not None: |
439 | - state = "released (%s)" % fixed_in |
440 | - elif fixed_in_release_version and release == fixed_in_release: |
441 | - state = "not-affected (%s)" % fixed_in_release_version |
442 | - higher_not_affected = True |
443 | - elif higher_not_affected: |
444 | - state = "not-affected" |
445 | - if release not in added: |
446 | - contents += "%s_%s: %s\n" % (release, pkgname, state) |
447 | - added.add(release) |
448 | - elif ref_pat.search(line): |
449 | - if not re.search(r'N', cve): |
450 | - contents += line + "\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=" + cve + "\n" |
451 | - else: |
452 | - contents += line + "\n" |
453 | - if options.ref_urls: |
454 | - for i in options.ref_urls: |
455 | - contents += " %s\n" % i |
456 | - elif options.bug_urls and bugs_pat.search(line): |
457 | - contents += line |
458 | - for i in options.bug_urls: |
459 | - contents += "\n %s\n" % i |
460 | - else: |
461 | - contents += line + '\n' |
462 | - |
463 | - # check for missing entries which aren't in the boilerplate |
464 | - for release in tmp_releases: |
465 | - if release == cve_lib.devel_release or release == '': |
466 | - release = 'devel' |
467 | - if release in added: |
468 | - continue |
469 | - if release in cve_lib.external_releases: |
470 | - continue |
471 | - # skip eol_releases releases without esm support |
472 | - if release in cve_lib.eol_releases \ |
473 | - and not cve_lib.is_active_esm_release(release): |
474 | - continue |
475 | - if pkg_in_rel(pkgname, release): |
476 | - state = "needs-triage" |
477 | - contents += "%s_%s: %s\n" % (release, pkgname, state) |
478 | - |
479 | - # for each line in contents, see if we need to supercede DNE status |
480 | - # with needs-triage as boilerplate entries may not be up-to-date with |
481 | - # xxx-supported.txt lists |
482 | - pkgstatus_pat = re.compile(r'^([/a-z/\-]+)_(.*): (.*)') |
483 | - old_contents = contents |
484 | - contents = "" |
485 | - for line in old_contents.splitlines(): |
486 | - match = pkgstatus_pat.search(line) |
487 | - if match is not None: |
488 | - release = match.group(1) |
489 | - pkgname = match.group(2) |
490 | - status = match.group(3) |
491 | - if status == 'DNE' and pkg_in_rel(pkgname, release): |
492 | - status = 'needs-triage' |
493 | - line = '%s_%s: %s' % (release, pkgname, status) |
494 | - contents += line + '\n' |
495 | - |
496 | - if not options.autoconfirm: |
497 | - print(contents) |
498 | - print("\nWrite the above to " + os.path.join(cvedir, cve) + " (y|N)? ") |
499 | - ans = sys.stdin.readline().lower() |
500 | - print("\n") |
501 | - else: |
502 | - ans = "y" |
503 | - |
504 | - if ans.startswith("y"): |
505 | - newfile = open(os.path.join(cvedir, cve), 'w') |
506 | - newfile.write(contents) |
507 | - newfile.close() |
508 | - else: |
509 | - print("Aborted\n") |
510 | |
511 | |
512 | +pkg_db = cve_lib.load_package_db() |
513 | |
514 | if not options.pkgs: |
515 | parser.print_help() |
516 | @@ -369,37 +255,5 @@ if not pat.search(cve): |
517 | print("Bad CVE entry. Should be CVE-XXXX-XXXX\n", file=sys.stderr) |
518 | sys.exit(1) |
519 | |
520 | -# more here |
521 | -pat = re.compile(r'\s') |
522 | -for p in pkgs: |
523 | - tmp_p = p.split(',') |
524 | - pkgname = tmp_p[0] |
525 | - fixed_in = None |
526 | - if len(tmp_p) > 1: |
527 | - fixed_in = tmp_p[1] |
528 | - fixed_in_release = None |
529 | - fixed_in_release_version = None |
530 | - if len(tmp_p) > 3: |
531 | - fixed_in_release = tmp_p[2] |
532 | - fixed_in_release_version = tmp_p[3] |
533 | - |
534 | - if pat.search(pkgname): |
535 | - print("Bad package name\n", file=sys.stderr) |
536 | - sys.exit(1) |
537 | - |
538 | - if not os.path.isfile(os.path.join(boilerplates, "00boilerplate")): |
539 | - print("Could not find 00boilerplate in " + cvedir + "\n", file=sys.stderr) |
540 | - sys.exit(1) |
541 | - |
542 | - if (os.path.isfile(os.path.join(cvedir, cve))): |
543 | - if not options.autoconfirm: |
544 | - print("Found existing " + cve + "...\n\n") |
545 | - update_cve(cve, pkgname, fixed_in, fixed_in_release, fixed_in_release_version) |
546 | - else: |
547 | - if not options.autoconfirm: |
548 | - print("Creating new " + cve + "...\n\n") |
549 | - create_cve(cve, pkgname, fixed_in, fixed_in_release, fixed_in_release_version) |
550 | - |
551 | - create_or_update_external_subproject_cves(cve, pkgname) |
552 | - |
553 | +create_or_update_cve(cve, pkgs, priority=options.priority, bug_urls=options.bug_urls, ref_urls=options.ref_urls, public_date=options.public_date, cvss=options.cvss) |
554 | sys.exit(0) |
555 | diff --git a/scripts/add-derived-kernel b/scripts/add-derived-kernel |
556 | index d219586..3c146ab 100755 |
557 | --- a/scripts/add-derived-kernel |
558 | +++ b/scripts/add-derived-kernel |
559 | @@ -67,19 +67,37 @@ if [ -z "${DERIVED_KERNEL}" ] ; then |
560 | DERIVED_KERNEL="linux" |
561 | fi |
562 | |
563 | -chunk=$(mktemp -t add-derived-kernel-XXXXXX) |
564 | +releases=$(PYTHONPATH=./scripts python3 -c "import cve_lib; print(' '.join([a for a in cve_lib.all_releases if a not in cve_lib.external_releases and a is not cve_lib.devel_release and (a not in cve_lib.eol_releases or cve_lib.is_active_esm_release(a))]))") |
565 | + |
566 | +releases="upstream ${releases} devel" |
567 | |
568 | +chunk=$(mktemp -t add-derived-kernel-XXXXXX) |
569 | echo "" > "$chunk" |
570 | -grep "^.*_linux:" active/00boilerplate.linux | sed -e "s#_linux:.*#_linux-${KERNEL}: DNE#" >> "$chunk" |
571 | -sed -i -e "s#^${RELEASE}_linux-${KERNEL}:.*#${RELEASE}_linux-${KERNEL}: needs-triage#" "$chunk" |
572 | -sed -i -e "s#^Patches_linux-${KERNEL}:.*#Patches_linux-${KERNEL}:#" "$chunk" |
573 | -sed -i -e "s#^upstream_linux-${KERNEL}:.*#upstream_linux-${KERNEL}: needs-triage#" "$chunk" |
574 | - |
575 | -if ! grep -q "^${RELEASE}_linux-${KERNEL}:" active/00boilerplate.linux ; then |
576 | - echo "Updating 00boilerplate.linux..." |
577 | - cat "$chunk" >> active/00boilerplate.linux |
578 | +echo "Patches_linux-${KERNEL}:" >> "$chunk" |
579 | + |
580 | +json="{" |
581 | + |
582 | +# add other releases |
583 | +for rel in $releases; do |
584 | + status="DNE" |
585 | + if [ "${rel}" = "${RELEASE}" ] || [ "${rel}" = "upstream" ] ; then |
586 | + status="needs-triage" |
587 | + fi |
588 | + json="$json\"${rel}_${DERIVED_KERNEL}\": [\"$status\", \"\"]," |
589 | + echo "${rel}_linux-${KERNEL}: ${status}" >> "$chunk" |
590 | +done |
591 | +# remove trailing comma |
592 | +json=${json%?} |
593 | +# close json objects |
594 | +json="$json}" |
595 | + |
596 | +# append the derived kernel to meta_lists/package-db.json if it doesn't exist already |
597 | +if jq -e ".linux.pkgs.\"linux-$KERNEL\"" meta_lists/package-db.json; then |
598 | + echo "kernel linux-${KERNEL} already exists in meta_lists/package-db.json, skipping update." |
599 | else |
600 | - echo "00boilerplate.linux already contains linux-${KERNEL}, skipping update." |
601 | + echo "adding kernel linux-${KERNEL} to meta_lists/package-db.json" |
602 | + jq -e ".linux.pkgs.\"linux-$KERNEL\" += $json" meta_lists/package-db.json > meta_lists/package-db.json.new |
603 | + mv meta_lists/package-db.json.new meta_lists/package-db.json |
604 | fi |
605 | |
606 | echo "Adding new backport to existing CVEs..." |
607 | @@ -105,6 +123,10 @@ echo "Updating status of released CVEs (ok if empty and prints help)..." |
608 | xargs --no-run-if-empty ./scripts/mass-cve-edit -p "linux-${KERNEL}" -r "${RELEASE}" -s not-affected |
609 | |
610 | printf "\n\nPlease update cve_lib.py, kernel_lib.py, sis-generate-usn, cve-alert.sh, and prepare-kernel-usn.py in scripts/\n" |
611 | -echo "Please add package description information if necessary to meta_lists/package_info_overrides.json" |
612 | +echo "Please add package description information if necessary to meta_lists/package-db.json" |
613 | echo "If this kernel has not been published yet, please add it to the unpublished_kernels list in check-syntax." |
614 | echo "Also, before publishing a USN against the new kernel, ensure packages_mirror has been run" |
615 | + |
616 | +# Local Variables: |
617 | +# sh-indentation: 4 |
618 | +# End: |
619 | diff --git a/scripts/add_meta_info.py b/scripts/add_meta_info.py |
620 | index 994c8ec..d5b78c3 100755 |
621 | --- a/scripts/add_meta_info.py |
622 | +++ b/scripts/add_meta_info.py |
623 | @@ -10,11 +10,13 @@ if len(sys.argv) != 4: |
624 | package = sys.argv[1] |
625 | title = sys.argv[2] |
626 | description = sys.argv[3] |
627 | -json_file = os.environ['UCT'] + "/meta_lists/package_info_overrides.json" |
628 | +json_file = os.environ['UCT'] + "/meta_lists/package-db.json" |
629 | |
630 | with open(json_file, 'r') as handle: |
631 | parsed = json.load(handle) |
632 | - parsed[package] = { "description": description, 'title': title } |
633 | + parsed.setdefault(package, {}) |
634 | + parsed[package]['title'] = title |
635 | + parsed[package]['description'] = description |
636 | new_content = json.dumps(parsed, indent=4, sort_keys=True) |
637 | |
638 | with open(json_file, 'w') as handle: |
639 | diff --git a/scripts/boilerplate-to-json.py b/scripts/boilerplate-to-json.py |
640 | new file mode 100755 |
641 | index 0000000..db2c735 |
642 | --- /dev/null |
643 | +++ b/scripts/boilerplate-to-json.py |
644 | @@ -0,0 +1,151 @@ |
645 | +#!/usr/bin/python3 |
646 | +import glob |
647 | +import json |
648 | +import os |
649 | +import re |
650 | +import sys |
651 | + |
652 | +import cve_lib |
653 | + |
654 | +def parse_boilerplate(filepath): |
655 | + cve_data = cve_lib.load_cve(filepath) |
656 | + # capture tags, Notes, and package relationships |
657 | + data = dict() |
658 | + data.setdefault("aliases", list()) |
659 | + # tags are a set but json can't serialise a set so convert to a list first |
660 | + data.setdefault("tags", list(cve_data.get("tags", list()))) |
661 | + data.setdefault("notes", cve_data.get("Notes", list())) |
662 | + data.setdefault("pkgs", cve_data.get("pkgs", dict())) |
663 | + return data |
664 | + |
665 | + |
666 | +def load_boilerplates(): |
667 | + data = dict() |
668 | + aliases = dict() |
669 | + for filepath in glob.glob("active/00boilerplate.*"): |
670 | + name = ".".join(filepath.split(".")[1:]) |
671 | + # check if is a symlink and if so don't bother loading the file |
672 | + # directly but add an entry as this is an alias |
673 | + if os.path.islink(filepath): |
674 | + orig = os.readlink(filepath) |
675 | + orig_name = ".".join(orig.split(".")[1:]) |
676 | + aliases.setdefault(orig_name, set()) |
677 | + aliases[orig_name].add(name) |
678 | + continue |
679 | + bpdata = parse_boilerplate(filepath) |
680 | + # having a package reference itself as we have in the boilerplates |
681 | + # is redundant - although this is not always the case as we may |
682 | + # have a boilerplate filename like openjdk yet there is no openjdk |
683 | + # package (just openjdk-8 etc) - so ignore any failures here |
684 | + try: |
685 | + del bpdata["pkgs"][name] |
686 | + except KeyError: |
687 | + pass |
688 | + data.setdefault(name, bpdata) |
689 | + for alias in aliases: |
690 | + data[alias]["aliases"] = list(aliases[alias]) |
691 | + return data |
692 | + |
693 | + |
694 | +def load_package_info_overrides(): |
695 | + with open("meta_lists/package_info_overrides.json", "r") as fp: |
696 | + data = json.load(fp) |
697 | + return data |
698 | + |
699 | + |
700 | +overrides_data = load_package_info_overrides() |
701 | +# turn this into empty data |
702 | +for pkg in overrides_data: |
703 | + for key in ["aliases", "tags", "notes"]: |
704 | + overrides_data[pkg][key] = [] |
705 | + overrides_data[pkg]["pkgs"] = {} |
706 | +bp_data = load_boilerplates() |
707 | +# merge the two data sources |
708 | +data = {**overrides_data, **bp_data} |
709 | + |
710 | +print(json.dumps(data, indent=2)) |
711 | +sys.exit(0) |
712 | + |
713 | +# TODO - decide if we want to keep this - for now leave it out |
714 | + |
715 | +def parse_embedded_code_copies(filepath): |
716 | + begin_re = re.compile(r"^---BEGIN$") |
717 | + pkg_re = re.compile(r"^([a-zA-Z0-9.+_-]+).*$") |
718 | + # [release] - srcpkg version|<status> (sort; bug #) |
719 | + embedding_re = re.compile(r"^\t(\[([a-z]+)\] )?- ([a-zA-Z0-9.+-]+) ([a-zA-Z0-9:~.+-]+|<(unfixed|removed|itp|not-affected|unknown|unfixable)>)( \((static|embed|modified-embed|fork|old-version)(; (.*))?\))?.*$") |
720 | + note_re = re.compile(r"^\tNOTE: (.*)$") |
721 | + data = dict() |
722 | + pkgs = dict() |
723 | + notes = dict() |
724 | + with open(filepath, "r") as fp: |
725 | + linenum = 0 |
726 | + begin = False |
727 | + pkg = None |
728 | + embedding_pkg = None |
729 | + for line in fp.readlines(): |
730 | + linenum += 1 |
731 | + # strip trailing space |
732 | + line = line.rstrip() |
733 | + if not begin: |
734 | + if begin_re.match(line): |
735 | + begin = True |
736 | + continue |
737 | + if len(line) == 0: |
738 | + continue |
739 | + # is this a new package |
740 | + m = pkg_re.match(line) |
741 | + if m is not None: |
742 | + pkg = m[1] |
743 | + pkgs[pkg] = dict() |
744 | + continue |
745 | + # is this an entry for a package |
746 | + m = embedding_re.match(line) |
747 | + if m is not None: |
748 | + assert pkg is not None |
749 | + if m[2] is not None: |
750 | + # release |
751 | + pass |
752 | + embedding_pkg = m[3] |
753 | + status = m[4] |
754 | + sort = m[7] |
755 | + pkgs[pkg][embedding_pkg] = (status, sort) |
756 | + continue |
757 | + m = note_re.match(line) |
758 | + if m is not None: |
759 | + assert pkg is not None |
760 | + assert embedding_pkg is not None |
761 | + notes.setdefault(pkg, dict()) |
762 | + notes[pkg][embedding_pkg] = m[1] |
763 | + continue |
764 | + print("%s: %d: Failed to parse: '%s'" % (filepath, linenum, line), file=sys.stderr) |
765 | + data["pkgs"] = pkgs |
766 | + data["notes"] = notes |
767 | + return data |
768 | + |
769 | +# also parse debian's embedded-code-copies and amalgate that into data |
770 | +config = cve_lib.read_config() |
771 | +debian_embedded_copies = os.path.join(config['secure_testing_path'], "data", "embedded-code-copies") |
772 | +code_copies = parse_embedded_code_copies(debian_embedded_copies) |
773 | +for pkg in code_copies["pkgs"]: |
774 | + data.setdefault(pkg, {"aliases": [], |
775 | + "tags": [], |
776 | + "notes": [], |
777 | + "pkgs": {}}) |
778 | + for embedding_pkg in code_copies["pkgs"][pkg]: |
779 | + data[pkg]["pkgs"].setdefault(embedding_pkg, ("needs-triage", "")) |
780 | + # use the sort to create a more informative description of the |
781 | + # relationship between pkg and embedding_pkg |
782 | + status, sort = code_copies["pkgs"][pkg][embedding_pkg] |
783 | + template = "%s embeds a copy of %s" |
784 | + if sort == "static": |
785 | + template = "%s statically links against %s so needs to be rebuilt" |
786 | + elif sort == "modified-embed": |
787 | + template = "%s embeds a modified copy of %s so should be checked if is affected" |
788 | + elif sort == "fork": |
789 | + template = "%s contains a fork of %s so should be checked if is affected" |
790 | + elif sort == "old-version": |
791 | + template = "%s contains an older version of %s so should be checked if is affected" |
792 | + note = template % (embedding_pkg, pkg) |
793 | + if pkg in code_copies["notes"] and embedding_pkg in code_copies["notes"][pkg]: |
794 | + note = note + " - " + code_copies["notes"][pkg][embedding_pkg] |
795 | + data[pkg]["notes"].append(("dst", note)) |
796 | diff --git a/scripts/check-cves b/scripts/check-cves |
797 | index fa6093d..963f7d3 100755 |
798 | --- a/scripts/check-cves |
799 | +++ b/scripts/check-cves |
800 | @@ -34,7 +34,6 @@ import xml.sax |
801 | import xml.sax.handler |
802 | import xml.sax.xmlreader |
803 | from html import escape |
804 | -from functools import reduce |
805 | import progressbar |
806 | |
807 | import cve_lib |
808 | @@ -75,9 +74,10 @@ for release in list(source.keys()): |
809 | # likely to sometimes contain these |
810 | common_words = ['an', 'and', 'context', 'file', 'modules', 'the', 'when'] |
811 | allsrcs.difference_update(set(common_words)) |
812 | -# add boilerplate names too so we can get mysql or postgresql |
813 | -# etc even if they don't exist as source package names |
814 | -allsrcs.update(set(cve_lib.load_boilerplates().keys())) |
815 | +# add names and aliases etc from package-db |
816 | +allsrcs.update(set(cve_lib.package_db.keys())) |
817 | +for pkg in cve_lib.package_db: |
818 | + allsrcs.update(set(cve_lib.package_db[pkg]["aliases"])) |
819 | |
820 | built_using_map = None |
821 | |
822 | @@ -124,24 +124,8 @@ def subtract_list(list1, list2): |
823 | list1.remove(item) |
824 | |
825 | |
826 | -def wordwrap(text, width): |
827 | - """ |
828 | - A word-wrap function that preserves existing line breaks |
829 | - and most spaces in the text. Expects that existing line |
830 | - breaks are posix newlines (\n). |
831 | - """ |
832 | - return reduce(lambda line, word, width=width: '%s%s%s' % |
833 | - (line, |
834 | - ' \n'[(len(line) - line.rfind('\n') - 1 + |
835 | - len(word.split('\n', 1)[0] |
836 | - ) >= width)], |
837 | - word), |
838 | - text.split(' ') |
839 | - ) |
840 | - |
841 | - |
842 | def _wrap_desc(desc): |
843 | - return wordwrap(desc, 75).replace(' \n', '\n') |
844 | + return cve_lib.wrap_text(desc) |
845 | |
846 | def _spawn_editor(path): |
847 | editor = os.getenv('EDITOR', 'vi') |
848 | @@ -164,104 +148,6 @@ def prompt_user(msg): |
849 | print(msg, flush=True, end='') |
850 | |
851 | |
852 | -def add_CVE_to_tracker(cve, info, packages, priority=None, bug_urls=[], ref_urls=[]): |
853 | - src = '%s/active/00boilerplate' % (destdir) |
854 | - |
855 | - # Use the first boilerplate for template |
856 | - first_boiler = "" |
857 | - for b in packages: |
858 | - if os.path.exists(src + "." + b): |
859 | - src = src + "." + b |
860 | - first_boiler = src |
861 | - break |
862 | - dst = '%s/active/%s' % (destdir, cve) |
863 | - with open(src) as f: |
864 | - template = f.readlines() |
865 | - cve_file = open(dst, 'w') |
866 | - orig_priority = "" |
867 | - for line in template: |
868 | - line = line.rstrip() |
869 | - if line.startswith('Candidate:'): |
870 | - print('Candidate: %s' % (cve), file=cve_file) |
871 | - elif info['public'] and line.startswith('PublicDate:'): |
872 | - print('PublicDate: %s' % (info['public']), file=cve_file) |
873 | - elif info['cvss'] and line.startswith('CVSS:'): |
874 | - print('CVSS:', file=cve_file) |
875 | - for entry in info['cvss']: |
876 | - print(' %s: %s [%s %s]' % (entry['source'], entry['vector'], entry['baseScore'], entry['baseSeverity']), file=cve_file) |
877 | - elif line.startswith('References:'): |
878 | - print('References:\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=%s' % (cve), file=cve_file) |
879 | - for i in ref_urls: |
880 | - print(" %s" % i, file=cve_file) |
881 | - elif line.startswith('Bugs:'): |
882 | - print(line, file=cve_file) |
883 | - for i in bug_urls: |
884 | - print(" %s" % i, file=cve_file) |
885 | - elif line.startswith('Priority:'): |
886 | - orig_priority = line.split()[1] |
887 | - if priority: |
888 | - print('Priority: %s' % priority, file=cve_file) |
889 | - else: |
890 | - print(line, file=cve_file) |
891 | - elif not line.startswith('#'): |
892 | - print(line, file=cve_file) |
893 | - |
894 | - if line.startswith('Description:'): |
895 | - for desc_line in _wrap_desc(info['desc']).split('\n'): |
896 | - print(" %s" % (desc_line), file=cve_file) |
897 | - |
898 | - # Now add package information (with Priority_<pkg>) from other boilers |
899 | - if len(packages) > 1: |
900 | - for p in packages: |
901 | - skip_emptyline = False |
902 | - boiler = '%s/active/00boilerplate.%s' % (destdir, p) |
903 | - if os.path.exists(boiler) and boiler != first_boiler: |
904 | - with open(boiler) as f: |
905 | - template = f.readlines() |
906 | - in_note_section = False |
907 | - for line in template: |
908 | - line = line.rstrip() |
909 | - # handle notes in templates |
910 | - if in_note_section: |
911 | - if line.startswith(' '): |
912 | - continue |
913 | - else: |
914 | - in_note_section = False |
915 | - if line.startswith('Notes:'): |
916 | - in_note_section = True |
917 | - continue |
918 | - if line.startswith('Candidate:') or \ |
919 | - line.startswith('PublicDate:') or \ |
920 | - line.startswith('References:') or \ |
921 | - line.startswith('Description:') or \ |
922 | - line.startswith('Ubuntu-Description:') or \ |
923 | - line.startswith('Bugs:') or \ |
924 | - line.startswith('Discovered-by:') or \ |
925 | - line.startswith('Assigned-to:') or \ |
926 | - line.startswith('#'): |
927 | - continue |
928 | - if line.startswith('Priority:'): |
929 | - new_priority = line.split()[1] |
930 | - if priority: |
931 | - continue |
932 | - elif new_priority == orig_priority: |
933 | - continue |
934 | - elif orig_priority == "": |
935 | - print(line, file=cve_file) |
936 | - else: |
937 | - print('Priority_%s: %s' % (p, new_priority), file=cve_file) |
938 | - skip_emptyline = True |
939 | - elif skip_emptyline and line == "": |
940 | - skip_emptyline = False |
941 | - continue |
942 | - else: |
943 | - print(line, file=cve_file) |
944 | - print("") |
945 | - |
946 | - cve_file.close() |
947 | - |
948 | - return dst |
949 | - |
950 | |
951 | class PercentageFile(object): |
952 | def __init__(self, filename): |
953 | @@ -1371,110 +1257,114 @@ class CVEHandler(xml.sax.handler.ContentHandler): |
954 | print('%s %s %s\n%s' % (action, cve, data, desc)) |
955 | |
956 | def add_cve(self, cve, packages, priority=None): |
957 | - # remove from not-for-us.txt if adding and ensure we remove any |
958 | - # mistriaged_hint from the description |
959 | - if cve in CVEIgnoreNotForUsList: |
960 | - cmd = ['sed', '-i', '/^%s #.*$/d' % cve, './ignored/not-for-us.txt'] |
961 | - subprocess.call(cmd) |
962 | - self.cve_data[cve]['desc'] = self.cve_data[cve]['desc'].replace(mistriaged_hint, '') |
963 | - |
964 | - # Build up list of reference urls |
965 | - ref_urls = [] |
966 | + # remove from not-for-us.txt if adding and ensure we remove any |
967 | + # mistriaged_hint from the description |
968 | + if cve in CVEIgnoreNotForUsList: |
969 | + cmd = ['sed', '-i', '/^%s #.*$/d' % cve, './ignored/not-for-us.txt'] |
970 | + subprocess.call(cmd) |
971 | + self.cve_data[cve]['desc'] = self.cve_data[cve]['desc'].replace(mistriaged_hint, '') |
972 | + |
973 | + # Build up list of reference urls |
974 | + ref_urls = [] |
975 | + if self.debian and \ |
976 | + cve in self.debian and \ |
977 | + 'note' in self.debian[cve]: |
978 | + for line in self.debian[cve]['note']: |
979 | + tmp = line.lstrip("NOTE: ") |
980 | + if tmp.startswith("http"): |
981 | + ref_urls.append(tmp) |
982 | + if 'refs' in self.cve_data[cve]: |
983 | + for ref in self.cve_data[cve]['refs']: |
984 | + url = "" |
985 | + if ref[1].strip().startswith("http"): |
986 | + url = ref[1].strip() |
987 | + elif ref[2] is not None and ref[2].strip().startswith("http"): |
988 | + url = ref[2].strip() |
989 | + else: # no urls |
990 | + continue |
991 | + |
992 | + if '//' not in url: # invalid url |
993 | + continue |
994 | + |
995 | + # ignore certain reference URLs which we don't use |
996 | + ignored_urls = ['www.securityfocus.com', 'www.osvdb.org'] |
997 | + if url.split('//')[1].split('/')[0] in ignored_urls: |
998 | + continue |
999 | + |
1000 | + if url not in ref_urls: |
1001 | + ref_urls.append(url) |
1002 | + |
1003 | + # Build up list of bug urls |
1004 | + bug_urls = [] |
1005 | + for pkg in packages: |
1006 | if self.debian and \ |
1007 | cve in self.debian and \ |
1008 | - 'note' in self.debian[cve]: |
1009 | - for line in self.debian[cve]['note']: |
1010 | - tmp = line.lstrip("NOTE: ") |
1011 | - if tmp.startswith("http"): |
1012 | - ref_urls.append(tmp) |
1013 | - if 'refs' in self.cve_data[cve]: |
1014 | - for ref in self.cve_data[cve]['refs']: |
1015 | - url = "" |
1016 | - if ref[1].strip().startswith("http"): |
1017 | - url = ref[1].strip() |
1018 | - elif ref[2] is not None and ref[2].strip().startswith("http"): |
1019 | - url = ref[2].strip() |
1020 | - else: # no urls |
1021 | - continue |
1022 | - |
1023 | - if '//' not in url: # invalid url |
1024 | - continue |
1025 | - |
1026 | - # ignore certain reference URLs which we don't use |
1027 | - ignored_urls = ['www.securityfocus.com', 'www.osvdb.org'] |
1028 | - if url.split('//')[1].split('/')[0] in ignored_urls: |
1029 | - continue |
1030 | - |
1031 | - if url not in ref_urls: |
1032 | - ref_urls.append(url) |
1033 | - |
1034 | - # Build up list of bug urls |
1035 | - bug_urls = [] |
1036 | - for pkg in packages: |
1037 | - if self.debian and \ |
1038 | - cve in self.debian and \ |
1039 | - self.debian[cve]['pkgs'] and \ |
1040 | - pkg in self.debian[cve]['pkgs']: |
1041 | - bug = None |
1042 | - if self.debian[cve]['pkgs'][pkg]['priority'] and \ |
1043 | - re.search(r'^bug #[0-9]+$', self.debian[cve]['pkgs'][pkg]['priority']): |
1044 | - bug = self.debian[cve]['pkgs'][pkg]['priority'].split('#')[1] |
1045 | - elif self.debian[cve]['pkgs'][pkg]['bug']: |
1046 | - bug = self.debian[cve]['pkgs'][pkg]['bug'] |
1047 | - if bug: |
1048 | - url = "http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%s" % bug |
1049 | - if url not in bug_urls: |
1050 | - bug_urls.append(url) |
1051 | - |
1052 | - # Add to tracker from 00boilerplate |
1053 | - dst = add_CVE_to_tracker(cve, self.cve_data[cve], packages, priority, bug_urls, ref_urls) |
1054 | - |
1055 | - # Build up command line |
1056 | - cmd = ['./scripts/active_edit', '-c', cve, '--yes'] |
1057 | - |
1058 | - # capture debian not-affected states |
1059 | - not_affected = [] |
1060 | - |
1061 | - for pkg in packages: |
1062 | - # The Debian convention is to specify the fixed version as the state |
1063 | - # with the bug number as the priority for fixed bugs. Leverage this |
1064 | - # with active_edit |
1065 | - fixed_in = "" |
1066 | - if self.debian and \ |
1067 | - cve in self.debian and \ |
1068 | - self.debian[cve]['pkgs'] and \ |
1069 | - pkg in self.debian[cve]['pkgs'] and \ |
1070 | - self.debian[cve]['pkgs'][pkg]['state']: |
1071 | - if re.search(r'^[0-9]', self.debian[cve]['pkgs'][pkg]['state']): |
1072 | - fixed_version = self.debian[cve]['pkgs'][pkg]['state'] |
1073 | - fixed_in = ",%s" % fixed_version |
1074 | - |
1075 | - # Now see if we can correlate this to an Ubuntu version |
1076 | - answer = source_map.madison(source, pkg) |
1077 | - for name in sorted(answer.keys()): |
1078 | - rel = name.split('/')[0].split('-')[0] # don't care about the pocket |
1079 | - version = answer[name][pkg] |
1080 | - # Try to compare apples to apples. Ie, if one of us has |
1081 | - # an epoch and the other doesn't, don't try to be smart |
1082 | - if (':' not in version and ':' not in fixed_version) or \ |
1083 | - (':' in version and ':' in fixed_version): |
1084 | - if dpkg_compare_versions(version, 'ge', fixed_version): |
1085 | - if rel == cve_lib.devel_release: |
1086 | - rel = 'devel' |
1087 | - fixed_in += ",%s,%s" % (rel, version) |
1088 | - break |
1089 | - elif self.debian[cve]['pkgs'][pkg]['state'].startswith('<not-affected>') and \ |
1090 | - len(self.debian[cve]['pkgs'][pkg]['priority']) > 0: |
1091 | - # capture that debian believes their version is unaffected |
1092 | - not_affected.append((pkg, "debian: %s" % self.debian[cve]['pkgs'][pkg]['priority'])) |
1093 | - cmd += ['-p', "%s%s" % (pkg, fixed_in)] |
1094 | - |
1095 | + self.debian[cve]['pkgs'] and \ |
1096 | + pkg in self.debian[cve]['pkgs']: |
1097 | + bug = None |
1098 | + if self.debian[cve]['pkgs'][pkg]['priority'] and \ |
1099 | + re.search(r'^bug #[0-9]+$', self.debian[cve]['pkgs'][pkg]['priority']): |
1100 | + bug = self.debian[cve]['pkgs'][pkg]['priority'].split('#')[1] |
1101 | + elif self.debian[cve]['pkgs'][pkg]['bug']: |
1102 | + bug = self.debian[cve]['pkgs'][pkg]['bug'] |
1103 | + if bug: |
1104 | + url = "http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%s" % bug |
1105 | + if url not in bug_urls: |
1106 | + bug_urls.append(url) |
1107 | + |
1108 | + |
1109 | + # Build up command line |
1110 | + cmd = ['./scripts/active_edit', '-c', cve, '--yes'] |
1111 | + for url in ref_urls: |
1112 | + cmd.extend(['--reference-url', url]) |
1113 | + for url in bug_urls: |
1114 | + cmd.extend(['--bug-url', url]) |
1115 | + if priority: |
1116 | + cmd.extend(['--priority', priority]) |
1117 | + |
1118 | + # capture debian not-affected states |
1119 | + not_affected = [] |
1120 | + |
1121 | + for pkg in packages: |
1122 | + # The Debian convention is to specify the fixed version as the state |
1123 | + # with the bug number as the priority for fixed bugs. Leverage this |
1124 | + # with active_edit |
1125 | + fixed_in = "" |
1126 | + if self.debian and \ |
1127 | + cve in self.debian and \ |
1128 | + self.debian[cve]['pkgs'] and \ |
1129 | + pkg in self.debian[cve]['pkgs'] and \ |
1130 | + self.debian[cve]['pkgs'][pkg]['state']: |
1131 | + if re.search(r'^[0-9]', self.debian[cve]['pkgs'][pkg]['state']): |
1132 | + fixed_version = self.debian[cve]['pkgs'][pkg]['state'] |
1133 | + fixed_in = ",%s" % fixed_version |
1134 | + |
1135 | + # Now see if we can correlate this to an Ubuntu version |
1136 | + answer = source_map.madison(source, pkg) |
1137 | + for name in sorted(answer.keys()): |
1138 | + rel = name.split('/')[0].split('-')[0] # don't care about the pocket |
1139 | + version = answer[name][pkg] |
1140 | + # Try to compare apples to apples. Ie, if one of us has |
1141 | + # an epoch and the other doesn't, don't try to be smart |
1142 | + if (':' not in version and ':' not in fixed_version) or \ |
1143 | + (':' in version and ':' in fixed_version): |
1144 | + if dpkg_compare_versions(version, 'ge', fixed_version): |
1145 | + if rel == cve_lib.devel_release: |
1146 | + rel = 'devel' |
1147 | + fixed_in += ",%s,%s" % (rel, version) |
1148 | + break |
1149 | + elif self.debian[cve]['pkgs'][pkg]['state'].startswith('<not-affected>') and \ |
1150 | + len(self.debian[cve]['pkgs'][pkg]['priority']) > 0: |
1151 | + # capture that debian believes their version is unaffected |
1152 | + not_affected.append((pkg, "debian: %s" % self.debian[cve]['pkgs'][pkg]['priority'])) |
1153 | + cmd += ['-p', "%s%s" % (pkg, fixed_in)] |
1154 | + |
1155 | + subprocess.call(cmd) |
1156 | + for (pkg, reason) in not_affected: |
1157 | + cmd = ['./scripts/mass-cve-edit', '-p', pkg, '-r', 'upstream', '-s', 'not-affected', '-v', reason, cve] |
1158 | subprocess.call(cmd) |
1159 | - for (pkg, reason) in not_affected: |
1160 | - cmd = ['./scripts/mass-cve-edit', '-p', pkg, '-r', 'upstream', '-s', 'not-affected', '-v', reason, cve] |
1161 | - subprocess.call(cmd) |
1162 | - self.num_added += 1 |
1163 | - return dst |
1164 | + self.num_added += 1 |
1165 | + return './active/%s' % cve |
1166 | |
1167 | def unembargo_cve(self, cve): |
1168 | # unembargo a cve now public |
1169 | diff --git a/scripts/check-syntax b/scripts/check-syntax |
1170 | index 9501d58..234c71a 100755 |
1171 | --- a/scripts/check-syntax |
1172 | +++ b/scripts/check-syntax |
1173 | @@ -279,7 +279,7 @@ if len(args) == 0: |
1174 | with open(opt.filelist) as fh: |
1175 | for line in fh: |
1176 | for dir in check_dirs: |
1177 | - if line.startswith("%s/CVE-" % dir) or line.startswith("%s/00boilerplate" % dir): |
1178 | + if line.startswith("%s/CVE-" % dir): |
1179 | args += [line.rstrip()] |
1180 | elif opt.modified: |
1181 | if opt.debug: |
1182 | @@ -299,11 +299,7 @@ if len(args) == 0: |
1183 | args += [filename] |
1184 | else: |
1185 | for dir in check_dirs: |
1186 | - # XXX might want to separate out boilerplate checks |
1187 | - # as they might need to be less restrictive than regular CVE |
1188 | - # file checks |
1189 | - for cve in sorted(glob.glob("%s/CVE-*" % dir) + |
1190 | - glob.glob("%s/00boilerplate.*" % dir)): |
1191 | + for cve in sorted(glob.glob("%s/CVE-*" % dir)): |
1192 | args += [cve] |
1193 | else: |
1194 | all_files = False |
1195 | @@ -404,7 +400,7 @@ for cve in args: |
1196 | cve_okay = False |
1197 | |
1198 | # verify candidate field matches the CVE file name |
1199 | - if "stdin" not in cve and "boilerplate" not in cve and not data["Candidate"] == cve: |
1200 | + if "stdin" not in cve and not data["Candidate"] == cve: |
1201 | filename = srcmap["Candidate"][0] |
1202 | linenum = srcmap["Candidate"][1] |
1203 | print( |
1204 | @@ -424,10 +420,9 @@ for cve in args: |
1205 | # place the generated error message on this release's line etc |
1206 | nearby_rel = list(listed_releases - missing_releases)[0] |
1207 | for rel in missing_releases: |
1208 | - # only warn on active CVEs but don't warn on boilerplate entries missing external |
1209 | - # releases as this is not supported |
1210 | + # only warn on active CVEs |
1211 | if is_active(cve) and \ |
1212 | - ("boilerplate" not in cve or rel not in cve_lib.external_releases) and \ |
1213 | + rel not in cve_lib.external_releases and \ |
1214 | rel in source and pkg in source[rel]: |
1215 | filename = srcmap["pkgs"][pkg][nearby_rel][0] |
1216 | linenum = srcmap["pkgs"][pkg][nearby_rel][1] |
1217 | @@ -658,10 +653,8 @@ for cve in args: |
1218 | ) |
1219 | cve_okay = False |
1220 | |
1221 | - # Verify priority for any CVE with a supported package and when this is |
1222 | - # not boilerplate |
1223 | + # Verify priority for any CVE with a supported package |
1224 | if ( |
1225 | - "boilerplate" not in cve and |
1226 | len(supported) |
1227 | and (is_active(cve) or is_embargoed(cve)) |
1228 | and ("Priority" not in data or data["Priority"] not in cve_lib.priorities) |
1229 | @@ -777,10 +770,10 @@ for cve in args: |
1230 | cve_okay = False |
1231 | |
1232 | # Either PublicDate or CRD must be set to something |
1233 | - if ("boilerplate" not in cve and |
1234 | - ("PublicDate" not in data or data["PublicDate"] == "") and ( |
1235 | - "CRD" not in data or data["CRD"] == "" |
1236 | - )): |
1237 | + if ( |
1238 | + "PublicDate" not in data or data["PublicDate"] == "" and |
1239 | + "CRD" not in data or data["CRD"] == "" |
1240 | + ): |
1241 | key = "PublicDate" if "PublicDate" in srcmap else "CRD" |
1242 | filename = srcmap[key][0] |
1243 | linenum = srcmap[key][1] |
1244 | diff --git a/scripts/check-syntax-fixup b/scripts/check-syntax-fixup |
1245 | index 24bf608..a38ad1e 100755 |
1246 | --- a/scripts/check-syntax-fixup |
1247 | +++ b/scripts/check-syntax-fixup |
1248 | @@ -147,11 +147,6 @@ for line in args.infile: |
1249 | # remove this hard-coded hack one-day... |
1250 | if rel in cve_lib.external_releases or \ |
1251 | (rel == "trusty/esm" and "DOES exist" in msg): |
1252 | - # ignore boilerplate files for external_releases |
1253 | - if "boilerplate" in cve: |
1254 | - # print unhandled lines |
1255 | - print(line, file=sys.stderr) |
1256 | - continue |
1257 | cve = os.path.join( |
1258 | cve_lib.get_external_subproject_cve_dir(rel), os.path.basename(cve) |
1259 | ) |
1260 | diff --git a/scripts/cve-mode.el b/scripts/cve-mode.el |
1261 | index b7413b0..0992e4c 100644 |
1262 | --- a/scripts/cve-mode.el |
1263 | +++ b/scripts/cve-mode.el |
1264 | @@ -333,7 +333,7 @@ Queries umt for PACKAGE as well as looking in overlay files." |
1265 | (defun cve-mode-insert-package (package) |
1266 | "Add PACKAGE as affected to this CVE. |
1267 | Calls out to active_edit to do the heavy lifting so that |
1268 | -boilerplate entries are handled automatically." |
1269 | +meta_lists/package-db.json entries are handled automatically." |
1270 | (interactive |
1271 | (list (completing-read "Package: " cve-mode--source-packages))) |
1272 | (let ((args (list "-c" (file-name-base (buffer-file-name)) |
1273 | @@ -683,8 +683,6 @@ cross boundaries of block literals." |
1274 | |
1275 | ;;;###autoload |
1276 | (add-to-list 'auto-mode-alist '("CVE-[[:digit:]]\\{4\\}-[[:digit:]]\\{4,\\}\\'" . cve-mode)) |
1277 | -;;;###autoload |
1278 | -(add-to-list 'auto-mode-alist '("00boilerplate.*\\'" . cve-mode)) |
1279 | |
1280 | (provide 'cve-mode) |
1281 | ;;; cve-mode.el ends here |
1282 | diff --git a/scripts/cve.vim b/scripts/cve.vim |
1283 | index bc89578..a13b158 100644 |
1284 | --- a/scripts/cve.vim |
1285 | +++ b/scripts/cve.vim |
1286 | @@ -7,7 +7,6 @@ |
1287 | " $ ln -s $UCT/scripts/cve.vim ~/.vim/syntax/cve.vim |
1288 | " Add to ~/.vimrc: |
1289 | " autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9N]\\\{4,\} set syntax=cve |
1290 | -" autocmd BufNewFile,BufRead 00boilerplate.* set syntax=cve |
1291 | " |
1292 | " TODO: |
1293 | " - turn the release names into variables so we only have to update in one |
1294 | diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py |
1295 | index fe8e4f5..60da772 100755 |
1296 | --- a/scripts/cve_lib.py |
1297 | +++ b/scripts/cve_lib.py |
1298 | @@ -23,6 +23,8 @@ import cache_urllib |
1299 | import json |
1300 | import yaml |
1301 | |
1302 | +from functools import reduce |
1303 | + |
1304 | def set_cve_dir(path): |
1305 | '''Return a path with CVEs in it. Specifically: |
1306 | - if 'path' has CVEs in it, return path |
1307 | @@ -912,11 +914,33 @@ kernel_srcs = set(['linux', |
1308 | kernel_topic_branches = kernel_srcs.difference(['linux']) |
1309 | |
1310 | # for sanity, try to keep these in alphabetical order in the json file |
1311 | -def load_package_info_overrides(list_dir): |
1312 | - package_info_overrides = dict() |
1313 | - with open(os.path.join(list_dir, "package_info_overrides.json")) as _file: |
1314 | - package_info_overrides = json.load(_file) |
1315 | - return package_info_overrides |
1316 | +def load_package_db(dir=meta_dir): |
1317 | + pkg_db = {} |
1318 | + pkg_db_json = os.path.join(dir, "package-db.json") |
1319 | + try: |
1320 | + with open(pkg_db_json, "r", encoding='utf-8') as fp: |
1321 | + pkg_db = json.load(fp) |
1322 | + # add lookups based on aliases - we can't iterate over pkg_db and |
1323 | + # modify it so collect aliases then add them manually |
1324 | + alias_info = {} |
1325 | + for p in pkg_db: |
1326 | + try: |
1327 | + aliases = pkg_db[p]["aliases"] |
1328 | + if len(aliases) > 0: |
1329 | + alias_info[p] = aliases |
1330 | + except KeyError: |
1331 | + pass |
1332 | + for p in alias_info.keys(): |
1333 | + for a in alias_info[p]: |
1334 | + if a not in pkg_db: |
1335 | + # use original info if already in pkg_db |
1336 | + pkg_db[a] = pkg_db[p] |
1337 | + except FileNotFoundError: |
1338 | + # TODO: remove this exception handling once we have a package-db.json |
1339 | + # checked into UCT git since in that case this should never occur so it |
1340 | + # should be a fatal error if it is missing |
1341 | + pass |
1342 | + return pkg_db |
1343 | |
1344 | |
1345 | # "arch_list" is all the physical architectures buildable |
1346 | @@ -1226,19 +1250,19 @@ def release_is_older_than(release_a, release_b): |
1347 | |
1348 | |
1349 | |
1350 | -package_info_overrides = load_package_info_overrides(meta_dir) |
1351 | +package_db = load_package_db() |
1352 | |
1353 | def lookup_package_override_title(source): |
1354 | - global package_info_overrides |
1355 | - res = package_info_overrides.get(source) |
1356 | + global package_db |
1357 | + res = package_db.get(source) |
1358 | if isinstance(res, dict): |
1359 | return(res.get("title")) |
1360 | |
1361 | return None |
1362 | |
1363 | def lookup_package_override_description(source): |
1364 | - global package_info_overrides |
1365 | - res = package_info_overrides.get(source) |
1366 | + global package_db |
1367 | + res = package_db.get(source) |
1368 | if isinstance(res, dict): |
1369 | return(res.get("description")) |
1370 | |
1371 | @@ -1964,9 +1988,6 @@ def load_cve(cve, strict=False, srcmap=None): |
1372 | nonempty = ['Candidate'] |
1373 | if strict: |
1374 | nonempty += ['PublicDate'] |
1375 | - # boilerplate files are special and can (should?) be empty |
1376 | - if "boilerplate" in cve: |
1377 | - nonempty = [] |
1378 | |
1379 | if field not in data or field not in fields_seen: |
1380 | msg += "%s: %d: missing field '%s'\n" % (cve, linenum, field) |
1381 | @@ -2008,17 +2029,6 @@ def load_cve(cve, strict=False, srcmap=None): |
1382 | raise ValueError(msg.strip()) |
1383 | return data |
1384 | |
1385 | -def load_boilerplates(verbose=False): |
1386 | - boilerplates = dict() |
1387 | - prefix = "00boilerplate." |
1388 | - for bp in glob.glob(os.path.join(active_dir, prefix + "*")): |
1389 | - # TODO - should we differentiate on symlinks so we can tell which |
1390 | - # are the "real" primary ones and which ones are secondary? |
1391 | - name = bp[bp.find(prefix) + len(prefix):] |
1392 | - info = load_cve(bp) |
1393 | - boilerplates.setdefault(name, info) |
1394 | - return boilerplates |
1395 | - |
1396 | def load_all(cves, uems, rcves=[]): |
1397 | table = dict() |
1398 | priority = dict() |
1399 | @@ -2865,3 +2875,23 @@ def parse_cvss(cvss): |
1400 | js['baseMetricV3']['exploitabilityScore'] = round(exploitability * 10) / 10 |
1401 | js['baseMetricV3']['impactScore'] = round(impact * 10) / 10 |
1402 | return js |
1403 | + |
1404 | +def wordwrap(text, width): |
1405 | + """ |
1406 | + A word-wrap function that preserves existing line breaks |
1407 | + and most spaces in the text. Expects that existing line |
1408 | + breaks are posix newlines (\n). |
1409 | + """ |
1410 | + return reduce(lambda line, word, width=width: |
1411 | + '%s%s%s' % |
1412 | + (line, |
1413 | + ' \n'[(len(line) - line.rfind('\n') - 1 + len(word.split('\n', 1)[0]) >= width)], |
1414 | + word), |
1415 | + text.split(' ') |
1416 | + ) |
1417 | + |
1418 | +def wrap_text(text, width=75): |
1419 | + """ |
1420 | + Wrap text to width chars wide. |
1421 | + """ |
1422 | + return wordwrap(text, width).replace(' \n', '\n') |
1423 | diff --git a/scripts/dup-status-for-pkg b/scripts/dup-status-for-pkg |
1424 | index fc32055..c15d669 100755 |
1425 | --- a/scripts/dup-status-for-pkg |
1426 | +++ b/scripts/dup-status-for-pkg |
1427 | @@ -24,7 +24,7 @@ fi |
1428 | for pkg in "$@"; do |
1429 | echo "Duplicating status from ${src} to ${dst} for ${pkg} where was DNE for ${dst}" |
1430 | # find files to change |
1431 | - for file in $(grep -l '^'"${dst}_${pkg}"': DNE' active/00boilerplate* active/CVE-* retired/CVE-*); do |
1432 | + for file in $(grep -l '^'"${dst}_${pkg}"': DNE' active/CVE-* retired/CVE-*); do |
1433 | status=$(grep '^'"${src}_${pkg}"': ' "${file}" | cut -f2- -d' ') |
1434 | if [ ! -z "${status}" ]; then |
1435 | echo "Duplicating status ${status} from ${src} to ${dst} in ${file}" |
1436 | diff --git a/scripts/kernel-triage-missing-break-fix b/scripts/kernel-triage-missing-break-fix |
1437 | index e60cb38..e21a949 100755 |
1438 | --- a/scripts/kernel-triage-missing-break-fix |
1439 | +++ b/scripts/kernel-triage-missing-break-fix |
1440 | @@ -35,7 +35,7 @@ if [ -z "${debian_kernel_cve_tracker}" ] ; then |
1441 | echo |
1442 | fi |
1443 | |
1444 | -_CVES=$(cd "$UCT" && grep -lr '^Patches_linux:' --exclude '*boilerplate*' active/ | xargs grep -L '^ break-fix:' | sort) |
1445 | +_CVES=$(cd "$UCT" && grep -lr '^Patches_linux:' active/ | xargs grep -L '^ break-fix:' | sort) |
1446 | for _CVE in ${_CVES} ; do |
1447 | echo "${_CVE}" |
1448 | CVE="${_CVE##active/}" |
1449 | diff --git a/scripts/release-cycle-devel-opens b/scripts/release-cycle-devel-opens |
1450 | index fccd5af..cf08707 100755 |
1451 | --- a/scripts/release-cycle-devel-opens |
1452 | +++ b/scripts/release-cycle-devel-opens |
1453 | @@ -18,7 +18,7 @@ if ! [ -d embargoed/ ]; then |
1454 | echo "Unable to find embargoed directory, is it set up?" >&2 |
1455 | fi |
1456 | |
1457 | -for f in active/{CVE-,00boilerplate}* embargoed/CVE-* ; do |
1458 | +for f in active/CVE-* embargoed/CVE-* ; do |
1459 | if egrep -q "^(#?)devel_" "$f" ; then |
1460 | echo "DEBUG: skipping $f" |
1461 | continue |
1462 | diff --git a/scripts/release-cycle-released b/scripts/release-cycle-released |
1463 | index b5cfc7a..c6d844f 100755 |
1464 | --- a/scripts/release-cycle-released |
1465 | +++ b/scripts/release-cycle-released |
1466 | @@ -22,7 +22,7 @@ if ! [ -d active/ ]; then |
1467 | exit 2 |
1468 | fi |
1469 | |
1470 | -for f in active/{00boilerplate,CVE-}*; do |
1471 | +for f in active/CVE-*; do |
1472 | # ignore symlinks so we don't replace twice in the same underlying file |
1473 | if [ ! -L "$f" ]; then |
1474 | perl -pi -e 's/^((#?)devel_(.*))/${2}'"$REL"'_$3\n$1/g' "$f" |
1475 | diff --git a/scripts/sync-from-eol.py b/scripts/sync-from-eol.py |
1476 | index b2a020c..dd6d9eb 100755 |
1477 | --- a/scripts/sync-from-eol.py |
1478 | +++ b/scripts/sync-from-eol.py |
1479 | @@ -24,16 +24,12 @@ import sys |
1480 | import cve_lib |
1481 | import source_map |
1482 | |
1483 | -import warnings |
1484 | -warnings.filterwarnings('ignore', 'apt API not stable yet', FutureWarning) |
1485 | -import apt |
1486 | |
1487 | parser = optparse.OptionParser() |
1488 | parser.add_option("-r", "--release", dest="release", default=None, help="release to modify") |
1489 | parser.add_option("-W", "--whole", dest="whole", help="End of life the whole release", action='store_true') |
1490 | parser.add_option("-U", "--universe", dest="universe", help="Modify packages in universe and multiverse", action='store_true') |
1491 | parser.add_option("-u", "--update", dest="update", help="Update CVEs with released package versions", action='store_true') |
1492 | -parser.add_option("-b", "--include-boilerplate", help="Update boilerplate files", action='store_true') |
1493 | (opt, args) = parser.parse_args() |
1494 | |
1495 | if not opt.release: |
1496 | @@ -52,16 +48,13 @@ pkgs = source_map.load(releases=[opt.release], skip_eol_releases=False) |
1497 | |
1498 | cves = glob.glob('%s/CVE-*' % cve_lib.active_dir) |
1499 | |
1500 | -if opt.include_boilerplate: |
1501 | - cves += glob.glob('%s/00boilerplate.*' % cve_lib.active_dir) |
1502 | - |
1503 | if os.path.islink('embargoed'): |
1504 | cves += glob.glob('embargoed/CVE-*') |
1505 | cves += glob.glob('embargoed/EMB-*') |
1506 | |
1507 | for filename in cves: |
1508 | - # we don't want to edit boilerplate symlinks as that will cause them |
1509 | - # to become unsymlinked |
1510 | + # we don't want to edit symlinks as that will cause them to become |
1511 | + # unsymlinked |
1512 | if os.path.islink(filename): |
1513 | continue |
1514 | |
1515 | @@ -88,17 +81,13 @@ for filename in cves: |
1516 | and cve_lib.is_active_esm_release(cve_lib.get_orig_rel_name(opt.release)) |
1517 | ): |
1518 | status = data['pkgs'][src][opt.release] |
1519 | - if cve.startswith('00boilerplate.'): |
1520 | - cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support') |
1521 | - elif status[1] != '': |
1522 | + if status[1] != '': |
1523 | cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support, was %s [%s]' % (status[0], status[1])) |
1524 | else: |
1525 | cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support, was %s' % (status[0])) |
1526 | elif 'LTS' in cve_lib.release_name(opt.release): |
1527 | status = data['pkgs'][src][opt.release] |
1528 | - if cve.startswith('00boilerplate.'): |
1529 | - cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support') |
1530 | - elif status[1] != '': |
1531 | + if status[1] != '': |
1532 | cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support, was %s [%s]' % (status[0], status[1])) |
1533 | else: |
1534 | cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support, was %s' % (status[0])) |
One thing I caught in the most briefest of surface level reviews:
On Wed, Aug 03, 2022 at 07:05:15AM -0000, Alex Murray wrote: code-copies and amalgate that into data tracker/ data/embedded- code-copies" ]:
> +# TODO - decide if we want to keep this - for now leave it out
> +
> +# also parse debian's embedded-
> +for f in ["../security-
If you do keep this, please use the `secure_ testing_ path` cve-tracker. conf config file (obtained via read_config( )) instead of hard-coding a relative path and code-copies data.
from the ~/.ubuntu-
cve_lib.
repo name when reading debian's embedded-
Thanks.
--
Steve Beattie
<email address hidden>