Merge ~alexmurray/ubuntu-cve-tracker:noboilerplates-3 into ubuntu-cve-tracker:master

[Alex Murray]

This series of commits is designed to allow us to replace the boilerplate files and the package_info_overrides.json with a single package-db.json which captures the details of all these data sources. I have left out the one remaining commit which would remove these old data sources and add the new package-db.json file so that we can generate this once this is merged so that it does not conflict too much.

To test this branch you will need to generate this package-db.json as:

./scripts/boilerplate-to-json.py > meta_lists/package-db.json

Then you should be able to do CVE triage and run active_edit etc etc as normal.

The main benefit this change gives is a better way to represent this data *plus* it ensures that when say updating a CVE with a boilerplate target (e.g. active_edit -C CVE-NNNN-XXXX -p openjdk) - that the packages captured by that boilerplate (ie openjdk-10 etc) all get added to the CVE file as expected *along with all the various subprojects*. This should mean we don't have to use check-syntax-fixup so much.

[Steve Beattie (sbeattie)]

One thing I caught in the most briefest of surface level reviews:

On Wed, Aug 03, 2022 at 07:05:15AM -0000, Alex Murray wrote:
> +# TODO - decide if we want to keep this - for now leave it out
> +
> +# also parse debian's embedded-code-copies and amalgate that into data
> +for f in ["../security-tracker/data/embedded-code-copies"]:

If you do keep this, please use the `secure_testing_path`
from the ~/.ubuntu-cve-tracker.conf config file (obtained via
cve_lib.read_config()) instead of hard-coding a relative path and
repo name when reading debian's embedded-code-copies data.

Thanks.

--
Steve Beattie
<email address hidden>

[Steve Beattie (sbeattie)]

Again, have not read in detail...

On Wed, Aug 03, 2022 at 07:05:15AM -0000, Alex Murray wrote:

> diff --git a/scripts/add-derived-kernel b/scripts/add-derived-kernel
> index d219586..f4c96c9 100755
> --- a/scripts/add-derived-kernel
> +++ b/scripts/add-derived-kernel
> @@ -105,6 +105,6 @@ echo "Updating status of released CVEs (ok if empty and prints help)..."
> xargs --no-run-if-empty ./scripts/mass-cve-edit -p "linux-${KERNEL}" -r "${RELEASE}" -s not-affected
>
> printf "\n\nPlease update cve_lib.py, kernel_lib.py, sis-generate-usn, cve-alert.sh, and prepare-kernel-usn.py in scripts/\n"
> -echo "Please add package description information if necessary to meta_lists/package_info_overrides.json"
> +echo "Please add package description information if necessary to meta_lists/package-db.json"
> echo "If this kernel has not been published yet, please add it to the unpublished_kernels list in check-syntax."
> echo "Also, before publishing a USN against the new kernel, ensure packages_mirror has been run"

Plesae note that one of the things the `add-derived-kernel` script does
for the new kernel being added for tracking in UCT is add it to the
boilerplate (I subsequently adjust the addition's location in the
boilerplate to try to keep them relatively organized). Is there a way
add-derived-kernel could be updated to do this under the new structure?

(Yes, there is post-hoc manual editing to be done after
add-derived-kernel does its thing, but my intent is to reduce that as
much as possible, to avoid errors.)

--
Steve Beattie
<email address hidden>

[Alex Murray (alexmurray)]

Thanks for the comments @sbeattie - see https://git.launchpad.net/~alexmurray/ubuntu-cve-tracker/commit/?id=ae277c8a754f907248feff4cb0fdd39e68136f37 which updates add-derived-kernel to now update package-db.json instead of 00boilerplate.linux, plus https://git.launchpad.net/~alexmurray/ubuntu-cve-tracker/commit/?id=81ff3b0a1caa32ae8bdce9952883fecbbf076fb8 which updates boilerplate-to-json to use the configured path to the debian secure_testing repo rather than assuming it.

[Alex Murray (alexmurray)]

So this merge request has had no real review for almost a whole month... I would really like to get it merged so will likely just merge it next week unless anyone wants to give it a look over... TIA 🙏

[Eduardo Barretto (ebarretto)]

Small fix in indentation, but the code looks good to me.
I believe you will send another PR, later when things are working in this new structure, to do the cleanup in cve_lib (load_boilerplates()) and remove the current boilerplates, right?

review: Needs Fixing

[Alex Murray (alexmurray)]

Thanks @ebarretto - I have fixed the indentation in https://git.launchpad.net/~alexmurray/ubuntu-cve-tracker/commit/?id=8ba68cbe6f16231c02947e553cd568a2b6b4cd04 and have removed load_boilerplates() from cve_lib in https://git.launchpad.net/~alexmurray/ubuntu-cve-tracker/commit/?id=1ec5cbbe7d208c2104d40505111b6947de29285f

Yes, once this is merged in then I plan to actually remove the boilerplate files *and* generate the new package-db.json and commit this to UCT (plus remove the now obsolete package_info_overrides.json since it has been subsumed into package-db.json) - I just didn't add that as part of this PR to keep this smaller and easier to review, otherwise the diff would have been huge.

[Eduardo Barretto (ebarretto)]

lgtm, thanks!

review: Approve

[Seth Arnold (seth-arnold)]

Sorry for not getting to this sooner :( I like quite a lot of it. I'd love to be rid of the boilerplates.

I've got a few comments inline.

Thanks

review: Approve

[Alex Murray (alexmurray)]

Thanks Seth - fixed the typo and moved the parse_embedded_code_copies() down so it was clear that is is not used currently.

Regarding wordwrap() etc - that is simply stolen from check-cves and moved into cve_lib now so that we can reuse it - so it will have the same functionality as it did before - I have not modified it.

[Alex Murray (alexmurray)]

I just noticed that I missed a few other scripts that still refer to the boilerplate files so they will need to be fixed up before this gets merged:

scripts/release-cycle-devel-opens
scripts/release-cycle-released
scripts/dup-status-for-pkg
scripts/sync-from-eol.py

Basically `git grep boilerplate` should return nothing when this MR is actually ready.

review: Needs Fixing

[Seth Arnold (seth-arnold)]

scripts/dup-status-for-pkg can probably just delete this word:
active/00boilerplate*

scripts/release-cycle-released can probably delete the paragraph.

scripts/release-cycle-devel-opens can probably just change:
active/{CVE-,00boilerplate}*
to:
active/CVE-*

scripts/sync-from-eol.py will require a lot more thinking.

Getting rid of the boilerplates looks like it'll simplify a lot :)

diff --git a/README b/README
index 433dde7..8817769 100644
--- a/README
+++ b/README
@@ -597,7 +597,7 @@ Stable Release Actions
 ----------------------
 When a stable release is published, the active CVEs need to be adjusted to
 reflect the new stable release.  e.g. when trusty was published:
-  perl -pi -e 's/^((#?)devel_(.*))/$2trusty_$3\n$1/g' active/{CVE-,00boilerplate}*
+  perl -pi -e 's/^((#?)devel_(.*))/$2trusty_$3\n$1/g' active/CVE-*
 The script tools will need to be adjusted as well.  There is usually some
 lag time between the new devel archive opening and the stable release
 getting published.  This means that "devel" will disappear from ubuntu-table
@@ -611,7 +611,7 @@ briefly:
 
   scripts/cve_lib.py should have an empty 'devel_release'.
 
-Move all active CVEs and boilerplates from "devel" to release state:
+Move all active CVEs from "devel" to release state:
 ./scripts/release-cycle-released $RELEASE
 
 
@@ -619,7 +619,7 @@ Development Release Actions
 ---------------------------
 Fill in releases and devel_release in ubuntu-cve-tools/scripts/cve_lib.py
 
-Move all active CVEs and boilerplates from latest release to devel state:
+Move all active CVEs  from latest release to devel state:
 ./scripts/release-cycle-devel-opens $LATEST_STABLE_RELEASE
 
 Add release to non-ports and ports section of
@@ -635,7 +635,6 @@ Here is how:
 
 2. update the CVEs:
 $ sed -i 's/^<release>_\(.*\): \(needed\|needs\-triage\)/<release>_\1: \2 (reached end-of-life)/g' ./active/CVE-*
-$ sed -i '/^<release>_\(.*\): /d' ./active/00boilerplate*
 
 3. retire the CVEs (see 'Retiring items', above)
 
@@ -678,7 +677,7 @@ New Backport Kernel
 When a new backport kernel is added, update scripts/cve_lib.py's
 kernel_srcs and description_overrides.
 
-Then update the 00boilerplate.linux with its entry, add that entry to each
+Then update the meta_lists/package-db.json with its entry, add that entry to each
 CVE with an entry for the LTS the kernel was added to, and update all the
 statuses for the newly added kernel, based off the version it will be
 branched from. For example, to add a new kernel backported from Vivid to
diff --git a/README.mozilla b/README.mozilla
index 5e06594..7c0f1f2 100644
--- a/README.mozilla
+++ b/README.mozilla
@@ -45,7 +45,7 @@ CVE Triage
 ----------
 CVEs in Firefox are tracked in the xulrunner source packages for builds that
 use the system xulrunner, and firefox source packages for those that use a
-static build. active/00boilerplate.firefox is used to capture the source
+static build. meta_lists/package-db.json is used to capture the source
 package relationships when triaging CVEs for firefox (ie, you only need to
 specify 'firefox' as the source package).
 
@@ -65,6 +65,7 @@ xulrunner-1.9.2: system xul for reverese dependencies that process web content.
 firefox: Ubuntu 8.04 LTS and higher (static build of 3.6.x or higher)
 
 Additionally, the following share a common codebase and are affected by the
-same CVE often enough that they warrant being part of the firefox boilerplate:
+same CVE often enough that they warrant being part of the firefox pkgs set in
+meta_lists/package-db.json:
 seamonkey
 thunkerbird
diff --git a/README.usn b/README.usn
index ac8156f..e5c3f58 100644
--- a/README.usn
+++ b/README.usn
@@ -30,7 +30,7 @@ versions.
 Make sure the source package description was properly set for every affected
 release (i.e. the value for the usn.py --source-description argument in the
 generated usn shell script is set). If it was not automatically populated, make
-sure the source package is present in $UCT/meta_list/package_info_overrides.json
+sure the source package is present in $UCT/meta_list/package-db.json
 and add it if it is not.
 
 ** CVEs references
diff --git a/README.webkit b/README.webkit
index 187d619..a52bf88 100644
--- a/README.webkit
+++ b/README.webkit
@@ -75,7 +75,7 @@ chromium-browser
 
 This package contains a fork of the WebKit code base (aka 'Blink'). It is
 maintained separately by Google and is tracked in the 'chromium-browser'
-boilerplate.
+entry in meta_lists/package-db.json.
 
 oxide
 -----
@@ -83,11 +83,11 @@ oxide
 Oxide is bindings for the chromium content api and therefore contains a fork
 of the webkit code base (aka, 'Blink'). The chromium content api is maintained
 separately by Google and Oxide is maintained by Canonical. Oxide is tracked in
-the 'chromium-browser' boilerplate.
+the 'chromium-browser' entry in meta_lists/package-db.json.
 
 CVE Triage
 ----------
-active/00boilerplate.webkit is used to capture the source package relationships
+meta_lists/package-db.json is used to capture the source package relationships
 when triaging CVEs for webkit (ie, you only need to specify 'webkit' as the
 source package).
 
diff --git a/scripts/active_edit b/scripts/active_edit
index a85f279..482446c 100755
--- a/scripts/active_edit
+++ b/scripts/active_edit
@@ -10,9 +10,9 @@
 
 import optparse
 import os
+import pathlib
 import re
 import sys
-import time
 
 import cve_lib
 import source_map
@@ -21,7 +21,6 @@ releases = ['upstream'] + cve_lib.all_releases
 
 max_file_size = 10 * 1024 * 1024  # 10MB
 cvedir = cve_lib.active_dir
-boilerplates = cvedir
 
 parser = optparse.OptionParser()
 parser.add_option("-p", "--package", dest="pkgs", help="Package name and optional version where package is fixed (with optional Ubuntu release and version in that release)", metavar="NAME[,VERSION[,RELEASE,RELEASE_VERSION]]", action="append")
@@ -30,7 +29,8 @@ parser.add_option("-r", "--reference-url", dest="ref_urls", help="URL references
 parser.add_option("-c", "--cve", dest="cve", help="CVE entry", metavar="CVE-YYYY-NNNN")
 parser.add_option("-e", "--embargoed", dest="embargoed", help="This is an embargoed entry", action="store_true")
 parser.add_option("-y", "--yes", dest="autoconfirm", help="Do not ask for confirmation", action="store_true")
-parser.add_option("-P", "--public", help="Record date the CVE went public", metavar="YYYY-MM-DD")
+parser.add_option("-P", "--public", dest="public_date", help="Record date the CVE went public", metavar="YYYY-MM-DD")
+parser.add_option("--priority", help="Record a priority for the CVE", default=None)
 parser.add_option("-C", "--cvss", help="CVSS3.1 rating", metavar="CVSS:3.1/AV:_/AC:_/PR:_/UI:_/S:_/C:_/I:_/A:_")
 (options, args) = parser.parse_args()
 
@@ -70,6 +70,7 @@ def create_or_update_external_subproject_cves(cve, pkgname):
             continue
         affected_releases.append(release)
 
+    ans = "y"
     if not options.autoconfirm:
         print("\n")
         for release in affected_releases:
@@ -101,245 +102,130 @@ def release_wants_dne(release):
     _, product, _, _ = cve_lib.get_subproject_details(release)
     return product != None and product == cve_lib.PRODUCT_UBUNTU
 
-def update_cve(cve, pkgname, fixed_in=None, fixed_in_release=None, fixed_in_release_version=None):
-    '''Update an existing CVE file'''
-    with open(os.path.join(cvedir, cve), "r") as f:
-        lines = f.read(max_file_size).split('\n')
-
-    skipped = []
-    added_lines = ""
-
-    tmp_releases = get_releases(pkgname)
-
-    # If we are using 00boilerplate.<pkgname> and the package is DNE on all
-    # current releases, then don't add the the stanza for this release. This
-    # allows us to use generic boilerplate names like 00boilerplate.gnutls or
-    # 00boilerplate.openjdk without adding useless extra stanzas.
-    # TODO: this still doesn't handle adding the contents of the boilerplate
-    # to an existing CVE (ie, ./scripts/sctive_edit -p openjdk -c CVE-YYYY-NNNN
-    # where CVE-YYYY-NNNN already exists)
-    if os.path.exists(os.path.join(boilerplates, '00boilerplate.%s' % pkgname)):
-        pkg_exists_somewhere = False
-        for r in tmp_releases:
-            if r == 'upstream' or (r in cve_lib.eol_releases \
-                    and not cve_lib.is_active_esm_release(r)):
-                continue
-            if pkg_in_rel(pkgname,r):
-                pkg_exists_somewhere = True
-                break
-        if not pkg_exists_somewhere:
-            print("skipping '" + pkgname + "' (DNE on all current releases)\n", file=sys.stderr)
-            return
-
-    for line in lines:
-        for r in tmp_releases:
-            if r == cve_lib.devel_release or r == '':
-                r = 'devel'
-            if not re.match(r'^' + r + ".*:", line):
-                continue
-            tmp = line.split(':')
-            match = "%s_%s" % (r, pkgname)
-            if match == tmp[0]:
-                skipped.append(r)
-
-    if len(skipped) == 0:
-        added_lines += '\nPatches_' + pkgname + ':\n'
-
-    higher_not_affected = False
-    for release in tmp_releases:
-        # don't add any external releases
-        if release in cve_lib.external_releases:
-            continue
-
-        r = release
-        if r == cve_lib.devel_release or r == '':
-            r = 'devel'
-
-        if r in skipped:
-             print("skipping '" + pkgname + "' for " + r + " (already included)\n", file=sys.stderr)
-        else:
-            # skip eol_releases releases without esm support
-            if r in cve_lib.eol_releases \
-                  and (not cve_lib.is_active_esm_release(r) or r == 'precise'):
-                continue
-            state = "needs-triage"
-            if not pkg_in_rel(pkgname, release):
-                # package doesn't exist in this release - see if it wants a
-                # DNE entry
-                if release_wants_dne(release):
-                    state = "DNE"
-                else:
-                    continue
-            elif cve_lib.is_active_esm_release(r):
-                state = "ignored (out of standard support)"
-            elif r == 'upstream' and fixed_in is not None:
-                state = "released (%s)" % fixed_in
-            elif fixed_in_release_version and r == fixed_in_release:
-                state = "not-affected (%s)" % fixed_in_release_version
-                higher_not_affected = True
-            elif higher_not_affected:
-                state = "not-affected"
-            added_lines += '%s_%s: %s\n' % (r, pkgname, state)
-
-    if len(releases) == len(skipped):
-        print("\nNothing to add!\n")
-        return
-
-    if not options.autoconfirm:
-        print("\n" + added_lines)
-        print("\nAppend the above to " + os.path.join(cvedir, cve) + " (y|N)? ")
-        ans = sys.stdin.readline().lower()
-        print("\n")
-    else:
-        ans = "y"
 
-    if ans.startswith("y"):
-        file = open(os.path.join(cvedir, cve), "a")
-        file.write(added_lines)
-        file.close()
+def create_or_update_cve(cve, packages, priority=None, bug_urls=None, ref_urls=None, public_date=None, desc=None, cvss=None):
+
+    pkgs = []
+    fixed = {}
+    # parse optional fixed_in release and version from package name
+    for p in packages:
+        tmp_p = p.split(',')
+        pkg = tmp_p[0]
+        pkgs.append(pkg)
+        fixed[pkg] = tmp_p[1:]
+
+    update = False
+    try:
+        dst = cve_lib.find_cve(cve)
+        update = True
+    except ValueError:
+        dst = os.path.join(pathlib.Path().parent.resolve(), "active", cve)
+
+    # collect notes from pkg_db and add any extra pkgs from pkg_db as well
+    notes = []
+    for p in pkgs:
+        if p in pkg_db:
+            notes = notes + pkg_db[p]["notes"]
+            pkgs = pkgs + list(pkg_db[p]["pkgs"].keys())
+
+    # normalise the list of packages
+    pkgs = sorted(list(set(pkgs)))
+
+    # remove any packages which don't actually exist in any release
+    for p in pkgs:
+        keep = False
+        for r in source.keys():
+            keep |= p in source[r]
+        if not keep:
+            pkgs.remove(p)
+
+    if update:
+        mode = "a"
     else:
-        print("Aborted\n")
-
-def create_cve(cve, pkgname, fixed_in=None, fixed_in_release=None, fixed_in_release_version=None):
-    '''Create a new CVE file'''
-    src = os.path.join(boilerplates, '00boilerplate')
-    if os.path.exists(src + "." + pkgname):
-        src = src + "." + pkgname
-    boiler = open(src, "r")
-    lines =  boiler.read(max_file_size).splitlines()
-    boiler.close()
-
-    cand_pat = re.compile(r'^Candidate:')
-    ref_pat = re.compile(r'^References:')
-    bugs_pat = re.compile(r'^Bugs:')
-    cvss_pat = re.compile(r'^CVSS:')
-    pkg_pat = re.compile(r'^#?[a-z/\-]+_(PKG|%s):' % pkgname)
-    patch_pat = re.compile(r'^#?Patches_(PKG|%s):' % pkgname)
-
-    tmp_releases = get_releases(pkgname)
-    added = set()
-
-    contents = ""
-    higher_not_affected = False
-    for line in lines:
-        if (cand_pat.search(line)):
-            contents += line + os.path.basename(cve) + '\n'
-        elif line.startswith('PublicDate:'):
-            if options.embargoed:
-                if options.public:
-                    # use public date as CRD
-                    contents += "CRD: %s\n" % options.public
-                else:
-                    contents += "CRD: <TBD>\n"
-            if options.public:
-                contents += "PublicDate: %s\n" % options.public
-            else:
-                # default to today-- this will be refreshed by check-cves
-                contents += "PublicDate: %s\n" % time.strftime("%Y-%m-%d", time.gmtime())
-        elif (cvss_pat.search(line)):
-            if options.cvss:
-                contents += 'CVSS: ' + options.cvss + '\n'
-            else:
-                contents += 'CVSS:\n'
-        elif (patch_pat.search(line)):
-            contents += '\nPatches_' + pkgname + ':\n'
-        elif (pkg_pat.search(line)):
-            for release in tmp_releases:
-                if release == cve_lib.devel_release or release == '':
-                    release = 'devel'
-
-                # skip eol_releases releases without esm support
-                if release in cve_lib.eol_releases \
-                        and not cve_lib.is_active_esm_release(release):
+        mode = "w"
+    with open(dst, mode, encoding="utf-8") as fp:
+        if not update:
+            print('Candidate: %s' % (cve), file=fp)
+            print('PublicDate: %s' % (public_date if public_date else "unknown"), file=fp)
+            print('References:\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=%s' % (cve), file=fp)
+            for url in (ref_urls if ref_urls else []):
+                print(" %s" % url, file=fp)
+            print('Description:', file=fp)
+            for desc_line in (cve_lib.wrap_text(desc).split('\n') if desc else []):
+                print(" %s" % (desc_line), file=fp)
+            print('Ubuntu-Description:', file=fp)
+            print('Notes:', file=fp)
+            for note in notes:
+                for note_line in cve_lib.wrap_text(note[1], 75 - len(note[0]) - 2).split('\n'):
+                    print(" %s> %s" % (note[0], note_line), file=fp)
+            print('Mitigation:', file=fp)
+            print('Bugs:', file=fp)
+            for url in (bug_urls if bug_urls else []):
+                print(" %s" % url, file=fp)
+            print('Priority: %s' % (priority if priority else "untriaged"), file=fp)
+            print('Discovered-by:', file=fp)
+            print('Assigned-to:', file=fp)
+            print('CVSS:', file=fp)
+            for entry in (cvss if cvss else []):
+                src, cvss = entry
+                print(' %s: %s' % (src, cvss), file=fp)
+
+        # add package info from pkg_db
+        for p in pkgs:
+            print('', file=fp)
+            print('Patches_%s:' % p, file=fp)
+            # find which releases p exists in
+            higher_not_affected = False
+            fixed_in = None
+            fixed_in_release = None
+            fixed_in_release_version = None
+            if p in fixed and len(fixed[p]) > 0:
+                fixed_in = fixed[p][0]
+                if len(fixed[p]) > 1:
+                    fixed_in_release = fixed[p][1]
+                if len(fixed[p]) > 2:
+                    fixed_in_release_version = fixed[p][2]
+            for rel in ['upstream'] + list(source.keys()):
+                # determine default state but override this if pkg_db has a
+                # better one
+                state = "needs-triage"
+                if not pkg_in_rel(p, rel):
+                    # package doesn't exist in this release - see if it wants a
+                    # DNE entry
+                    if release_wants_dne(rel):
+                        state = "DNE"
+                    else:
+                        continue
+                if rel == cve_lib.devel_release:
+                    # devel is present in source.keys() so use that instead
+                    # of the codename
                     continue
+                if cve_lib.is_active_esm_release(rel):
+                    state = "ignored (out of standard support)"
+                elif rel == 'upstream' and fixed_in is not None:
+                    state = "released (%s)" % fixed_in
+                elif fixed_in_release_version and rel == fixed_in_release:
+                    state = "not-affected (%s)" % fixed_in_release_version
+                    higher_not_affected = True
+                elif higher_not_affected:
+                    state = "not-affected"
+
+                # use pkg_db state if one exists
+                if p in pkg_db and p in pkg_db[p]["pkgs"] and rel in pkg_db[p]["pkgs"][p]:
+                    state_tuple = pkg_db[p]["pkgs"][p][rel]
+                    state = state_tuple[0]
+                    if len(state_tuple[1]) > 0:
+                        state = state + " (%s)" % state_tuple[1]
+                if rel not in cve_lib.external_releases:
+                    print('%s_%s: %s' % (rel, p, state), file=fp)
+                else:
+                    # add this to subprojects for rel
+                    with open(os.path.join(cve_lib.get_external_subproject_cve_dir(rel), cve), "a") as f:
+                        print('%s_%s: %s' % (rel, p, state), file=f)
 
-                rel_pat = re.compile('#?' + release + '_')
-
-                if (rel_pat.search(line)):
-                    state = "needs-triage"
-                    if not pkg_in_rel(pkgname, release):
-                        # package doesn't exist in this release - see if it wants a
-                        # DNE entry
-                        if release_wants_dne(release):
-                            state = "DNE"
-                        else:
-                            continue
-                    elif cve_lib.is_active_esm_release(release):
-                        state = "ignored (out of standard support)"
-                    elif release == 'upstream' and fixed_in is not None:
-                        state = "released (%s)" % fixed_in
-                    elif fixed_in_release_version and release == fixed_in_release:
-                        state = "not-affected (%s)" % fixed_in_release_version
-                        higher_not_affected = True
-                    elif higher_not_affected:
-                        state = "not-affected"
-                    if release not in added:
-                        contents += "%s_%s: %s\n" % (release, pkgname, state)
-                        added.add(release)
-        elif ref_pat.search(line):
-            if not re.search(r'N', cve):
-                contents += line + "\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=" + cve + "\n"
-            else:
-                contents += line + "\n"
-            if options.ref_urls:
-                for i in options.ref_urls:
-                   contents += " %s\n" % i
-        elif options.bug_urls and bugs_pat.search(line):
-            contents += line
-            for i in options.bug_urls:
-                contents += "\n %s\n" % i
-        else:
-            contents += line + '\n'
-
-    # check for missing entries which aren't in the boilerplate
-    for release in tmp_releases:
-        if release == cve_lib.devel_release or release == '':
-            release = 'devel'
-        if release in added:
-            continue
-        if release in cve_lib.external_releases:
-            continue
-        # skip eol_releases releases without esm support
-        if release in cve_lib.eol_releases \
-           and not cve_lib.is_active_esm_release(release):
-            continue
-        if pkg_in_rel(pkgname, release):
-            state = "needs-triage"
-            contents += "%s_%s: %s\n" % (release, pkgname, state)
-
-    # for each line in contents, see if we need to supercede DNE status
-    # with needs-triage as boilerplate entries may not be up-to-date with
-    # xxx-supported.txt lists
-    pkgstatus_pat = re.compile(r'^([/a-z/\-]+)_(.*): (.*)')
-    old_contents = contents
-    contents = ""
-    for line in old_contents.splitlines():
-        match = pkgstatus_pat.search(line)
-        if match is not None:
-            release = match.group(1)
-            pkgname = match.group(2)
-            status = match.group(3)
-            if status == 'DNE' and pkg_in_rel(pkgname, release):
-                status = 'needs-triage'
-                line = '%s_%s: %s' % (release, pkgname, status)
-        contents += line + '\n'
-
-    if not options.autoconfirm:
-        print(contents)
-        print("\nWrite the above to " + os.path.join(cvedir, cve) + " (y|N)? ")
-        ans = sys.stdin.readline().lower()
-        print("\n")
-    else:
-        ans = "y"
-
-    if ans.startswith("y"):
-        newfile = open(os.path.join(cvedir, cve), 'w')
-        newfile.write(contents)
-        newfile.close()
-    else:
-        print("Aborted\n")
 
 
+pkg_db = cve_lib.load_package_db()
 
 if not options.pkgs:
     parser.print_help()
@@ -369,37 +255,5 @@ if not pat.search(cve):
         print("Bad CVE entry.  Should be CVE-XXXX-XXXX\n", file=sys.stderr)
     sys.exit(1)
 
-# more here
-pat = re.compile(r'\s')
-for p in pkgs:
-    tmp_p = p.split(',')
-    pkgname = tmp_p[0]
-    fixed_in = None
-    if len(tmp_p) > 1:
-        fixed_in = tmp_p[1]
-    fixed_in_release = None
-    fixed_in_release_version = None
-    if len(tmp_p) > 3:
-        fixed_in_release = tmp_p[2]
-        fixed_in_release_version = tmp_p[3]
-
-    if pat.search(pkgname):
-        print("Bad package name\n", file=sys.stderr)
-        sys.exit(1)
-
-    if not os.path.isfile(os.path.join(boilerplates, "00boilerplate")):
-        print("Could not find 00boilerplate in " + cvedir + "\n", file=sys.stderr)
-        sys.exit(1)
-
-    if (os.path.isfile(os.path.join(cvedir, cve))):
-        if not options.autoconfirm:
-            print("Found existing " + cve + "...\n\n")
-        update_cve(cve, pkgname, fixed_in, fixed_in_release, fixed_in_release_version)
-    else:
-        if not options.autoconfirm:
-            print("Creating new " + cve + "...\n\n")
-        create_cve(cve, pkgname, fixed_in, fixed_in_release, fixed_in_release_version)
-
-    create_or_update_external_subproject_cves(cve, pkgname)
-
+create_or_update_cve(cve, pkgs, priority=options.priority, bug_urls=options.bug_urls, ref_urls=options.ref_urls, public_date=options.public_date, cvss=options.cvss)
 sys.exit(0)
diff --git a/scripts/add-derived-kernel b/scripts/add-derived-kernel
index d219586..3c146ab 100755
--- a/scripts/add-derived-kernel
+++ b/scripts/add-derived-kernel
@@ -67,19 +67,37 @@ if [ -z "${DERIVED_KERNEL}" ] ; then
     DERIVED_KERNEL="linux"
 fi
 
-chunk=$(mktemp -t add-derived-kernel-XXXXXX)
+releases=$(PYTHONPATH=./scripts python3 -c "import cve_lib; print(' '.join([a for a in cve_lib.all_releases if a not in cve_lib.external_releases and a is not cve_lib.devel_release and (a not in cve_lib.eol_releases or cve_lib.is_active_esm_release(a))]))")
+
+releases="upstream ${releases} devel"
 
+chunk=$(mktemp -t add-derived-kernel-XXXXXX)
 echo "" > "$chunk"
-grep "^.*_linux:" active/00boilerplate.linux | sed -e "s#_linux:.*#_linux-${KERNEL}: DNE#" >> "$chunk"
-sed -i -e "s#^${RELEASE}_linux-${KERNEL}:.*#${RELEASE}_linux-${KERNEL}: needs-triage#" "$chunk"
-sed -i -e "s#^Patches_linux-${KERNEL}:.*#Patches_linux-${KERNEL}:#" "$chunk"
-sed -i -e "s#^upstream_linux-${KERNEL}:.*#upstream_linux-${KERNEL}: needs-triage#" "$chunk"
-
-if ! grep -q "^${RELEASE}_linux-${KERNEL}:" active/00boilerplate.linux ; then
-    echo "Updating 00boilerplate.linux..."
-    cat "$chunk" >> active/00boilerplate.linux
+echo "Patches_linux-${KERNEL}:" >> "$chunk"
+
+json="{"
+
+# add other releases
+for rel in $releases; do
+    status="DNE"
+    if [ "${rel}" = "${RELEASE}" ] || [ "${rel}" = "upstream" ] ; then
+        status="needs-triage"
+    fi
+    json="$json\"${rel}_${DERIVED_KERNEL}\": [\"$status\", \"\"],"
+    echo "${rel}_linux-${KERNEL}: ${status}" >> "$chunk"
+done
+# remove trailing comma
+json=${json%?}
+# close json objects
+json="$json}"
+
+# append the derived kernel to meta_lists/package-db.json if it doesn't exist already
+if jq -e ".linux.pkgs.\"linux-$KERNEL\"" meta_lists/package-db.json; then
+    echo "kernel linux-${KERNEL} already exists in meta_lists/package-db.json, skipping update."
 else
-    echo "00boilerplate.linux already contains linux-${KERNEL}, skipping update."
+    echo "adding kernel linux-${KERNEL} to meta_lists/package-db.json"
+    jq -e ".linux.pkgs.\"linux-$KERNEL\" += $json" meta_lists/package-db.json > meta_lists/package-db.json.new
+    mv meta_lists/package-db.json.new meta_lists/package-db.json
 fi
 
 echo "Adding new backport to existing CVEs..."
@@ -105,6 +123,10 @@ echo "Updating status of released CVEs (ok if empty and prints help)..."
  xargs --no-run-if-empty  ./scripts/mass-cve-edit -p "linux-${KERNEL}" -r "${RELEASE}" -s not-affected
 
 printf "\n\nPlease update cve_lib.py, kernel_lib.py, sis-generate-usn, cve-alert.sh, and prepare-kernel-usn.py in scripts/\n"
-echo "Please add package description information if necessary to meta_lists/package_info_overrides.json"
+echo "Please add package description information if necessary to meta_lists/package-db.json"
 echo "If this kernel has not been published yet, please add it to the unpublished_kernels list in check-syntax."
 echo "Also, before publishing a USN against the new kernel, ensure packages_mirror has been run"
+
+# Local Variables:
+# sh-indentation: 4
+# End:
diff --git a/scripts/add_meta_info.py b/scripts/add_meta_info.py
index 994c8ec..d5b78c3 100755
--- a/scripts/add_meta_info.py
+++ b/scripts/add_meta_info.py
@@ -10,11 +10,13 @@ if len(sys.argv) != 4:
 package = sys.argv[1]
 title = sys.argv[2]
 description = sys.argv[3]
-json_file = os.environ['UCT'] + "/meta_lists/package_info_overrides.json"
+json_file = os.environ['UCT'] + "/meta_lists/package-db.json"
 
 with open(json_file, 'r') as handle:
     parsed = json.load(handle)
-    parsed[package] = { "description": description, 'title': title }
+    parsed.setdefault(package, {})
+    parsed[package]['title'] = title
+    parsed[package]['description'] = description
     new_content = json.dumps(parsed, indent=4, sort_keys=True)
 
 with open(json_file, 'w') as handle:
diff --git a/scripts/boilerplate-to-json.py b/scripts/boilerplate-to-json.py
 new file mode 100755
index 0000000..db2c735
--- /dev/null
+++ b/scripts/boilerplate-to-json.py
@@ -0,0 +1,151 @@
+#!/usr/bin/python3
+import glob
+import json
+import os
+import re
+import sys
+
+import cve_lib
+
+def parse_boilerplate(filepath):
+    cve_data = cve_lib.load_cve(filepath)
+    # capture tags, Notes, and package relationships
+    data = dict()
+    data.setdefault("aliases", list())
+    # tags are a set but json can't serialise a set so convert to a list first
+    data.setdefault("tags", list(cve_data.get("tags", list())))
+    data.setdefault("notes", cve_data.get("Notes", list()))
+    data.setdefault("pkgs", cve_data.get("pkgs", dict()))
+    return data
+
+
+def load_boilerplates():
+    data = dict()
+    aliases = dict()
+    for filepath in glob.glob("active/00boilerplate.*"):
+        name = ".".join(filepath.split(".")[1:])
+        # check if is a symlink and if so don't bother loading the file
+        # directly but add an entry as this is an alias
+        if os.path.islink(filepath):
+            orig = os.readlink(filepath)
+            orig_name = ".".join(orig.split(".")[1:])
+            aliases.setdefault(orig_name, set())
+            aliases[orig_name].add(name)
+            continue
+        bpdata = parse_boilerplate(filepath)
+        # having a package reference itself as we have in the boilerplates
+        # is redundant - although this is not always the case as we may
+        # have a boilerplate filename like openjdk yet there is no openjdk
+        # package (just openjdk-8 etc) - so ignore any failures here
+        try:
+            del bpdata["pkgs"][name]
+        except KeyError:
+            pass
+        data.setdefault(name, bpdata)
+    for alias in aliases:
+        data[alias]["aliases"] = list(aliases[alias])
+    return data
+
+
+def load_package_info_overrides():
+    with open("meta_lists/package_info_overrides.json", "r") as fp:
+        data = json.load(fp)
+        return data
+
+
+overrides_data = load_package_info_overrides()
+# turn this into empty data
+for pkg in overrides_data:
+    for key in ["aliases", "tags", "notes"]:
+        overrides_data[pkg][key] = []
+    overrides_data[pkg]["pkgs"] = {}
+bp_data = load_boilerplates()
+# merge the two data sources
+data = {**overrides_data, **bp_data}
+
+print(json.dumps(data, indent=2))
+sys.exit(0)
+
+# TODO - decide if we want to keep this - for now leave it out
+
+def parse_embedded_code_copies(filepath):
+    begin_re = re.compile(r"^---BEGIN$")
+    pkg_re = re.compile(r"^([a-zA-Z0-9.+_-]+).*$")
+    #	[release] - srcpkg version|<status> (sort; bug #)
+    embedding_re = re.compile(r"^\t(\[([a-z]+)\] )?- ([a-zA-Z0-9.+-]+) ([a-zA-Z0-9:~.+-]+|<(unfixed|removed|itp|not-affected|unknown|unfixable)>)( \((static|embed|modified-embed|fork|old-version)(; (.*))?\))?.*$")
+    note_re = re.compile(r"^\tNOTE: (.*)$")
+    data = dict()
+    pkgs = dict()
+    notes = dict()
+    with open(filepath, "r") as fp:
+        linenum = 0
+        begin = False
+        pkg = None
+        embedding_pkg = None
+        for line in fp.readlines():
+            linenum += 1
+            # strip trailing space
+            line = line.rstrip()
+            if not begin:
+                if begin_re.match(line):
+                    begin = True
+                continue
+            if len(line) == 0:
+                continue
+            # is this a new package
+            m = pkg_re.match(line)
+            if m is not None:
+                pkg = m[1]
+                pkgs[pkg] = dict()
+                continue
+            # is this an entry for a package
+            m = embedding_re.match(line)
+            if m is not None:
+                assert pkg is not None
+                if m[2] is not None:
+                    # release
+                    pass
+                embedding_pkg = m[3]
+                status = m[4]
+                sort = m[7]
+                pkgs[pkg][embedding_pkg] = (status, sort)
+                continue
+            m = note_re.match(line)
+            if m is not None:
+                assert pkg is not None
+                assert embedding_pkg is not None
+                notes.setdefault(pkg, dict())
+                notes[pkg][embedding_pkg] = m[1]
+                continue
+            print("%s: %d: Failed to parse: '%s'" % (filepath, linenum, line), file=sys.stderr)
+    data["pkgs"] = pkgs
+    data["notes"] = notes
+    return data
+
+# also parse debian's embedded-code-copies and amalgate that into data
+config = cve_lib.read_config()
+debian_embedded_copies = os.path.join(config['secure_testing_path'], "data", "embedded-code-copies")
+code_copies = parse_embedded_code_copies(debian_embedded_copies)
+for pkg in code_copies["pkgs"]:
+    data.setdefault(pkg, {"aliases": [],
+                          "tags": [],
+                          "notes": [],
+                          "pkgs": {}})
+    for embedding_pkg in code_copies["pkgs"][pkg]:
+        data[pkg]["pkgs"].setdefault(embedding_pkg, ("needs-triage", ""))
+        # use the sort to create a more informative description of the
+        # relationship between pkg and embedding_pkg
+        status, sort = code_copies["pkgs"][pkg][embedding_pkg]
+        template = "%s embeds a copy of %s"
+        if sort == "static":
+            template = "%s statically links against %s so needs to be rebuilt"
+        elif sort == "modified-embed":
+            template = "%s embeds a modified copy of %s so should be checked if is affected"
+        elif sort == "fork":
+            template = "%s contains a fork of %s so should be checked if is affected"
+        elif sort == "old-version":
+            template = "%s contains an older version of %s so should be checked if is affected"
+        note = template % (embedding_pkg, pkg)
+        if pkg in code_copies["notes"] and embedding_pkg in code_copies["notes"][pkg]:
+            note = note + " - " + code_copies["notes"][pkg][embedding_pkg]
+        data[pkg]["notes"].append(("dst", note))
diff --git a/scripts/check-cves b/scripts/check-cves
index fa6093d..963f7d3 100755
--- a/scripts/check-cves
+++ b/scripts/check-cves
@@ -34,7 +34,6 @@ import xml.sax
 import xml.sax.handler
 import xml.sax.xmlreader
 from html import escape
-from functools import reduce
 import progressbar
 
 import cve_lib
@@ -75,9 +74,10 @@ for release in list(source.keys()):
 # likely to sometimes contain these
 common_words = ['an', 'and', 'context', 'file', 'modules', 'the', 'when']
 allsrcs.difference_update(set(common_words))
-# add boilerplate names too so we can get mysql or postgresql
-# etc even if they don't exist as source package names
-allsrcs.update(set(cve_lib.load_boilerplates().keys()))
+# add names and aliases etc from package-db
+allsrcs.update(set(cve_lib.package_db.keys()))
+for pkg in cve_lib.package_db:
+    allsrcs.update(set(cve_lib.package_db[pkg]["aliases"]))
 
 built_using_map = None
 
@@ -124,24 +124,8 @@ def subtract_list(list1, list2):
             list1.remove(item)
 
 
-def wordwrap(text, width):
-    """
-    A word-wrap function that preserves existing line breaks
-    and most spaces in the text. Expects that existing line
-    breaks are posix newlines (\n).
-    """
-    return reduce(lambda line, word, width=width: '%s%s%s' %
-                  (line,
-                   ' \n'[(len(line) - line.rfind('\n') - 1 +
-                          len(word.split('\n', 1)[0]
-                              ) >= width)],
-                   word),
-                  text.split(' ')
-                 )
-
-
 def _wrap_desc(desc):
-    return wordwrap(desc, 75).replace(' \n', '\n')
+    return cve_lib.wrap_text(desc)
 
 def _spawn_editor(path):
     editor = os.getenv('EDITOR', 'vi')
@@ -164,104 +148,6 @@ def prompt_user(msg):
         print(msg, flush=True, end='')
 
 
-def add_CVE_to_tracker(cve, info, packages, priority=None, bug_urls=[], ref_urls=[]):
-    src = '%s/active/00boilerplate' % (destdir)
-
-    # Use the first boilerplate for template
-    first_boiler = ""
-    for b in packages:
-        if os.path.exists(src + "." + b):
-            src = src + "." + b
-            first_boiler = src
-            break
-    dst = '%s/active/%s' % (destdir, cve)
-    with open(src) as f:
-        template = f.readlines()
-    cve_file = open(dst, 'w')
-    orig_priority = ""
-    for line in template:
-        line = line.rstrip()
-        if line.startswith('Candidate:'):
-            print('Candidate: %s' % (cve), file=cve_file)
-        elif info['public'] and line.startswith('PublicDate:'):
-            print('PublicDate: %s' % (info['public']), file=cve_file)
-        elif info['cvss'] and line.startswith('CVSS:'):
-            print('CVSS:', file=cve_file)
-            for entry in info['cvss']:
-                print(' %s: %s [%s %s]' % (entry['source'], entry['vector'], entry['baseScore'], entry['baseSeverity']), file=cve_file)
-        elif line.startswith('References:'):
-            print('References:\n https://cve.mitre.org/cgi-bin/cvename.cgi?name=%s' % (cve), file=cve_file)
-            for i in ref_urls:
-                print(" %s" % i, file=cve_file)
-        elif line.startswith('Bugs:'):
-            print(line, file=cve_file)
-            for i in bug_urls:
-                print(" %s" % i, file=cve_file)
-        elif line.startswith('Priority:'):
-            orig_priority = line.split()[1]
-            if priority:
-                print('Priority: %s' % priority, file=cve_file)
-            else:
-                print(line, file=cve_file)
-        elif not line.startswith('#'):
-            print(line, file=cve_file)
-
-        if line.startswith('Description:'):
-            for desc_line in _wrap_desc(info['desc']).split('\n'):
-                print(" %s" % (desc_line), file=cve_file)
-
-    # Now add package information (with Priority_<pkg>) from other boilers
-    if len(packages) > 1:
-        for p in packages:
-            skip_emptyline = False
-            boiler = '%s/active/00boilerplate.%s' % (destdir, p)
-            if os.path.exists(boiler) and boiler != first_boiler:
-                with open(boiler) as f:
-                    template = f.readlines()
-                in_note_section = False
-                for line in template:
-                    line = line.rstrip()
-                    # handle notes in templates
-                    if in_note_section:
-                        if line.startswith(' '):
-                            continue
-                        else:
-                            in_note_section = False
-                    if line.startswith('Notes:'):
-                        in_note_section = True
-                        continue
-                    if line.startswith('Candidate:') or \
-                       line.startswith('PublicDate:') or \
-                       line.startswith('References:') or \
-                       line.startswith('Description:') or \
-                       line.startswith('Ubuntu-Description:') or \
-                       line.startswith('Bugs:') or \
-                       line.startswith('Discovered-by:') or \
-                       line.startswith('Assigned-to:') or \
-                       line.startswith('#'):
-                        continue
-                    if line.startswith('Priority:'):
-                        new_priority = line.split()[1]
-                        if priority:
-                            continue
-                        elif new_priority == orig_priority:
-                            continue
-                        elif orig_priority == "":
-                            print(line, file=cve_file)
-                        else:
-                            print('Priority_%s: %s' % (p, new_priority), file=cve_file)
-                            skip_emptyline = True
-                    elif skip_emptyline and line == "":
-                        skip_emptyline = False
-                        continue
-                    else:
-                        print(line, file=cve_file)
-                print("")
-
-    cve_file.close()
-
-    return dst
-
 
 class PercentageFile(object):
     def __init__(self, filename):
@@ -1371,110 +1257,114 @@ class CVEHandler(xml.sax.handler.ContentHandler):
         print('%s %s %s\n%s' % (action, cve, data, desc))
 
     def add_cve(self, cve, packages, priority=None):
-            # remove from not-for-us.txt if adding and ensure we remove any
-            # mistriaged_hint from the description
-            if cve in CVEIgnoreNotForUsList:
-                cmd = ['sed', '-i', '/^%s #.*$/d' % cve, './ignored/not-for-us.txt']
-                subprocess.call(cmd)
-                self.cve_data[cve]['desc'] = self.cve_data[cve]['desc'].replace(mistriaged_hint, '')
-
-            # Build up list of reference urls
-            ref_urls = []
+        # remove from not-for-us.txt if adding and ensure we remove any
+        # mistriaged_hint from the description
+        if cve in CVEIgnoreNotForUsList:
+            cmd = ['sed', '-i', '/^%s #.*$/d' % cve, './ignored/not-for-us.txt']
+            subprocess.call(cmd)
+            self.cve_data[cve]['desc'] = self.cve_data[cve]['desc'].replace(mistriaged_hint, '')
+
+        # Build up list of reference urls
+        ref_urls = []
+        if self.debian and \
+           cve in self.debian and \
+           'note' in self.debian[cve]:
+            for line in self.debian[cve]['note']:
+                tmp = line.lstrip("NOTE: ")
+                if tmp.startswith("http"):
+                    ref_urls.append(tmp)
+        if 'refs' in self.cve_data[cve]:
+            for ref in self.cve_data[cve]['refs']:
+                url = ""
+                if ref[1].strip().startswith("http"):
+                    url = ref[1].strip()
+                elif ref[2] is not None and ref[2].strip().startswith("http"):
+                    url = ref[2].strip()
+                else:  # no urls
+                    continue
+
+                if '//' not in url:  # invalid url
+                    continue
+
+                # ignore certain reference URLs which we don't use
+                ignored_urls = ['www.securityfocus.com', 'www.osvdb.org']
+                if url.split('//')[1].split('/')[0] in ignored_urls:
+                    continue
+
+                if url not in ref_urls:
+                    ref_urls.append(url)
+
+        # Build up list of bug urls
+        bug_urls = []
+        for pkg in packages:
             if self.debian and \
                cve in self.debian and \
-               'note' in self.debian[cve]:
-                for line in self.debian[cve]['note']:
-                    tmp = line.lstrip("NOTE: ")
-                    if tmp.startswith("http"):
-                        ref_urls.append(tmp)
-            if 'refs' in self.cve_data[cve]:
-                for ref in self.cve_data[cve]['refs']:
-                    url = ""
-                    if ref[1].strip().startswith("http"):
-                        url = ref[1].strip()
-                    elif ref[2] is not None and ref[2].strip().startswith("http"):
-                        url = ref[2].strip()
-                    else:  # no urls
-                        continue
-
-                    if '//' not in url:  # invalid url
-                        continue
-
-                    # ignore certain reference URLs which we don't use
-                    ignored_urls = ['www.securityfocus.com', 'www.osvdb.org']
-                    if url.split('//')[1].split('/')[0] in ignored_urls:
-                        continue
-
-                    if url not in ref_urls:
-                        ref_urls.append(url)
-
-            # Build up list of bug urls
-            bug_urls = []
-            for pkg in packages:
-                if self.debian and \
-                   cve in self.debian and \
-                   self.debian[cve]['pkgs'] and \
-                   pkg in self.debian[cve]['pkgs']:
-                    bug = None
-                    if self.debian[cve]['pkgs'][pkg]['priority'] and \
-                       re.search(r'^bug #[0-9]+$', self.debian[cve]['pkgs'][pkg]['priority']):
-                        bug = self.debian[cve]['pkgs'][pkg]['priority'].split('#')[1]
-                    elif self.debian[cve]['pkgs'][pkg]['bug']:
-                        bug = self.debian[cve]['pkgs'][pkg]['bug']
-                    if bug:
-                        url = "http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%s" % bug
-                        if url not in bug_urls:
-                            bug_urls.append(url)
-
-            # Add to tracker from 00boilerplate
-            dst = add_CVE_to_tracker(cve, self.cve_data[cve], packages, priority, bug_urls, ref_urls)
-
-            # Build up command line
-            cmd = ['./scripts/active_edit', '-c', cve, '--yes']
-
-            # capture debian not-affected states
-            not_affected = []
-
-            for pkg in packages:
-                # The Debian convention is to specify the fixed version as the state
-                # with the bug number as the priority for fixed bugs. Leverage this
-                # with active_edit
-                fixed_in = ""
-                if self.debian and \
-                   cve in self.debian and \
-                   self.debian[cve]['pkgs'] and \
-                   pkg in self.debian[cve]['pkgs'] and \
-                   self.debian[cve]['pkgs'][pkg]['state']:
-                    if re.search(r'^[0-9]', self.debian[cve]['pkgs'][pkg]['state']):
-                        fixed_version = self.debian[cve]['pkgs'][pkg]['state']
-                        fixed_in = ",%s" % fixed_version
-
-                        # Now see if we can correlate this to an Ubuntu version
-                        answer = source_map.madison(source, pkg)
-                        for name in sorted(answer.keys()):
-                            rel = name.split('/')[0].split('-')[0]  # don't care about the pocket
-                            version = answer[name][pkg]
-                            # Try to compare apples to apples. Ie, if one of us has
-                            # an epoch and the other doesn't, don't try to be smart
-                            if (':' not in version and ':' not in fixed_version) or \
-                               (':' in version and ':' in fixed_version):
-                                if dpkg_compare_versions(version, 'ge', fixed_version):
-                                    if rel == cve_lib.devel_release:
-                                        rel = 'devel'
-                                    fixed_in += ",%s,%s" % (rel, version)
-                                    break
-                    elif self.debian[cve]['pkgs'][pkg]['state'].startswith('<not-affected>') and \
-                         len(self.debian[cve]['pkgs'][pkg]['priority']) > 0:
-                        # capture that debian believes their version is unaffected
-                        not_affected.append((pkg, "debian: %s" % self.debian[cve]['pkgs'][pkg]['priority']))
-                cmd += ['-p', "%s%s" % (pkg, fixed_in)]
-
+               self.debian[cve]['pkgs'] and \
+               pkg in self.debian[cve]['pkgs']:
+                bug = None
+                if self.debian[cve]['pkgs'][pkg]['priority'] and \
+                   re.search(r'^bug #[0-9]+$', self.debian[cve]['pkgs'][pkg]['priority']):
+                    bug = self.debian[cve]['pkgs'][pkg]['priority'].split('#')[1]
+                elif self.debian[cve]['pkgs'][pkg]['bug']:
+                    bug = self.debian[cve]['pkgs'][pkg]['bug']
+                if bug:
+                    url = "http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=%s" % bug
+                    if url not in bug_urls:
+                        bug_urls.append(url)
+
+
+        # Build up command line
+        cmd = ['./scripts/active_edit', '-c', cve, '--yes']
+        for url in ref_urls:
+            cmd.extend(['--reference-url', url])
+        for url in bug_urls:
+            cmd.extend(['--bug-url', url])
+        if priority:
+            cmd.extend(['--priority', priority])
+
+        # capture debian not-affected states
+        not_affected = []
+
+        for pkg in packages:
+            # The Debian convention is to specify the fixed version as the state
+            # with the bug number as the priority for fixed bugs. Leverage this
+            # with active_edit
+            fixed_in = ""
+            if self.debian and \
+               cve in self.debian and \
+               self.debian[cve]['pkgs'] and \
+               pkg in self.debian[cve]['pkgs'] and \
+               self.debian[cve]['pkgs'][pkg]['state']:
+                if re.search(r'^[0-9]', self.debian[cve]['pkgs'][pkg]['state']):
+                    fixed_version = self.debian[cve]['pkgs'][pkg]['state']
+                    fixed_in = ",%s" % fixed_version
+
+                    # Now see if we can correlate this to an Ubuntu version
+                    answer = source_map.madison(source, pkg)
+                    for name in sorted(answer.keys()):
+                        rel = name.split('/')[0].split('-')[0]  # don't care about the pocket
+                        version = answer[name][pkg]
+                        # Try to compare apples to apples. Ie, if one of us has
+                        # an epoch and the other doesn't, don't try to be smart
+                        if (':' not in version and ':' not in fixed_version) or \
+                           (':' in version and ':' in fixed_version):
+                            if dpkg_compare_versions(version, 'ge', fixed_version):
+                                if rel == cve_lib.devel_release:
+                                    rel = 'devel'
+                                fixed_in += ",%s,%s" % (rel, version)
+                                break
+                elif self.debian[cve]['pkgs'][pkg]['state'].startswith('<not-affected>') and \
+                        len(self.debian[cve]['pkgs'][pkg]['priority']) > 0:
+                    # capture that debian believes their version is unaffected
+                    not_affected.append((pkg, "debian: %s" % self.debian[cve]['pkgs'][pkg]['priority']))
+            cmd += ['-p', "%s%s" % (pkg, fixed_in)]
+
+        subprocess.call(cmd)
+        for (pkg, reason) in not_affected:
+            cmd = ['./scripts/mass-cve-edit', '-p', pkg, '-r', 'upstream', '-s', 'not-affected', '-v', reason, cve]
             subprocess.call(cmd)
-            for (pkg, reason) in not_affected:
-                cmd = ['./scripts/mass-cve-edit', '-p', pkg, '-r', 'upstream', '-s', 'not-affected',  '-v', reason, cve]
-                subprocess.call(cmd)
-            self.num_added += 1
-            return dst
+        self.num_added += 1
+        return './active/%s' % cve
 
     def unembargo_cve(self, cve):
         # unembargo a cve now public
diff --git a/scripts/check-syntax b/scripts/check-syntax
index 9501d58..234c71a 100755
--- a/scripts/check-syntax
+++ b/scripts/check-syntax
@@ -279,7 +279,7 @@ if len(args) == 0:
         with open(opt.filelist) as fh:
             for line in fh:
                 for dir in check_dirs:
-                    if line.startswith("%s/CVE-" % dir) or line.startswith("%s/00boilerplate" % dir):
+                    if line.startswith("%s/CVE-" % dir):
                         args += [line.rstrip()]
     elif opt.modified:
         if opt.debug:
@@ -299,11 +299,7 @@ if len(args) == 0:
                     args += [filename]
     else:
         for dir in check_dirs:
-            # XXX might want to separate out boilerplate checks
-            # as they might need to be less restrictive than regular CVE
-            # file checks
-            for cve in sorted(glob.glob("%s/CVE-*" % dir) +
-                              glob.glob("%s/00boilerplate.*" % dir)):
+            for cve in sorted(glob.glob("%s/CVE-*" % dir)):
                 args += [cve]
 else:
     all_files = False
@@ -404,7 +400,7 @@ for cve in args:
             cve_okay = False
 
     # verify candidate field matches the CVE file name
-    if "stdin" not in cve and "boilerplate" not in cve and not data["Candidate"] == cve:
+    if "stdin" not in cve and not data["Candidate"] == cve:
         filename = srcmap["Candidate"][0]
         linenum = srcmap["Candidate"][1]
         print(
@@ -424,10 +420,9 @@ for cve in args:
         # place the generated error message on this release's line etc
         nearby_rel = list(listed_releases - missing_releases)[0]
         for rel in missing_releases:
-            # only warn on active CVEs but don't warn on boilerplate entries missing external
-            # releases as this is not supported
+            # only warn on active CVEs
             if is_active(cve) and \
-               ("boilerplate" not in cve or rel not in cve_lib.external_releases) and \
+               rel not in cve_lib.external_releases and \
                rel in source and pkg in source[rel]:
                 filename = srcmap["pkgs"][pkg][nearby_rel][0]
                 linenum = srcmap["pkgs"][pkg][nearby_rel][1]
@@ -658,10 +653,8 @@ for cve in args:
                 )
                 cve_okay = False
 
-    # Verify priority for any CVE with a supported package and when this is
-    # not boilerplate
+    # Verify priority for any CVE with a supported package
     if (
-        "boilerplate" not in cve and
         len(supported)
         and (is_active(cve) or is_embargoed(cve))
         and ("Priority" not in data or data["Priority"] not in cve_lib.priorities)
@@ -777,10 +770,10 @@ for cve in args:
                     cve_okay = False
 
     # Either PublicDate or CRD must be set to something
-    if ("boilerplate" not in cve and
-        ("PublicDate" not in data or data["PublicDate"] == "") and (
-        "CRD" not in data or data["CRD"] == ""
-    )):
+    if (
+            "PublicDate" not in data or data["PublicDate"] == "" and
+            "CRD" not in data or data["CRD"] == ""
+    ):
         key = "PublicDate" if "PublicDate" in srcmap else "CRD"
         filename = srcmap[key][0]
         linenum = srcmap[key][1]
diff --git a/scripts/check-syntax-fixup b/scripts/check-syntax-fixup
index 24bf608..a38ad1e 100755
--- a/scripts/check-syntax-fixup
+++ b/scripts/check-syntax-fixup
@@ -147,11 +147,6 @@ for line in args.infile:
         # remove this hard-coded hack one-day...
         if rel in cve_lib.external_releases or \
            (rel == "trusty/esm" and "DOES exist" in msg):
-            # ignore boilerplate files for external_releases
-            if "boilerplate" in cve:
-                # print unhandled lines
-                print(line, file=sys.stderr)
-                continue
             cve = os.path.join(
                 cve_lib.get_external_subproject_cve_dir(rel), os.path.basename(cve)
             )
diff --git a/scripts/cve-mode.el b/scripts/cve-mode.el
index b7413b0..0992e4c 100644
--- a/scripts/cve-mode.el
+++ b/scripts/cve-mode.el
@@ -333,7 +333,7 @@ Queries umt for PACKAGE as well as looking in overlay files."
 (defun cve-mode-insert-package (package)
   "Add PACKAGE as affected to this CVE.
 Calls out to active_edit to do the heavy lifting so that
-boilerplate entries are handled automatically."
+meta_lists/package-db.json entries are handled automatically."
   (interactive
    (list (completing-read "Package: " cve-mode--source-packages)))
   (let ((args (list "-c" (file-name-base (buffer-file-name))
@@ -683,8 +683,6 @@ cross boundaries of block literals."
 
 ;;;###autoload
 (add-to-list 'auto-mode-alist '("CVE-[[:digit:]]\\{4\\}-[[:digit:]]\\{4,\\}\\'" . cve-mode))
-;;;###autoload
-(add-to-list 'auto-mode-alist '("00boilerplate.*\\'" . cve-mode))
 
 (provide 'cve-mode)
 ;;; cve-mode.el ends here
diff --git a/scripts/cve.vim b/scripts/cve.vim
index bc89578..a13b158 100644
--- a/scripts/cve.vim
+++ b/scripts/cve.vim
@@ -7,7 +7,6 @@
 " $ ln -s $UCT/scripts/cve.vim ~/.vim/syntax/cve.vim
 " Add to ~/.vimrc:
 " autocmd BufNewFile,BufRead CVE-[0-9][0-9][0-9][0-9]-[0-9N]\\\{4,\} set syntax=cve
-" autocmd BufNewFile,BufRead 00boilerplate.* set syntax=cve
 "
 " TODO:
 " - turn the release names into variables so we only have to update in one
diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py
index fe8e4f5..60da772 100755
--- a/scripts/cve_lib.py
+++ b/scripts/cve_lib.py
@@ -23,6 +23,8 @@ import cache_urllib
 import json
 import yaml
 
+from functools import reduce
+
 def set_cve_dir(path):
     '''Return a path with CVEs in it. Specifically:
        - if 'path' has CVEs in it, return path
@@ -912,11 +914,33 @@ kernel_srcs = set(['linux',
 kernel_topic_branches = kernel_srcs.difference(['linux'])
 
 # for sanity, try to keep these in alphabetical order in the json file
-def load_package_info_overrides(list_dir):
-    package_info_overrides = dict()
-    with open(os.path.join(list_dir, "package_info_overrides.json")) as _file:
-        package_info_overrides = json.load(_file)
-    return package_info_overrides
+def load_package_db(dir=meta_dir):
+    pkg_db = {}
+    pkg_db_json = os.path.join(dir, "package-db.json")
+    try:
+        with open(pkg_db_json, "r", encoding='utf-8') as fp:
+            pkg_db = json.load(fp)
+            # add lookups based on aliases - we can't iterate over pkg_db and
+            # modify it so collect aliases then add them manually
+            alias_info = {}
+            for p in pkg_db:
+                try:
+                    aliases = pkg_db[p]["aliases"]
+                    if len(aliases) > 0:
+                        alias_info[p] = aliases
+                except KeyError:
+                    pass
+            for p in alias_info.keys():
+                for a in alias_info[p]:
+                    if a not in pkg_db:
+                        # use original info if already in pkg_db
+                        pkg_db[a] = pkg_db[p]
+    except FileNotFoundError:
+        # TODO: remove this exception handling once we have a package-db.json
+        # checked into UCT git since in that case this should never occur so it
+        # should be a fatal error if it is missing
+        pass
+    return pkg_db
 
 
 # "arch_list" is all the physical architectures buildable
@@ -1226,19 +1250,19 @@ def release_is_older_than(release_a, release_b):
 
 
 
-package_info_overrides = load_package_info_overrides(meta_dir)
+package_db = load_package_db()
 
 def lookup_package_override_title(source):
-    global package_info_overrides
-    res = package_info_overrides.get(source)
+    global package_db
+    res = package_db.get(source)
     if isinstance(res, dict):
         return(res.get("title"))
 
     return None
 
 def lookup_package_override_description(source):
-    global package_info_overrides
-    res = package_info_overrides.get(source)
+    global package_db
+    res = package_db.get(source)
     if isinstance(res, dict):
         return(res.get("description"))
 
@@ -1964,9 +1988,6 @@ def load_cve(cve, strict=False, srcmap=None):
         nonempty = ['Candidate']
         if strict:
             nonempty += ['PublicDate']
-        # boilerplate files are special and can (should?) be empty
-        if "boilerplate" in cve:
-            nonempty = []
 
         if field not in data or field not in fields_seen:
             msg += "%s: %d: missing field '%s'\n" % (cve, linenum, field)
@@ -2008,17 +2029,6 @@ def load_cve(cve, strict=False, srcmap=None):
         raise ValueError(msg.strip())
     return data
 
-def load_boilerplates(verbose=False):
-    boilerplates = dict()
-    prefix = "00boilerplate."
-    for bp in glob.glob(os.path.join(active_dir, prefix + "*")):
-        # TODO - should we differentiate on symlinks so we can tell which
-        # are the "real" primary ones and which ones are secondary?
-        name = bp[bp.find(prefix) + len(prefix):]
-        info = load_cve(bp)
-        boilerplates.setdefault(name, info)
-    return boilerplates
-
 def load_all(cves, uems, rcves=[]):
     table = dict()
     priority = dict()
@@ -2865,3 +2875,23 @@ def parse_cvss(cvss):
     js['baseMetricV3']['exploitabilityScore'] = round(exploitability * 10) / 10
     js['baseMetricV3']['impactScore'] = round(impact * 10) / 10
     return js
+
+def wordwrap(text, width):
+    """
+    A word-wrap function that preserves existing line breaks
+    and most spaces in the text. Expects that existing line
+    breaks are posix newlines (\n).
+    """
+    return reduce(lambda line, word, width=width:
+                  '%s%s%s' %
+                  (line,
+                   ' \n'[(len(line) - line.rfind('\n') - 1 + len(word.split('\n', 1)[0]) >= width)],
+                   word),
+                  text.split(' ')
+                  )
+
+def wrap_text(text, width=75):
+    """
+    Wrap text to width chars wide.
+    """
+    return wordwrap(text, width).replace(' \n', '\n')
diff --git a/scripts/dup-status-for-pkg b/scripts/dup-status-for-pkg
index fc32055..c15d669 100755
--- a/scripts/dup-status-for-pkg
+++ b/scripts/dup-status-for-pkg
@@ -24,7 +24,7 @@ fi
 for pkg in "$@"; do
   echo "Duplicating status from ${src} to ${dst} for ${pkg} where was DNE for ${dst}"
   # find files to change
-  for file in $(grep -l '^'"${dst}_${pkg}"': DNE' active/00boilerplate* active/CVE-* retired/CVE-*); do
+  for file in $(grep -l '^'"${dst}_${pkg}"': DNE' active/CVE-* retired/CVE-*); do
     status=$(grep '^'"${src}_${pkg}"': ' "${file}" | cut -f2- -d' ')
     if [ ! -z "${status}" ]; then
       echo "Duplicating status ${status} from ${src} to ${dst} in ${file}"
diff --git a/scripts/kernel-triage-missing-break-fix b/scripts/kernel-triage-missing-break-fix
index e60cb38..e21a949 100755
--- a/scripts/kernel-triage-missing-break-fix
+++ b/scripts/kernel-triage-missing-break-fix
@@ -35,7 +35,7 @@ if [ -z "${debian_kernel_cve_tracker}" ] ; then
     echo
 fi
 
-_CVES=$(cd "$UCT" && grep -lr '^Patches_linux:' --exclude '*boilerplate*' active/ | xargs grep -L '^ break-fix:' | sort)
+_CVES=$(cd "$UCT" && grep -lr '^Patches_linux:' active/ | xargs grep -L '^ break-fix:' | sort)
 for _CVE in ${_CVES} ; do
     echo "${_CVE}"
     CVE="${_CVE##active/}"
diff --git a/scripts/release-cycle-devel-opens b/scripts/release-cycle-devel-opens
index fccd5af..cf08707 100755
--- a/scripts/release-cycle-devel-opens
+++ b/scripts/release-cycle-devel-opens
@@ -18,7 +18,7 @@ if ! [ -d embargoed/ ]; then
     echo "Unable to find embargoed directory, is it set up?" >&2
 fi
 
-for f in active/{CVE-,00boilerplate}* embargoed/CVE-* ; do
+for f in active/CVE-* embargoed/CVE-* ; do
     if egrep -q "^(#?)devel_" "$f" ; then
         echo "DEBUG: skipping $f"
         continue
diff --git a/scripts/release-cycle-released b/scripts/release-cycle-released
index b5cfc7a..c6d844f 100755
--- a/scripts/release-cycle-released
+++ b/scripts/release-cycle-released
@@ -22,7 +22,7 @@ if ! [ -d active/ ]; then
     exit 2
 fi
 
-for f in active/{00boilerplate,CVE-}*; do
+for f in active/CVE-*; do
   # ignore symlinks so we don't replace twice in the same underlying file
   if [ ! -L "$f" ]; then
     perl -pi -e 's/^((#?)devel_(.*))/${2}'"$REL"'_$3\n$1/g' "$f"
diff --git a/scripts/sync-from-eol.py b/scripts/sync-from-eol.py
index b2a020c..dd6d9eb 100755
--- a/scripts/sync-from-eol.py
+++ b/scripts/sync-from-eol.py
@@ -24,16 +24,12 @@ import sys
 import cve_lib
 import source_map
 
-import warnings
-warnings.filterwarnings('ignore', 'apt API not stable yet', FutureWarning)
-import apt
 
 parser = optparse.OptionParser()
 parser.add_option("-r", "--release", dest="release", default=None, help="release to modify")
 parser.add_option("-W", "--whole", dest="whole", help="End of life the whole release", action='store_true')
 parser.add_option("-U", "--universe", dest="universe", help="Modify packages in universe and multiverse", action='store_true')
 parser.add_option("-u", "--update", dest="update", help="Update CVEs with released package versions", action='store_true')
-parser.add_option("-b", "--include-boilerplate", help="Update boilerplate files", action='store_true')
 (opt, args) = parser.parse_args()
 
 if not opt.release:
@@ -52,16 +48,13 @@ pkgs = source_map.load(releases=[opt.release], skip_eol_releases=False)
 
 cves = glob.glob('%s/CVE-*' % cve_lib.active_dir)
 
-if opt.include_boilerplate:
-    cves += glob.glob('%s/00boilerplate.*' % cve_lib.active_dir)
-
 if os.path.islink('embargoed'):
     cves += glob.glob('embargoed/CVE-*')
     cves += glob.glob('embargoed/EMB-*')
 
 for filename in cves:
-    # we don't want to edit boilerplate symlinks as that will cause them
-    # to become unsymlinked
+    # we don't want to edit symlinks as that will cause them to become
+    # unsymlinked
     if os.path.islink(filename):
         continue
 
@@ -88,17 +81,13 @@ for filename in cves:
                         and cve_lib.is_active_esm_release(cve_lib.get_orig_rel_name(opt.release))
                     ):
                         status = data['pkgs'][src][opt.release]
-                        if cve.startswith('00boilerplate.'):
-                            cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support')
-                        elif status[1] != '':
+                        if status[1] != '':
                             cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support, was %s [%s]' % (status[0], status[1]))
                         else:
                             cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of ESM support, was %s' % (status[0]))
                     elif 'LTS' in cve_lib.release_name(opt.release):
                         status = data['pkgs'][src][opt.release]
-                        if cve.startswith('00boilerplate.'):
-                            cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support')
-                        elif status[1] != '':
+                        if status[1] != '':
                             cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support, was %s [%s]' % (status[0], status[1]))
                         else:
                             cve_lib.update_state(filename, src, opt.release, 'ignored', 'end of standard support, was %s' % (status[0]))