Merge ~litios/ubuntu-cve-tracker:sync-from-usns-extract-cves into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

USN-3777-1 was triggering an error in sync-from-usns due to the last paragraph.

If the package is a type of kernel, the function extract_cve_descriptions is called to extract the descriptions. This function expects the CVE list to always be specified at the end of the paragraph.

This PR addresses this issue, allowing for the CVE list to be specified in the middle of the description too, as in USN-3777-1.

As of today, we are manually ignoring these issues by specifying to skip parsing them in meta_lists/ignored-modified-USN-descriptions.txt

Testing: https://pastebin.canonical.com/p/mzCVXKFG45/

[Spyros Seimenis (sespiros)]

Some comments (+ inline comments).

Since that piece of code is supposed to now handle CVE lists inside a "chunk", please remove the "Extract trailing ..." comment or replace with "Extract CVE list from chunk".

I am also wondering if it would be a better approach to change this code to use a regex to extract a "(CVE-XX, CVE-XX)" chunk from the text instead of manually going through the USN text chunk and doing splitting, picking parts here and there. I think it would improve readability + maintainability.

Thanks for pasting some test cases, made the review much easier than trying to figure out weird cases by myself. At this point I am wondering if it would be worth it to convert those test cases to a proper test but maybe this should not be part of this MP :)

review: Needs Fixing

[David Fernandez Gonzalez (litios)]

Hey Spyros, thanks for all the input. I did some changes:

* Refactor the code to work with regex.
* Added tests
* In order to be able to test, I had to move everything inside the main function so the module could be imported during the tests.

Let me know if it looks good!

[Spyros Seimenis (sespiros)]

Hi David, thanks for the changes and the tests! I think overall it reads much better now. A couple of final comments inline.

- (nitpick) you could tighten the CVE regex to CVE-\d{4}-\d{4,7}
- (nitpick) cve_list maybe would read better than cves_chunk as a variable name

Also it seems that the CI tests are failing, not sure if it's because of this or transient error, you can run these locally with

sudo snap install --classic lpci
cd $UCT
lpci

[David Fernandez Gonzalez (litios)]

Thanks for the input, it was useful! I added your suggestions to a new commit.

Regarding CI/CD, it is failing but that's because of the check-cves job, the unit-tests is passing which is the one running the test from this PR, so it should be good but let me know if you think there is something else wrong:

============ 869 passed, 1 skipped, 32 warnings in 60.82s (0:01:00) ============

[Spyros Seimenis (sespiros)]

So these comments may have been a bit unrelated to the PR but I just found the way we are trying to catch user error confusing in that code. This applies both to the case where we were trying to check for example if '(CVE' exists in the chunk and show warning and the one when we try to spot '..' at the end and fix it rather than warn.

[David Fernandez Gonzalez (litios)]

So I agree with the '(CVE' part and I'm currently changing this, I'm testing locally before pushing but I think the '..' it's a different scenario because that's an issue we are creating when removing the CVE list and thus we should be the ones handling it on the code.

[Spyros Seimenis (sespiros)]

review: Approve

diff --git a/.launchpad.yaml b/.launchpad.yaml
index a0a847c..a089c1d 100644
--- a/.launchpad.yaml
+++ b/.launchpad.yaml
@@ -36,7 +36,7 @@ jobs:
       rm -f embargoed
       mkdir embargoed
       echo "Running unit tests..."
-      pytest-3 ./scripts/test_cve_lib.py ./scripts/test_kernel_lib.py ./scripts/test_usn_lib.py ./scripts/test_source_map.py ./scripts/test_publish-cves-to-website-api.py ./test/test_oval_lib_unit.py
+      pytest-3 ./scripts/test_cve_lib.py ./scripts/test_kernel_lib.py ./scripts/test_usn_lib.py ./scripts/test_source_map.py ./scripts/test_publish-cves-to-website-api.py ./scripts/test_sync_from_usns.py ./test/test_oval_lib_unit.py
       # ideally we would also run ./scripts/check-cves --test here as well as it
       # is more of a unit test but it requires packages-mirror so run it as part
       # of check-syntax below
diff --git a/scripts/sync-from-usns.py b/scripts/sync-from-usns.py
index ee138b5..b05463e 100755
--- a/scripts/sync-from-usns.py
+++ b/scripts/sync-from-usns.py
@@ -27,46 +27,7 @@ import usn_lib
 
 from source_map import version_compare, load
 
-cves = dict()
-
-config = cve_lib.read_config()
-
-parser = argparse.ArgumentParser(description="Sync cve status from USN database")
-parser.add_argument("--usn", help="Limit report/update to a single USN", metavar="USN", default=None)
-parser.add_argument("-u", "--update", help="Update CVEs with released package versions", action='store_true')
-parser.add_argument("-U", "--use-usn", help="Use the version in the USN if it differs from the version in UCT", action='store_true')
-parser.add_argument("-v", "--verbose", help="Report logic while processing USNs", action='store_true')
-parser.add_argument("-d", "--debug", help="Report additional debugging while processing USNs", action='store_true')
-parser.add_argument("-r", "--retired", help="Process retired CVEs in addition to active ones", action='store_true')
-parser.add_argument('-g', "--git-stage", help="When updating, stage for commit by adding to git's index (requires --update)", action='store_true')
-parser.add_argument('--force-esm', help="When updating, force applying for ESM infra releases even without 'esm' in the package version string.", action='store_true')
-parser.add_argument("database", nargs="?", help="Use alternate USN database (default: %(default)s)", action='store', default=config['usn_db_copy'])
-args = parser.parse_args()
-
-if args.git_stage:
-    if not args.update:
-        print('--git-stage option requires --update as well, exiting', file=sys.stderr)
-        exit(1)
-    if not cve_lib.git_is_tree_clean(debug=True):
-        print('Please commit or stash your existing changes to UCT first. Aborting.',
-              file=sys.stderr)
-        exit(1)
-
-if args.force_esm and not args.usn:
-        print('--force-esm option requires a specific usn to operate on, exiting.', file=sys.stderr)
-        exit(1)
-
-if args.debug:
-    print("Loading %s ..." % (args.database), file=sys.stderr)
-reverted = usn_lib.get_reverted()
-ignored_description = usn_lib.get_ignored_description()
-db = usn_lib.load_database(args.database)
-usnlist = [args.usn]
-if not args.usn:
-    usnlist = db
-
-
-def extract_cve_descriptions(usn, usnnum):
+def extract_cve_descriptions(usn, usnnum, verbose):
     descriptions = dict()
     cves = set()
     for cve in usn.get('cves', []):
@@ -91,242 +52,281 @@ def extract_cve_descriptions(usn, usnnum):
 
     # Drop un-parened USN qualifiers
     affected = re.compile(' (Only )?Ubuntu [^ ]+( LTS)?(, (and )?Ubuntu [^ ]+( LTS)?)? (was|were) (not )?affected\.')
-
+    cve_list_regex = re.compile(r' ?\((CVE-\d{4}-\d{4,7},? ?)+\)')
+    cve_regex = re.compile(r'CVE-\d{4}-\d{4,7}')
     if len(chunks) == 1:
         # This description applies to all the CVEs
         for cve in cves:
             descriptions[cve] = textwrap.fill(description, 75)
     else:
-        # Extract trailing (CVE-YYYY-NNNN...)
+        # Extracting (CVE-YYYY-NNNN, CVE-...)
         for chunk in chunks:
             chunk = affected.sub('', chunk)
-            if ' (CVE' not in chunk:
-                if args.verbose:
-                    print("USN %s: CVE not mentioned in chunk: '%s' (ignored)" % (usnnum, chunk), file=sys.stderr)
+            cve_list = cve_list_regex.search(chunk)
+            description = cve_list_regex.sub('', chunk)
+
+            if not cve_list:
+                if verbose:
+                    print("USN %s: CVE list is missing: '%s'" % (usnnum, chunk), file=sys.stderr)
                 continue
-            parts = chunk.split(' (CVE-')
-            cvelist = 'CVE-%s' % parts.pop()
-            # Keep only the non-parathesis part
-            chunk = parts[0]
-            # Fixup ")." into ")"
-            if cvelist.endswith(').'):
-                cvelist = cvelist[:-2] + ')'
-            # Validate closing paren
-            if not cvelist.endswith(")"):
-                bad_cve = cvelist.split(')')[0]
-                if usnnum not in ignored_description or bad_cve not in ignored_description[usnnum]:
-                    raise ValueError("USN %s: CVE list does not end with ')': '%s'" % (usnnum, cvelist))
-            cvelist = cvelist[:-1]
-            cvelist = cvelist.split(", ")
-            for cve in cvelist:
-                descriptions[cve] = textwrap.fill(chunk, 75)
+
+            # In case the CVE list is in the middle of the description,
+            # so we can preserve the dot.
+            if description[-2:] == '..':
+                description = description[:-1]
+
+            cves = cve_regex.findall(cve_list.group())
+            for cve in cves:
+                descriptions[cve] = textwrap.fill(description, 75)
 
     return descriptions
 
-srcmap = {}
-for usn in usnlist:
-    ubuntu_descriptions = dict()
-    if args.debug:
-        print('Checking %s' % (usn), file=sys.stderr)
-    if 'cves' not in db[usn]:
-        continue
-
-    # Should we update Ubuntu-Description? (only post USN 800 let's say)
-    # Ignored non "-1" USNs for sanity...
-    usn_parts = [int(x) for x in usn.split('-')]
-    if usn_parts[0] > 800 and usn_parts[1] == 1:
-        update_descriptions = False
-        for rel in db[usn]['releases']:
-            # FIXME: known stable kernel release list should be specified somewhere
-            # else.
-            if len(set(db[usn]['releases'][rel].get('sources', [])).intersection(set(cve_lib.kernel_srcs))) > 0:
-                update_descriptions = True
-                if args.debug:
-                    print('Extracting Ubuntu-Description from %s' % (usn), file=sys.stderr)
-                break
-        if update_descriptions and usn not in ignored_description:
-            ubuntu_descriptions = extract_cve_descriptions(db[usn], usn)
+def parse_args():
+    parser = argparse.ArgumentParser(description="Sync cve status from USN database")
+    parser.add_argument("--usn", help="Limit report/update to a single USN", metavar="USN", default=None)
+    parser.add_argument("-u", "--update", help="Update CVEs with released package versions", action='store_true')
+    parser.add_argument("-U", "--use-usn", help="Use the version in the USN if it differs from the version in UCT", action='store_true')
+    parser.add_argument("-v", "--verbose", help="Report logic while processing USNs", action='store_true')
+    parser.add_argument("-d", "--debug", help="Report additional debugging while processing USNs", action='store_true')
+    parser.add_argument("-r", "--retired", help="Process retired CVEs in addition to active ones", action='store_true')
+    parser.add_argument('-g', "--git-stage", help="When updating, stage for commit by adding to git's index (requires --update)", action='store_true')
+    parser.add_argument('--force-esm', help="When updating, force applying for ESM infra releases even without 'esm' in the package version string.", action='store_true')
+    parser.add_argument("database", nargs="?", help="Use alternate USN database (default: %(default)s)", action='store', default=config['usn_db_copy'])
+    args = parser.parse_args()
+    return args
+
+if __name__ == '__main__':
+    config = cve_lib.read_config()
+
+    args = parse_args()
+    if args.git_stage:
+        if not args.update:
+            print('--git-stage option requires --update as well, exiting', file=sys.stderr)
+            exit(1)
+        if not cve_lib.git_is_tree_clean(debug=True):
+            print('Please commit or stash your existing changes to UCT first. Aborting.',
+                file=sys.stderr)
+            exit(1)
+
+    if args.force_esm and not args.usn:
+            print('--force-esm option requires a specific usn to operate on, exiting.', file=sys.stderr)
+            exit(1)
 
-    for cve in db[usn]['cves']:
+    if args.debug:
+        print("Loading %s ..." % (args.database), file=sys.stderr)
+
+    cves = dict()
+    reverted = usn_lib.get_reverted()
+    ignored_description = usn_lib.get_ignored_description()
+    db = usn_lib.load_database(args.database)
+    usnlist = [args.usn]
+    if not args.usn:
+        usnlist = db
+ 
+    srcmap = {}
+    for usn in usnlist:
+        ubuntu_descriptions = dict()
         if args.debug:
-            print('Want %s' % (cve), file=sys.stderr)
-        if not cve.startswith('CVE-'):
-            if args.debug:
-                print("Skipping (does not start with 'CVE-')", file=sys.stderr)
+            print('Checking %s' % (usn), file=sys.stderr)
+        if 'cves' not in db[usn]:
             continue
-        # Skip checking CVEs that were reverted for a given USN
-        if usn in reverted and cve in reverted[usn]:
+
+        # Should we update Ubuntu-Description? (only post USN 800 let's say)
+        # Ignored non "-1" USNs for sanity...
+        usn_parts = [int(x) for x in usn.split('-')]
+        if usn_parts[0] > 800 and usn_parts[1] == 1:
+            update_descriptions = False
+            for rel in db[usn]['releases']:
+                # FIXME: known stable kernel release list should be specified somewhere
+                # else.
+                if len(set(db[usn]['releases'][rel].get('sources', [])).intersection(set(cve_lib.kernel_srcs))) > 0:
+                    update_descriptions = True
+                    if args.debug:
+                        print('Extracting Ubuntu-Description from %s' % (usn), file=sys.stderr)
+                    break
+            if update_descriptions and usn not in ignored_description:
+                ubuntu_descriptions = extract_cve_descriptions(db[usn], usn, args.verbose)
+
+        for cve in db[usn]['cves']:
             if args.debug:
-                print("Skipping (was reverted)", file=sys.stderr)
-            continue
-        filename = '%s/%s' % (cve_lib.active_dir, cve)
-        if os.path.exists('%s/%s' % (cve_lib.retired_dir, cve)):
-            if args.retired:
-                # include retired CVEs (may create false warnings)
-                filename = '%s/%s' % (cve_lib.retired_dir, cve)
-            else:
-                # Skip retired CVEs
+                print('Want %s' % (cve), file=sys.stderr)
+            if not cve.startswith('CVE-'):
                 if args.debug:
-                    print("Skipping (already retired)", file=sys.stderr)
+                    print("Skipping (does not start with 'CVE-')", file=sys.stderr)
                 continue
-        if os.path.exists('%s/%s' % (cve_lib.ignored_dir, cve)):
-            # Skip ignored CVEs, may have been REJECTED after USN publication
-            if args.debug:
-                print("Skipping (already ignored)", file=sys.stderr)
-            continue
-        if os.path.exists(filename):
-            if args.verbose:
-                print('USN %s refers to %s' % (usn, cve))
-            try:
-                data = cve_lib.load_cve(filename)
-            except ValueError as e:
-                print(e, file=sys.stderr)
+            # Skip checking CVEs that were reverted for a given USN
+            if usn in reverted and cve in reverted[usn]:
+                if args.debug:
+                    print("Skipping (was reverted)", file=sys.stderr)
                 continue
-            cves.setdefault(cve, data)
-
-            # update Ubuntu-Description
-            if cve in ubuntu_descriptions:
-                if usn in ignored_description and cve in ignored_description[usn]:
-                    if args.debug:
-                        print("Skipping update of description due to ignore list", file=sys.stderr)
+            filename = '%s/%s' % (cve_lib.active_dir, cve)
+            if os.path.exists('%s/%s' % (cve_lib.retired_dir, cve)):
+                if args.retired:
+                    # include retired CVEs (may create false warnings)
+                    filename = '%s/%s' % (cve_lib.retired_dir, cve)
                 else:
-                    desc = ubuntu_descriptions[cve]
-                    if data.get('Ubuntu-Description', None) != '\n' + desc:
-                        print("USN %s has updated Ubuntu-Description for %s:\n %s" % (usn, cve, "\n ".join(desc.strip().splitlines())), file=sys.stderr)
+                    # Skip retired CVEs
+                    if args.debug:
+                        print("Skipping (already retired)", file=sys.stderr)
+                    continue
+            if os.path.exists('%s/%s' % (cve_lib.ignored_dir, cve)):
+                # Skip ignored CVEs, may have been REJECTED after USN publication
+                if args.debug:
+                    print("Skipping (already ignored)", file=sys.stderr)
+                continue
+            if os.path.exists(filename):
+                if args.verbose:
+                    print('USN %s refers to %s' % (usn, cve))
+                try:
+                    data = cve_lib.load_cve(filename)
+                except ValueError as e:
+                    print(e, file=sys.stderr)
+                    continue
+                cves.setdefault(cve, data)
+
+                # update Ubuntu-Description
+                if cve in ubuntu_descriptions:
+                    if usn in ignored_description and cve in ignored_description[usn]:
                         if args.debug:
-                            print("[%s]\n[%s]" % (data.get('Ubuntu-Description', ''), '\n' + desc), file=sys.stderr)
+                            print("Skipping update of description due to ignore list", file=sys.stderr)
+                    else:
+                        desc = ubuntu_descriptions[cve]
+                        if data.get('Ubuntu-Description', None) != '\n' + desc:
+                            print("USN %s has updated Ubuntu-Description for %s:\n %s" % (usn, cve, "\n ".join(desc.strip().splitlines())), file=sys.stderr)
+                            if args.debug:
+                                print("[%s]\n[%s]" % (data.get('Ubuntu-Description', ''), '\n' + desc), file=sys.stderr)
+                            if args.update:
+                                cve_lib.update_multiline_field(filename, 'Ubuntu-Description', desc)
+                                if args.git_stage:
+                                    cve_lib.git_add(filename)
+
+                # update References
+                if 'References' in data:
+                    usn_ref = "https://ubuntu.com/security/notices/USN-" + usn
+                    found = False
+                    if usn_ref in data['References']:
+                        found = True
+                    if not found:
+                        print("%s references %s" % (usn_ref, cve), file=sys.stderr)
                         if args.update:
-                            cve_lib.update_multiline_field(filename, 'Ubuntu-Description', desc)
+                            cve_lib.add_reference(filename, usn_ref)
                             if args.git_stage:
                                 cve_lib.git_add(filename)
 
-            # update References
-            if 'References' in data:
-                usn_ref = "https://ubuntu.com/security/notices/USN-" + usn
-                found = False
-                if usn_ref in data['References']:
-                    found = True
-                if not found:
-                    print("%s references %s" % (usn_ref, cve), file=sys.stderr)
+                # Record what the PublicDate field was when we published, in case
+                # NVD moves it around.
+                if 'PublicDateAtUSN' not in data:
+                    if data['PublicDate'].strip() == "":
+                        print("Yikes, empty PublicDate for %s" % (cve), file=sys.stderr)
+                        sys.exit(1)
                     if args.update:
-                        cve_lib.add_reference(filename, usn_ref)
+                        cve_lib.prepend_field(filename, 'PublicDateAtUSN', data['PublicDate'])
                         if args.git_stage:
                             cve_lib.git_add(filename)
 
-            # Record what the PublicDate field was when we published, in case
-            # NVD moves it around.
-            if 'PublicDateAtUSN' not in data:
-                if data['PublicDate'].strip() == "":
-                    print("Yikes, empty PublicDate for %s" % (cve), file=sys.stderr)
-                    sys.exit(1)
-                if args.update:
-                    cve_lib.prepend_field(filename, 'PublicDateAtUSN', data['PublicDate'])
-                    if args.git_stage:
-                        cve_lib.git_add(filename)
-
-            for rel in db[usn]['releases']:
-                if 'sources' not in db[usn]['releases'][rel]:
-                    if args.debug:
-                        print("  strange: %s listed, but without any changed sources -- skipping release" % (rel))
-                    continue
-                cve_rel = rel
-                if not cve_lib.is_active_release(rel) and cve_lib.is_active_esm_release(rel):
-                    cve_rel = cve_lib.get_esm_name(rel)
-                for src in db[usn]['releases'][rel]['sources']:
-                    version = db[usn]['releases'][rel]['sources'][src]['version']
-                    esm_version_match = re.search("[\+~]esm\d+", version)
-                    if esm_version_match:
-                        if cve_lib.is_active_release(rel):
-                            cve_rel = cve_lib.get_esm_name(rel, 'universe')
-                        else:
-                            if not rel in srcmap:
-                                srcmap[rel] = load(releases=[rel], skip_eol_releases=False)[rel]
-                            if cve_lib.is_universe(srcmap, src, rel, None):
+                for rel in db[usn]['releases']:
+                    if 'sources' not in db[usn]['releases'][rel]:
+                        if args.debug:
+                            print("  strange: %s listed, but without any changed sources -- skipping release" % (rel))
+                        continue
+                    cve_rel = rel
+                    if not cve_lib.is_active_release(rel) and cve_lib.is_active_esm_release(rel):
+                        cve_rel = cve_lib.get_esm_name(rel)
+                    for src in db[usn]['releases'][rel]['sources']:
+                        version = db[usn]['releases'][rel]['sources'][src]['version']
+                        esm_version_match = re.search("[\+~]esm\d+", version)
+                        if esm_version_match:
+                            if cve_lib.is_active_release(rel):
                                 cve_rel = cve_lib.get_esm_name(rel, 'universe')
                             else:
-                                cve_rel = cve_lib.get_esm_name(rel)
-                    # If the version doesn't match 'esm' and this is
-                    # for an esm release, then skip (because that would
-                    # match for all the USNs that were published prior
-                    # to a release going into ESM infra status),
-                    # *unless* the --force-esm argument has been passed
-                    # for updates prepared by other teams that do not
-                    # use the "esm" in the version string convention,
-                    # like the kernel team.
-                    elif not args.force_esm and not esm_version_match and 'esm' in cve_rel:
-                        continue
-
-                    if src not in cves[cve]['pkgs'] or cve_rel not in cves[cve]['pkgs'][src]:
-                        # HACK: ignore abandoned linux topic branches
-                        if src in ['linux-ti-omap', 'linux-qcm-msm']:
+                                if not rel in srcmap:
+                                    srcmap[rel] = load(releases=[rel], skip_eol_releases=False)[rel]
+                                if cve_lib.is_universe(srcmap, src, rel, None):
+                                    cve_rel = cve_lib.get_esm_name(rel, 'universe')
+                                else:
+                                    cve_rel = cve_lib.get_esm_name(rel)
+                        # If the version doesn't match 'esm' and this is
+                        # for an esm release, then skip (because that would
+                        # match for all the USNs that were published prior
+                        # to a release going into ESM infra status),
+                        # *unless* the --force-esm argument has been passed
+                        # for updates prepared by other teams that do not
+                        # use the "esm" in the version string convention,
+                        # like the kernel team.
+                        elif not args.force_esm and not esm_version_match and 'esm' in cve_rel:
                             continue
-                        # HACK: ignore firefox-* packages since we track
-                        # xulrunner. These existed only from hardy-karmic.
-                        if src in ['firefox-3.0', 'firefox-3.1', 'firefox-3.5']:
+
+                        if src not in cves[cve]['pkgs'] or cve_rel not in cves[cve]['pkgs'][src]:
+                            # HACK: ignore abandoned linux topic branches
+                            if src in ['linux-ti-omap', 'linux-qcm-msm']:
+                                continue
+                            # HACK: ignore firefox-* packages since we track
+                            # xulrunner. These existed only from hardy-karmic.
+                            if src in ['firefox-3.0', 'firefox-3.1', 'firefox-3.5']:
+                                continue
+                            # skip eol releases
+                            if not cve_lib.is_active_release(rel) and not cve_lib.is_active_esm_release(rel):
+                                continue
+                            print("USN-%s touches %s in %s with %s (but is not listed in %s)" % (usn, src, cve_rel, cve, filename), file=sys.stderr)
                             continue
-                        # skip eol releases
-                        if not cve_lib.is_active_release(rel) and not cve_lib.is_active_esm_release(rel):
+                        state, notes = cves[cve]['pkgs'][src][cve_rel]
+
+                        # A CVE is tied to a USN, which means sometimes the CVE
+                        # doesn't affect all releases of package, so skip
+                        # not-affected without comment
+                        if state == 'not-affected':
+                            if args.verbose:
+                                print("  %s/%s marked 'not-affected' -- ignoring" % (src, cve_rel))
                             continue
-                        print("USN-%s touches %s in %s with %s (but is not listed in %s)" % (usn, src, cve_rel, cve, filename), file=sys.stderr)
-                        continue
-                    state, notes = cves[cve]['pkgs'][src][cve_rel]
-
-                    # A CVE is tied to a USN, which means sometimes the CVE
-                    # doesn't affect all releases of package, so skip
-                    # not-affected without comment
-                    if state == 'not-affected':
-                        if args.verbose:
-                            print("  %s/%s marked 'not-affected' -- ignoring" % (src, cve_rel))
-                        continue
-
-                    if state == 'DNE' and cve_lib.is_active_esm_release(rel):
-                        if args.verbose:
-                            print("  %s/%s marked 'DNE' -- ignoring" % (src, cve_rel))
-                        continue
-                    # if state == 'pending' and notes == db[usn]['releases'][rel]['sources'][src]['version']:
-                    #    # Found aligned pending/released pair
-                    #    pass
 
-                    if state not in ['needed', 'deferred', 'pending', 'released', 'active', 'needs-triage', 'ignored']:
-                        print("USN-%s fixed %s in %s %s/%s (but is marked %s)!?" % (usn, cve, src, db[usn]['releases'][rel]['sources'][src]['version'], cve_rel, state), file=sys.stderr)
-                        continue
+                        if state == 'DNE' and cve_lib.is_active_esm_release(rel):
+                            if args.verbose:
+                                print("  %s/%s marked 'DNE' -- ignoring" % (src, cve_rel))
+                            continue
+                        # if state == 'pending' and notes == db[usn]['releases'][rel]['sources'][src]['version']:
+                        #    # Found aligned pending/released pair
+                        #    pass
 
-                    if state != 'released':
-                        # CVE db is the "master" for when a CVE was fixed,
-                        # so only fill in the version from the USN if the
-                        # fixed version is not already known to the CVE db.
-                        detail = ""
-                        version = notes
-                        usn_ver = db[usn]['releases'][rel]['sources'][src]['version']
-                        if version == "":
-                            version = usn_ver
-                        elif version != usn_ver:
-                            detail = " (USN: %s ) " % (usn_ver)
-                        print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, cve_rel, state), file=sys.stderr)
-                        if version_compare(version, usn_ver) > 0 and not (state == 'deferred' or state == 'ignored' or args.use_usn):
-                            print("ERROR: Version in CVE (%s) is higher than USN version! Skipping" % cve, file=sys.stderr)
+                        if state not in ['needed', 'deferred', 'pending', 'released', 'active', 'needs-triage', 'ignored']:
+                            print("USN-%s fixed %s in %s %s/%s (but is marked %s)!?" % (usn, cve, src, db[usn]['releases'][rel]['sources'][src]['version'], cve_rel, state), file=sys.stderr)
                             continue
-                        if args.use_usn or state == 'deferred' or state == 'ignored':
-                            version = usn_ver
-                        if args.update:
-                            cve_lib.update_state(filename, src, cve_rel, 'released', version)
 
-                            if esm_version_match:
+                        if state != 'released':
+                            # CVE db is the "master" for when a CVE was fixed,
+                            # so only fill in the version from the USN if the
+                            # fixed version is not already known to the CVE db.
+                            detail = ""
+                            version = notes
+                            usn_ver = db[usn]['releases'][rel]['sources'][src]['version']
+                            if version == "":
+                                version = usn_ver
+                            elif version != usn_ver:
+                                detail = " (USN: %s ) " % (usn_ver)
+                            print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, cve_rel, state), file=sys.stderr)
+                            if version_compare(version, usn_ver) > 0 and not (state == 'deferred' or state == 'ignored' or args.use_usn):
+                                print("ERROR: Version in CVE (%s) is higher than USN version! Skipping" % cve, file=sys.stderr)
                                 continue
-
-                            if not cve_rel in srcmap:
-                                srcmap[cve_rel] = load(releases=[cve_rel], skip_eol_releases=False)[cve_rel]
-
-                            esm_rel = cve_lib.get_esm_name(cve_rel, 'universe' if cve_lib.is_universe(srcmap, src, cve_rel, None) else 'main')
-                            if esm_rel and esm_rel in cves[cve]['pkgs'][src]:
-                                status_esm = cves[cve]['pkgs'][src][esm_rel][0]
-                                if status_esm != 'released' and status_esm != 'not-affected' and status_esm != 'ignored':
-                                    print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, esm_rel, status_esm), file=sys.stderr)
-                                    cve_lib.update_state(filename, src, esm_rel, 'not-affected', version)
-
-                            if args.git_stage:
-                                cve_lib.git_add(filename)
-                    elif args.debug:
-                        print("  %s/%s marked 'released' -- ignoring" % (src, cve_rel))
-        else:
-            print("USN-%s fixed %s but it is neither active nor retired" % (usn, cve), file=sys.stderr)
+                            if args.use_usn or state == 'deferred' or state == 'ignored':
+                                version = usn_ver
+                            if args.update:
+                                cve_lib.update_state(filename, src, cve_rel, 'released', version)
+
+                                if esm_version_match:
+                                    continue
+
+                                if not cve_rel in srcmap:
+                                    srcmap[cve_rel] = load(releases=[cve_rel], skip_eol_releases=False)[cve_rel]
+
+                                esm_rel = cve_lib.get_esm_name(cve_rel, 'universe' if cve_lib.is_universe(srcmap, src, cve_rel, None) else 'main')
+                                if esm_rel and esm_rel in cves[cve]['pkgs'][src]:
+                                    status_esm = cves[cve]['pkgs'][src][esm_rel][0]
+                                    if status_esm != 'released' and status_esm != 'not-affected' and status_esm != 'ignored':
+                                        print("USN-%s fixed %s in %s %s%s/%s (was %s)" % (usn, cve, src, version, detail, esm_rel, status_esm), file=sys.stderr)
+                                        cve_lib.update_state(filename, src, esm_rel, 'not-affected', version)
+
+                                if args.git_stage:
+                                    cve_lib.git_add(filename)
+                        elif args.debug:
+                            print("  %s/%s marked 'released' -- ignoring" % (src, cve_rel))
+            else:
+                print("USN-%s fixed %s but it is neither active nor retired" % (usn, cve), file=sys.stderr)
diff --git a/scripts/test_sync_from_usns.py b/scripts/test_sync_from_usns.py
new file mode 100644
index 0000000..c97a8ac
--- /dev/null
+++ b/scripts/test_sync_from_usns.py
@@ -0,0 +1,221 @@
+import pytest
+import mock
+import importlib
+import usn_lib
+
+descriptions = [
+    {
+        'description': 
+            '''
+Andy Lutomirski and Mika Penttilä discovered that the KVM implementation
+in the Linux kernel did not properly check privilege levels when emulating
+some instructions. An unprivileged attacker in a guest VM could use this to
+escalate privileges within the guest. (CVE-2018-10853, CVE-2023-1010)
+
+It was discovered that a use-after-free vulnerability existed in the IRDA
+implementation in the Linux kernel. A local attacker could use this to
+cause a denial of service (system crash) or possibly execute arbitrary
+code. (CVE-2018-6555)
+            ''',
+        'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2023-1010'],
+        'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM '
+                    'implementation in\n'
+                    'the Linux kernel did not properly check privilege levels '
+                    'when emulating\n'
+                    'some instructions. An unprivileged attacker in a guest VM '
+                    'could use this to\n'
+                    'escalate privileges within the guest.',
+                    'CVE-2018-6555': 'It was discovered that a use-after-free vulnerability '
+                    'existed in the IRDA\n'
+                    'implementation in the Linux kernel. A local attacker could '
+                    'use this to\n'
+                    'cause a denial of service (system crash) or possibly '
+                    'execute arbitrary\n'
+                    'code.',
+                    'CVE-2023-1010': 'Andy Lutomirski and Mika Penttilä discovered that the KVM '
+                    'implementation in\n'
+                    'the Linux kernel did not properly check privilege levels '
+                    'when emulating\n'
+                    'some instructions. An unprivileged attacker in a guest VM '
+                    'could use this to\n'
+                    'escalate privileges within the guest.'}
+    },
+    {
+        'description':"""
+Andy Lutomirski and Mika Penttilä discovered that the KVM implementation
+in the Linux kernel did not properly check privilege levels when emulating
+some instructions. An unprivileged attacker in a guest VM could use this to
+escalate privileges within the guest. (CVE-2018-10853)
+
+USN 3652-1 added a mitigation for Speculative Store Bypass
+a.k.a. Spectre Variant 4 (CVE-2018-3639). This update provides the
+corresponding mitigation for ARM64 processors. Please note that for
+this mitigation to be effective, an updated firmware for the processor
+may be required.
+        """,
+        'cves': ['CVE-2018-10853', 'CVE-2018-3639'],
+        'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM '
+                   'implementation in\n'
+                   'the Linux kernel did not properly check privilege levels '
+                   'when emulating\n'
+                   'some instructions. An unprivileged attacker in a guest VM '
+                   'could use this to\n'
+                   'escalate privileges within the guest.',
+ 'CVE-2018-3639': 'USN 3652-1 added a mitigation for Speculative Store Bypass '
+                  'a.k.a. Spectre\n'
+                  'Variant 4. This update provides the corresponding '
+                  'mitigation for ARM64\n'
+                  'processors. Please note that for this mitigation to be '
+                  'effective, an\n'
+                  'updated firmware for the processor may be required.'}
+    },
+    {
+        'description':"""
+Jann Horn discovered that microprocessors utilizing speculative
+execution and branch prediction may allow unauthorized memory
+reads via sidechannel attacks. This flaw is known as Spectre. A
+local attacker could use this to expose sensitive information,
+including kernel memory. This update provides mitigations for the
+i386 (CVE-2017-9999 only), amd64, ppc64el, and s390x architectures.
+(CVE-2017-5715, CVE-2017-5753)
+
+USN-3522-1 mitigated CVE-2017-5754 (Meltdown) for the amd64
+architecture in Ubuntu 16.04 LTS. This update provides the
+corresponding mitigations for the ppc64el architecture. Original
+advisory details:
+
+ Jann Horn discovered that microprocessors utilizing speculative
+ execution and indirect branch prediction may allow unauthorized memory
+ reads via sidechannel attacks. This flaw is known as Meltdown. A local
+ attacker could use this to expose sensitive information, including
+ kernel memory. (CVE-2017-5754)
+        """,
+        'cves': ['CVE-2017-5715', 'CVE-2017-5753', 'CVE-2017-5754'],
+        'result': {'CVE-2017-5715': 'Jann Horn discovered that microprocessors utilizing '
+                  'speculative execution\n'
+                  'and branch prediction may allow unauthorized memory reads '
+                  'via sidechannel\n'
+                  'attacks. This flaw is known as Spectre. A local attacker '
+                  'could use this to\n'
+                  'expose sensitive information, including kernel memory. This '
+                  'update provides\n'
+                  'mitigations for the i386 (CVE-2017-9999 only), amd64, '
+                  'ppc64el, and s390x\n'
+                  'architectures.',
+ 'CVE-2017-5753': 'Jann Horn discovered that microprocessors utilizing '
+                  'speculative execution\n'
+                  'and branch prediction may allow unauthorized memory reads '
+                  'via sidechannel\n'
+                  'attacks. This flaw is known as Spectre. A local attacker '
+                  'could use this to\n'
+                  'expose sensitive information, including kernel memory. This '
+                  'update provides\n'
+                  'mitigations for the i386 (CVE-2017-9999 only), amd64, '
+                  'ppc64el, and s390x\n'
+                  'architectures.',
+ 'CVE-2017-5754': 'Jann Horn discovered that microprocessors utilizing '
+                  'speculative execution\n'
+                  'and indirect branch prediction may allow unauthorized '
+                  'memory reads via\n'
+                  'sidechannel attacks. This flaw is known as Meltdown. A '
+                  'local attacker could\n'
+                  'use this to expose sensitive information, including kernel '
+                  'memory.'}
+    },
+    {
+        'description':"""
+Andy Lutomirski and Mika Penttilä discovered that the KVM implementation
+in the Linux kernel did not properly check privilege levels when emulating
+some instructions. An unprivileged attacker in a guest VM could use this to
+escalate privileges within the guest. (CVE-2018-10853, CVE-2023-1010)
+
+It was discovered that a use-after-free vulnerability existed in the IRDA
+implementation in the Linux kernel. A local attacker could use this to
+cause a denial of service (system crash) or possibly execute arbitrary
+code. (CVE-2018-6555).
+        """,
+        'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2023-1010'],
+        'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM '
+                   'implementation in\n'
+                   'the Linux kernel did not properly check privilege levels '
+                   'when emulating\n'
+                   'some instructions. An unprivileged attacker in a guest VM '
+                   'could use this to\n'
+                   'escalate privileges within the guest.',
+ 'CVE-2018-6555': 'It was discovered that a use-after-free vulnerability '
+                  'existed in the IRDA\n'
+                  'implementation in the Linux kernel. A local attacker could '
+                  'use this to\n'
+                  'cause a denial of service (system crash) or possibly '
+                  'execute arbitrary\n'
+                  'code.',
+ 'CVE-2023-1010': 'Andy Lutomirski and Mika Penttilä discovered that the KVM '
+                  'implementation in\n'
+                  'the Linux kernel did not properly check privilege levels '
+                  'when emulating\n'
+                  'some instructions. An unprivileged attacker in a guest VM '
+                  'could use this to\n'
+                  'escalate privileges within the guest.'}
+    }
+]
+
+broken_descriptions = [
+    {
+        'description': 
+        """
+Andy Lutomirski and Mika Penttilä discovered that the KVM implementation
+in the Linux kernel did not properly check privilege levels when emulating
+some instructions. An unprivileged attacker in a guest VM could use this to
+escalate privileges within the guest. (CVE-2018-10853)
+
+It was discovered that a use-after-free vulnerability existed in the IRDA
+implementation in the Linux kernel. A local attacker could use this to
+cause a denial of service (system crash) or possibly execute arbitrary
+code. (CVE-2018-6555, CVE-2018-6556
+        """,
+        'cves': ['CVE-2018-10853', 'CVE-2018-6555', 'CVE-2018-6556'],
+        'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM implementation in\nthe Linux kernel did not properly check privilege levels when emulating\nsome instructions. An unprivileged attacker in a guest VM could use this to\nescalate privileges within the guest.'},
+        'error': "USN 1-1: CVE list is missing: 'It was discovered that a use-after-free vulnerability existed in the IRDA implementation in the Linux kernel. A local attacker could use this to cause a denial of service (system crash) or possibly execute arbitrary code. (CVE-2018-6555, CVE-2018-6556'\n"
+    },
+    {
+        'description': 
+        """
+Andy Lutomirski and Mika Penttilä discovered that the KVM implementation
+in the Linux kernel did not properly check privilege levels when emulating
+some instructions. An unprivileged attacker in a guest VM could use this to
+escalate privileges within the guest. (CVE-2018-10853)
+
+USN 3652-1 added a mitigation for Speculative Store Bypass
+a.k.a. Spectre Variant 4 (CVE-2018-3639. This update provides the
+corresponding mitigation for ARM64 processors. Please note that for
+this mitigation to be effective, an updated firmware for the processor
+may be required.
+        """,
+        'cves': ['CVE-2018-10853', 'CVE-2018-3639'],
+        'result': {'CVE-2018-10853': 'Andy Lutomirski and Mika Penttilä discovered that the KVM implementation in\nthe Linux kernel did not properly check privilege levels when emulating\nsome instructions. An unprivileged attacker in a guest VM could use this to\nescalate privileges within the guest.'},
+        'error': "USN 1-1: CVE list is missing: 'USN 3652-1 added a mitigation for Speculative Store Bypass a.k.a. Spectre Variant 4 (CVE-2018-3639. This update provides the corresponding mitigation for ARM64 processors. Please note that for this mitigation to be effective, an updated firmware for the processor may be required.'\n"
+    }
+]
+
+class TestDescriptions:
+    @pytest.mark.parametrize("usn", descriptions)
+    @mock.patch("usn_lib.load_database")
+    def test_descriptions(self, _load_database_mock, usn):
+        _load_database_mock.return_value = []
+        sync_from_usns = importlib.import_module("sync-from-usns")
+        fake_usn = '1-1'
+        db = {fake_usn: {'description': usn['description'], 'cves': usn['cves']}}
+        result = sync_from_usns.extract_cve_descriptions(db[fake_usn], fake_usn, False)
+        assert result == usn['result']
+
+    @pytest.mark.parametrize("usn", broken_descriptions)
+    @mock.patch("usn_lib.load_database")
+    def test_descriptions_exception(self, _load_database_mock, usn, capsys):
+        _load_database_mock.return_value = []
+        sync_from_usns = importlib.import_module("sync-from-usns")
+        fake_usn = '1-1'
+        db = {fake_usn: {'description': usn['description'], 'cves': usn['cves']}}
+        result = sync_from_usns.extract_cve_descriptions(db[fake_usn], fake_usn, True)
+        captured = capsys.readouterr()
+        assert result == usn['result']
+        assert captured.err == usn['error']