Merge ~sbeattie/ubuntu-cve-tracker/+git/ubuntu-cve-tracker:create_uct_module_refactoring into ubuntu-cve-tracker:master

[Steve Beattie]

[WIP] refactor UCT layout and add coverage and script syntax check reporting

This branch is motivated by a few different things:

- wanting to break up cve_lib.py into smaller components, as well as support moving things implemented in individual scripts into common importable places
- wanting to know about the deprecation warnings from newer versions of python (like the recent invalid escape sequence warnings that came from python 3.12)
- wanting to see test coverage reports

To do this, I made a couple of changes that we can bikeshed about:

- created a scripts/uct directory / module that can be imported and moved a few things in there.
- created an empty cache/ directory and adjusted process_cves to use it to store and look for external data files during triage

The motivation for the latter was to (a) reduce the number of things that we look at when identifying python or shell scripts to check for syntax or other errors, and (b) I would like to reduce the clutter from the top level of UCT. I have not done it yet, but would like to move the *supported.txt files to a subdirectory (likely the meta_lists/ directory but we can argue about the name/location).

Some results out of this series:

- there are new make targets:
    - check-python: invokes the same tests that the launchpad.yaml unit test job does (these should probably be unified)
    - check-syntax-scripts: calls specific targets for running the syntax checkers on python and shell scripts in the tree

These targets are added as dependencies to the 'check' target but that also runs `check-syntax` on the cve files, so they need to be individually invokable if we're going to keep the separate test jobs in lpci.

Also a `coverage` make target was added that generates an html reoprt of the python test coverage in the coverage_html subdirectory, which reports details about which lines in each python have or have not been exercised. Interestingly, it identifies that a couple of small portions of the existing tests that are supposed to look at the subprojects tree don't get exercised (despite the symlink to the external repo being present on my system); not sure what's going wrong there. In an ideal world the test coverage pages would get published somewhere.

One thing to think about is how to integrate the rest of the oval functional and semi-end-to-end tests that are in the toplevel test/ directory, and work around that some of the data sources may not be available from lpci.

Another note: the read_config() and read_config_file() implementation cannot yet be removed, as there are a number of scripts in repos outsde of UCT Tthat need to be adjusted first.

UCT layout changes and add test coverage and syntax warning reports

* [988c62cb224] uct module: create, reproduce cve_lib.read_config() functionality
* [3aa4f0a5186] UCT/scripts: remove unneeded calls to cve_lib.read_config()
* [a691753e5c9] UCT/scripts: convert most scripts to use uct.config.read_uct_config()
* [26fcf6446ec] cve_lib.py: convert read_config to use uct.config.read_uct_config()
* [657e0da2753] check-cves: move ignored_cache.py under scripts/uct
* [1ef963e634f] check-cves: move ignore suggestion reasons to separate file/class
* [4a803a179ca] launchpad.yaml: run pytest against scripts/test_*.py
* [6578c2f6e15] launchpad.yaml: drop calling check-cves --test
* [85aa9d5031e] check-cves: drop self unit test argument and implementation
* [1c539b24ed7] launchpad.yaml: generate a coverage report for the unit tests
* [6345465aed2] Makefile: add python check and coverage report targets
* [e62116181d9] Add empty cache dir
* [e01c4c53ac5] process_cves: use cache directory for triage
* [2161889cd62] Make: add targets for checking script syntax
* [196a1bff689] launchpad.yaml: add syntax warning to lpci unit test job

[Marc Deslauriers (mdeslaur)]

Do you need to add the new cache directory to .gitignore?

34c2e14... by Steve Beattie on 2024-03-10

gitignore: ignore cache directory

Signed-off-by: Steve Beattie <email address hidden>

[Steve Beattie (sbeattie)]

On Sun, Mar 10, 2024 at 07:21:57PM -0000, Marc Deslauriers wrote:
> Do you need to add the new cache directory to .gitignore?

Not *strictly* necessary, but useful for not committing additional files
to that directory. Added a commit for that.

--
Steve Beattie
<email address hidden>

[Marc Deslauriers (mdeslaur)]

I agree with the reasoning behind this, and everything looks good to me. ACK

review: Approve

Unmerged commits

34c2e14... by Steve Beattie on 2024-03-10

gitignore: ignore cache directory

Signed-off-by: Steve Beattie <email address hidden>

	unit-tests:0 (build)
	check-cves:0 (build)

1 → 2 of 2 results

First • Previous • Next • Last

196a1bf... by Steve Beattie on 2024-03-10

launchpad.yaml: add syntax warning to lpci unit test job

Add needed dependencies and invoke make check-syntax-scripts

Signed-off-by: Steve Beattie <email address hidden>

	unit-tests:0 (build)
	check-cves:0 (build)

1 → 2 of 2 results

First • Previous • Next • Last

2161889... by Steve Beattie on 2024-03-10

Make: add targets for checking script syntax

Add targets for checking scripts syntax, currently using pyflakes and
shellcheck; I tried to use flake8 but it has an issue with the version
of pycodestyle in noble.

These invocations have their error codes suppressed, so that they are
informational only for now, as there are a large number of items
reported that need to be investigated.

This is the reason for the prior commit to try to move the nvd files
into a cache subdirectory to make pruning directories easier so that
e.g. file isn't invoked on all the individual nvd cve json files.

Signed-off-by: Steve Beattie <email address hidden>

e01c4c5... by Steve Beattie on 2024-02-27

process_cves: use cache directory for triage

Signed-off-by: Steve Beattie <email address hidden>

	unit-tests:0 (build)
	check-cves:0 (build)

1 → 2 of 2 results

First • Previous • Next • Last

e621161... by Steve Beattie on 2024-02-27

Add empty cache dir

Signed-off-by: Steve Beattie <email address hidden>

6345465... by Steve Beattie on 2024-03-10

Makefile: add python check and coverage report targets

Use the coverage module to generate an html coverage report. As part of
this:

- Adjust the .coveragerc config file to omit files that have not been
  covnerted to be python3 parseable.
- add the coverage_html created directory to .gitignore

Signed-off-by: Steve Beattie <email address hidden>

1c539b2... by Steve Beattie on 2024-03-10

launchpad.yaml: generate a coverage report for the unit tests

Signed-off-by: Steve Beattie <email address hidden>

85aa9d5... by Steve Beattie on 2024-03-10

check-cves: drop self unit test argument and implementation

Signed-off-by: Steve Beattie <email address hidden>

6578c2f... by Steve Beattie on 2024-03-10

launchpad.yaml: drop calling check-cves --test

As the same tests are now covered in the unit tests section, via
test_uct_suggestions.py

Signed-off-by: Steve Beattie <email address hidden>

4a803a1... by Steve Beattie on 2024-03-10

launchpad.yaml: run pytest against scripts/test_*.py

This way it will automatically pick up any new test scripts, including
the most recent scripts/test_uct_suggestions.py

Signed-off-by: Steve Beattie <email address hidden>

diff --git a/.coveragerc b/.coveragerc
index 3ad560c..d59ccbe 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -3,3 +3,10 @@ branch = True
 include =
     scripts/generate-oval
     scripts/oval_lib.py
+# the follow scripts have not been converted to be parseable by a
+# python3 intepreter
+omit =
+    scripts/pull-in-progress.py
+    scripts/report-date.py
+    scripts/report-mismatched-cve-fixes.py
+    scripts/report_universe_cves.py
diff --git a/.gitignore b/.gitignore
index 551e1e4..d8a74d4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -70,7 +70,9 @@ graphing/cosmic.data
 ignore_package_regexes.txt
 .login
 .coverage
+coverage_html
 /subprojects
 /pkg_cache.json
 .vscode/
 /.venv
+/cache/
diff --git a/.launchpad.yaml b/.launchpad.yaml
index a089c1d..9bcdb5d 100644
--- a/.launchpad.yaml
+++ b/.launchpad.yaml
@@ -13,7 +13,9 @@ jobs:
     architectures: amd64
     packages:
       - distro-info
+      - file
       - lsb-release
+      - pyflakes3
       - python3
       - python3-apt
       - python3-configobj
@@ -21,7 +23,9 @@ jobs:
       - python3-mock
       - python3-progressbar
       - python3-pytest
+      - python3-pytest-cov
       - python3-yaml
+      - shellcheck
     run-before: |
       # configure a basic ~/.ubuntu-cve-tracker.conf
       echo plb_authentication=/dev/null > ~/.ubuntu-cve-tracker.conf
@@ -35,11 +39,10 @@ jobs:
       # new one
       rm -f embargoed
       mkdir embargoed
+      echo "reporting syntax issues on scripts"
+      make check-syntax-scripts
       echo "Running unit tests..."
-      pytest-3 ./scripts/test_cve_lib.py ./scripts/test_kernel_lib.py ./scripts/test_usn_lib.py ./scripts/test_source_map.py ./scripts/test_publish-cves-to-website-api.py ./scripts/test_sync_from_usns.py ./test/test_oval_lib_unit.py
-      # ideally we would also run ./scripts/check-cves --test here as well as it
-      # is more of a unit test but it requires packages-mirror so run it as part
-      # of check-syntax below
+      pytest-3 --cov=scripts ./scripts/test_*.py ./test/test_oval_lib_unit.py
   check-cves:
     series: jammy
     architectures: amd64
@@ -75,6 +78,5 @@ jobs:
       rm -f embargoed
       mkdir embargoed
 
-      ./scripts/check-cves --test
       echo "Checking syntax..."
       ./scripts/check-syntax
diff --git a/Makefile b/Makefile
index 2c0abb7..254b074 100644
--- a/Makefile
+++ b/Makefile
@@ -13,6 +13,9 @@ ifneq ($(UCT_REVIEWED)$(SCRIPTS_RELDIR),$(SCRIPTS_RELDIR))
 endif
 export SCRIPTS=$(UCT_SCRIPTS)
 export ACTIVE=$(shell pwd)/active
+PYTEST=pytest-3
+PYTHON=python3
+PYFLAKES=pyflakes3
 
 all: cves pkgs tables
 
@@ -79,7 +82,33 @@ dev_setup:
 	        echo '*** $$UQT is not set, unable to find location of lpl_common.py ***' ; \
 	fi
 
-check:
+syntax-shell:
+	# run shellcheck on the specific types of shell scripts we have
+	# this does not propagate the error code yet because we have a
+	# a bunch of warnings, some of which because of how things are
+	# used, just need to be silenced.
+	find -type d \( -name cache -o -name active -o -name retired -o -name ignored -o -name nvd-database -o -name .git \) -prune   -o -exec file {} \+ | grep "Bourne-Again shell script" | cut -d ":" -f 1 | xargs shellcheck -s bash || true
+	find -type d \( -name cache -o -name active -o -name retired -o -name ignored -o -name nvd-database -o -name .git \) -prune   -o -exec file {} \+ | grep -i "posix shell script" | cut -d ":" -f 1 | xargs shellcheck -s dash || true
+
+syntax-python:
+	# run pyflakes on the python scripts we have
+	# this does not propagate the error code yet because there are
+	# some scripts that need to be made python3 compliant and some
+	# imports cleaned up
+	find -type d \( -name cache -o -name active -o -name retired -o -name ignored -o -name nvd-database -o -name .git \) -prune -o -exec file {} \+  | grep "Python script" | cut -d ":" -f 1 | xargs $(PYFLAKES) || true
+
+check-syntax-scripts: syntax-shell syntax-python
+
+coverage-python: check-python .coverage
+	$(PYTHON) -m coverage html --dir coverage_html
+
+.coverage: check-python
+
+check-python:
+	$(PYTEST) --cov=scripts ./scripts/test_*.py ./test/test_oval_lib_unit.py
+
+check: check-syntax-scripts check-python
 	$(SCRIPTS)/check-syntax
 
-.PHONY: dev_setup check
+.PHONY: dev_setup check check-python coverage-python check-syntax-scripts \
+syntax-shell syntax-python
diff --git a/cache/.empty_dir b/cache/.empty_dir
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/cache/.empty_dir
diff --git a/scripts/boilerplate-to-json.py b/scripts/boilerplate-to-json.py
index 06d800f..4066116 100755
--- a/scripts/boilerplate-to-json.py
+++ b/scripts/boilerplate-to-json.py
@@ -6,6 +6,7 @@ import re
 import sys
 
 import cve_lib
+from uct.config import read_uct_config
 
 def parse_boilerplate(filepath):
     cve_data = cve_lib.load_cve(filepath)
@@ -131,7 +132,7 @@ def parse_embedded_code_copies(filepath):
     return data
 
 # also parse debian's embedded-code-copies and amalgate that into data
-config = cve_lib.read_config()
+config = read_uct_config()
 debian_embedded_copies = os.path.join(config['secure_testing_path'], "data", "embedded-code-copies")
 code_copies = parse_embedded_code_copies(debian_embedded_copies)
 for pkg in code_copies["pkgs"]:
diff --git a/scripts/check-cves b/scripts/check-cves
index 8a9e8fc..020693f 100755
--- a/scripts/check-cves
+++ b/scripts/check-cves
@@ -28,7 +28,6 @@ import subprocess
 import sys
 import tempfile
 import time
-import unittest
 import urllib.request
 import xml.sax
 import xml.sax.handler
@@ -38,10 +37,12 @@ import progressbar
 
 import cve_lib
 import source_map
-from check_cves.ignored_cache import IgnoredCache
+from uct.ignored_cache import IgnoredCache
+from uct.suggestions import IgnoreSuggestions
+from uct.config import read_uct_config
 
 # load settings, if any
-cve_lib.read_config()
+uct_config = read_uct_config()
 
 parser = optparse.OptionParser()
 parser.add_option("-r", "--report", help="Just report CVEs that need checking", action="store_true")
@@ -50,7 +51,7 @@ parser.add_option("-k", "--known", help="Only report CVEs already known", action
 parser.add_option("-N", "--skip-nfu", help="Skip any CVEs marked as NFU (used with -k)", action="store_true")
 parser.add_option("-R", "--refresh", help="Refresh CVE descriptions", action="store_true")
 parser.add_option("-S", "--score-refresh", help="Refresh CVSS scores values only", action="store_true")
-parser.add_option("", "--test", help="Run regression tests", action="store_true")
+parser.add_option("", "--test", help="NO LONGER SUPPORTED, see test_uct_suggestions.py", action="help")
 parser.add_option("--untriaged", help="Process untriaged CVEs from output of locate_cves.py", metavar="FILE")
 parser.add_option("--mbox", help="Process untriaged CVEs from mbox file", metavar="FILE")
 parser.add_option("--rhel8oval", help="Process untriaged RHEL8 CVEs", metavar="FILE")
@@ -242,7 +243,7 @@ def import_debian(handler):
         return False
 
     # pull in CVEs from data/DSA/list
-    dsas = cve_lib.load_debian_dsas(cve_lib.config['secure_testing_path'] + '/data/DSA/list', opt.verbose)
+    dsas = cve_lib.load_debian_dsas(uct_config['secure_testing_path'] + '/data/DSA/list', opt.verbose)
     for dsa in dsas:
         for cve in dsas[dsa]['cves']:
             if not cve_lib.CVE_RE.match(cve):
@@ -581,12 +582,14 @@ class CVEHandler(xml.sax.handler.ContentHandler):
         self.saved_cve = ""
         self.debian = None
 
+        # get ignore suggestion
+        self.ignore_suggestion = IgnoreSuggestions()
         # File-type detection
 
 
         # Load debian CVE states, if configured
-        if 'secure_testing_path' in cve_lib.config:
-            self.debian = cve_lib.load_debian_cves(cve_lib.config['secure_testing_path'] + '/data/CVE/list')
+        if 'secure_testing_path' in uct_config:
+            self.debian = cve_lib.load_debian_cves(uct_config['secure_testing_path'] + '/data/CVE/list')
 
     def updateTimestamp(self):
         # Get UTC time
@@ -792,117 +795,6 @@ class CVEHandler(xml.sax.handler.ContentHandler):
     def cves(self):
         return self.cve_list
 
-    def get_ignore_suggestion(self, text):
-        '''Try to find a reasonable suggestion for the user.'''
-        suggestion = ""
-
-        # remove any whitespace
-        rev_text = text.strip()
-        # strip out the added mailing list stuff (locate_cves.py importing)
-        rev_text = re.sub(r'^ML-Date: .* ML-Subject: ', '', rev_text)
-        rev_text = re.sub(r'^(|Re: )CVE (r|R)equest: ', '', rev_text)
-
-        first_sentence = re.split(r'\. ', rev_text)[0]
-
-        # hunt for module/component
-        nouns = ["library", "templates?", "components?", "modules?", "[Pp]lug-?ins?", "extension", "application", "[Tt]hemes?"]
-        products = ["Joomla!", "Drupal", "WordPress", "TYPO3", "Mambo", "Android", "jQuery"]
-        match = re.search(r'(?: in t|^T)he (.*) (%s) (?:.* )?for (%s)' % ("|".join(nouns), "|".join(products)),
-                          first_sentence)
-        preposition = "for"
-        if not match:
-            # try swapping order of product and noun in which case there is no preposition
-            match = re.search(r'(?: in t|^T)he (.*) (%s) (%s) (?:.* )' % ("|".join(products), "|".join(nouns)),
-                              first_sentence)
-            preposition = ""
-        if match:
-            module = match.group(1)
-            noun = match.group(2)
-            product = match.group(3)
-            for marker in [" module", " ("]:
-                if marker in module:
-                    module = module.split(marker)[0]
-            return " ".join(filter(lambda x: len(x) > 0, [module, noun, preposition, product]))
-
-        # hunt for a description of a thing written in a language or for creating something else
-        match = re.search(r'(.*) is a (?:.*) (written in|for creating) (?:.*)', first_sentence)
-        if match:
-            return match.group(1)
-
-        # hunt for a product on a platform
-        match = re.search(r'.* for (Windows|Linux|Mac OS|iOS|Android|iPhone|iPad|BlackBerry|Symbian)', first_sentence)
-        if match:
-            return match.group(0)
-
-        # look for common pattern
-        match = re.search(r'A vulnerability(?:, which was)? classified as (?:.*),? was found in (.*)', first_sentence)
-        if match:
-            return match.group(1)
-
-        # drop commas-extensions
-        if ',' in first_sentence:
-            first_sentence = re.split(r',', first_sentence)[0]
-        phrases = re.split(r' [io]n ', first_sentence)
-
-        # default to the last phrase
-        suggestion = phrases[-1]
-        # move to earlier phrase if suggestion starts with "a"
-        if suggestion.startswith('a ') and len(phrases) > 1:
-            suggestion = phrases[-2]
-
-        version_preps = r'(\s+(before|through|prior to|versions?|[<>]=?))+\s*'
-        version_regex = r'\s+([a-zA-Z\._\-]*[0-9]+[a-zA-Z\._\-]*)+'
-        # prefer 'Apple iOS before <version>' or 'Apple Mac OS X through
-        # <version' in the last phrase over other suggestions
-        if not re.search(r'' + version_preps + version_regex, suggestion):
-            # grab the first phrase with something that may be a version number
-            for p in phrases:
-                if re.search(r'' + version_regex, p):
-                    suggestion = p
-                    break
-                if re.search(r'^[^,]+\s+for\s+', p):
-                    suggestion = p
-                    break
-
-        # try to find a good suggestion from the phrase (ie suggest 'Linux
-        # kernel' from 'the Linux kernel before 2.6.27')
-        suggestion = re.split(r'\s+([a-zA-Z\._\-]*[0-9]+[a-zA-Z\._\-]*)+', suggestion)[0]
-        # "blah in component for Software"
-        if re.search(r'^[^,]+\s+for\s+', suggestion):
-            suggestion = re.split(r'[^,]+\s+for\s+', suggestion)[1]
-
-        # Chop off action verbs
-        cleanup_regexes = [
-            # clean up leading "in" or "the"
-            r'^\s*([tT]he|[iI]n)\s+',
-            # clean up trailing version prepositions like "before" or "through"
-            # from version details
-            r'' + version_preps + '$',
-            # clean up trailing parens
-            r'\s+\([^\)]+\)\s*$',
-            # action verbs
-            r'\s+(has|creates|allows|could|contains)($|\s+.*)',
-            # "vulnerbale installations of"
-            r'(^|\s+)vulnerable\sinstallations?\sof($|\s+)',
-            # This affects all versions of package
-            r'^This affects all versions of package\s+',
-            # This affects the package
-            r'^This affects the package\s+',
-        ]
-
-        for regex in cleanup_regexes:
-            if re.search(regex, suggestion):
-                suggestion = re.sub(regex, '', suggestion)
-
-        # if the phrase is too long, truncate it to max_length, but make
-        # sure we don't have a partial word at the end
-        max_length = 64
-        if len(suggestion) > max_length:
-            suggestion = suggestion[:max_length]
-            suggestion = re.sub(r'\s+\w+$', '', suggestion)
-
-        return suggestion
-
     def display_command_file_usage(self, f, line_prefix=''):
         f.write('%sThe following commands can be used in this file:\n' % (line_prefix))
         f.write('%s\n' % (line_prefix))
@@ -1000,13 +892,13 @@ class CVEHandler(xml.sax.handler.ContentHandler):
             # no debian info, display possible commented ignore command when
             # using command file (i.e. wrap_desc is true)
             if (self.debian[cve]['state'] == 'RESERVED' or self.debian[cve]['state'] == None)  and wrap_desc:
-                proposed_ignore = self.get_ignore_suggestion(self.cve_data[cve]['desc'])
+                proposed_ignore = self.ignore_suggestion.get_ignore_suggestion(self.cve_data[cve]['desc'])
                 print('%s ignore "%s"' % (cve, proposed_ignore))
             # debian rejected, so offer to reject by ignoring when using command file (i.e. wrap_desc is true)
             if (self.debian[cve]['state'] == 'REJECTED' and wrap_desc):
                 print('%s ignore "%s"' % (cve, "REJECTED"))
         else:
-            proposed_ignore = self.get_ignore_suggestion(self.cve_data[cve]['desc'])
+            proposed_ignore = self.ignore_suggestion.get_ignore_suggestion(self.cve_data[cve]['desc'])
             print('%s ignore %s' % (cve, proposed_ignore))
 
         software_hints_from_cve_desc = self.get_software_hints_from_cve_description(self.cve_data[cve]["desc"])
@@ -1096,7 +988,7 @@ class CVEHandler(xml.sax.handler.ContentHandler):
         return (action, reason, packages)
 
     def get_software_hints_from_cve_description(self, cve_description):
-        desc = self.get_ignore_suggestion(cve_description)
+        desc = self.ignore_suggestion.get_ignore_suggestion(cve_description)
         # split based on brackets and other chars which aren't going to be
         # helpful in matching a package name etc
         words = set(filter(None, re.split("[\\s\\(\\){}\\[\\]!\\?\\\\/]+", desc.lower())))
@@ -1166,7 +1058,7 @@ class CVEHandler(xml.sax.handler.ContentHandler):
                 # Show debian reason first, then automatically determined
                 # reason, then saved reasons. This makes more sense
                 # than sorting the reasons and is more predictable
-                choices = [reason, self.get_ignore_suggestion(self.cve_data[cve]['desc'])]
+                choices = [reason, self.ignore_suggestion.get_ignore_suggestion(self.cve_data[cve]['desc'])]
                 choices.extend(self.saved_ignore_cache.get())
                 for choice in choices:
                     if choice != "" and choice not in prompts:
@@ -1478,113 +1370,7 @@ class CVEHandler(xml.sax.handler.ContentHandler):
     def skip_cve(self):
         self.num_skipped += 1
 
-class CheckCVETest(unittest.TestCase):
-    def test_get_ignore_suggestion(self):
-        '''"Ignore" suggestion text extraction'''
-
-        # Re-use the global handler
-        h = handler
-
-        self.assertEqual("Courier-Authlib", h.get_ignore_suggestion('''SQL injection vulnerability in authpgsqllib.c in Courier-Authlib before 0.62.0, when a non-Latin locale Postgres database is used, allows remote attackers to execute arbitrary SQL commands via query parameters containing apostrophes.'''))
-
-        self.assertEqual("Apple Mac OS X", h.get_ignore_suggestion('''Buffer overflow in the DirectoryService Proxy in DirectoryService in Apple Mac OS X through 10.6.8 allows remote attackers to execute arbitrary code or cause a denial of service (application crash) via unspecified vectors.'''))
-
-        self.assertEqual("KDE", h.get_ignore_suggestion('''HTMLTokenizer::scriptHandler in Konqueror in KDE 3.5.9 and 3.5.10 allows remote attackers to cause a denial of service (application crash) via an invalid document.load call that triggers use of a deleted object.  NOTE: some of these details are obtained from third party information.'''))
-
-        self.assertEqual("Sun Solaris", h.get_ignore_suggestion('''The name service cache daemon (nscd) in Sun Solaris 10 and OpenSolaris snv_50 through snv_104 does not properly check permissions, which allows local users to gain privileges and obtain sensitive information via unspecified vectors.'''))
-
-        self.assertEqual("Linux kernel", h.get_ignore_suggestion('''libata in the Linux kernel before 2.6.27.9 does not set minimum timeouts for SG_IO requests, which allows local users to cause a denial of service (Programmed I/O mode on drives) via multiple simultaneous invocations of an unspecified test program.'''))
-
-        self.assertEqual("iGaming", h.get_ignore_suggestion('''Multiple SQL injection vulnerabilities in iGaming 1.5 and earlier allow remote attackers to execute arbitrary SQL commands via the browse parameter to (1) previews.php and (2) reviews.php, and the (3) id parameter to index.php in a viewarticle action.'''))
-
-        self.assertEqual("PHP iCalendar", h.get_ignore_suggestion('''PHP iCalendar 2.24 and earlier allows remote attackers to bypass authentication by setting the phpicalendar and phpicalendar_login cookies to 1.'
-'''))
-
-        # Test length truncation, tweaked to avoid "has" matcher
-        self.assertEqual("** TEST CVE ** This candidate HAS been reserved by an", h.get_ignore_suggestion('''** TEST CVE ** This candidate HAS been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided.'''))
-
-        self.assertEqual("Sun OpenSolaris", h.get_ignore_suggestion('''Unspecified vulnerability in the root/boot archive tool in Sun OpenSolaris has unknown impact and local attack vectors, related to a "Temporary file vulnerability," aka Bug ID 6653455.'''))
-
-        self.assertEqual("Red Hat Certificate System", h.get_ignore_suggestion('''Red Hat Certificate System 7.2 uses world-readable permissions for password.conf and unspecified other configuration files, which allows local users to discover passwords by reading these files.'''))
-
-        self.assertEqual("Microsoft Internet Explorer", h.get_ignore_suggestion('''An unspecified function in the JavaScript implementation in Microsoft Internet Explorer creates and exposes a "temporary footprint" when there is a current login to a web site, which makes it easier for remote attackers to trick a user into acting upon a spoofed pop-up message, aka an "in-session phishing attack." NOTE: as of 20090116, the only disclosure is a vague pre-advisory with no actionable information. However, because it is from a well-known researcher, it is being assigned a CVE identifier for tracking purposes.'''))
-
-        self.assertEqual("Umer Inc Songs Portal", h.get_ignore_suggestion('''SQL injection vulnerability in albums.php in Umer Inc Songs Portal allows remote attackers to execute arbitrary SQL commands via the id parameter.'''))
-
-        self.assertEqual("Limbo CMS", h.get_ignore_suggestion('''SQL injection vulnerability in open.php in the Private Messaging (com_privmsg) component for Limbo CMS allows remote attackers to execute arbitrary SQL commands via the id parameter in a pms action to index.php.'''))
-
-        self.assertEqual("phpscripts Ranking Script", h.get_ignore_suggestion('''phpscripts Ranking Script allows remote attackers to bypass authentication and gain administrative access by sending an admin=ja cookie.'''))
-
-        self.assertEqual("A4Desk Event Calendar", h.get_ignore_suggestion('''PHP remote file inclusion vulnerability in index.php in A4Desk Event Calendar, when magic_quotes_gpc is disabled, allows remote attackers to execute arbitrary PHP code via a URL in the v parameter.'''))
-
-        self.assertEqual("Galatolo WebManager", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in result.php in Galatolo WebManager (GWM) 1.0 allows remote attackers to inject arbitrary web script or HTML via the key parameter.'''))
-
-        self.assertEqual("Sun OpenSolaris", h.get_ignore_suggestion('''Unspecified vulnerability in the process (aka proc) filesystem in Sun OpenSolaris snv_85 through snv_100 allows local users to gain privileges via vectors related to the contract filesystem.'''))
-
-        self.assertEqual("Dreampics Gallery Builder", h.get_ignore_suggestion('''SQL injection vulnerability in index.php in Dreampics Gallery Builder allows remote attackers to execute arbitrary SQL commands via the exhibition_id parameter in a gallery.viewPhotos action.'''))
-
-        self.assertEqual("Omilen Photo Gallery component for Joomla!", h.get_ignore_suggestion('''Directory traversal vulnerability in the Omilen Photo Gallery (com_omphotogallery) component Beta 0.5 for Joomla! allows remote attackers to include and execute arbitrary local files via directory traversal sequences in the controller parameter to index.php.'''))
-
-        self.assertEqual("Webform module for Drupal", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the Webform module 5.x before 5.x-2.7 and 6.x before 6.x-2.7, a module for Drupal, allows remote attackers to inject arbitrary web script or HTML via a submission.'''))
-
-        self.assertEqual("Itamar Elharar MusicGallery component for Joomla!", h.get_ignore_suggestion('''SQL injection vulnerability in the Itamar Elharar MusicGallery (com_musicgallery) component for Joomla! allows remote attackers to execute arbitrary SQL commands via the id parameter in an itempage action to index.php.  NOTE: the provenance of this information is unknown; the details are obtained solely from third party information.'''))
-
-        self.assertEqual("Kide Shoutbox component for Joomla!", h.get_ignore_suggestion('''The Kide Shoutbox (com_kide) component 0.4.6 for Joomla! does not properly perform authentication, which allows remote attackers to post messages with an arbitrary account name via an insertar action to index.php.  NOTE: the provenance of this information is unknown; the details are obtained solely from third party information.'''))
 
-        self.assertEqual("WP-Forum plugin for WordPress", h.get_ignore_suggestion('''Multiple SQL injection vulnerabilities in the WP-Forum plugin before 2.4 for WordPress allow remote attackers to execute arbitrary SQL commands via (1) the search_max parameter in a search action to the default URI, related to wpf.class.php; (2) the forum parameter to an unspecified component, related to wpf.class.php; (3) the topic parameter in a viewforum action to the default URI, related to the remove_topic function in wpf.class.php; or the id parameter in a (4) editpost or (5) viewtopic action to the default URI, related to wpf-post.php.'''))
-
-        self.assertEqual("ListMan extension for TYPO3", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the ListMan (nl_listman) extension 1.2.1 for TYPO3 allows remote attackers to inject arbitrary web script or HTML via unspecified vectors.'''))
-
-        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: Re: CVE request: multiple status.net issues'''))
-        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: CVE request: multiple status.net issues'''))
-        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: multiple status.net issues'''))
-        self.assertEqual("Mambo CMS", h.get_ignore_suggestion('''ML-Date: 2011-06-28 16:24:28, ML-Subject: Re: CVE Request: Mambo CMS 4.6.x | Multiple Cross Site Scripting Vulnerabilities'''))
-
-        self.assertEqual("Apple iOS", h.get_ignore_suggestion('''The DNAv4 protocol implementation in the DHCP component in Apple iOS before 6 sends Wi-Fi packets containing a MAC address of a host on a previously used network, which might allow remote attackers to obtain sensitive information about previous device locations by sniffing an unencrypted Wi-Fi network for these packets.'''))
-
-        self.assertEqual("Conceptronic", h.get_ignore_suggestion('''Multiple open redirect vulnerabilities on the Conceptronic C54APM access point with runtime code 1.26 allow remote attackers to redirect users to arbitrary web sites and conduct phishing attacks via (1) the submit-url parameter in a Refresh action to goform/formWlSiteSurvey or (2) the wlan-url parameter to goform/formWlanSetup.'''))
-
-        self.assertEqual("Tapjoy library for Android", h.get_ignore_suggestion('''The Tapjoy library for Android does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.'''))
-
-        self.assertEqual("kamkomesan application for Android", h.get_ignore_suggestion('''The kamkomesan (aka com.anek.kamkomesan) application 1.0 for Android does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.'''))
-
-        self.assertEqual("DataTables plugin for jQuery", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the DataTables plugin 1.10.8 and earlier for jQuery allows remote attackers to inject arbitrary web script or HTML via the scripts parameter to media/unit_testing/templates/6776.php.'''))
-
-        self.assertEqual("Cisco Application Policy Infrastructure Controller (APIC)", h.get_ignore_suggestion('''A vulnerability in Cisco Application Policy Infrastructure Controller (APIC) could allow an authenticated, remote attacker to gain higher privileges than the account is assigned.'''))
-
-        self.assertEqual("Redirection", h.get_ignore_suggestion('''Redirection version 2.7.3 contains a ACE via file inclusion vulnerability in Pass-through mode that can result in allows admins to execute any PHP file in the filesystem.'''))
-
-        self.assertEqual("topydo", h.get_ignore_suggestion('''topydo contains a CWE-20: Improper Input Validation vulnerability in ListFormatParser::parse, file topydo/lib/ListFormat.py line 292 as of d4f843dac71308b2f29a7c2cdc76f055c3841523 that can result in Injection of arbitrary bytes to the terminal,'''))
-
-        self.assertEqual("Joplin", h.get_ignore_suggestion('''Joplin version prior to 1.0.90 contains a XSS evolving into code execution due to enabled nodeIntegration.'''))
-
-        self.assertEqual("Foxit Reader", h.get_ignore_suggestion('''This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Foxit Reader 9.2.0.9297.'''))
-
-        self.assertEqual("Exquisite Ultimate Newspaper theme for WordPress", h.get_ignore_suggestion('''The Exquisite Ultimate Newspaper theme 1.3.3 for WordPress has XSS via the anchor identifier to assets/js/jquery.foundation.plugins.js.'''))
-        self.assertEqual("Weaver Xtreme Theme for WordPress", h.get_ignore_suggestion('''The Weaver Xtreme Theme for WordPress is vulnerable to stored Cross-Site Scripting due to insufficient escaping of the profile display name in versions up to, and including, 5.0.7.'''))
-        self.assertEqual("Weaver Show Posts Plugin for WordPress", h.get_ignore_suggestion('''The Weaver Show Posts Plugin for WordPress is vulnerable to stored Cross-Site Scripting due to insufficient escaping of the profile display name in versions up to, and including, 1.6.'''))
-        self.assertEqual("WordPress File Upload and WordPress File Upload Pro plugins for WordPress", h.get_ignore_suggestion('''The WordPress File Upload and WordPress File Upload Pro plugins for WordPress are vulnerable to Path Traversal in versions up to, and including, 4.19.1 via the vulnerable parameter wfu_newpath.'''))
-        self.assertEqual("browserless-chrome", h.get_ignore_suggestion('''This affects all versions of package browserless-chrome. User input flowing from the workspace endpoint gets used to create a file path filePath and this is fetched and then sent back to a user. This can be escaped to fetch arbitrary files from a server.'''))
-        self.assertEqual("@absolunet/kafe", h.get_ignore_suggestion('''This affects the package @absolunet/kafe before 3.2.10. It allows cause a denial of service when validating crafted invalid emails.'''))
-        self.assertEqual("Magneticlab Sàrl Homepage Pop-up plugin", h.get_ignore_suggestion('''Cross-Site Request Forgery (CSRF) vulnerability in Magneticlab Sàrl Homepage Pop-up plugin <= 1.2.5 versions.'''))
-
-        # handle odd whitespace
-        self.assertEqual("PTC Vuforia Studio does not require a token.", h.get_ignore_suggestion('''
-
-
-
-
- PTC Vuforia Studio does not require a token.
-                                                                  '''))
-        self.assertEqual("zxcvbn-ts", h.get_ignore_suggestion('''zxcvbn-ts is a password strength estimator written in typescript.'''))
-        self.assertEqual("VMware Tools for Windows", h.get_ignore_suggestion('''VMware Tools for Windows (10.x.y prior to 10.3.10, 11.x.y prior to 11.1.5) contains a denial-of-service vulnerability due to NULL pointer dereference in vm3dmp driver. Successful exploitation of this issue may allow attackers with normal user privileges to create a denial-of-service condition on the guest machine where VMware Tools is installed.'''))
-
-        self.assertEqual("3DPrint WordPress plugin", h.get_ignore_suggestion("The 3DPrint WordPress plugin before 3.5.6.9 does not protect against CSRF attacks in the modified version of Tiny File Manager included with the plugin, allowing an attacker to craft a malicious request that will create an archive of any files or directories on the target server by tricking a logged in admin into submitting a form."))
-
-        self.assertEqual("Feathersjs", h.get_ignore_suggestion('''Feathersjs is a framework for creating web APIs and real-time applications with TypeScript or JavaScript.'''))
-
-        self.assertEqual("PaulPrinting CMS 2018", h.get_ignore_suggestion("A vulnerability, which was classified as problematic, was found in PaulPrinting CMS 2018"))
-        
 ignored_notforus_path = 'ignored/not-for-us.txt'
 if destdir != './' and destdir != '.':
     ignored_notforus_path = os.path.join(destdir, ignored_notforus_path)
@@ -1625,12 +1411,6 @@ parser = xml.sax.make_parser()
 handler = CVEHandler(CVEIgnoreList)
 parser.setContentHandler(handler)
 
-if opt.test:
-    suite = unittest.TestSuite()
-    suite.addTest(unittest.TestLoader().loadTestsFromTestCase(CheckCVETest))
-    unittest.TextTestRunner(verbosity=2).run(suite)
-    sys.exit(0)
-
 # if has specified to triage only specific CVEs, check these are not
 # ignored
 specific_cves = None
diff --git a/scripts/check-unreplaced-templates.py b/scripts/check-unreplaced-templates.py
index 4723688..3e438d9 100755
--- a/scripts/check-unreplaced-templates.py
+++ b/scripts/check-unreplaced-templates.py
@@ -16,6 +16,7 @@ import cve_lib
 import json
 import optparse
 import usn_lib
+from uct.config import read_uct_config
 
 parser = optparse.OptionParser()
 parser.add_option("-d", "--debug", help="Report additional debugging while loading USNs", action='store_true')
@@ -26,7 +27,7 @@ parser.add_option("-j", "--json", help="Check json instead of default pickle dat
 
 cves = dict()
 
-config = cve_lib.read_config()
+config = read_uct_config()
 
 changed = False
 if opt.json:
diff --git a/scripts/cve_lib.py b/scripts/cve_lib.py
index a3104fa..e982b55 100755
--- a/scripts/cve_lib.py
+++ b/scripts/cve_lib.py
@@ -24,6 +24,7 @@ import json
 import yaml
 import urllib.error
 import urllib.request
+from uct.config import read_uct_config
 
 from functools import reduce
 from functools import lru_cache
@@ -1698,24 +1699,16 @@ def read_config_file(config_file):
 
     return ConfigObj(config_file)
 
-def read_config():
-    config_file = os.path.join(os.path.expanduser("~"), ".ubuntu-cve-tracker.conf")
-
-    if not os.path.exists(config_file):
-        raise ValueError("Could not find '%s'" % (config_file))
 
-    # FIXME: Why does this need to be defined as "global" when other globals
-    # like "releases" and "EXIT_OKAY" don't need it??
+# this function has been deprecated in favor of
+# uct.config.read_uct_config() and callers should be converted to use
+# that directly
+def read_config():
     global config
-    config = read_config_file(config_file)
-
-    # Validate required arguments
-    if "plb_authentication" not in config:
-        raise ValueError(("Could not find 'plb_authentication' entry in %s." % (config_file)))
-    if not os.path.exists(config["plb_authentication"]):
-        raise ValueError(("Could not find file specified by 'plb_authentication' in %s." % (config_file)))
+    config = read_uct_config()
     return config
 
+
 def drop_dup_release(cve, rel):
     output = codecs.open(cve + ".new", 'w', encoding="utf-8")
     saw = set()
diff --git a/scripts/debian-cve-update b/scripts/debian-cve-update
index 07c1bea..37856fa 100755
--- a/scripts/debian-cve-update
+++ b/scripts/debian-cve-update
@@ -10,12 +10,13 @@ from __future__ import print_function
 
 from cve_lib import (
     check_mirror_timestamp, get_cve_list, load_debian_cves,
-    load_ignored_reasons, prepend_debian_cve, read_config,
+    load_ignored_reasons, prepend_debian_cve,
     update_debian_todo_cves,
 )
 import source_map
 import optparse
 import os
+from uct.config import read_uct_config
 
 parser = optparse.OptionParser()
 parser.add_option("-p", "--packages", help="Include known packages in the updates", action='store_true')
@@ -23,7 +24,7 @@ parser.add_option("-n", "--dry-run", help="Do not actually make changes", action
 parser.add_option("-q", "--quiet", help="Report actions verbosely", dest="verbose", default=True, action='store_false')
 (opt, args) = parser.parse_args()
 
-config = read_config()
+config = read_uct_config()
 if len(args) == 1:
     debian_cve_list = args[0]
 else:
diff --git a/scripts/dump-features b/scripts/dump-features
index 30ca715..5dccc03 100755
--- a/scripts/dump-features
+++ b/scripts/dump-features
@@ -29,8 +29,6 @@ if opt.editmoin:
     else:
         sys.exit(1)
 
-config = cve_lib.read_config()
-
 features = dict()
 
 UNIMPLEMENTED = 0
diff --git a/scripts/gen-source-map-cache b/scripts/gen-source-map-cache
index 1c84cc6..d6fd4cf 100755
--- a/scripts/gen-source-map-cache
+++ b/scripts/gen-source-map-cache
@@ -23,10 +23,11 @@ if not "UCT" in os.environ and not os.path.islink('subprojects'):
 
 import cve_lib
 import source_map
+from uct.config import read_uct_config
 
 def main():
 
-    config = source_map.read_config_file(os.path.expanduser("~/.ubuntu-cve-tracker.conf"))
+    config = read_uct_config()
     if 'packages_mirror' not in config:
         sys.stderr.write("\rERROR: Could not find packages_mirror in config file!\n")
         sys.exit(1)
diff --git a/scripts/generate-graphs.py b/scripts/generate-graphs.py
index e699f04..cceb55c 100755
--- a/scripts/generate-graphs.py
+++ b/scripts/generate-graphs.py
@@ -61,8 +61,6 @@ if opt.target == None:
     print("Must specify --target", file=sys.stderr)
     sys.exit(1)
 
-config = cve_lib.read_config()
-
 if not os.path.exists(opt.database):
     print("Cannot find '%s'" % opt.database, file=sys.stderr)
     sys.exit(1)
diff --git a/scripts/plot-usns.py b/scripts/plot-usns.py
index b6e8e2b..527bb22 100755
--- a/scripts/plot-usns.py
+++ b/scripts/plot-usns.py
@@ -8,6 +8,7 @@
 # License: GPLv3
 import sys, time, usn_lib, cve_lib
 import optparse
+from uct.config import read_uct_config
 
 import pprint
 pp = pprint.PrettyPrinter(indent=4)
@@ -23,7 +24,7 @@ if opt.target not in ['usn','src','bin','cve']:
     print("Unknown target '%s'" % (opt.target), file=sys.stderr)
     sys.exit(1)
 
-config = cve_lib.read_config()
+config = read_uct_config()
 db = None
 db_filename = config['usn_db_copy']
 if len(args) > 0:
@@ -33,9 +34,6 @@ db = usn_lib.load_database(db_filename)
 columns = ['total', 'untriaged'] + cve_lib.priorities
 
 cves = dict()
-if opt.target == 'cve':
-    cve_lib.read_config()
-
 months = dict()
 month_cves = dict()
 for usn in sorted(db.keys()):
diff --git a/scripts/process_cves b/scripts/process_cves
index 97d15cc..334ae9b 100755
--- a/scripts/process_cves
+++ b/scripts/process_cves
@@ -11,6 +11,7 @@
 set -e
 
 loc=$(readlink -f "$(dirname "$0")/..")
+cache_dir="${loc}/cache/"
 
 uctconf="$HOME/.ubuntu-cve-tracker.conf"
 if [ -s "$uctconf" ]; then
@@ -26,6 +27,8 @@ is_http_url() {
 }
 
 download() {
+    local url
+
     url="$1"
     # if http, then download it with wget, otherwise, try to rsync it
     if is_http_url "$url" ; then
@@ -35,11 +38,11 @@ download() {
         file=$(basename "$url")
 
         echo "wget -N \"$url\""
-        wget -N "$url" ||:
-        gunzip -f "$file"
+        wget -N "$url" -P "${cache_dir}" ||:
+        gunzip -f "${cache_dir}/$file"
     else
-        echo "rsync -avz --progress \"$url\" ."
-        rsync -avz --progress "$url" .
+        echo "rsync -avz --progress \"$url\" ${cache_dir}"
+        rsync -avz --progress "$url" "${cache_dir}"
     fi
     echo "---"
 }
@@ -107,18 +110,18 @@ update_files() {
 
 full_refresh() {
     # get PubDate
-    echo "update PubDate (check-cves --refresh nvdcve-*.json) ..."
-    ./scripts/check-cves --refresh nvdcve-*.json
+    echo "update PubDate (check-cves --refresh ${cache_dir}/nvdcve-*.json) ..."
+    ./scripts/check-cves --refresh "${cache_dir}"/nvdcve-*.json
 
     # get authoritative descriptions
-    echo "get authoritative descriptions (check-cves --refresh ./allitems.xml) ..."
-    ./scripts/check-cves --refresh ./allitems.xml
+    echo "get authoritative descriptions (check-cves --refresh ${cache_dir}/allitems.xml) ..."
+    ./scripts/check-cves --refresh "${cache_dir}/allitems.xml"
 }
 
 refresh_cvss_score() {
     # get CVSS score from NVD
-    echo "update CVSS score (check-cves --score-refresh nvdcve-*.json) ..."
-    ./scripts/check-cves --score-refresh nvdcve-*.json
+    echo "update CVSS score (check-cves --score-refresh ${cache_dir}/nvdcve-*.json) ..."
+    ./scripts/check-cves --score-refresh "${cache_dir}"/nvdcve-*.json
 }
 
 kernel_team_merge() {
@@ -303,7 +306,7 @@ case "$action" in
         # kernel_team_merge
         echo "Skipping merge from kernel team"
         echo "check-cves nvdcve-*.json"
-        ./scripts/check-cves nvdcve-*.json
+        ./scripts/check-cves "${cache_dir}"/nvdcve-*.json
         # this will also do missing debian
         mistriaged
 
@@ -321,10 +324,10 @@ case "$action" in
         echo "Skipping merge from kernel team"
 
         # process new ones
-        echo "check-cves ./nvdcve-1.1-recent.json"
-        ./scripts/check-cves ./nvdcve-1.1-recent.json
-        echo "check-cves ./allitems.xml"
-        ./scripts/check-cves ./allitems.xml
+        echo "check-cves ${cache_dir}/nvdcve-1.1-recent.json"
+        ./scripts/check-cves "${cache_dir}/nvdcve-1.1-recent.json"
+        echo "check-cves ${cache_dir}/allitems.xml"
+        ./scripts/check-cves "${cache_dir}/allitems.xml"
 
         process_missing_debian
         # Disabling this due to https://listman.redhat.com/archives/rhsa-announce/2023-October/012854.html
@@ -396,7 +399,7 @@ case "$action" in
         update_files
         echo "check-cves --cve ${action}"
         # add all NVD details as well so we can get date etc for it
-        ./scripts/check-cves --cve ${action} nvdcve-*.json
+        ./scripts/check-cves --cve ${action} "${cache_dir}"/nvdcve-*.json
         check_syntax
         ;;
     *.json)
@@ -420,8 +423,8 @@ case "$action" in
         update_files
         # kernel_team_merge
         echo "Skipping merge from kernel team"
-        echo "check-cves ./nvdcve-1.1-recent.json"
-        ./scripts/check-cves ./nvdcve-1.1-recent.json
+        echo "check-cves ${cache_dir}/nvdcve-1.1-recent.json"
+        ./scripts/check-cves "${cache_dir}/nvdcve-1.1-recent.json"
         process_missing_debian
         refresh_cvss_score
         check_syntax
diff --git a/scripts/pull-in-progress.py b/scripts/pull-in-progress.py
index a4a57d9..0426e74 100755
--- a/scripts/pull-in-progress.py
+++ b/scripts/pull-in-progress.py
@@ -7,6 +7,7 @@
 import sys, os, os.path, re, urllib, tempfile
 import optparse, glob
 import cve_lib
+from utc.config import read_uct_config
 
 from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
 from launchpadlib.credentials import Credentials
@@ -57,8 +58,8 @@ def get_cves(bug):
 #
 from launchpadbugs.connector import ConnectBugList
 BugList = ConnectBugList()
-cve_lib.read_config()
-BugList.authentication = cve_lib.config["plb_authentication"]
+config = read_uct_config()
+BugList.authentication = config["plb_authentication"]
 
 url="https://bugs.launchpad.net/~ubuntu-security/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=INPROGRESS&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&field.bug_commenter=&field.subscriber=&field.status_upstream-empty-marker=1&field.omit_dupes.used=&field.has_patch.used=&field.has_patch=on&field.has_cve.used=&field.tag=&field.tags_combinator=ANY&search=Search"
 if opt.any:
diff --git a/scripts/report-bugs b/scripts/report-bugs
index bb2d78c..de6684a 100755
--- a/scripts/report-bugs
+++ b/scripts/report-bugs
@@ -14,6 +14,7 @@ import re
 import subprocess
 import sys
 import time
+from uct.config import read_uct_config
 
 parser = optparse.OptionParser()
 parser.add_option("--debug", help="Verbose processing output", action='store_true')
@@ -43,14 +44,14 @@ except:
     sys.exit(1)
 
 # Load configuration
-cve_lib.read_config()
+config = read_uct_config()
 
 # API interface
 debug("Connecting to LP ...\n")
 lp = lpl_common.connect()
 
 # Get authenticated URL fetcher
-opener = lpl_common.opener_with_cookie(cve_lib.config["plb_authentication"])
+opener = lpl_common.opener_with_cookie(config["plb_authentication"])
 if not opener:
     raise ValueError("Could not open cookies")
 
diff --git a/scripts/report-bugs-by-assignee b/scripts/report-bugs-by-assignee
index e34ba4b..2fc5d5e 100755
--- a/scripts/report-bugs-by-assignee
+++ b/scripts/report-bugs-by-assignee
@@ -30,9 +30,6 @@ def debug(s):
     if opt.debug:
         print( "DEBUG: %s" % s, file=sys.stderr)
 
-# Load configuration
-cve_lib.read_config()
-
 # API interface
 debug("Connecting to LP ...")
 lp = UCTLaunchpad(opt)
diff --git a/scripts/report-bugs-by-team b/scripts/report-bugs-by-team
index e3b71ec..da7ae74 100755
--- a/scripts/report-bugs-by-team
+++ b/scripts/report-bugs-by-team
@@ -19,6 +19,7 @@ import shutil
 import subprocess
 import sys
 import time
+from uct.config import read_uct_config
 
 try:
     import pickle
@@ -377,7 +378,7 @@ if all_bugs == None:
         sys.exit(1)
 
     # Load configuration
-    cve_lib.read_config()
+    config = read_uct_config()
 
     # API interface
     debug("Team: %s" % opt.team)
@@ -385,7 +386,7 @@ if all_bugs == None:
     lp = lpl_common.connect()
 
     # Get authenticated URL fetcher
-    opener = lpl_common.opener_with_cookie(cve_lib.config["plb_authentication"])
+    opener = lpl_common.opener_with_cookie(config["plb_authentication"])
     if not opener:
         raise ValueError("Could not open cookies")
 
diff --git a/scripts/report-bugs-for-eol b/scripts/report-bugs-for-eol
index 2cc2895..81e3397 100755
--- a/scripts/report-bugs-for-eol
+++ b/scripts/report-bugs-for-eol
@@ -17,6 +17,7 @@ import optparse
 import os
 import re
 import sys
+from uct.config import read_uct_config
 
 parser = optparse.OptionParser()
 parser.add_option("--debug", help="Verbose processing output", action='store_true')
@@ -138,14 +139,14 @@ if bugs == None:
         sys.exit(1)
 
     # Load configuration
-    cve_lib.read_config()
+    config = read_uct_config()
 
     # API interface
     print("Connecting to LP ...", end=' ', file=sys.stderr)
     lp = lpl_common.connect()
 
     # Get authenticated URL fetcher
-    opener = lpl_common.opener_with_cookie(cve_lib.config["plb_authentication"])
+    opener = lpl_common.opener_with_cookie(config["plb_authentication"])
     if not opener:
         raise ValueError("Could not open cookies")
 
diff --git a/scripts/report-cve-age.py b/scripts/report-cve-age.py
index 26c2925..7d28820 100755
--- a/scripts/report-cve-age.py
+++ b/scripts/report-cve-age.py
@@ -18,11 +18,12 @@ import os
 import sys
 import optparse
 import cve_lib, usn_lib
+from uct.config import read_uct_config
 
 import source_map
 source_map = source_map.load()
 releases = cve_lib.releases
-config = cve_lib.read_config()
+config = read_uct_config()
 
 default_db = config['usn_db_copy']
 if '-all' not in default_db:
diff --git a/scripts/report-mistriaged-cves.py b/scripts/report-mistriaged-cves.py
index 352a104..90e717f 100755
--- a/scripts/report-mistriaged-cves.py
+++ b/scripts/report-mistriaged-cves.py
@@ -9,6 +9,7 @@
 # Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
 # for details.
 import sys, os, os.path, optparse, cve_lib, re, subprocess, signal
+from uct.config import read_uct_config
 
 def subprocess_setup():
     # Python installs a SIGPIPE handler by default. This is usually not what
@@ -63,7 +64,7 @@ def check_apt_cache(package):
 
 
 # get base config for path to debian security tracker etc
-config = cve_lib.read_config()
+config = read_uct_config()
 
 parser = optparse.OptionParser()
 parser.add_option("-v", "--verbose", dest="verbose",
@@ -139,7 +140,7 @@ if len(dups) > 0:
     print('Duplicate entries in not-for-us.txt: ' + ' '.join(dups))
 
 # get the list of CVEs which Debian knows about
-debian = cve_lib.load_debian_cves(cve_lib.config['secure_testing_path'] +
+debian = cve_lib.load_debian_cves(config['secure_testing_path'] +
                                   '/data/CVE/list')
 
 cache = {}
diff --git a/scripts/report-packages.py b/scripts/report-packages.py
index bcc3406..33eb8a8 100755
--- a/scripts/report-packages.py
+++ b/scripts/report-packages.py
@@ -23,9 +23,10 @@ import cve_lib
 import usn_lib
 
 import source_map
+from uct.config import read_uct_config
 source_map = source_map.load()
 releases = cve_lib.releases
-config = cve_lib.read_config()
+config = read_uct_config()
 
 parser = optparse.OptionParser()
 parser.add_option("-S", "--skip-devel", help="Show only those CVEs *not* in the current devel release", action="store_true")
diff --git a/scripts/report-todo-mir b/scripts/report-todo-mir
index d598fb6..e820a43 100755
--- a/scripts/report-todo-mir
+++ b/scripts/report-todo-mir
@@ -14,7 +14,6 @@ import argparse
 import re
 import sys
 from launchpadlib.launchpad import Launchpad
-import cve_lib
 
 # by default we want all not in Fix Released, Fix Committed, Opinion, Invalid,
 # Won't Fix & Expired which leaves the following
@@ -225,9 +224,6 @@ def lookup_tasks_for_assignees(taskdb, assignees):
             taskdb.add(mir_task)
 
 
-# Load configuration
-cve_lib.read_config()
-
 # API interface
 debug("Connecting to LP ...")
 lp = Launchpad.login_anonymously("report-todo-mir", "production", version="devel")
diff --git a/scripts/report-todo-sponsoring b/scripts/report-todo-sponsoring
index 14c87d3..9294a40 100755
--- a/scripts/report-todo-sponsoring
+++ b/scripts/report-todo-sponsoring
@@ -151,9 +151,6 @@ except:
     print("lpl_common.py seems to be missing.  Please create a symlink from $UQT/common/lpl_common.py to $UCT/scripts/", file=sys.stderr)
     sys.exit(1)
 
-# Load configuration
-cve_lib.read_config()
-
 # API interface
 debug("Connecting to LP ...")
 lp = lpl_common.connect()
diff --git a/scripts/report-updates.py b/scripts/report-updates.py
index ed69604..4d939ca 100755
--- a/scripts/report-updates.py
+++ b/scripts/report-updates.py
@@ -13,11 +13,12 @@ import os
 import sys
 import time
 import usn_lib
+from uct.config import read_uct_config
 
 import source_map
 source_map = source_map.load()
 
-config = cve_lib.read_config()
+config = read_uct_config()
 default_db = config['usn_db_copy']
 if '-all' not in default_db:
     tmp = os.path.splitext(default_db)
diff --git a/scripts/report-version.py b/scripts/report-version.py
index 381c724..c1fdb7d 100755
--- a/scripts/report-version.py
+++ b/scripts/report-version.py
@@ -18,13 +18,14 @@ import sys
 import optparse
 import cve_lib
 import usn_lib
+from uct.config import read_uct_config
 
 import apt_pkg
 apt_pkg.init_system()
 
 releases = cve_lib.releases
 esm_releases = cve_lib.esm_releases + cve_lib.esm_infra_releases
-config = cve_lib.read_config()
+config = read_uct_config()
 
 parser = optparse.OptionParser()
 parser.add_option("--db", help="Specify the USN database to load", metavar="FILENAME", default=config['usn_db_copy'])
diff --git a/scripts/sis-changes b/scripts/sis-changes
index d33da15..fc398f4 100755
--- a/scripts/sis-changes
+++ b/scripts/sis-changes
@@ -23,6 +23,7 @@ import urllib.request, urllib.error, urllib.parse
 import cve_lib
 import json
 from source_map import version_compare, load
+from uct.config import read_uct_config
 
 try:
     import lpl_common
@@ -148,13 +149,13 @@ parser.add_option("--distribution", help="Distribution to use (eg, 'ubuntu-rtm')
 (opt, args) = parser.parse_args()
 
 # Load configuration
-cve_lib.read_config()
+config = read_uct_config()
 
 # API interface
 lp = lpl_common.connect(beta=opt.beta, uri=opt.uri)
 
 # Get authenticated URL fetcher
-opener = lpl_common.opener_with_cookie(cve_lib.config["plb_authentication"])
+opener = lpl_common.opener_with_cookie(config["plb_authentication"])
 if not opener:
     raise ValueError("Could not open cookies")
 
diff --git a/scripts/sis-generate-usn b/scripts/sis-generate-usn
index 8947779..9ae7a8b 100755
--- a/scripts/sis-generate-usn
+++ b/scripts/sis-generate-usn
@@ -23,6 +23,7 @@ import subprocess
 import sys
 import tempfile
 import cve_lib
+from uct.config import read_uct_config
 
 opter = optparse.OptionParser()
 opter.add_option("--debug", help="Verbose processing output", action='store_true')
@@ -60,7 +61,7 @@ for rel in cve_lib.releases:
 # components (-updates has gotten wrecked by things in -proposed before).
 release_suffixes = ['','-security','-updates']
 
-config = cve_lib.read_config()
+config = read_uct_config()
 archive = config['packages_mirror']
 cve_lib.check_mirror_timestamp(config, mirror='packages_mirror')
 
diff --git a/scripts/source_map.py b/scripts/source_map.py
index c5ec9ad..b6377fb 100755
--- a/scripts/source_map.py
+++ b/scripts/source_map.py
@@ -19,6 +19,7 @@ import sys
 import cve_lib
 import yaml
 import pickle
+from uct.config import read_uct_config
 
 built_using_tags = ["Built-Using", "Static-Built-Using", "X-Cargo-Built-Using"]
 apt_pkg.init_system()
@@ -27,28 +28,9 @@ apt_pkg.init_system()
 def version_compare(one, two):
     return apt_pkg.version_compare(one, two)
 
-def read_config_file(config_file):
-    '''Read in and do basic validation on config file'''
-    try:
-        from configobj import ConfigObj
-    except ImportError:
-        # Some releases lack this class, so reimplement it quickly
-        class ConfigObj(dict):
-            def __init__(self, filepath):
-                for line in open(filepath).readlines():
-                    line = line.strip()
-                    if line.startswith('#') or len(line) == 0:
-                        continue
-                    name, stuff = line.strip().split('=', 1)
-                    self[name] = eval(stuff)
-
-            def __attr__(self, name):
-                return self.stuff[name]
-
-    return ConfigObj(config_file)
 
 def _find_sources(pockets=None, releases=None, skip_eol_releases=True, arch='amd64'):
-    config = read_config_file(os.path.expanduser("~/.ubuntu-cve-tracker.conf"))
+    config = read_uct_config()
     if 'packages_mirror' in config:
         cve_lib.check_mirror_timestamp(config)
         return _find_from_mirror(config['packages_mirror'],
@@ -62,7 +44,7 @@ def _find_sources(pockets=None, releases=None, skip_eol_releases=True, arch='amd
 
 
 def _find_packages(pockets=None, releases=None, skip_eol_releases=True, arch='amd64'):
-    config = read_config_file(os.path.expanduser("~/.ubuntu-cve-tracker.conf"))
+    config = read_uct_config()
     if 'packages_mirror' in config:
         cve_lib.check_mirror_timestamp(config)
         return _find_from_mirror(config['packages_mirror'],
@@ -218,7 +200,7 @@ def load(data_type='sources', pockets=None, releases=None, skip_eol_releases=Tru
         raise ValueError("'data_type' should be either 'sources' or 'packages'")
 
     # Attempt to load from cached pickle files
-    config = read_config_file(os.path.expanduser("~/.ubuntu-cve-tracker.conf"))
+    config = read_uct_config()
     if 'packages_mirror' in config:
         mirror = config['packages_mirror']
 
diff --git a/scripts/sync-bugs-kernel.py b/scripts/sync-bugs-kernel.py
index 67b1bf9..21e1c32 100755
--- a/scripts/sync-bugs-kernel.py
+++ b/scripts/sync-bugs-kernel.py
@@ -29,6 +29,7 @@ import sys
 import cve_lib
 import urlparse
 from lp_lib import UCTLaunchpad
+from uct.config import read_uct_config
 
 priority_to_importance = {
     'critical':   'Critical',
@@ -61,7 +62,7 @@ uctlp = UCTLaunchpad(opt)
 # Use devel version so we get the bugtask delete API
 uctlp.lp_version = "devel"
 
-config = cve_lib.read_config()
+config = read_uct_config()
 ktools = config.get('kernel_team_tools_path', None)
 if not ktools:
     raise ValueError("'kernel_team_tools_path' missing in ~/.ubuntu-cve-tracker.conf")
diff --git a/scripts/sync-from-usns.py b/scripts/sync-from-usns.py
index c2a2a27..91a2ef3 100755
--- a/scripts/sync-from-usns.py
+++ b/scripts/sync-from-usns.py
@@ -24,6 +24,7 @@ import textwrap
 
 import cve_lib
 import usn_lib
+from uct.config import read_uct_config
 
 from source_map import version_compare, load
 
@@ -96,7 +97,7 @@ def parse_args():
     return args
 
 if __name__ == '__main__':
-    config = cve_lib.read_config()
+    config = read_uct_config()
 
     args = parse_args()
     if args.git_stage:
diff --git a/scripts/test_uct_config.py b/scripts/test_uct_config.py
new file mode 100755
index 0000000..c69fbda
--- /dev/null
+++ b/scripts/test_uct_config.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env pytest-3
+# -*- coding: utf-8 -*-
+#
+# Test functions for scripts/uct/config.py
+#
+# Author: Steve Beattie <[e-mail]>
+# Copyright (C) 2005-2024 Canonical Ltd.
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 2 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+import os
+import pytest
+
+# _read_config and _read_config_cached are private functions in
+# uct/config.py for testing
+# users of uct/config.py should only use read_config()
+from uct.config import _read_config, _read_config_cached
+
+TEST_DATA_DIR = "test/uct/config/"
+
+
+class TestUCTConfig:
+
+    def test_config_file_not_exist(self):
+        with pytest.raises(ValueError):
+            _read_config(os.path.join(TEST_DATA_DIR, "does-not-exist.conf"))
+
+    def test_plb_auth_config_not_exist(self):
+        with pytest.raises(ValueError):
+            _read_config(os.path.join(TEST_DATA_DIR, "plb_auth_does_not_exist.conf"))
+
+    def test_no_plb_auth_entry(self):
+        with pytest.raises(ValueError):
+            _read_config(os.path.join(TEST_DATA_DIR, "no_plb_auth_entry.conf"))
+
+    def test_plb_auth_config_simple(self):
+        expected = {
+            'plb_authentication': '/dev/null'
+        }
+        config = _read_config(os.path.join(TEST_DATA_DIR, "plb_auth_dev_null.conf"))
+        assert config == expected
+
+    def test_returned_config_is_singleton(self):
+        first_config = _read_config_cached(os.path.join(TEST_DATA_DIR, "uct.conf"))
+        second_config = _read_config_cached(os.path.join(TEST_DATA_DIR, "uct.conf"))
+        assert first_config is second_config
diff --git a/scripts/test_uct_suggestions.py b/scripts/test_uct_suggestions.py
new file mode 100755
index 0000000..43cfb75
--- /dev/null
+++ b/scripts/test_uct_suggestions.py
@@ -0,0 +1,124 @@
+#!/usr/bin/env pytest-3
+# -*- coding: utf-8 -*-
+
+# Author: Martin Pitt <[e-mail]>
+# Author: Kees Cook <[e-mail]>
+# Author: Jamie Strandboge <[e-mail]>
+# Author: Marc Deslauriers <[e-mail]>
+# Author: Steve Beattie <[e-mail]>
+# Copyright (C) 2005-2024 Canonical Ltd.
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 2 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+import unittest
+from uct.suggestions import IgnoreSuggestions
+
+class CheckCVETest(unittest.TestCase):
+    def test_get_ignore_suggestion(self):
+        '''"Ignore" suggestion text extraction'''
+
+        # instantiate the suggestion handler
+        h = IgnoreSuggestions()
+
+        self.assertEqual("Courier-Authlib", h.get_ignore_suggestion('''SQL injection vulnerability in authpgsqllib.c in Courier-Authlib before 0.62.0, when a non-Latin locale Postgres database is used, allows remote attackers to execute arbitrary SQL commands via query parameters containing apostrophes.'''))
+
+        self.assertEqual("Apple Mac OS X", h.get_ignore_suggestion('''Buffer overflow in the DirectoryService Proxy in DirectoryService in Apple Mac OS X through 10.6.8 allows remote attackers to execute arbitrary code or cause a denial of service (application crash) via unspecified vectors.'''))
+
+        self.assertEqual("KDE", h.get_ignore_suggestion('''HTMLTokenizer::scriptHandler in Konqueror in KDE 3.5.9 and 3.5.10 allows remote attackers to cause a denial of service (application crash) via an invalid document.load call that triggers use of a deleted object.  NOTE: some of these details are obtained from third party information.'''))
+
+        self.assertEqual("Sun Solaris", h.get_ignore_suggestion('''The name service cache daemon (nscd) in Sun Solaris 10 and OpenSolaris snv_50 through snv_104 does not properly check permissions, which allows local users to gain privileges and obtain sensitive information via unspecified vectors.'''))
+
+        self.assertEqual("Linux kernel", h.get_ignore_suggestion('''libata in the Linux kernel before 2.6.27.9 does not set minimum timeouts for SG_IO requests, which allows local users to cause a denial of service (Programmed I/O mode on drives) via multiple simultaneous invocations of an unspecified test program.'''))
+
+        self.assertEqual("iGaming", h.get_ignore_suggestion('''Multiple SQL injection vulnerabilities in iGaming 1.5 and earlier allow remote attackers to execute arbitrary SQL commands via the browse parameter to (1) previews.php and (2) reviews.php, and the (3) id parameter to index.php in a viewarticle action.'''))
+
+        self.assertEqual("PHP iCalendar", h.get_ignore_suggestion('''PHP iCalendar 2.24 and earlier allows remote attackers to bypass authentication by setting the phpicalendar and phpicalendar_login cookies to 1.'
+'''))
+
+        # Test length truncation, tweaked to avoid "has" matcher
+        self.assertEqual("** TEST CVE ** This candidate HAS been reserved by an", h.get_ignore_suggestion('''** TEST CVE ** This candidate HAS been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided.'''))
+
+        self.assertEqual("Sun OpenSolaris", h.get_ignore_suggestion('''Unspecified vulnerability in the root/boot archive tool in Sun OpenSolaris has unknown impact and local attack vectors, related to a "Temporary file vulnerability," aka Bug ID 6653455.'''))
+
+        self.assertEqual("Red Hat Certificate System", h.get_ignore_suggestion('''Red Hat Certificate System 7.2 uses world-readable permissions for password.conf and unspecified other configuration files, which allows local users to discover passwords by reading these files.'''))
+
+        self.assertEqual("Microsoft Internet Explorer", h.get_ignore_suggestion('''An unspecified function in the JavaScript implementation in Microsoft Internet Explorer creates and exposes a "temporary footprint" when there is a current login to a web site, which makes it easier for remote attackers to trick a user into acting upon a spoofed pop-up message, aka an "in-session phishing attack." NOTE: as of 20090116, the only disclosure is a vague pre-advisory with no actionable information. However, because it is from a well-known researcher, it is being assigned a CVE identifier for tracking purposes.'''))
+
+        self.assertEqual("Umer Inc Songs Portal", h.get_ignore_suggestion('''SQL injection vulnerability in albums.php in Umer Inc Songs Portal allows remote attackers to execute arbitrary SQL commands via the id parameter.'''))
+
+        self.assertEqual("Limbo CMS", h.get_ignore_suggestion('''SQL injection vulnerability in open.php in the Private Messaging (com_privmsg) component for Limbo CMS allows remote attackers to execute arbitrary SQL commands via the id parameter in a pms action to index.php.'''))
+
+        self.assertEqual("phpscripts Ranking Script", h.get_ignore_suggestion('''phpscripts Ranking Script allows remote attackers to bypass authentication and gain administrative access by sending an admin=ja cookie.'''))
+
+        self.assertEqual("A4Desk Event Calendar", h.get_ignore_suggestion('''PHP remote file inclusion vulnerability in index.php in A4Desk Event Calendar, when magic_quotes_gpc is disabled, allows remote attackers to execute arbitrary PHP code via a URL in the v parameter.'''))
+
+        self.assertEqual("Galatolo WebManager", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in result.php in Galatolo WebManager (GWM) 1.0 allows remote attackers to inject arbitrary web script or HTML via the key parameter.'''))
+
+        self.assertEqual("Sun OpenSolaris", h.get_ignore_suggestion('''Unspecified vulnerability in the process (aka proc) filesystem in Sun OpenSolaris snv_85 through snv_100 allows local users to gain privileges via vectors related to the contract filesystem.'''))
+
+        self.assertEqual("Dreampics Gallery Builder", h.get_ignore_suggestion('''SQL injection vulnerability in index.php in Dreampics Gallery Builder allows remote attackers to execute arbitrary SQL commands via the exhibition_id parameter in a gallery.viewPhotos action.'''))
+
+        self.assertEqual("Omilen Photo Gallery component for Joomla!", h.get_ignore_suggestion('''Directory traversal vulnerability in the Omilen Photo Gallery (com_omphotogallery) component Beta 0.5 for Joomla! allows remote attackers to include and execute arbitrary local files via directory traversal sequences in the controller parameter to index.php.'''))
+
+        self.assertEqual("Webform module for Drupal", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the Webform module 5.x before 5.x-2.7 and 6.x before 6.x-2.7, a module for Drupal, allows remote attackers to inject arbitrary web script or HTML via a submission.'''))
+
+        self.assertEqual("Itamar Elharar MusicGallery component for Joomla!", h.get_ignore_suggestion('''SQL injection vulnerability in the Itamar Elharar MusicGallery (com_musicgallery) component for Joomla! allows remote attackers to execute arbitrary SQL commands via the id parameter in an itempage action to index.php.  NOTE: the provenance of this information is unknown; the details are obtained solely from third party information.'''))
+
+        self.assertEqual("Kide Shoutbox component for Joomla!", h.get_ignore_suggestion('''The Kide Shoutbox (com_kide) component 0.4.6 for Joomla! does not properly perform authentication, which allows remote attackers to post messages with an arbitrary account name via an insertar action to index.php.  NOTE: the provenance of this information is unknown; the details are obtained solely from third party information.'''))
+
+        self.assertEqual("WP-Forum plugin for WordPress", h.get_ignore_suggestion('''Multiple SQL injection vulnerabilities in the WP-Forum plugin before 2.4 for WordPress allow remote attackers to execute arbitrary SQL commands via (1) the search_max parameter in a search action to the default URI, related to wpf.class.php; (2) the forum parameter to an unspecified component, related to wpf.class.php; (3) the topic parameter in a viewforum action to the default URI, related to the remove_topic function in wpf.class.php; or the id parameter in a (4) editpost or (5) viewtopic action to the default URI, related to wpf-post.php.'''))
+
+        self.assertEqual("ListMan extension for TYPO3", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the ListMan (nl_listman) extension 1.2.1 for TYPO3 allows remote attackers to inject arbitrary web script or HTML via unspecified vectors.'''))
+
+        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: Re: CVE request: multiple status.net issues'''))
+        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: CVE request: multiple status.net issues'''))
+        self.assertEqual("multiple status.net issues", h.get_ignore_suggestion('''ML-Date: 2011-01-25 12:08:05, ML-Subject: multiple status.net issues'''))
+        self.assertEqual("Mambo CMS", h.get_ignore_suggestion('''ML-Date: 2011-06-28 16:24:28, ML-Subject: Re: CVE Request: Mambo CMS 4.6.x | Multiple Cross Site Scripting Vulnerabilities'''))
+
+        self.assertEqual("Apple iOS", h.get_ignore_suggestion('''The DNAv4 protocol implementation in the DHCP component in Apple iOS before 6 sends Wi-Fi packets containing a MAC address of a host on a previously used network, which might allow remote attackers to obtain sensitive information about previous device locations by sniffing an unencrypted Wi-Fi network for these packets.'''))
+
+        self.assertEqual("Conceptronic", h.get_ignore_suggestion('''Multiple open redirect vulnerabilities on the Conceptronic C54APM access point with runtime code 1.26 allow remote attackers to redirect users to arbitrary web sites and conduct phishing attacks via (1) the submit-url parameter in a Refresh action to goform/formWlSiteSurvey or (2) the wlan-url parameter to goform/formWlanSetup.'''))
+
+        self.assertEqual("Tapjoy library for Android", h.get_ignore_suggestion('''The Tapjoy library for Android does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.'''))
+
+        self.assertEqual("kamkomesan application for Android", h.get_ignore_suggestion('''The kamkomesan (aka com.anek.kamkomesan) application 1.0 for Android does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.'''))
+
+        self.assertEqual("DataTables plugin for jQuery", h.get_ignore_suggestion('''Cross-site scripting (XSS) vulnerability in the DataTables plugin 1.10.8 and earlier for jQuery allows remote attackers to inject arbitrary web script or HTML via the scripts parameter to media/unit_testing/templates/6776.php.'''))
+
+        self.assertEqual("Cisco Application Policy Infrastructure Controller (APIC)", h.get_ignore_suggestion('''A vulnerability in Cisco Application Policy Infrastructure Controller (APIC) could allow an authenticated, remote attacker to gain higher privileges than the account is assigned.'''))
+
+        self.assertEqual("Redirection", h.get_ignore_suggestion('''Redirection version 2.7.3 contains a ACE via file inclusion vulnerability in Pass-through mode that can result in allows admins to execute any PHP file in the filesystem.'''))
+
+        self.assertEqual("topydo", h.get_ignore_suggestion('''topydo contains a CWE-20: Improper Input Validation vulnerability in ListFormatParser::parse, file topydo/lib/ListFormat.py line 292 as of d4f843dac71308b2f29a7c2cdc76f055c3841523 that can result in Injection of arbitrary bytes to the terminal,'''))
+
+        self.assertEqual("Joplin", h.get_ignore_suggestion('''Joplin version prior to 1.0.90 contains a XSS evolving into code execution due to enabled nodeIntegration.'''))
+
+        self.assertEqual("Foxit Reader", h.get_ignore_suggestion('''This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Foxit Reader 9.2.0.9297.'''))
+
+        self.assertEqual("Exquisite Ultimate Newspaper theme for WordPress", h.get_ignore_suggestion('''The Exquisite Ultimate Newspaper theme 1.3.3 for WordPress has XSS via the anchor identifier to assets/js/jquery.foundation.plugins.js.'''))
+        self.assertEqual("Weaver Xtreme Theme for WordPress", h.get_ignore_suggestion('''The Weaver Xtreme Theme for WordPress is vulnerable to stored Cross-Site Scripting due to insufficient escaping of the profile display name in versions up to, and including, 5.0.7.'''))
+        self.assertEqual("Weaver Show Posts Plugin for WordPress", h.get_ignore_suggestion('''The Weaver Show Posts Plugin for WordPress is vulnerable to stored Cross-Site Scripting due to insufficient escaping of the profile display name in versions up to, and including, 1.6.'''))
+        self.assertEqual("WordPress File Upload and WordPress File Upload Pro plugins for WordPress", h.get_ignore_suggestion('''The WordPress File Upload and WordPress File Upload Pro plugins for WordPress are vulnerable to Path Traversal in versions up to, and including, 4.19.1 via the vulnerable parameter wfu_newpath.'''))
+        self.assertEqual("browserless-chrome", h.get_ignore_suggestion('''This affects all versions of package browserless-chrome. User input flowing from the workspace endpoint gets used to create a file path filePath and this is fetched and then sent back to a user. This can be escaped to fetch arbitrary files from a server.'''))
+        self.assertEqual("@absolunet/kafe", h.get_ignore_suggestion('''This affects the package @absolunet/kafe before 3.2.10. It allows cause a denial of service when validating crafted invalid emails.'''))
+        self.assertEqual("Magneticlab Sàrl Homepage Pop-up plugin", h.get_ignore_suggestion('''Cross-Site Request Forgery (CSRF) vulnerability in Magneticlab Sàrl Homepage Pop-up plugin <= 1.2.5 versions.'''))
+
+        # handle odd whitespace
+        self.assertEqual("PTC Vuforia Studio does not require a token.", h.get_ignore_suggestion('''
+
+
+
+
+ PTC Vuforia Studio does not require a token.
+                                                                  '''))
+        self.assertEqual("zxcvbn-ts", h.get_ignore_suggestion('''zxcvbn-ts is a password strength estimator written in typescript.'''))
+        self.assertEqual("VMware Tools for Windows", h.get_ignore_suggestion('''VMware Tools for Windows (10.x.y prior to 10.3.10, 11.x.y prior to 11.1.5) contains a denial-of-service vulnerability due to NULL pointer dereference in vm3dmp driver. Successful exploitation of this issue may allow attackers with normal user privileges to create a denial-of-service condition on the guest machine where VMware Tools is installed.'''))
+
+        self.assertEqual("3DPrint WordPress plugin", h.get_ignore_suggestion("The 3DPrint WordPress plugin before 3.5.6.9 does not protect against CSRF attacks in the modified version of Tiny File Manager included with the plugin, allowing an attacker to craft a malicious request that will create an archive of any files or directories on the target server by tricking a logged in admin into submitting a form."))
+
+        self.assertEqual("Feathersjs", h.get_ignore_suggestion('''Feathersjs is a framework for creating web APIs and real-time applications with TypeScript or JavaScript.'''))
+
+        self.assertEqual("PaulPrinting CMS 2018", h.get_ignore_suggestion("A vulnerability, which was classified as problematic, was found in PaulPrinting CMS 2018"))
+        
diff --git a/scripts/uct/config.py b/scripts/uct/config.py
new file mode 100644
index 0000000..9476199
--- /dev/null
+++ b/scripts/uct/config.py
@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#
+#  Copyright (C) 2008-2024 Canonical Ltd.
+#  Author: Jamie Strandboge <[e-mail]>
+#  Author: Kees Cook <[e-mail]>
+#  Author: Steve Beattie <[e-mail]>
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 3 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+from __future__ import print_function
+
+import os
+
+# nobody should be using ubuntu 14.04 LTS with this code, so we don't
+# need a fallback implementation to ConfigObj
+from configobj import ConfigObj
+
+# private singleton object, use read_config() to access
+_config = None
+
+
+# _read_config: does the actual work without writing to the global
+# singleton. This can be used in tests where read_config() will always
+# return the same thing.
+def _read_config(_config_file=None):
+
+    if _config_file:
+        config_file = _config_file
+    else:
+        config_file = os.path.join(
+            os.path.expanduser("~"), ".ubuntu-cve-tracker.conf"
+        )
+
+    if not os.path.exists(config_file):
+        raise ValueError("Could not find '%s'" % (config_file))
+
+    config = ConfigObj(config_file)
+
+    validate_config(config, config_file)
+
+    return config
+
+
+# _read_config_cached: if the results of parsing the config file have
+# not already been cached, save it; otherwise return the singleton.
+# Takes an optional config file path as an argument. This can be used in tests
+# where read_config() will only use ~/.ubuntu-cve-tracker.conf
+def _read_config_cached(_config_file=None):
+    global _config
+
+    # if we've already parsed the config file, return what we already
+    # have
+    if not _config:
+        _config = _read_config()
+
+    return _config
+
+
+# read_uct_config: call with no arguments to get the contents of
+# ~/.ubuntu-cve-tracker.conf
+def read_uct_config():
+    return _read_config_cached()
+
+
+# XXX-FIXME this should use configobj's Validator stuff
+def validate_config(config, config_file):
+
+    # Validate required arguments
+    if "plb_authentication" not in config:
+        raise ValueError(("Could not find 'plb_authentication' entry in %s." % (config_file)))
+    if not os.path.exists(config["plb_authentication"]):
+        raise ValueError(("Could not find file specified by 'plb_authentication' in %s." % (config_file)))
diff --git a/scripts/check_cves/ignored_cache.py b/scripts/uct/ignored_cache.py
similarity index 100%
rename from scripts/check_cves/ignored_cache.py
rename to scripts/uct/ignored_cache.py
diff --git a/scripts/uct/suggestions.py b/scripts/uct/suggestions.py
new file mode 100755
index 0000000..fe496fc
--- /dev/null
+++ b/scripts/uct/suggestions.py
@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+# suggestions module
+#
+# Author: Martin Pitt <[e-mail]>
+# Author: Kees Cook <[e-mail]>
+# Author: Jamie Strandboge <[e-mail]>
+# Author: Marc Deslauriers <[e-mail]>
+# Author: Steve Beattie <[e-mail]>
+# Copyright (C) 2005-2024 Canonical Ltd.
+#
+# This script is distributed under the terms and conditions of the GNU General
+# Public License, Version 2 or later. See http://www.gnu.org/copyleft/gpl.html
+# for details.
+
+import re
+
+class IgnoreSuggestions:
+
+    def get_ignore_suggestion(self, text):
+        '''Try to find a reasonable suggestion for the user.'''
+        suggestion = ""
+
+        # remove any whitespace
+        rev_text = text.strip()
+        # strip out the added mailing list stuff (locate_cves.py importing)
+        rev_text = re.sub(r'^ML-Date: .* ML-Subject: ', '', rev_text)
+        rev_text = re.sub(r'^(|Re: )CVE (r|R)equest: ', '', rev_text)
+
+        first_sentence = re.split(r'\. ', rev_text)[0]
+
+        # hunt for module/component
+        nouns = ["library", "templates?", "components?", "modules?", "[Pp]lug-?ins?", "extension", "application", "[Tt]hemes?"]
+        products = ["Joomla!", "Drupal", "WordPress", "TYPO3", "Mambo", "Android", "jQuery"]
+        match = re.search(r'(?: in t|^T)he (.*) (%s) (?:.* )?for (%s)' % ("|".join(nouns), "|".join(products)),
+                          first_sentence)
+        preposition = "for"
+        if not match:
+            # try swapping order of product and noun in which case there is no preposition
+            match = re.search(r'(?: in t|^T)he (.*) (%s) (%s) (?:.* )' % ("|".join(products), "|".join(nouns)),
+                              first_sentence)
+            preposition = ""
+        if match:
+            module = match.group(1)
+            noun = match.group(2)
+            product = match.group(3)
+            for marker in [" module", " ("]:
+                if marker in module:
+                    module = module.split(marker)[0]
+            return " ".join(filter(lambda x: len(x) > 0, [module, noun, preposition, product]))
+
+        # hunt for a description of a thing written in a language or for creating something else
+        match = re.search(r'(.*) is a (?:.*) (written in|for creating) (?:.*)', first_sentence)
+        if match:
+            return match.group(1)
+
+        # hunt for a product on a platform
+        match = re.search(r'.* for (Windows|Linux|Mac OS|iOS|Android|iPhone|iPad|BlackBerry|Symbian)', first_sentence)
+        if match:
+            return match.group(0)
+
+        # look for common pattern
+        match = re.search(r'A vulnerability(?:, which was)? classified as (?:.*),? was found in (.*)', first_sentence)
+        if match:
+            return match.group(1)
+
+        # drop commas-extensions
+        if ',' in first_sentence:
+            first_sentence = re.split(r',', first_sentence)[0]
+        phrases = re.split(r' [io]n ', first_sentence)
+
+        # default to the last phrase
+        suggestion = phrases[-1]
+        # move to earlier phrase if suggestion starts with "a"
+        if suggestion.startswith('a ') and len(phrases) > 1:
+            suggestion = phrases[-2]
+
+        version_preps = r'(\s+(before|through|prior to|versions?|[<>]=?))+\s*'
+        version_regex = r'\s+([a-zA-Z\._\-]*[0-9]+[a-zA-Z\._\-]*)+'
+        # prefer 'Apple iOS before <version>' or 'Apple Mac OS X through
+        # <version' in the last phrase over other suggestions
+        if not re.search(r'' + version_preps + version_regex, suggestion):
+            # grab the first phrase with something that may be a version number
+            for p in phrases:
+                if re.search(r'' + version_regex, p):
+                    suggestion = p
+                    break
+                if re.search(r'^[^,]+\s+for\s+', p):
+                    suggestion = p
+                    break
+
+        # try to find a good suggestion from the phrase (ie suggest 'Linux
+        # kernel' from 'the Linux kernel before 2.6.27')
+        suggestion = re.split(r'\s+([a-zA-Z\._\-]*[0-9]+[a-zA-Z\._\-]*)+', suggestion)[0]
+        # "blah in component for Software"
+        if re.search(r'^[^,]+\s+for\s+', suggestion):
+            suggestion = re.split(r'[^,]+\s+for\s+', suggestion)[1]
+
+        # Chop off action verbs
+        cleanup_regexes = [
+            # clean up leading "in" or "the"
+            r'^\s*([tT]he|[iI]n)\s+',
+            # clean up trailing version prepositions like "before" or "through"
+            # from version details
+            r'' + version_preps + '$',
+            # clean up trailing parens
+            r'\s+\([^\)]+\)\s*$',
+            # action verbs
+            r'\s+(has|creates|allows|could|contains)($|\s+.*)',
+            # "vulnerbale installations of"
+            r'(^|\s+)vulnerable\sinstallations?\sof($|\s+)',
+            # This affects all versions of package
+            r'^This affects all versions of package\s+',
+            # This affects the package
+            r'^This affects the package\s+',
+        ]
+
+        for regex in cleanup_regexes:
+            if re.search(regex, suggestion):
+                suggestion = re.sub(regex, '', suggestion)
+
+        # if the phrase is too long, truncate it to max_length, but make
+        # sure we don't have a partial word at the end
+        max_length = 64
+        if len(suggestion) > max_length:
+            suggestion = suggestion[:max_length]
+            suggestion = re.sub(r'\s+\w+$', '', suggestion)
+
+        return suggestion
+
diff --git a/scripts/usn_lib.py b/scripts/usn_lib.py
index 3e45239..d8c7622 100644
--- a/scripts/usn_lib.py
+++ b/scripts/usn_lib.py
@@ -16,7 +16,7 @@ import subprocess
 import sys
 import urllib
 from source_map import version_compare
-import cve_lib
+from uct.config import read_uct_config
 
 try:
     import cPickle
@@ -164,7 +164,7 @@ class USNdb(object):
         if db:
             db_name = db
         else:
-            config = cve_lib.read_config()
+            config = read_uct_config()
             if config['usn_db_copy']:
                 db_name = config['usn_db_copy']
 
diff --git a/test/uct/config/no_plb_auth_entry.conf b/test/uct/config/no_plb_auth_entry.conf
new file mode 100644
index 0000000..755ad8c
--- /dev/null
+++ b/test/uct/config/no_plb_auth_entry.conf
@@ -0,0 +1,2 @@
+# python-launchpad-bugs authentication cookies file (sis-changes)
+not_plb_authentication="/bin/true"
diff --git a/test/uct/config/plb_auth_dev_null.conf b/test/uct/config/plb_auth_dev_null.conf
new file mode 100644
index 0000000..1e33174
--- /dev/null
+++ b/test/uct/config/plb_auth_dev_null.conf
@@ -0,0 +1,2 @@
+# python-launchpad-bugs authentication cookies file (sis-changes)
+plb_authentication="/dev/null"
diff --git a/test/uct/config/plb_auth_does_not_exist.conf b/test/uct/config/plb_auth_does_not_exist.conf
new file mode 100644
index 0000000..9a81f39
--- /dev/null
+++ b/test/uct/config/plb_auth_does_not_exist.conf
@@ -0,0 +1,2 @@
+# python-launchpad-bugs authentication cookies file (sis-changes)
+plb_authentication="/does/not/exist"
diff --git a/test/uct/config/uct.conf b/test/uct/config/uct.conf
new file mode 100644
index 0000000..8bf6334
--- /dev/null
+++ b/test/uct/config/uct.conf
@@ -0,0 +1,15 @@
+# python-launchpad-bugs authentication cookies file (sis-changes)
+plb_authentication="/dev/null"
+
+# path to Debian "security-tracker" GIT tree (check-cves)
+secure_testing_path='/home/ubuntu-security/git/cve_trackers/debian-security-tracker'
+
+# path to archive-layout mirror of supported architectures
+# sis-generate-usn, packages-mirror)
+packages_mirror='/home/ubuntu-security/archive-metadata/ubuntu-archive-metadata'
+
+# same as packages_mirror, but for Debian testing repository
+debian_mirror='/home/ubuntu-security/archive-metadata/debian-testing-archive-metadata'
+
+# path to usn-tool bzr tree, used to manipulate USN databases
+usn_tool='/home/ubuntu-security/git/usn-tool'