Merge ~eslerm/ubuntu-cve-tracker:sync-nvd into ubuntu-cve-tracker:master

[Mark Esler]

update_files() code to download NVD 2.0 API data

Archive mode is added to rsync for recursion. Doesn't alter other, non-recursive, downloads.

Eventually, nvd2_loc will replace nvd_loc and 1.0 data and tooling can be discarded.

If approved, I'll update ~/.ubuntu-cve-tracker.conf documentation before merging.

[Alex Murray (alexmurray)]

nvd2_loc should have a default value please.

review: Needs Fixing

[Mark Esler (eslerm)]

Alex, nvd2_loc is defined in ~/.ubuntu-cve-tracker.conf as:

nvd2_loc="people.canonical.com:/home/ubuntu-security/nvd-database"

Before merging, I'd update https://wiki.ubuntu.com/SecurityTeam/UbuntuCVETracker

[Alex Murray (alexmurray)]

Updating documentation is not sufficient IMO - the code should be robust and define sensible default since not everyone will have updated their local configuration before using this new code. FWIW I found this since I tried testing this and it failed as I do not have a nvd2_loc defined in my ~/.ubuntu-cve-tracker.conf

[Mark Esler (eslerm)]

I'm happy to set a new convention. Existing code (for `mitre_loc` and `nvd_loc`) is not defined this way, but has URL alternatives (which will not work for NVD 2.0).

For a sensible default, I could hardcode the location to people, which could be overridden by a definition in ~/.ubuntu-cve-tracker.conf

Does that sound good to you?

[Alex Murray (alexmurray)]

Yes I think hard-coding people is the best option here - thanks.

[Mark Esler (eslerm)]

Done.

Tested without defining variable, setting variable correctly, and setting to the pre-existing `deleteme.txt` path on people.

[Alex Murray (alexmurray)]

LGTM.

review: Approve

diff --git a/scripts/process_cves b/scripts/process_cves
index d42c160..06bdb20 100755
--- a/scripts/process_cves
+++ b/scripts/process_cves
@@ -38,8 +38,8 @@ download() {
         wget -N "$url" ||:
         gunzip -f "$file"
     else
-        echo "rsync -vz --progress \"$url\" ."
-        rsync -vz --progress "$url" .
+        echo "rsync -avz --progress \"$url\" ."
+        rsync -avz --progress "$url" .
     fi
     echo "---"
 }
@@ -97,9 +97,12 @@ update_debian() {
 }
 
 update_files() {
+    # deprecate NVD 1.0 data in 2024
     download "${mitre_loc:-https://cve.mitre.org/data/downloads}/allitems.xml"
     download_yearly_files
     download "https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json"
+    # NVD 2.0 API will replace and become nvd_loc in December 2023
+    download "${nvd2_loc:-people.canonical.com:/home/ubuntu-security/nvd-database}"
 }
 
 full_refresh() {