Merge ~alexmurray/ubuntu-cve-tracker:check-syntax-n-jobs into ubuntu-cve-tracker:master

[Alex Murray]

check-syntax can quite easily monopolise your whole CPU making it quite unresponsive - add an option so we can change the number of processes it uses in parallel to limit this impact

https://chat.canonical.com/canonical/pl/4td4fgdp1tg7bmzbfc7wz8yxtr

[Eduardo Barretto (ebarretto)]

lgtm, thanks!

the CI issue is unrelated to this and was fixed already, so a rebase on top of master should make the CI pass.

review: Approve

diff --git a/scripts/check-syntax b/scripts/check-syntax
index 28a62a4..37c21cf 100755
--- a/scripts/check-syntax
+++ b/scripts/check-syntax
@@ -21,6 +21,7 @@ import re
 import sys
 import subprocess
 import signal
+import concurrent.futures
 import multiprocessing
 
 import cve_lib
@@ -249,6 +250,13 @@ parser.add_option(
     action="store_true",
     default=False,
 )
+parser.add_option(
+    "-j",
+    "--jobs",
+    type=int,
+    help="Number of jobs to run simultaneously (default: %d)" % multiprocessing.cpu_count(),
+    default=multiprocessing.cpu_count(),
+)
 # parser.add_option("-c", "--cna", help="Ensure every CVE assigned by Canonical's CNA has an entry", action='store_true')
 (opt, args) = parser.parse_args()
 
@@ -1055,7 +1063,10 @@ def check_cve(cve):
     debug(data)
     return cve_okay
 
-with multiprocessing.Pool(multiprocessing.cpu_count()) as p:
+# use ProcessPoolExecutor for parallel processing since this is more efficient
+# than multiple threads via ThreadPoolExecutor as we can keep each at 100% CPU
+# utilisation
+with concurrent.futures.ProcessPoolExecutor(max_workers=opt.jobs) as p:
     results = p.map(check_cve, args)
     all_okay = all(results)