Merge ~emitorino/ubuntu-cve-tracker:updating_report_scripts into ubuntu-cve-tracker:master

[Emilia Torino]

Existing scripts to report USNs & CVEs including main + universe did not have a --before option. So I am adding --before to the main bash file as well as to the supporting report_universe_cves python script

[Steve Beattie (sbeattie)]

It might be useful to support passing a friendlier time format, but that's a bit out of scope for this feature. One comment below; otherwise, LGTM.

Thanks.

review: Approve

[Steve Beattie (sbeattie)]

Okay, I managed to not save my inline comment, and now lp won't let me add one.

All I pointed out was that, because you already set a default in the add_argument() call, you don't need to do the if else logic around args.before (becuase args.before will always be set); you can just assign before_date the value (or choose to use args.before directly).

Thanks.

798f47b... by Emilia Torino on 2019-11-06

adding code review suggestions

93c0a17... by Emilia Torino on 2019-11-06

merge

6cb9aeb... by Emilia Torino on 2019-11-06

removing --before arg validation as we have the default value

[Emilia Torino (emitorino)]

> It might be useful to support passing a friendlier time format, but that's a
> bit out of scope for this feature. One comment below; otherwise, LGTM.

Yeah, I agree. I did it this way as other scripts work that way as well. An option would be to ask for releases names and then lookup in the release_stamps dict from cve_lib.py.

I will add a card to our trello

Thanks!

> Thanks.

diff --git a/scripts/report_universe_cves.py b/scripts/report_universe_cves.py
index 98876bc..c5fd172 100755
--- a/scripts/report_universe_cves.py
+++ b/scripts/report_universe_cves.py
@@ -20,12 +20,13 @@ import os
 import glob
 import sys
 import subprocess
+import time
 
 arg_parser = argparse.ArgumentParser(description='Counts CVEs for trusty/esm')
-group = arg_parser.add_mutually_exclusive_group(required=True)
-group.add_argument('-a', '--all-time', action='store_true', help="get all CVEs since ESM inception")
-group.add_argument('-s', '--since', action='store', help="Report only CVEs fixed after this date (in seconds "
+arg_parser.add_argument('-s', '--since', action='store', help="Report only CVEs fixed after this date (in seconds "
                                                                "from UTC)", default=0)
+arg_parser.add_argument('-b', '--before', action='store', help="Report only CVEs fixed before this date (in seconds "
+                                                               "from UTC)", default=time.time())
 args, extra_arg = arg_parser.parse_known_args()
 release = extra_arg[0]
 
@@ -44,21 +45,17 @@ if not os.path.isdir(private_esm_dir):
 os.chdir(private_esm_dir)
 private_cves = [f for f in glob.glob1(".", "CVE-*")]
 report_cves=[]
-if args.since:
-    since_date = args.since
-else:
-    since_date = 0
 
 for cve in private_cves:
     if os.path.isfile('../esm-universe/' + cve):
         if release in open('../esm-universe/' + cve).read():
-            args = ['git', 'log', '-1', '--format=%at', '../esm-universe/' + cve]
+            cmd_args = ['git', 'log', '-1', '--format=%at', '../esm-universe/' + cve]
         else:
-            args = ['git', 'log', '-1', '--format=%at', cve]
+            cmd_args = ['git', 'log', '-1', '--format=%at', cve]
     else:
-        args = ['git', 'log', '-1', '--format=%at', cve]
-    cve_creation_timestamp = subprocess.check_output(args)
-    if cve_creation_timestamp >= since_date:
+        cmd_args = ['git', 'log', '-1', '--format=%at', cve]
+    cve_creation_timestamp = float(subprocess.check_output(cmd_args))
+    if float(args.since) <= cve_creation_timestamp <= float(args.before):
         report_cves.append(cve)
 
 print "Total CVEs fixed for %s ESM universe: %d" % (release, len(report_cves))
diff --git a/scripts/report_usns_and_cves_including_universe b/scripts/report_usns_and_cves_including_universe
index de63315..c155b1c 100644
--- a/scripts/report_usns_and_cves_including_universe
+++ b/scripts/report_usns_and_cves_including_universe
@@ -27,11 +27,13 @@ EOM
 }
 
 since=0
+before=$( date +%s )
 details="no"
-while getopts "hdr:s:" opt ; do
+while getopts "hdr:s:b:" opt ; do
     case "$opt" in
         r) release_name="$OPTARG";;
         s) since="$OPTARG";;
+        b) before="$OPTARG";;
         d) details="yes";;
         h) help ; exit 0;;
         ?) help ; exit 1;;
@@ -50,9 +52,8 @@ IFS=':'
 
 # cd to $UCT directory if not there already
 cd $UCT
-
 #Get main USNs and CVEs stats.
-MAIN_USNS_AND_CVES=$(./scripts/report-fixes-by-usn.py "${release_name}" --since "${since}" --summary 2>&1)
+MAIN_USNS_AND_CVES=$(./scripts/report-fixes-by-usn.py "${release_name}" --since "${since}" --before "${before}" --summary 2>&1)
 while read -r line; do
   read -ra ADDR <<< "$line"
   if [ "${ADDR[0]}" == "CVEs" ]; then
@@ -65,7 +66,7 @@ while read -r line; do
 done <<< "$MAIN_USNS_AND_CVES"
 
 #Universe CVEs are not present in USNs so we need to query them separately
-UNIVERSE_CVES_INFO=$(./scripts/report_universe_cves.py "${release_name}" --since "${since}"2>&1)
+UNIVERSE_CVES_INFO=$(./scripts/report_universe_cves.py "${release_name}" --since "${since}" --before "${before}"2>&1)
 while read -r line; do
   read -ra ADDR <<< "$line"
   if [ "${ADDR[0]}" == "CVEs" ]; then