Merge lp:~pkern/ubuntu-cve-tracker/master into lp:~ubuntu-security/ubuntu-cve-tracker/master

[Philipp Kern]

The JSON USN database is huge and can benefit a lot from compression. It does not look like it already serves a compressed copy. Furthermore the cron parts are missing in the source, hence I simply patched the pickle to JSON converter.

[Philipp Kern (pkern)]

FWIW, for instance Cloud services like AppEngine can only download 32 MiB per chunk over HTTP, hence this proposal.

[Jamie Strandboge (jdstrand)]

We already offer compressed (bz2) alongside the uncompressed files.

From IRC:
07:37 < jdstrand> mdeslaur, and pkern: so, there are already bz2 files created:
                  cd $UCT/scripts/fetch-db database.pickle.bz2
07:37 < jdstrand> ...
07:38 < jdstrand> = Fetching database.pickle.bz2 =
07:38 < jdstrand> --2014-09-10 07:37:36--
                  https://usn.ubuntu.com/usn-db/database.pickle.bz2
07:38 < jdstrand> ...
07:38 < jdstrand> = Fetching database.pickle.bz2.sha256 =
07:38 < jdstrand> --2014-09-10 07:37:49--
                  https://usn.ubuntu.com/usn-db/database.pickle.bz2.sha256
07:38 < jdstrand> ...
07:38 < jdstrand> = Verifiying database.pickle.bz2.sha256 =
07:38 < jdstrand> ./database.pickle.bz2: OK
07:38 < jdstrand> = Uncompressing database.pickle.bz2 =
07:38 < jdstrand> Download complete. File saved to 'database.pickle'
07:38 < jdstrand> so that is the pickle file
07:38 < jdstrand> but the json is there too
07:38 < jdstrand> $ $UCT/scripts/fetch-db database.json.bz2
07:38 < jdstrand> = Fetching database.json.bz2 =
07:38 < jdstrand> --2014-09-10 07:38:47--
                  https://usn.ubuntu.com/usn-db/database.json.bz2
07:38 < jdstrand> ...
07:39 < jdstrand> = Fetching database.json.bz2.sha256 =
07:39 < jdstrand> --2014-09-10 07:38:55--
                  https://usn.ubuntu.com/usn-db/database.json.bz2.sha256
07:39 < jdstrand> ...
07:39 < jdstrand> = Verifiying database.json.bz2.sha256 =
07:39 < jdstrand> ./database.json.bz2: OK
07:39 < jdstrand> = Uncompressing database.json.bz2 =
07:39 < jdstrand> Download complete. File saved to 'database.json'

Unmerged revisions

8455. By pkern <pkern@localhost> on 2014-09-08

Save the JSON database gzip'ed as well, which makes for much smaller files.

=== modified file 'scripts/convert-pickle.py'
--- scripts/convert-pickle.py	2012-04-26 21:27:31 +0000
+++ scripts/convert-pickle.py	2014-09-08 07:37:44 +0000
@@ -9,16 +9,28 @@
 
 import codecs
 import cPickle
+import cStringIO
+import gzip
 import json
 import optparse
 import os
+import shutil
 import sys
 
 # TODO: move these to usn_lib.py
 def save_database_json(database, db_filename):
     '''Save usn database'''
     filename = os.path.expanduser(db_filename)
-    json.dump(database, open(filename, 'w'), -1, encoding="utf-8")
+    buf = cStringIO.StringIO()
+    json.dump(database, buf, -1, encoding="utf-8")
+    buf.seek(0)
+    print >>sys.stderr, "INFO: Saving %s..." % (filename)
+    with open(filename, 'w') as outf:
+        shutil.copyfileobj(buf, outf)
+    buf.seek(0)
+    print >>sys.stderr, "INFO: Saving %s..." % (filename + '.gz')
+    with gzip.GzipFile(filename + '.gz', 'w') as outf:
+        shutil.copyfileobj(buf, outf)
 
 def convert_pickle_to_json(indb, outdb):
     '''Convert a pickle database into a json'''
@@ -47,7 +59,6 @@
     if count > 0:
         print >>sys.stderr, "WARN: performed %d pickle decode conversions" % count
 
-    print >>sys.stderr, "INFO: Saving %s..." % (outdb)
     save_database_json(new_db, outdb)
 
 #