Merge lp:~mvo/daisy/problem-type-column-family into lp:daisy

[Michael Vogt]

This is my first jump into the world of cassandra, so please be gentle with me :)

This branch adds a new ColumnFamily of type "ProblemReport" so that its quicker to send a query that filters out all problem reports of type "Snap".

Mostly pushing this to get things started and to see if I am vaguely in the right direction.

[Brian Murray (brian-murray)]

In reflection, I think Evan's idea of having a Snaps ColumnFamily similar to the OOPS ColumnFamily makes sense, but that we should also add a DaySnaps CF similar to DayOOPS CF.

[lp:~mvo/daisy/problem-type-column-family]

740. By Michael Vogt on 2017-02-21

fix ordering

Unmerged revisions

740. By Michael Vogt on 2017-02-21

fix ordering

739. By Michael Vogt on 2017-02-21

add ColumnFamily for ProblemType

738. By Michael Vogt on 2017-02-21

add ProblemType ColumnFamily

=== modified file 'daisy/schema.py'
--- daisy/schema.py	2014-08-05 17:52:36 +0000
+++ daisy/schema.py	2017-02-21 17:37:20 +0000
@@ -93,6 +93,10 @@
             workaround_1779(mgr.create_column_family, keyspace, 'SystemImages',
                 key_validation_class=UTF8_TYPE,
                 comparator_type=UTF8_TYPE)
+        if 'ProblemType' not in cfs:
+            workaround_1779(mgr.create_column_family, keyspace, 'ProblemType',
+                key_validation_class=UTF8_TYPE,
+                comparator_type=UTF8_TYPE)
     finally:
         mgr.close()
 

=== modified file 'daisy/submit.py'
--- daisy/submit.py	2016-11-03 16:39:25 +0000
+++ daisy/submit.py	2017-02-21 17:37:20 +0000
@@ -300,6 +300,7 @@
     oops_cf = pycassa.ColumnFamily(_pool, 'OOPS')
     stacktrace_cf = pycassa.ColumnFamily(_pool, 'Stacktrace')
     images_cf = pycassa.ColumnFamily(_pool, 'SystemImages')
+    problem_type_cf = pycassa.ColumnFamily(_pool, 'ProblemType')
     report = create_report_from_bson(data)
 
     # gather and insert image information in the SystemImages CF
@@ -333,6 +334,11 @@
         except NotFoundException:
             images_cf.insert('device_image', {device_image : ''})
 
+    # make problem-type easily accessible
+    problem_type = report.get('ProblemType')
+    if problem_type:
+        problem_type_cf.insert(problem_type, {oops_id: ''})
+
     # Recoverable Problem, Package Install Failure, Suspend Resume
     crash_signature = report.get('DuplicateSignature')
     if crash_signature:

=== added file 'tools/back_populate_problem_type.py'
--- tools/back_populate_problem_type.py	1970-01-01 00:00:00 +0000
+++ tools/back_populate_problem_type.py	2017-02-21 17:37:20 +0000
@@ -0,0 +1,46 @@
+#!/usr/bin/python
+
+import sys
+import pycassa
+from pycassa.cassandra.ttypes import NotFoundException
+from collections import defaultdict
+from daisy import config
+
+creds = {'username': config.cassandra_username,
+         'password': config.cassandra_password}
+pool = pycassa.ConnectionPool(config.cassandra_keyspace,
+                              config.cassandra_hosts, timeout=600,
+                              credentials=creds)
+
+dayoops_cf = pycassa.ColumnFamily(pool, 'DayOOPS')
+oops_cf = pycassa.ColumnFamily(pool, 'OOPS')
+problem_type_cf = pycassa.ColumnFamily(pool, 'ProblemType')
+
+# Main
+
+if __name__ == '__main__':
+    if len(sys.argv) != 2:
+        print >>sys.stderr, "Usage: [date]"
+        sys.exit(1)
+    oopses = set()
+    start = ''
+    date = sys.argv[1]
+    while True:
+        try:
+            buf = dayoops_cf.get(date, column_start=start, column_count=1000)
+        except NotFoundException:
+            break
+        start = buf.keys()[-1]
+        buf = buf.values()
+        oopses.update(buf)
+        if len(buf) < 1000:
+            break
+    for oops_id in oopses:
+        try:
+            data = oops_cf.get(str(oops_id), columns=['ProblemType'])
+            problem_type = data['ProblemType']
+            problem_type_cf.insert(problem_type, {oops_id: ''})
+        except (NotFoundException, KeyError):
+            # Sometimes we didn't insert the full OOPS. I have no idea why.
+            #print 'could not find', uuid
+            continue