Merge lp:~brian-murray/daisy/src-version-buckets into lp:daisy

[Brian Murray]

This branch will start recording to SourceVersionBuckets during the bucketing of Oopses. Additionally, there is a tool to populate SourceVersionBuckets with data from the OOPS column family.

[Evan (ev)]

Mostly looks good. Feel free to merge once you've addressed the points
below.

On Tue, Mar 26, 2013 at 9:57 PM, Brian Murray <email address hidden> wrote:
>
> + oopses.update_source_version_buckets(oops_config, src_package,
> + version, crash_signature)
>

Can you make this a if hasattr(oopses, 'update_source_version_buckets') so
that we don't crash on processing new crashes while we're updating
oops-repository and daisy on production.

> if version:
> oopses.update_bucket_versions(oops_config, crash_signature,
> version)
>
>
> +for key, oops in oops_cf.get_range(columns=cols):
> + count += 1
> + if count % 10000 == 0:
> + break
>

I suspect you had this in for debugging?

> + release = oops['DistroRelease'].encode('utf8')
>

oops.get. You're not guaranteed to get rows with all the column names you
specified.

> +
> + if not release.startswith('Ubuntu '):
> + continue
> + package_data = oops['Package'].split(' ')

+ if len(package_data) < 2:
> + continue
> + version = package_data[1]
>

Please use daisy.utils.split_package_and_version instead, as it catches
some corner cases (hopefully).

+ src_package = oops['SourcePackage']
>

oops.get.

> + srcversbucketsinsert((src_package, version), {oops_id : ''})
>

Typo :)

[lp:~brian-murray/daisy/src-version-buckets]

280. By Brian Murray on 2013-03-27

changes based on evan's feedback

=== modified file 'daisy/utils.py'
--- daisy/utils.py	2013-03-26 23:58:48 +0000
+++ daisy/utils.py	2013-03-27 17:13:24 +0000
@@ -38,6 +38,7 @@
 def bucket(oops_config, oops_id, crash_signature, report_dict):
     release = report_dict.get('DistroRelease', '')
     package = report_dict.get('Package', '')
+    src_package = report_dict.get('SourcePackage', '')
     problem_type = report_dict.get('ProblemType', '')
     dependencies = report_dict.get('Dependencies', '')
     system_uuid = report_dict.get('SystemIdentifier', '')
@@ -78,6 +79,9 @@
         oopses.update_bucket_metadata(oops_config, crash_signature, package,
                                       version, apt.apt_pkg.version_compare,
                                       release)
+        if hasattr(oopses, 'update_source_version_buckets'):
+            oopses.update_source_version_buckets(oops_config, src_package,
+                                                 version, crash_signature)
     if version:
         oopses.update_bucket_versions(oops_config, crash_signature, version)
 
 
=== added file 'tools/build_src_version_buckets.py'
--- tools/build_src_version_buckets.py	1970-01-01 00:00:00 +0000
+++ tools/build_src_version_buckets.py	2013-03-27 17:13:24 +0000
@@ -0,0 +1,42 @@
+#!/usr/bin/python
+
+import pycassa
+import uuid
+from daisy import config
+from utils import split_package_and_version
+from collections import Counter
+
+creds = {'username': config.cassandra_username,
+         'password': config.cassandra_password}
+pool = pycassa.ConnectionPool(config.cassandra_keyspace,
+                              config.cassandra_hosts, timeout=600,
+                              max_retries=100, credentials=creds)
+
+oops_cf = pycassa.ColumnFamily(pool, 'OOPS')
+srcversbuckets = pycassa.ColumnFamily(pool, 'SourceVersionBuckets')
+
+cols = ['SourcePackage', 'Package', 'DistroRelease']
+count = 0
+for key, oops in oops_cf.get_range(columns=cols):
+    count += 1
+    if count % 100000 == 0:
+        print 'processed', count
+
+    if Counter(cols) != Counter(oops.keys()):
+        continue
+
+    release = oops.get('DistroRelease', '')
+    if not release.startswith('Ubuntu ') or release == '':
+        continue
+    package = oops.get('Package', '')
+    if package:
+        package, version = split_package_and_version(package)
+    src_package = oops.get('SourcePackage', '')
+    if src_package == '' or version == '':
+        continue
+    oops_id = uuid.UUID(key)
+    #print('Would insert (%s, %s) = {%s, ""}' % (src_package, version,
+    #       oops_id))
+    srcversbuckets.insert((src_package, version), {oops_id : ''})
+
+print 'total processed', count