Merge lp:~brian-murray/errors/search-by-sourcepackage into lp:errors

[Brian Murray]

I've set it up so that a check is made to see if package is a source package and if it is then the binary packages making up that source package are found which are then used to find problems pertaining to all of those packages. I tested this in juju and it worked well.

I also did some python cleanup of the launchpad.py code.

The thing most needing review is the change to json_request in launchpad.py. I changed it so that now in the event of a KeyError the result of json.loads(content) is returned. This was necessary as 'ubuntu/current_series' does not return entries but a dictionary (with information about the current series) instead.

[Brian Murray (brian-murray)]

I also wonder if it might be a good idea to modify the 'Package' column label in the most common problems table to be 'Binary Package'.

[Evan (ev)]

Yes, I think changing it to 'Binary package' is a good idea.

My general concern with changing the column headers is that we're already tight on space. If the header gets longer than the value for any of the rows, then we're reducing the amount of the signature column that can be read. Since compizconfig-settings-manager exists, I think we'll be okay in this instance.

[Evan (ev)]

This mostly looks good.

The rank will start at 1 for the most common problems in each package, which will look confusing when displayed in the table. You'll have repeating number for each binary package.

Instead, we should do a sorted insertion (to results) by count in and apply the rankings.

review: Needs Fixing

[lp:~brian-murray/errors/search-by-sourcepackage]

244. By Brian Murray on 2013-01-04

for the most common problems set the rank after adding results for all the packages

[Brian Murray (brian-murray)]

I've updated the branch so that a list of counts is kept and sorted. This is then used to determine the rank of each result. This has the effect of creating a ranking like [1, 1, 3, 3, 3, 6] which I think is actually more accurate as the items have the same count and should not have a ranking indicating one is "higher" than another when they are not.

[Evan (ev)]

Generally I would agree, but in this case "rank" is used to allow teams talking over the phone to identify a specific issue without a lot of added complexity. For example, "Brian, can you take issue #4 off the top crashes for software-center."

https://bugs.launchpad.net/errors/+bug/1046277

[lp:~brian-murray/errors/search-by-sourcepackage]

245. By Brian Murray on 2013-01-08

in the most common problems create a rank by sorting count and then function name

246. By Brian Murray on 2013-01-08

in the most common problems include the package name when the determining the rank

[Brian Murray (brian-murray)]

I've modified the branch so that rank is determined using frequency, then package name and then the function. I added in the package name as I found it odd to have /usr/sbin/cups appear after /usr/bin/usb-creator-gtk.

[lp:~brian-murray/errors/search-by-sourcepackage]

247. By Brian Murray on 2013-01-08

most_common_problems.js: set column label to binary package

[Evan (ev)]

I've merged this in with some fixes. Thanks for the hard work on this!

review: Approve

=== modified file 'api/resources.py'
--- api/resources.py	2012-10-24 15:59:24 +0000
+++ api/resources.py	2013-01-08 23:19:21 +0000
@@ -142,44 +142,58 @@
         period = request.GET.get('period', None)
         from_date = str(request.GET.get('from', '')).translate(None, '/-')
         to_date = str(request.GET.get('to', '')).translate(None, '/-')
-
-        buckets = cassandra.get_bucket_counts(release, package,
-                                              version, period,
-                                              show_failed=True,
-                                              from_date=from_date,
-                                              to_date=to_date)
-
-        # Since fetching the metadata is an expensive operation as it involves
-        # seeking the disk and searching other nodes for multiple rows, we
-        # filter the set of buckets we're concerned about first.
-        buckets = buckets[start:finish]
         results = []
-        metadata = cassandra.get_metadata_for_buckets([x[0] for x in buckets], release)
-        i = 1
-        for bucket, count in buckets:
-            try:
-                m = metadata[bucket]
-            except KeyError:
-                m = {}
-            package = m.get('Source', '')
-            if not package:
-                # Because we weren't always writing the Source field in
-                # BucketMetadata, there are going to be a few buckets with an
-                # empty value here. Use the old method for looking up the
-                # package for now.
-                package, version = cassandra.get_package_for_bucket(bucket)
-
-            last_seen = m.get('LastSeen', '')
-            report = m.get('CreatedBug', '') or m.get('LaunchpadBug', '')
-            results.append(ResultObject({
-                 'rank': i,
-                 'count': count,
-                 'package': package.decode('utf-8'),
-                 'first_seen' : m.get('FirstSeen', ''),
-                 'last_seen' : last_seen,
-                 'function': bucket.decode('utf-8'),
-                 'report': report }))
-            i += 1
+        packages = []
+        # XXX: is returning source package info if package is a binary package
+        # the best?
+        if package and launchpad.is_source_package(package):
+            for binary in launchpad.get_binaries_in_source_package(package):
+                packages.append(binary)
+        else:
+            packages.append(package)
+        counts = []
+        for package in packages:
+            buckets = cassandra.get_bucket_counts(release, package,
+                                                  version, period,
+                                                  show_failed=True,
+                                                  from_date=from_date,
+                                                  to_date=to_date)
+
+            # Since fetching the metadata is an expensive operation as it involves
+            # seeking the disk and searching other nodes for multiple rows, we
+            # filter the set of buckets we're concerned about first.
+            buckets = buckets[start:finish]
+            metadata = cassandra.get_metadata_for_buckets([x[0] for x in buckets], release)
+            for bucket, count in buckets:
+                try:
+                    m = metadata[bucket]
+                except KeyError:
+                    m = {}
+                package = m.get('Source', '')
+                if not package:
+                    # Because we weren't always writing the Source field in
+                    # BucketMetadata, there are going to be a few buckets with an
+                    # empty value here. Use the old method for looking up the
+                    # package for now.
+                    package, version = cassandra.get_package_for_bucket(bucket)
+
+                last_seen = m.get('LastSeen', '')
+                report = m.get('CreatedBug', '') or m.get('LaunchpadBug', '')
+                results.append(ResultObject({
+                     'rank': 0,
+                     'count': count,
+                     'package': package.decode('utf-8'),
+                     'first_seen': m.get('FirstSeen', ''),
+                     'last_seen': last_seen,
+                     'function': bucket.decode('utf-8'),
+                     'report': report}))
+                counts.append((count, package.decode('utf-8'),
+                    bucket.decode('utf-8')))
+
+        counts.sort(key=lambda t: (-int(t[0]), t[1], t[2]))
+        for result in results:
+            result.rank = counts.index((result.count, result.package,
+                result.function)) + 1
 
         return results
 

=== modified file 'launchpad.py'
--- launchpad.py	2012-10-18 13:59:42 +0000
+++ launchpad.py	2013-01-08 23:19:21 +0000
@@ -1,12 +1,12 @@
-from oauth import oauth
+import apt
+import httplib2
+import json
+import sys
 import urllib
 import urllib2
-import httplib2
-import json
-import apt
-import lazr.uri
+
 from lazr.restfulclient._browser import AtomicFileCache
-import sys
+from oauth import oauth
 
 configuration = None
 try:
@@ -59,7 +59,8 @@
 _file_cache = AtomicFileCache(configuration.http_cache_dir)
 _http = httplib2.Http(_file_cache)
 
-def json_request (url):
+
+def json_request(url):
     try:
         response, content = _http.request(url)
     except httplib2.ServerNotFoundError:
@@ -67,21 +68,32 @@
 
     try:
         return json.loads(content)['entries']
-    except (ValueError, KeyError):
+    except KeyError:
+        # current_series has no entries
+        return json.loads(content)
+    except ValueError:
         return ''
 
-def get_all_codenames ():
-    url = _launchpad_base + '/ubuntu/series'
-    return [entry['name'] for entry in json_request (url)]
-
-def get_codename_for_version (version):
-    url = _launchpad_base + '/ubuntu/series'
-    for entry in json_request (url):
+
+def get_all_codenames():
+    url = _launchpad_base + '/ubuntu/series'
+    return [entry['name'] for entry in json_request(url)]
+
+
+def get_codename_for_version(version):
+    url = _launchpad_base + '/ubuntu/series'
+    for entry in json_request(url):
         if 'name' in entry and entry.get('version', None) == version:
             return entry['name']
     return None
 
 
+def get_devel_series_codename():
+    url = _launchpad_base + '/ubuntu/current_series'
+    current_series = json_request(url)
+    return current_series['name']
+
+
 def get_version_for_codename(codename):
     url = _launchpad_base + '/ubuntu/series'
     for entry in json_request(url):
@@ -90,7 +102,7 @@
     return None
 
 
-def get_versions_for_binary (binary_package, ubuntu_version):
+def get_versions_for_binary(binary_package, ubuntu_version):
     if not ubuntu_version:
         codenames = get_all_codenames()
     else:
@@ -112,7 +124,8 @@
     results |= set([get_version_for_codename(x['display_name'].split(' ')[3]) for x in json_request(url)])
     return results
 
-def binaries_are_most_recent (specific_packages, release=None):
+
+def binaries_are_most_recent(specific_packages, release=None):
     '''For each (package, version) tuple supplied, determine if that is the
     most recent version of the binary package.
 
@@ -142,7 +155,8 @@
             result.append(True)
     return result
 
-def _get_most_recent_binary_version (package, release):
+
+def _get_most_recent_binary_version(package, release):
     url = _get_published_binaries_url % urllib.quote(package)
     if release:
         # TODO cache this by pushing it into the above function and instead
@@ -155,7 +169,8 @@
     except (KeyError, IndexError):
         return ''
 
-def binary_is_most_recent (package, version):
+
+def binary_is_most_recent(package, version):
     # FIXME we need to factor in the release, otherwise this is often going to
     # look like the issue has disappeared when filtering the most common
     # problems view to a since-passed release.
@@ -167,6 +182,7 @@
     # we haven't seen it in the new version it may be fixed.
     return apt.apt_pkg.version_compare(version, latest_version) != -1
 
+
 def bug_is_fixed(bug, release=None):
     url = _get_bug_tasks_url % urllib.quote(bug)
     if release:
@@ -205,8 +221,32 @@
     except (ValueError, KeyError):
         return False
 
+
+def is_source_package(package_name):
+    dev_series = get_devel_series_codename()
+    url = _launchpad_base + '/ubuntu/' + dev_series + \
+        ('/?ws.op=getSourcePackage&name=%s' % package_name)
+    request = json_request(url)
+    if request:
+        return True
+
+
+def get_binaries_in_source_package(package_name):
+    dev_series = get_devel_series_codename()
+    ma_url = _launchpad_base + '/ubuntu/' + dev_series + \
+        '/main_archive'
+    ma = json_request(ma_url)['self_link']
+    ps_url = ma + ('/?ws.op=getPublishedSources&exact_match=true&source_name=%s'
+        % package_name)
+    # just use the first one, since they are unordered
+    ps = json_request(ps_url)[0]['self_link']
+    pb_url = ps + '/?ws.op=getPublishedBinaries'
+    pbs = set(pb['binary_package_name'] for pb in json_request(pb_url))
+    return pbs
+
 # Bug creation.
 
+
 def _generate_operation(title, description, target=_ubuntu_target):
     _ubuntu_target = 'https://api.qastaging.launchpad.net/devel/ubuntu'
 
@@ -216,6 +256,7 @@
                   'title' : title }
     return urllib.urlencode(operation)
 
+
 def _generate_headers(oauth_token, oauth_secret):
     a = (('OAuth realm="%s", '
           'oauth_consumer_key="testing", '
@@ -232,6 +273,7 @@
                 'Content-Type' : 'application/x-www-form-urlencoded' }
     return headers
 
+
 def create_bug(signature, source=''):
     '''Returns a tuple of (bug number, url)'''
 
@@ -264,12 +306,14 @@
     except KeyError:
         return (None, None)
 
+
 def _generate_subscription(user):
-    operation = { 'ws.op' : 'subscribe',
-                  'level' : 'Discussion',
-                  'person' : _person_url + user }
+    operation = {'ws.op': 'subscribe',
+                 'level': 'Discussion',
+                 'person': _person_url + user}
     return urllib.urlencode(operation)
 
+
 def subscribe_user(bug, user):
     operation = _generate_subscription(user)
     headers = _generate_headers(configuration.lp_oauth_token,

=== modified file 'static/js/most_common_problems.js'
--- static/js/most_common_problems.js	2012-10-24 15:59:24 +0000
+++ static/js/most_common_problems.js	2013-01-08 23:19:21 +0000
@@ -173,7 +173,7 @@
     var cols = [
         {key: "rank", label: "Rank", sortable:true },
         {key: "count", label: "Frequency", sortable:true, formatter:chartFormatter, allowHTML: true},
-        {key: "package", label: "Package", sortable:true, formatter:genericFormatter, allowHTML: true},
+        {key: "package", label: "Binary Package", sortable:true, formatter:genericFormatter, allowHTML: true},
         {key: "first_seen", label: "First seen", formatter:genericFormatter, allowHTML: true},
         {key: "last_seen", label: "Last seen", formatter:genericFormatter, allowHTML: true},
         {key: "function", label: "Function", formatter:functionFormatter, allowHTML: true},