Merge lp:~achiang/ubuntu-nexus7/valgrind-ubuntu-dbg-packages into lp:~ubuntu-nexus7/ubuntu-nexus7/valgrind-ubuntu-dbg-packages

[Alex Chiang]

  Parallel python and go implementations of valgrind-ubuntu-dbg-packages

  These tools parse a valgrind log looking for unknown symbols. It then
  attempts to find a corresponding Ubuntu package that might provide the
  proper debug symbols, and finally recommends them for installation by
  the user.

  Usage:
    <script> <valgrind.log>

  To build the go version:
    go build valgrind-ubuntu-dbg-packages.go

[lp:~achiang/ubuntu-nexus7/valgrind-ubuntu-dbg-packages]

3. By Alex Chiang on 2012-11-19

Re-implement in functional style

I found the multiple "found = True" statements to be a bit ugly so
refactored into a functional implementation.

At the bottom, you see that I compare the package sets returned by
the two implementations, and the output on the console is:

set([])

Therefore, I claim the functional implementation is equivalent to the
iterative implementation.

4. By Alex Chiang on 2012-11-19

Remove iterative implementation of find_pkgs()

The previous commit proved they were equivalent, so now let's get rid
of the old one.

5. By Alex Chiang on 2012-11-19

Fix debug print statements

- Remove superfluous prints that existed to print empty lines
- When no debug package found, print full name of original package

[Alex Chiang (achiang)]

To understand the purpose of this helper program, you can run it on the log attached to comment #52 of a recent bug I worked on:

https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/780602/comments/52

[Scott Sweeny (ssweeny)]

Python script looks reasonable, and pylint/pep8 only show nitpicks, so I approve.

I'm not really qualified to review Go code, but the output of the Go program is functionally equivalent to the python script, so I feel good about it too :)

review: Approve

[Dimitri John Ledkov (xnox)]

Reviewing the python bits only.
Might be easier to use python-apt and/or python-debian bindings.
The code is not python3-safe.

General:
Only small subset of packages have -dbg packages. Most debug packages are generated as ddebs and are have -dbgsym suffix. https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages

We already have daisy and apport re-tracers that can find missing debug packages, can we not adapt/use those?

review: Needs Fixing

[Martin Pitt (pitti)]

It depends on how far you want to go with those. Installing -dbg packages only is insufficient for a lot of cases (as many/most packages don't build them), but is "safe" in the sense that they will always stay installable and upgradeable. Using -dbgsym isn't, due to the hackish and convoluted way that we (have to) use to build the ddebs archive. I would _not_ recommend to permanently install those.

Instead, what I recommend is to install apport-retrace and if you have a crash, use it to debug a .crash file in a temporary sandbox (-S system) which will include all available dbgsyms, not only for the transitive dependencies of a package, but also from the dynamically loaded plugins of the program.

What's the purpose of the script, i. e. in which cases would you run this? Is that an use case which should be supported by apport-retrace?

[Alex Chiang (achiang)]

The goal of this script was not to help debug crashers, but rather with the aim of providing a convenience script for community members as the Ubuntu Nexus 7 memory slimming program gets under way.

The idea was that as people run valgrind on individual programs, they might want to install the -dbg packages relevant to the program they are tracing. They might valgrind a program once, briefly; capture the log; then run this convenience script on the log.

The tool tells them which extra -dbg packages are available, user installs them, and then user re-runs valgrind for real.

[Martin Pitt (pitti)]

Right, so for permanent installation I indeed recommend installing -dbg only, not ddebs. For the "full" symbol experience the ddebs should be installed only temporarily (they are not very stable on upgrades and also take a ton of space), so something similar like apport-retrace ("apport-valgrind package" or so) would be more appropriate, if the need arises.

[Mike Carifio (carifio)]

As per our recent discussion in #premium, the golang portion is fine idomatic go. I didn't test it however.

If you want convert the go implementation to do the equivalent of python generators (the functions with yield in them), then I suggest golang channels and "go routines", see http://jnwhiteh.net/posts/2010/09/go-examples-1-channels-and-goroutines.html for a quick tutorial-by-example.

review: Abstain

[Alex Chiang (achiang)]

Hi Martin,

Thanks for the feedback, but I'm a little confused. Are you suggesting to fold this work into apport instead?

Hi Dmitrijs,

Based on Martin's feedback, I'll either fix the python up to be python3 compliant or figure out how to make it work with apport.

[Martin Pitt (pitti)]

> Thanks for the feedback, but I'm a little confused. Are you suggesting to fold
> this work into apport instead?

I was saying "if the need arises". If you prefer having permanently installed debug packages, and the number of -dbg that we have are sufficient for what you are trying to do, it's fine.

Unmerged revisions

5. By Alex Chiang on 2012-11-19

Fix debug print statements

- Remove superfluous prints that existed to print empty lines
- When no debug package found, print full name of original package

4. By Alex Chiang on 2012-11-19

Remove iterative implementation of find_pkgs()

The previous commit proved they were equivalent, so now let's get rid
of the old one.

3. By Alex Chiang on 2012-11-19

Re-implement in functional style

I found the multiple "found = True" statements to be a bit ugly so
refactored into a functional implementation.

At the bottom, you see that I compare the package sets returned by
the two implementations, and the output on the console is:

set([])

Therefore, I claim the functional implementation is equivalent to the
iterative implementation.

2. By Alex Chiang on 2012-11-19

Parallel python and go implementations of valgrind-ubuntu-dbg-packages

These tools parse a valgrind log looking for unknown symbols. It then
attempts to find a corresponding Ubuntu package that might provide the
proper debug symbols, and finally recommends them for installation by
the user.

Usage:
  <script> <valgrind.log>

To build the go version:
  go build valgrind-ubuntu-dbg-packages.go

=== added file 'valgrind-ubuntu-dbg-packages.go'
--- valgrind-ubuntu-dbg-packages.go	1970-01-01 00:00:00 +0000
+++ valgrind-ubuntu-dbg-packages.go	2012-11-19 16:50:28 +0000
@@ -0,0 +1,92 @@
+package main
+
+import (
+    "fmt"
+    "os"
+    "io"
+    "flag"
+    "bufio"
+    "strings"
+    "path/filepath"
+    "os/exec"
+)
+
+func find_syms(log *os.File) map[string]string {
+    syms := make(map[string]string)
+
+    f := bufio.NewReader(log)
+    for {
+        line, err := f.ReadString('\n')
+        if err == io.EOF {
+            break
+        }
+
+        if strings.Contains(line, "??? (in") {
+            fields := strings.Split(line, " ")
+            path := strings.TrimRight(fields[8], ")\n")
+            base := filepath.Base(path)
+            if _, ok := syms[base]; !ok {
+                syms[base] = path
+            }
+        }
+    }
+    return syms
+}
+
+func find_pkgs(syms map[string]string) map[string]bool {
+    pkgs := make(map[string]bool)
+
+    for k, v := range syms {
+        out, _:= exec.Command("dpkg-query",  "-S", v).Output()
+        dbg := ""
+        found := false
+
+        deb_package := strings.Split(string(out), ":")
+        pkg_parts := strings.Split(deb_package[0], "-")
+
+        fmt.Println("File", k, "provided by", deb_package[0])
+
+        for i := range pkg_parts {
+            dbg += pkg_parts[i]
+            fmt.Println("Searching for", dbg + "-dbg")
+            out, _ := exec.Command("apt-cache", "search", dbg + "-dbg").Output()
+            if string(out) != "" {
+                pkg := strings.Split(strings.TrimRight(string(out), "\n"), " ")[0]
+                if _, ok := pkgs[pkg]; ok {
+                    fmt.Println("Already found", pkg)
+                    found = true
+                    break
+                }
+                pkgs[pkg] = true
+                fmt.Println("Found", pkg)
+                found = true
+                break
+            }
+            dbg += "-"
+        }
+        dbg = ""
+        if found == false {
+            fmt.Println("No debug package for", k)
+        }
+        fmt.Println()
+    }
+    return pkgs
+}
+
+func main() {
+    flag.Parse()
+
+    log, err := os.Open(flag.Arg(0))
+    if err != nil {
+        fmt.Println("Error:", err)
+        os.Exit(1)
+    }
+
+    syms := find_syms(log)
+    pkgs := find_pkgs(syms)
+    fmt.Println("It is recommended to install the following packages:")
+    for k := range pkgs {
+        fmt.Printf("%s ", k)
+    }
+    fmt.Println()
+}

=== added file 'valgrind-ubuntu-dbg-packages.py'
--- valgrind-ubuntu-dbg-packages.py	1970-01-01 00:00:00 +0000
+++ valgrind-ubuntu-dbg-packages.py	2012-11-19 16:50:28 +0000
@@ -0,0 +1,65 @@
+#!/usr/bin/python
+
+import sys
+import re
+import os
+
+def find_syms(log):
+    syms = {}
+
+    f = open(log)
+    for l in f.readlines():
+        if re.search('\?\?\?\ \(in', l):
+            fields = re.split(' ', l)
+            path = fields[8].rstrip('\)\r\n')
+            lib = os.path.basename(path)
+            if lib not in syms:
+                syms[lib] = path
+            else:
+                continue
+    return syms
+
+def find_pkgs(syms):
+    from subprocess import Popen, PIPE
+    def find_pkg(k, v):
+        out = Popen(("dpkg-query -S %s" % v), stdout=PIPE, shell=True).stdout.read()
+        deb_package = out.split(":")
+        parts = deb_package[0].split("-")
+        print "File %s provided by %s " % (k, deb_package[0])
+        yield parts
+
+    def find_dbg(pkg_parts):
+        dbg = ""
+        for i in pkg_parts:
+            dbg += i
+            print "Searching for %s" % dbg + "-dbg"
+            out = Popen(("apt-cache search %s" % dbg + "-dbg"), stdout=PIPE, shell=True).stdout.read()
+            if out != "":
+                pkg = out.rstrip("\n").split(" ")[0]
+                print "Found %s\n" % pkg
+                yield pkg
+                return
+            dbg += "-"
+        print "No debug package for %s\n" % "-".join(pkg_parts)
+        return
+
+    pkgs = dict((dbg, True)
+            for k, v in syms.items()
+            for pkg in find_pkg(k, v)
+            for dbg in find_dbg(pkg))
+
+    return pkgs
+
+if __name__ == "__main__":
+    try:
+        log = sys.argv[1]
+    except:
+        print "No log file supplied"
+        sys.exit(1)
+
+    syms = find_syms(log)
+    pkgs = find_pkgs(syms)
+    print "It is recommended to install the following packages:"
+    for p in pkgs:
+        print p,
+    print