Merge lp:~mwhudson/ubuntu-archive-scripts/find-rcbuggy-problem-packages into lp:ubuntu-archive-scripts

[Michael Hudson-Doyle]

Generate a report of packages that have been removed from testing whose removal from Ubuntu would aid other packages though proposed migration.

This branch implements this idea:

Packages that have been removed from Debian testing should not automatically be removed from Ubuntu devel, because a package may be buggy in Debian but not in Ubuntu (or vice-versa). However, packages that have been removed from Debian testing for bugs, and which are also blocking things in the proposed-migration queue, are candidates for fast-tracking for removal from devel + devel-proposed.

Currently we waste too much time identifying packages in this category.

Create a report that shows packages that are in this situation, so that they can be more quickly identified. Report should provide recommended commands for their removal. Iterate until the report is correct and a simple button-pushing exercise for the archive admins, then automate.

Should be added to lp:ubuntu-archive-scripts

Example of a package removed from testing that we then manually removed from devel:

https://launchpad.net/debian/+source/twitter-bootstrap/+publishinghistory

[Michael Hudson-Doyle (mwhudson)]

The output looks like this fwiw: https://paste.ubuntu.com/p/6NC4VPMJTw/

[lp:~mwhudson/ubuntu-archive-scripts/find-rcbuggy-problem-packages]

247. By Michael Hudson-Doyle on 2019-09-02

print remove-package command for packages not in proposed (oops)

248. By Michael Hudson-Doyle on 2019-09-06

html output

249. By Michael Hudson-Doyle on 2019-09-09

make html a bit nicer

250. By Michael Hudson-Doyle on 2019-09-09

MOAR CSS

251. By Michael Hudson-Doyle on 2019-09-09

we do not care about a package blocking its own migration!

252. By Michael Hudson-Doyle on 2019-09-09

more css tweakery

253. By Michael Hudson-Doyle on 2019-09-09

update run-proposed-migration

[Michael Hudson-Doyle (mwhudson)]

Updated to produce HTML output, preview at http://michael.hudsondoyle.geek.nz/rcbuggy-problem-packages.html

=== added file 'find-rcbuggy-problem-packages'
--- find-rcbuggy-problem-packages	1970-01-01 00:00:00 +0000
+++ find-rcbuggy-problem-packages	2019-09-09 01:50:04 +0000
@@ -0,0 +1,244 @@
+#!/usr/bin/python3
+
+# This script highlights packages that are being kept out of Debian
+# testing by release critical bugs whose removal from the Ubuntu devel
+# series would aid the migration of other packages out of proposed.
+#
+# The packages that are being kept out of Debian testing by release
+# critical bugs can be found fairly easily by looking at the output
+# from Debian's britney runs. We do this first (and call these
+# "rc-gone packages").
+#
+# Such packages can inhibit the migration of other packages in two
+# ways:
+#
+# 1) autopkgtest regressions
+# 2) by becoming uninstallable
+#
+# The first is fairly easy to find: scan through Ubuntu's excuses.yaml
+# and look for the rc-gone package in the list of autopkgtest
+# regressions for any package.
+#
+# The second is a bit more mind-bending to detect. If the package is
+# caught up in a transition, a rebuild will be attempted as a matter
+# of course, and if this succeeds and passes its autopkgtests then
+# removing the package will not aid proposed migration. So we look for
+# rc-gone packages in proposed that are missing builds or failing
+# autopkgtests. For such source packages, we see if any of their
+# binary packages are reported as being made uninstallable by the
+# migration of any source package in proposed. If so, removing the
+# rc-gone package will help proposed migration.
+
+import argparse
+import os
+import shlex
+import subprocess
+
+import attr
+from jinja2 import Environment, FileSystemLoader
+import yaml
+
+env = Environment(
+    loader=FileSystemLoader(os.path.dirname(os.path.abspath(__file__)) + '/templates'),
+    autoescape=True,
+    extensions=['jinja2.ext.i18n'],
+)
+env.install_null_translations(True)
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--ubuntu-excuses', action='store')
+    parser.add_argument('--ubuntu-update_output', action='store')
+    parser.add_argument('--debian-excuses', action='store')
+    parser.add_argument('--output', action='store')
+    return parser.parse_args()
+
+args = parse_args()
+
+def run_output(*cmd, **extra):
+    kw = dict(check=True, encoding='ascii', stdout=subprocess.PIPE)
+    kw.update(extra)
+    cp = subprocess.run(cmd, **kw)
+    return cp.stdout.strip()
+
+def extract_uninstallable_to_src_pkg(output_fp):
+    # Extract a mapping from binary package name to the set of source
+    # packages whose migration from proposed makes that package
+    # uninstallable.
+
+    # We're looking for sequences of lines like this:
+
+    # skipped: camlp5 (0, 3, 57)
+    #     got: 13+0: a-1:a-0:a-0:i-6:p-0:s-6
+    #      * s390x: hol-light, libaac-tactics-ocaml-dev, libcoq-ocaml-dev, libledit-ocaml-dev, ocaml-ulex08
+
+    # (Britney tries to migrate batches of packages but it always
+    # tries each package on its own as well).
+
+    r = {}
+    srcpkg = None
+    for line in output_fp:
+        parts = line.split()
+        if len(parts) >= 2:
+            if parts[0] == 'skipped:':
+                srcpkg = None
+                # If parts[2] is '(' then this line is about trying to
+                # migrate a single package, which is what we are
+                # looking for.
+                if parts[2].startswith('('):
+                    srcpkg = parts[1]
+            if srcpkg is not None and parts[0] == '*':
+                # parts[1] is "${arch}:"
+                # parts[2:] is a comma+space separated list of binary package names.
+                for binpkg in parts[2:]:
+                    binpkg = binpkg.strip(',')
+                    r.setdefault(binpkg, set()).add(srcpkg)
+    return r
+
+
+@attr.s
+class RCGone:
+    source_package_name = attr.ib()
+    bugs = attr.ib()
+    block_by_regression = attr.ib(default=attr.Factory(set))
+    block_by_uninstallability = attr.ib(default=attr.Factory(list))
+    suites = attr.ib(default=attr.Factory(set))
+    _rdeps_lines = attr.ib(default=None)
+
+    @property
+    def comment(self):
+        comment = "removed from testing (Debian bug"
+        if len(self.bugs) > 1:
+            comment += "s"
+        comment += " " + ", ".join('#' + b for b in self.bugs) + ")"
+        if self.block_by_regression:
+            comment += ', blocks {} by regression'.format(', '.join(sorted(self.block_by_regression)))
+        if self.block_by_uninstallability:
+            comment += ', blocks {} by uninstallability'.format(', '.join(sorted(self.block_by_uninstallability)))
+        return comment
+
+    @property
+    def rdeps_lines(self):
+        if self._rdeps_lines is None:
+            rdeps = run_output("reverse-depends", 'src:' + self.source_package_name)
+            rbdeps = run_output("reverse-depends", '-b', 'src:' + self.source_package_name)
+
+            if rdeps == 'No reverse dependencies found':
+                rdeps = ''
+            if rbdeps == 'No reverse dependencies found':
+                rbdeps = ''
+
+            if rdeps or rbdeps:
+                if rdeps and rbdeps:
+                    self._rdeps_lines = (rdeps + '\n' + rbdeps).splitlines()
+                else:
+                    self._rdeps_lines = (rdeps + rbdeps).splitlines()
+            else:
+                self._rdeps_lines = []
+        return self._rdeps_lines
+
+    @property
+    def rdeps_text_short(self):
+        return "\n".join(self.rdeps_lines[:10])
+
+    @property
+    def rdeps_text_more(self):
+        return "\n".join(self.rdeps_lines[10:])
+
+    @property
+    def removal_commands(self):
+        suites = self.suites
+        if not suites:
+            suites = [series]
+        cmds = []
+        for suite in suites:
+            cmd = " ".join(["remove-package", "-s", suite, "-m", shlex.quote(self.comment), self.source_package_name])
+            if self.rdeps_lines:
+                cmd = '#' + cmd
+            cmds.append(cmd)
+        return cmds
+
+    @property
+    def css_class(self):
+        if self.rdeps_lines:
+            return 'removal-not-ok'
+        else:
+            return 'removal-ok'
+
+
+if 'SERIES' in os.environ:
+    series = os.environ['SERIES']
+else:
+    series = run_output('distro-info', '-d')
+
+
+def main():
+    print("loading data")
+    with open(args.ubuntu_excuses) as fp:
+        ubuntu_excuses = yaml.load(fp, Loader=yaml.CSafeLoader)
+    with open(args.ubuntu_update_output) as fp:
+        uninstallable_to_src_pkg = extract_uninstallable_to_src_pkg(fp)
+    with open(args.debian_excuses) as fp:
+        debian_excuses = yaml.load(fp, Loader=yaml.CSafeLoader)
+
+    print("finding rcgone packages")
+    rc_gones = {}
+
+    for source in debian_excuses['sources']:
+        if source['old-version'] == '-':
+            info = source['policy_info']
+            rc_bugs = info.get('rc-bugs', {})
+            if rc_bugs.get('verdict') == "REJECTED_PERMANENTLY":
+                bugs = []
+                for k in 'shared-bugs', 'unique-source-bugs', 'unique-target-bugs':
+                    bugs.extend(rc_bugs[k])
+                rc_gones[source['item-name']] = RCGone(
+                    source_package_name=source['item-name'],
+                    bugs=bugs)
+    in_proposed_by_autopkgtest_or_missing_binaries = set()
+    print("checking autopkgtests")
+    for source in ubuntu_excuses['sources']:
+        item = source['item-name']
+        if 'autopkgtest' in source['reason']:
+            in_proposed_by_autopkgtest_or_missing_binaries.add(item)
+            for package, results in sorted(source['policy_info']['autopkgtest'].items()):
+                package = package.split('/')[0]
+                if package not in rc_gones:
+                    continue
+                for arch, result in sorted(results.items()):
+                    outcome, log, history, wtf1, wtf2 = result
+                    if outcome == "REGRESSION" and package != item:
+                        rc_gones[package].block_by_regression.add(item)
+                        break
+        if 'missing-builds' in source:
+            in_proposed_by_autopkgtest_or_missing_binaries.add(item)
+        if item in rc_gones:
+            if source['new-version'] != '-':
+                rc_gones[item].suites.add(series+"-proposed")
+            if source['old-version'] != '-':
+                rc_gones[item].suites.add(series)
+    print("checking uninstallability")
+    for pkg in rc_gones:
+        if pkg not in in_proposed_by_autopkgtest_or_missing_binaries:
+            continue
+        bin_pkgs = run_output("chdist", "grep-dctrl-packages", series, "-SwnsPackage", pkg, check=False)
+        for bin_pkg in set(bin_pkgs.splitlines()):
+            rc_gones[pkg].block_by_uninstallability.extend(uninstallable_to_src_pkg.get(bin_pkg, []))
+    print("finding reverse-deps")
+    packages = []
+    for _, rc_gone in sorted(rc_gones.items()):
+        if not rc_gone.block_by_regression and not rc_gone.block_by_uninstallability:
+            continue
+        rc_gone.rdeps_lines
+        packages.append(rc_gone)
+
+    print("rendering")
+    t = env.get_template('rcbuggy-problem-packages.html')
+    with open(args.output, 'w', encoding='utf-8') as fp:
+        fp.write(t.render(
+            packages=packages,
+            now=ubuntu_excuses["generated-date"].strftime("%Y.%m.%d %H:%M:%S")))
+
+
+if __name__ == '__main__':
+    main()

=== modified file 'run-proposed-migration'
--- run-proposed-migration	2019-04-20 00:23:18 +0000
+++ run-proposed-migration	2019-09-09 01:50:04 +0000
@@ -52,4 +52,10 @@
     generate-team-p-m --excuses-yaml $SERIES/update_excuses.yaml \
                       --subscribers-json ~/public_html/package-team-mapping.json \
                       $SERIES/update_excuses_by_team.html $SERIES/update_excuses_by_team.yaml
+    rm -f /tmp/debian-excuses.yaml
+    wget  -O /tmp/debian-excuses.yaml https://release.debian.org/britney/excuses.yaml
+    find-rcbuggy-problem-packages --ubuntu-excuses $SERIES/update_excuses.yaml \
+                                  --ubuntu-update_output $SERIES/update_output.txt \
+                                  --debian-excuses /tmp/debian-excuses.yaml \
+                                  --output $SERIES/rcbuggy-problem-packages.html
 fi

=== added file 'templates/rcbuggy-problem-packages.html'
--- templates/rcbuggy-problem-packages.html	1970-01-01 00:00:00 +0000
+++ templates/rcbuggy-problem-packages.html	2019-09-09 01:50:04 +0000
@@ -0,0 +1,90 @@
+<!DOCTYPE HTML>
+<html>
+  <head>
+    <meta charset="utf-8">
+    <title>rcbuggy problem packages</title>
+    <style>
+      body {
+          font-family: ubuntu;
+          background-color: #ffd;
+      }
+      .package {
+          margin: 0.5em;
+          padding: 0.5em;
+      }
+      .removal-ok {
+          background-color: #dfd;
+      }
+      .removal-not-ok {
+          background-color: #ddd;
+      }
+      h2 {
+          margin: 0em;
+      }
+      .rdeps {
+          margin-left: 1em;
+      }
+      .commands {
+          white-space: pre-wrap;
+          display: inline-block;
+          margin: 0em;
+          padding: 0.2em;
+          border: solid;
+      }
+      .removal-ok .commands {
+          border-color: #afa;
+          background-color: #cfc;
+      }
+      .removal-not-ok .commands {
+          border-color: #aaa;
+          background-color: #ccc;
+      }
+    </style>
+  </head>
+  <body lang="en">
+    <h1>rcbuggy problem packages</h1>
+    <p>
+      Generated: {{ now }}
+    </p>
+    {% for package in packages %}
+    <div class="package {{ package.css_class }}">
+      <h2 id="{{ package.source_package_name }}">{{ package.source_package_name }}</h2>
+      <div class="details">
+        <p>
+          Kept out of Debian testing by Debian bug{% if package.bugs|length > 1 %}s{% endif %}
+          {% for bug in package.bugs %}
+          <a href="https://bugs.debian.org/{{bug}}">#{{bug}}</a>{% if not loop.last %},{% endif %}{% endfor %}.
+        </p>
+        {% if package.block_by_regression %}
+        <p>
+          The failures of this package's autopkgtests block migration of {{ package.block_by_regression|join(", ") }}.
+        </p>
+        {% endif %}
+        {% if package.block_by_uninstallability %}
+        <p>
+          At least one binary from this package is made uninstallable by the versions of {{ package.block_by_uninstallability|join(", ") }} in -proposed.
+        </p>
+        {% endif %}
+        {% if package.rdeps_lines %}
+        <pre class="rdeps">
+{{ package.rdeps_text_short }}
+{% if package.rdeps_text_more -%}
+<details><summary>more...</summary>
+{{ package.rdeps_text_more }}
+</details>
+{% endif -%}
+        </pre>
+        {% endif %}
+        <p>
+          Suggested removal commands:
+        </p>
+        <pre class="commands">
+{% for cmd in package.removal_commands -%}
+{{ cmd }}
+{% endfor -%}
+        </pre>
+      </div>
+    </div>
+    {% endfor %}
+  </body>
+</html>