Merge lp:~sergiodj/lp-ftbfs-report/fix-internal-pkgset-team-links into lp:lp-ftbfs-report

[Sergio Durigan Junior]

When the packageset and the team names are the same, the internal links for each of these will end up also being the same. Currently, because the packageset section comes first in the page, that means that the team internal link will point to it.

A simple fix is to use distinct suffixes to compose the internal link names, which help differentiate them.

Unmerged revisions

65. By Sergio Durigan Junior on 2021-08-13

Use suffixes for pkgset and team internal links

When a packageset has the same name of a team, the internal links will
point to the same place, which currently is the packageset section
(because it comes first in the page). We can solve that by using
distinct suffixes when creating the links.

64. By William Grant on 2018-06-06

Exclude copy archive failures that also failed in the primary archive.

63. By William Grant on 2018-06-05

Update source link.

62. By William Grant on 2018-06-05

Include only packages in main in team sections.

61. By William Grant on 2018-04-13

Generate per team lists.

60. By William Grant on 2018-02-20

Port to modern python-apt.

59. By William Grant on 2016-01-29

Add --release-only which skips packages that only fail outside the release pocket. Useful for port bootstrapping.

58. By William Grant on 2016-01-29

Add space before pocket name.

57. By William Grant on 2016-01-29

Fix spacing with more architectures.

56. By William Grant on 2016-01-29

Don't always use the current series when comparing test rebuilds to the primary archive. We rebuild old series too.

=== modified file 'source/build_status.html'
--- source/build_status.html	2013-05-28 21:39:21 +0000
+++ source/build_status.html	2021-08-13 16:24:09 +0000
@@ -4,7 +4,7 @@
 {#- Macro definitions -#}
 {% macro legend_row(row_class, field, field_desc) -%}
     <tr class="{{ row_class }}" style="text-align: center;">
-        <td class="{{ field }}">arch ({{ field[0] }})</td>
+        <td class="{{ field }}">arch ({{ abbrs[field] }})</td>
         <td style="text-align: left;">{{ field_desc }}</td>
         {% for arch in arch_list -%}
             {% if stats[field][arch].cnt -%}
@@ -26,9 +26,9 @@
                 <col width="25%" />
                 <col width="24%" />
                 <col width="2%" />
-                <col width="11%" span="{{ arch_list|count }}" />
-                <col width="5%" />
-                <col width="5%" />
+                <col width="8%" span="{{ arch_list|count }}" />
+                <col width="6%" />
+                <col width="6%" />
             </colgroup>
             <thead>
                 <tr>
@@ -79,7 +79,7 @@
                                 {% if arch in ver.logs -%}
                                     <td class="{{ ver.logs[arch].buildstate }}"
                                         onmouseover="Tip('{{ ver.logs[arch].tooltip|e }}')" onmouseout="UnTip()">
-                                        <a href="{{ ver.logs[arch].url }}">{{ arch }} ({{ ver.logs[arch].buildstate[0] }})</a>
+                                        <a href="{{ ver.logs[arch].url }}">{{ arch }} ({{ abbrs[ver.logs[arch].buildstate] }})</a>
                                         &rarr;
                                         <a href="{{ ver.logs[arch].log }}">Log</a>
                                     </td>
@@ -130,16 +130,25 @@
 
     <p>Jump to packageset:
         {% for (ps, lst) in packagesets.items()|sort if lst|count > 0 -%}
-        <a href="#{{ ps }}">{{ ps }}</a> ({{ lst|count }})
-        {% endfor -%}
-    </p>
+        <a href="#{{ ps }}-pkgset">{{ ps }}</a> ({{ lst|count }})
+        {% endfor -%}
+    </p>
+    <p>Jump to team:
+        {% for (team, lst) in teams.items()|sort if lst|count > 0 -%}
+        <a href="#{{ team }}-team">{{ team }}</a> ({{ lst|count }})
+        {% endfor -%}
+    </p>
+
+    {% if notice %}
+    {{ notice }}
+    {% endif %}
 
     <h2>Legend and statistics for the displayed data</h2>
     <table class="grid" width="95%">
         <colgroup>
             <col width="6%" />
             <col width="45%" />
-            <col width="11%" span="{{ arch_list|count }}" />
+            <col width="8%" span="{{ arch_list|count }}" />
         </colgroup>
         <thead>
             <tr>
@@ -159,6 +168,7 @@
         </thead>
         <tbody>
             {{ legend_row('even', 'FAILEDTOBUILD', 'Package failed to build') }}
+            {{ legend_row('odd', 'CANCELLED', 'Cancelled build') }}
             {{ legend_row('odd', 'MANUALDEPWAIT', 'Package is waiting on another package') }}
             {{ legend_row('even', 'CHROOTWAIT', 'Failure in the chroot') }}
             {{ legend_row('odd', 'UPLOADFAIL', 'Failed to upload') }}
@@ -179,10 +189,16 @@
 
     <h2>Packagesets</h2>
     {% for (ps, lst) in packagesets.items()|sort if lst|count > 0 -%}
-        <h3 id="{{ ps }}">{{ ps }}: {{ lst|count }} packages (<a href="#top">top</a>)</h3>
+        <h3 id="{{ ps }}-pkgset">{{ ps }}: {{ lst|count }} packages (<a href="#top">top</a>)</h3>
         {{ table(lst, ps, 'Also belongs to another packageset?') }}
     {% endfor -%}
 
+    <h2>Teams (main only)</h2>
+    {% for (team, lst) in teams.items()|sort if lst|count > 0 -%}
+        <h3 id="{{ team }}-team">{{ team }}: {{ lst|count }} packages (<a href="#top">top</a>)</h3>
+        {{ table(lst, team, 'Also belongs to another team?') }}
+    {% endfor -%}
+
     {% if main_archive %}
     <h2 id="main_superseded">main (superseded): {{ main_superseded|count }} packages (<a href="#top">top</a>)</h2>
     {{ table(main_superseded) }}
@@ -217,6 +233,7 @@
         alt="Valid XHTML 1.1" height="31" width="88"
         style="border-width:0px" /></a>
     </p>
+<<<<<<< TREE
     <p style="font-size:smaller">Source: <a href="https://code.launchpad.net/lp-ftbfs-report">lp:lp-ftbfs-report</a></p>
     <script type="text/javascript" src="source/wz_tooltip.js"></script>
     <script type="text/javascript" src="source/jquery.min.js"></script>
@@ -357,5 +374,8 @@
     /* Build failures raw data */
     var json_data={{ historical_json }};
     </script>
+=======
+    <p style="font-size:smaller">Source: <a href="https://code.launchpad.net/~wgrant/lp-ftbfs-report/production">lp:~wgrant/lp-ftbfs-report/production</a></p>
+>>>>>>> MERGE-SOURCE
 </body>
 </html>
 
=== modified file 'source/build_status.py'
--- source/build_status.py	2012-04-14 17:11:03 +0000
+++ source/build_status.py	2021-08-13 16:24:09 +0000
@@ -20,6 +20,8 @@
 #import httplib2
 #httplib2.debuglevel = 1
 
+import os
+import requests
 import sys
 import time
 import apt_pkg
@@ -30,12 +32,13 @@
 from launchpadlib.errors import HTTPError
 from launchpadlib.launchpad import Launchpad
 from operator import (attrgetter, methodcaller)
+from optparse import OptionParser
 
 lp_service = 'production'
 api_version = 'devel'
 default_arch_list = []
 find_tagged_bugs = 'ftbfs'
-apt_pkg.InitSystem()
+apt_pkg.init_system()
 
 class PersonTeam(object):
     _cache = dict()
@@ -77,7 +80,7 @@
     class VersionList(list):
         def append(self, item):
             super(SourcePackage.VersionList, self).append(item)
-            self.sort(key = attrgetter('version'), cmp = apt_pkg.VersionCompare)
+            self.sort(key = attrgetter('version'), cmp = apt_pkg.version_compare)
 
     def __new__(cls, spph):
         try:
@@ -99,6 +102,10 @@
             for ps in srcpkg.packagesets:
                 packagesets_ftbfs[ps].append(srcpkg)
 
+            srcpkg.teams = set([team for (team, srcpkglist) in teams.items() if spph.source_package_name in srcpkglist and spph.component_name == "main"])
+            for team in srcpkg.teams:
+                teams_ftbfs[team].append(srcpkg)
+
             # add to cache
             cls._cache[spph.source_package_name] = srcpkg
 
@@ -136,6 +143,33 @@
         else:
             return list(self.packagesets.difference((name,)))
 
+class MainArchiveBuilds(object):
+    _cache = dict()
+
+    def __new__(cls, main_archive, source, version):
+        try:
+            return cls._cache["%s,%s" % (source, version)]
+        except KeyError:
+            bfm = super(MainArchiveBuilds, cls).__new__(cls)
+            results = {}
+            sourcepubs = main_archive.getPublishedSources(
+                exact_match=True, source_name=source, version=version)
+            for pub in sourcepubs:
+                for build in pub.getBuilds():
+                    # assumes sourcepubs are sorted latest release to oldest,
+                    # so first record wins
+                    if build.arch_tag not in results:
+                        results[build.arch_tag] = build.buildstate
+            bfm.results = results
+            # add to cache
+            cls._cache["%s,%s" % (source, version)] = bfm
+
+            return bfm
+
+    @classmethod
+    def clear(cls):
+        cls._cache.clear()
+
 class SPPH(object):
     _cache = dict() # dict with all SPPH objects
 
@@ -172,6 +206,7 @@
                     'Dependency wait': 'MANUALDEPWAIT',
                     'Chroot problem': 'CHROOTWAIT',
                     'Failed to upload': 'UPLOADFAIL',
+                    'Cancelled build': 'CANCELLED',
                     }
             self.buildstate = buildstates[build.buildstate]
             self.url = build.web_link
@@ -203,7 +238,7 @@
         return u'Changed-By: %s' % (self.changed_by)
 
 
-def fetch_pkg_list(archive, series, state, last_published, arch_list=default_arch_list, main_archive=None, main_series=None):
+def fetch_pkg_list(archive, series, state, last_published, arch_list=default_arch_list, main_archive=None, main_series=None, release_only=False):
     print "Processing '%s'" % state
 
     cur_last_published = None
@@ -250,30 +285,59 @@
                     version=spph._lp.source_package_version,
                     status='Published')
                 spph.current = len(main_publications[:1]) > 0
+            elif release_only:
+                release_publications = archive.getPublishedSources(
+                    distro_series=series,
+                    pocket='Release',
+                    exact_match=True,
+                    source_name=spph._lp.source_package_name,
+                    version=spph._lp.source_package_version,
+                    status='Published')
+                spph.current = len(release_publications[:1]) > 0
+                if not spph.current:
+                    release_publications = archive.getPublishedSources(
+                        distro_series=series,
+                        pocket='Release',
+                        exact_match=True,
+                        source_name=spph._lp.source_package_name,
+                        version=spph._lp.source_package_version,
+                        status='Pending')
+                    spph.current = len(release_publications[:1]) > 0
             else:
                 spph.current = True
 
         if not spph.current:
             print "    superseded"
+
+        if main_archive:
+            # If this build failure is not a regression versus the
+            # main archive, do not report it.
+            main_builds = MainArchiveBuilds(main_archive,
+                                            spph._lp.source_package_name,
+                                            spph._lp.source_package_version)
+            try:
+                if main_builds.results[arch] != 'Successfully built':
+                    print "  Skipping %s" % build.title
+                    continue
+            except KeyError:
+                pass
+
         SPPH(csp_link).addBuildLog(build)
 
     return cur_last_published
 
 
-def generate_page(archive, series, archs_by_archive, main_archive, template = 'build_status.html', arch_list = default_arch_list):
-    try:
-        out = open('../%s-%s.html' % (archive.name, series.name), 'w')
-    except IOError:
-        return
-
+def generate_page(name, archive, series, archs_by_archive, main_archive, template = 'build_status.html', arch_list = default_arch_list, notice=None, release_only=False):
     # sort the package lists
     filter_ftbfs = lambda pkglist, current: filter(methodcaller('isFTBFS', arch_list, current), sorted(pkglist))
     data = {}
     for comp in ('main', 'restricted', 'universe', 'multiverse'):
         data[comp] = filter_ftbfs(components[comp], True)
-        data['%s_superseded' % comp] = filter_ftbfs(components[comp], False)
+        data['%s_superseded' % comp] = filter_ftbfs(components[comp], False) if not release_only else []
     for pkgset, pkglist in packagesets_ftbfs.items():
         packagesets_ftbfs[pkgset] = filter_ftbfs(pkglist, True)
+    for team, pkglist in teams_ftbfs.items():
+        teams_ftbfs[team] = filter_ftbfs(pkglist, True)
 
     # container object to hold the counts and the tooltip
     class StatData(object):
@@ -284,7 +348,7 @@
 
     # compute some statistics (number of packages for each build failure type)
     stats = {}
-    for state in ('FAILEDTOBUILD', 'MANUALDEPWAIT', 'CHROOTWAIT', 'UPLOADFAIL'):
+    for state in ('FAILEDTOBUILD', 'MANUALDEPWAIT', 'CHROOTWAIT', 'UPLOADFAIL', 'CANCELLED'):
         stats[state] = {}
         for arch in arch_list:
             tooltip = []
@@ -313,31 +377,47 @@
     data['archs_by_archive'] = archs_by_archive
     data['lastupdate'] = time.strftime('%F %T %z')
     data['packagesets'] = packagesets_ftbfs
+<<<<<<< TREE
     data['historical_json'] = generate_historical_json(archive, series)
+=======
+    data['teams'] = teams_ftbfs
+    data['notice'] = notice
+    data['abbrs'] = {
+        'FAILEDTOBUILD': 'F',
+        'CANCELLED': 'X',
+        'MANUALDEPWAIT': 'M',
+        'CHROOTWAIT': 'C',
+        'UPLOADFAIL': 'U',
+        }
+>>>>>>> MERGE-SOURCE
 
     env = Environment(loader=FileSystemLoader('.'))
     template = env.get_template('build_status.html')
     stream = template.render(**data)
+
+    fn = '../%s.html' % name
+    out = open('%s.new' % fn, 'w')
     out.write(stream.encode('utf-8'))
     out.close()
+    os.rename('%s.new' % fn, fn)
 
-def generate_csvfile(archive, series, arch_list = default_arch_list):
-    csvout = open('../%s-%s.csv' % (archive.name, series.name), 'w')
+def generate_csvfile(name, arch_list = default_arch_list):
+    csvout = open('../%s.csv' % name, 'w')
     linetemplate = '%(name)s,%(link)s,%(explain)s\n'
     for comp in components.values():
         for pkg in comp:
             for ver in pkg.versions:
-                for state in ('FAILEDTOBUILD', 'MANUALDEPWAIT', 'CHROOTWAIT', 'UPLOADFAIL'):
+                for state in ('FAILEDTOBUILD', 'MANUALDEPWAIT', 'CHROOTWAIT', 'UPLOADFAIL', 'CANCELLED'):
                     archs = [ arch for (arch, log) in ver.logs.items() if log.buildstate == state ]
                     if archs:
                         log = ver.logs[archs[0]].log
                         csvout.write(linetemplate  % {'name': pkg.name, 'link': log,
                             'explain':"[%s] %s" %(', '.join(archs), state)})
 
-def load_timestamps(archive, series):
+def load_timestamps(name):
     '''Load the saved timestamps about the last still published FTBFS build record.'''
     try:
-        timestamp_file = file('%s-%s.json' % (archive.name, series.name), 'r')
+        timestamp_file = file('%s.json' % name, 'r')
         tmp = json.load(timestamp_file)
         timestamps = {}
         for state, timestamp in tmp.items():
@@ -352,11 +432,12 @@
             'Dependency wait': None,
             'Chroot problem': None,
             'Failed to upload': None,
+            'Cancelled build': None,
         }
 
-def save_timestamps(archive, series, timestamps):
+def save_timestamps(name, timestamps):
     '''Save the timestamps of the last still published FTBFS build record into a JSON file.'''
-    timestamp_file = file('%s-%s.json' % (archive.name, series.name), 'w')
+    timestamp_file = file('%s.json' % name, 'w')
     tmp = {}
     for state, timestamp in timestamps.items():
         if timestamp is not None:
@@ -439,6 +520,7 @@
     launchpad = Launchpad.login_anonymously('qa-ftbfs', lp_service, version=api_version)
 
     ubuntu = launchpad.distributions['ubuntu']
+<<<<<<< TREE
     assert len(sys.argv) >= 4
 
     try:
@@ -451,15 +533,44 @@
     except HTTPError:
         print 'Error: %s is not a valid series.' % sys.argv[2]
         sys.exit(1)
+=======
+
+    usage = "usage: %prog [options] <archive> <series> <arch> [<arch> ...]"
+    parser = OptionParser(usage=usage)
+    parser.add_option(
+        "-f", "--filename", dest="name",
+        help="File name prefix for the result.")
+    parser.add_option(
+        "-n", "--notice", dest="notice_file",
+        help="HTML notice file to include in the page header.")
+    parser.add_option(
+        "--release-only", dest="release_only", action="store_true",
+        help="Only include sources currently published in the release pocket.")
+    (options, args) = parser.parse_args()
+    if len(args) < 3:
+        parser.error("Need at least 4 arguments.")
+
+    try:
+        archive = ubuntu.getArchive(name=args[0])
+    except HTTPError:
+        print 'Error: %s is not a valid archive.' % args[0]
+    try:
+        series = ubuntu.getSeries(name_or_version=args[1])
+    except HTTPError:
+            print 'Error: %s is not a valid series.' % args[1]
+
+    if options.name is None:
+        options.name = '%s-%s' % (archive.name, series.name)
+>>>>>>> MERGE-SOURCE
 
     if archive.name != 'primary':
         main_archive = ubuntu.main_archive
-        main_series = ubuntu.current_series
+        main_series = series
     else:
         main_archive = main_series = None
 
     archs_by_archive = dict(main=[], ports=[])
-    for arch in sys.argv[3:]:
+    for arch in args[2:]:
         das = series.getDistroArchSeries(archtag=arch)
         archs_by_archive[das.official and 'main' or 'ports'].append(arch)
     default_arch_list.extend(archs_by_archive['main'])
@@ -472,7 +583,7 @@
         PersonTeam.clear()
         SourcePackage.clear()
         SPPH.clear()
-        last_published = load_timestamps(archive, series)
+        last_published = load_timestamps(options.name)
 
         # list of SourcePackages for each component
         components = {
@@ -491,14 +602,24 @@
                 packagesets[ps.name] = ps.getSourcesIncluded(direct_inclusion=False)
                 packagesets_ftbfs[ps.name] = [] # empty list to add FTBFS for each package set later
 
-        for state in ('Failed to build', 'Dependency wait', 'Chroot problem', 'Failed to upload'):
-            last_published[state] = fetch_pkg_list(archive, series, state, last_published[state], default_arch_list, main_archive, main_series)
-
-        save_timestamps(archive, series, last_published)
+        teams = requests.get('https://people.canonical.com/~ubuntu-archive/package-team-mapping.json').json()
+
+        # Per team list of FTBFS
+        teams_ftbfs = {team: [] for team in teams}
+
+        for state in ('Failed to build', 'Dependency wait', 'Chroot problem', 'Failed to upload', 'Cancelled build'):
+            last_published[state] = fetch_pkg_list(archive, series, state, last_published[state], default_arch_list, main_archive, main_series, options.release_only)
+
+        save_timestamps(options.name, last_published)
+
+        if options.notice_file:
+            notice = open(options.notice_file).read()
+        else:
+            notice = None
 
         print "Updating Historical database..."
         store_historical(archive, series)
         print "Generating HTML page..."
-        generate_page(archive, series, archs_by_archive, main_archive)
+        generate_page(options.name, archive, series, archs_by_archive, main_archive, notice=notice, release_only=options.release_only)
         print "Generating CSV file..."
-        generate_csvfile(archive, series)
+        generate_csvfile(options.name)
 
=== modified file 'source/style.css'
--- source/style.css	2011-10-16 14:22:34 +0000
+++ source/style.css	2021-08-13 16:24:09 +0000
@@ -52,6 +52,10 @@
 	background-color:#cc0000;
 }
 
+.CANCELLED {
+	background-color:#aa0000;
+}
+
 .MANUALDEPWAIT {
 	background:#ff9900;
 }