Merge ~vorlon/ubuntu-dev-tools:pm-helper into ubuntu-dev-tools:main

[Steve Langasek]

As part of my +1 maintenance shift this week I scratched the itch of wanting a commandline tool to streamline identifying a package to work on in update_excuses and taking a lock on it. Initially this just takes a package name and manages update-excuse bugs for it but eventually it should be extended to scroll through the list of packages in update_excuses (oldest first) and pick one, then grab information from various sources (launchpad, autopkgtest.u.c, Debian archive, buildd.debian.org, ci.debian.net) to make suggestions on fixing.

Very raw at the moment - no tests yet.

[Bryce Harrington (bryce)]

Cool, hopefully my suggestions inline are useful. I look forward to how this develops.

review: Needs Fixing

[Steve Langasek (vorlon)]

Thanks for the review, Bryce! I've implemented most of your suggestions, and responded to comments. Would you care to re-review?

[Bryce Harrington (bryce)]

Looks great Steve, the updates all look great. This'll make a solid start for the tool. The man page is a nice touch, I'm often too lazy about making those. :-)

review: Approve

[Benjamin Drung (bdrung)]

This script has zero test cases which makes maintaining it hard. I would expect new code to come with test cases.

[Simon Quigley (tsimonq2)]

I'm no longer going to block on test cases. Let's move this forward, if there is interest in a test case we can address that later.

I apologize for not looking at this sooner, given the last uploads to ubuntu-dev-tools are from me. I will take care of the upload tomorrow, along with some testing and any necessary fixes.

I noticed two potential pedantic items, I'll check them further then fix as appropriate.

Thank you all for your hard work on this! I'm certain this will improve collaboration.

review: Approve

diff --git a/debian/control b/debian/control
index 3f2d647..1345a0d 100644
--- a/debian/control
+++ b/debian/control
@@ -20,6 +20,7 @@ Build-Depends:
  pylint <!nocheck>,
  python3-all,
  python3-apt,
+ python3-dateutil,
  python3-debian,
  python3-debianbts,
  python3-distro-info,
@@ -132,6 +133,7 @@ Package: python3-ubuntutools
 Architecture: all
 Section: python
 Depends:
+ python3-dateutil,
  python3-debian,
  python3-distro-info,
  python3-httplib2,
diff --git a/doc/pm-helper.1 b/doc/pm-helper.1
new file mode 100644
index 0000000..c40934b
--- /dev/null
+++ b/doc/pm-helper.1
@@ -0,0 +1,44 @@
+.\" Copyright (C) 2023, Canonical Ltd.
+.\"
+.\" This program is free software; you can redistribute it and/or
+.\" modify it under the terms of the GNU General Public License, version 3.
+.\"
+.\" This program is distributed in the hope that it will be useful,
+.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
+.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+.\" General Public License for more details.
+.\"
+.\" You should have received a copy of the GNU General Public License
+.\" along with this program.  If not, see <http://www.gnu.org/licenses/>.
+.TH pm\-helper 1 "June 2023" ubuntu\-dev\-tools
+
+.SH NAME
+pm\-helper \- helper to guide a developer through proposed\-migration work
+
+.SH SYNOPSIS
+.B pm\-helper \fR[\fIoptions\fR] [\fIpackage\fR]
+
+.SH DESCRIPTION
+Claim a package from proposed\-migration to work on and get additional
+information (such as the state of the package in Debian) that may be helpful
+in unblocking it.
+.PP
+This tool is incomplete and under development.
+
+.SH OPTIONS
+.TP
+.B \-l \fIINSTANCE\fR, \fB\-\-launchpad\fR=\fIINSTANCE\fR
+Use the specified instance of Launchpad (e.g. "staging"), instead of
+the default of "production".
+.TP
+.B \-v\fR, \fB--verbose\fR
+be more verbose
+.TP
+\fB\-h\fR, \fB\-\-help\fR
+Display a help message and exit
+
+.SH AUTHORS
+\fBpm\-helper\fR and this manpage were written by Steve Langasek
+<[e-mail]>.
+.PP
+Both are released under the GPLv3 license.
diff --git a/pm-helper b/pm-helper
new file mode 100755
index 0000000..9cdc176
--- /dev/null
+++ b/pm-helper
@@ -0,0 +1,149 @@
+#!/usr/bin/python3
+# Find the next thing to work on for proposed-migration
+# Copyright (C) 2023 Canonical Ltd.
+# Author: Steve Langasek <[e-mail]>
+
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License, version 3.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import lzma
+from argparse import ArgumentParser
+import sys
+import webbrowser
+import yaml
+
+from launchpadlib.launchpad import Launchpad
+
+from ubuntutools.utils import get_url
+
+
+# proposed-migration is only concerned with the devel series; unlike other
+# tools, don't make this configurable
+excuses_url = 'https://ubuntu-archive-team.ubuntu.com/proposed-migration/' \
+              + 'update_excuses.yaml.xz'
+
+
+def get_proposed_version(excuses, package):
+    for k in excuses['sources']:
+        if k['source'] == package:
+            return k.get('new-version')
+    return None
+
+
+def claim_excuses_bug(launchpad, bug, package):
+    print("LP: #%d: %s" % (bug.id, bug.title))
+    ubuntu = launchpad.distributions['ubuntu']
+    series = ubuntu.current_series.fullseriesname
+
+    for task in bug.bug_tasks:
+        # targeting to a series doesn't make the default task disappear,
+        # it just makes it useless
+        if task.bug_target_name == "%s (%s)" % (package, series):
+            our_task = task
+            break
+        elif task.bug_target_name == "%s (Ubuntu)" % package:
+            our_task = task
+
+    if our_task.assignee == launchpad.me:
+        print("Bug already assigned to you.")
+        return True
+    elif our_task.assignee:
+        print("Currently assigned to %s" % our_task.assignee.name)
+
+    print('''Do you want to claim this bug? [yN] ''', end="")
+    sys.stdout.flush()
+    response = sys.stdin.readline()
+    if response.strip().lower().startswith('y'):
+        our_task.assignee = launchpad.me
+        our_task.lp_save()
+        return True
+
+    return False
+
+
+def create_excuses_bug(launchpad, package, version):
+    print("Will open a new bug")
+    bug = launchpad.bugs.createBug(
+        title = 'proposed-migration for %s %s' % (package, version),
+        tags = ('update-excuse'),
+        target = 'https://api.launchpad.net/devel/ubuntu/+source/%s' % package,
+        description = '%s %s is stuck in -proposed.' % (package, version)
+    )
+
+    task = bug.bug_tasks[0]
+    task.assignee = launchpad.me
+    task.lp_save()
+
+    print("Opening %s in browser" % bug.web_link)
+    webbrowser.open(bug.web_link)
+    return bug
+
+
+def has_excuses_bugs(launchpad, package):
+    ubuntu = launchpad.distributions['ubuntu']
+    pkg = ubuntu.getSourcePackage(name=package)
+    if not pkg:
+        raise ValueError(f"No such source package: {package}")
+
+    tasks = pkg.searchTasks(tags=['update-excuse'], order_by=['id'])
+
+    bugs = [task.bug for task in tasks]
+    if not bugs:
+        return False
+
+    if len(bugs) == 1:
+        print("There is 1 open update-excuse bug against %s" % package)
+    else:
+        print("There are %d open update-excuse bugs against %s" \
+              % (len(bugs), package))
+
+    for bug in bugs:
+        if claim_excuses_bug(launchpad, bug, package):
+            return True
+
+    return True
+
+
+def main():
+    parser = ArgumentParser()
+    parser.add_argument(
+        "-l", "--launchpad", dest="launchpad_instance", default="production")
+    parser.add_argument(
+        "-v", "--verbose", default=False, action="store_true",
+        help="be more verbose")
+    parser.add_argument(
+        'package', nargs='?', help="act on this package only")
+    args = parser.parse_args()
+
+    args.launchpad = Launchpad.login_with(
+        "pm-helper", args.launchpad_instance, version="devel")
+
+    f = get_url(excuses_url, False)
+    with lzma.open(f) as lzma_f:
+        excuses = yaml.load(lzma_f, Loader=yaml.CSafeLoader)
+
+    if args.package:
+        try:
+            if not has_excuses_bugs(args.launchpad, args.package):
+                proposed_version = get_proposed_version(excuses, args.package)
+                if not proposed_version:
+                    print("Package %s not found in -proposed." % args.package)
+                    sys.exit(1)
+                create_excuses_bug(args.launchpad, args.package,
+                                   proposed_version)
+        except ValueError as e:
+            sys.stderr.write(f"{e}\n")
+    else:
+        pass # for now
+
+
+if __name__ == '__main__':
+    sys.exit(main())
diff --git a/requirements.txt b/requirements.txt
index 34b16ff..ba91ac5 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,6 @@
 python-debian
 python-debianbts
+dateutil
 distro-info
 httplib2
 launchpadlib
diff --git a/ubuntutools/utils.py b/ubuntutools/utils.py
new file mode 100644
index 0000000..dcc354e
--- /dev/null
+++ b/ubuntutools/utils.py
@@ -0,0 +1,82 @@
+# Copyright (C) 2019-2023 Canonical Ltd.
+# Author: Brian Murray <[e-mail]> et al.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Portions of archive related code that is re-used by various tools."""
+
+from datetime import datetime
+import os
+import re
+import urllib.request
+
+import dateutil.parser
+from dateutil.tz import tzutc
+
+
+def get_cache_dir():
+    cache_dir = os.environ.get('XDG_CACHE_HOME',
+                               os.path.expanduser(os.path.join('~', '.cache')))
+    uat_cache = os.path.join(cache_dir, 'ubuntu-archive-tools')
+    os.makedirs(uat_cache, exist_ok=True)
+    return uat_cache
+
+
+def get_url(url, force_cached):
+    ''' Return file to the URL, possibly caching it
+    '''
+    cache_file = None
+
+    # ignore bileto urls wrt caching, they're usually too small to matter
+    # and we don't do proper cache expiry
+    m = re.search('ubuntu-archive-team.ubuntu.com/proposed-migration/'
+                  '([^/]*)/([^/]*)',
+                  url)
+    if m:
+        cache_dir = get_cache_dir()
+        cache_file = os.path.join(cache_dir, '%s_%s' % (m.group(1), m.group(2)))
+    else:
+        # test logs can be cached, too
+        m = re.search(
+            'https://autopkgtest.ubuntu.com/results/autopkgtest-[^/]*/([^/]*)/([^/]*)'
+            '/[a-z0-9]*/([^/]*)/([_a-f0-9]*)@/log.gz',
+            url)
+        if m:
+            cache_dir = get_cache_dir()
+            cache_file = os.path.join(
+                cache_dir, '%s_%s_%s_%s.gz' % (
+                    m.group(1), m.group(2), m.group(3), m.group(4)))
+
+    if cache_file:
+        try:
+            prev_mtime = os.stat(cache_file).st_mtime
+        except FileNotFoundError:
+            prev_mtime = 0
+        prev_timestamp = datetime.fromtimestamp(prev_mtime, tz=tzutc())
+        new_timestamp = datetime.now(tz=tzutc()).timestamp()
+        if force_cached:
+            return open(cache_file, 'rb')
+
+    f = urllib.request.urlopen(url)
+
+    if cache_file:
+        remote_ts = dateutil.parser.parse(f.headers['last-modified'])
+        if remote_ts > prev_timestamp:
+            with open('%s.new' % cache_file, 'wb') as new_cache:
+                for line in f:
+                    new_cache.write(line)
+            os.rename('%s.new' % cache_file, cache_file)
+            os.utime(cache_file, times=(new_timestamp, new_timestamp))
+        f.close()
+        f = open(cache_file, 'rb')
+    return f