Merge lp:~nskaggs/juju-release-tools/generate-release-notes into lp:juju-release-tools

[Nicholas Skaggs]

Add a script to generate a release checklist. This is initially based around juju, but is generic enough to support other projects. The checklist can be used to gate releases.

[Nicholas Skaggs (nskaggs)]

If someone knows why I can't import assignee from a bug without unicode issues, that would be a welcome improvement on below.

[Aaron Bentley (abentley)]

review: Needs Fixing

[Curtis Hovey (sinzui)]

I see you got hold of some of my scripts, but you are missing a few crucial trick to level-up. This script doesn't scale or work for bots, see juju-reports/jujureports/jobs/lptasks.py which is also gave to Chris an Torsten last year, and Several other managers over the last 5 years.

The "display_name" and "title" attributes causes problems for everyone. We often do this for reports

    bugtask.assignee.display_name.encode('ascii', 'ignore')

More inline

review: Needs Information (code)

[lp:~nskaggs/juju-release-tools/generate-release-notes]

377. By Nicholas Skaggs on 2017-03-28

cleanups per review

378. By Nicholas Skaggs on 2017-03-28

suggested changes from release meeting

379. By Nicholas Skaggs on 2017-03-28

add stakeholder output

380. By Nicholas Skaggs on 2017-03-28

add lp credentials for bot use, simplify get_*_bugs

381. By Nicholas Skaggs on 2017-03-28

add safe_encode, fix tag typo

Unmerged revisions

381. By Nicholas Skaggs on 2017-03-28

add safe_encode, fix tag typo

380. By Nicholas Skaggs on 2017-03-28

add lp credentials for bot use, simplify get_*_bugs

379. By Nicholas Skaggs on 2017-03-28

add stakeholder output

378. By Nicholas Skaggs on 2017-03-28

suggested changes from release meeting

377. By Nicholas Skaggs on 2017-03-28

cleanups per review

376. By Nicholas Skaggs on 2017-03-21

v1, manual updating of CDO and CI results required (sso + vpn requirements)

375. By Nicholas Skaggs on 2017-03-14

Working version with somewhat poor printing, unicode issue on assignee, flake8 issues, and no details for CI or CDO because of SSO requirements

374. By Nicholas Skaggs on 2017-03-14

working tags

373. By Nicholas Skaggs on 2017-03-14

naming

372. By Nicholas Skaggs on 2017-03-14

first pass

=== added file 'generate_release_checklist.py'
--- generate_release_checklist.py	1970-01-01 00:00:00 +0000
+++ generate_release_checklist.py	2017-03-28 20:06:32 +0000
@@ -0,0 +1,293 @@
+#!/usr/bin/python
+
+from __future__ import print_function
+
+from argparse import ArgumentParser
+import sys
+
+from launchpadlib.launchpad import Launchpad
+
+TEMPLATE = """\
+Release checklist for {project}-{milestone}
+{milestone_link}
+
+Meeting Attendees:
+Key StakeHolders:  {stakeholders}
+Meeting Date: {meeting_date}
+Release Date: {release_date}
+
+Test Results:
+CI: {project_ci_link}
+{project_ci_results}
+
+CDO QA: {cdo_qa_link}
+
+Compatibility Matrix:
+{cdo_qa_results}
+
+Main Features of this release:
+
+{features}
+
+Release Notes Available:
+
+{milestone}: {release_notes}
+
+Documentation Ready:
+
+{docs}
+
+Issues Resolved in this milestone: {num_fixed_bugs}
+All Issues fixed during this release: {all_fixed_bugs}
+
+{fixed_bugs}
+
+Open Release-Blocking Issues (stable-blocker, rc-blocker): {num_blocker_bugs}
+
+{blocker_tag_bugs}
+
+Open CDO-Blocking Issues (cdo-qa-blocker): {num_cdo_blocker_bugs}
+
+{cdo_blocker_bugs}
+
+Open Critical Issues: {num_critical_bugs}
+
+{critical_bugs}
+
+Other Issues Anyone?
+"""
+
+
+def _as_list(value):
+    """Return None or the list."""
+    if value and not isinstance(value, list):
+        value = [value]
+    return value
+
+
+def _check_for_stakeholder_tag(bug):
+    # tags in use
+    # Conjure-up: conjure-up
+    # OIL: oil-2.0, oil {would be nice to settle on one :D}
+    # CDO QA: cdo-qa
+    # CPE: cpe, cpe-sa, cpec {would be nice to settle on one :D}
+    # IS: canonical-is
+    # Bootstack: canonical-bootstack
+    # STS: sts
+    # JAAS: jaas
+    # Landscape: landscape
+    # Openstack and UOSCI: uosci
+    # Juju QA: jujuqa
+    # Charm Eco: charmers
+
+    # proposed tags
+    # LXD: canonical-lxd
+    # MAAS: canonical-maas
+    # Docs: canonical-docs
+    # Omnibus: omnibus
+    # GUI: juju-gui
+    stakeholders = []
+    stakeholder_tags = ['conjure-up', 'oil-.20', 'oil', 'cdo-qa', 'cpe',
+                        'cpe-sa', 'cpec', 'canonical-is',
+                        'canonical-bootstack', 'sts', 'jaas', 'landscape',
+                        'uosci', 'jujuqa', 'charmers', 'canonical-lxd',
+                        'canonical-maas', 'canonical-docs', 'omnibus',
+                        'juju-gui']
+    for tag in stakeholder_tags:
+        if tag in bug.tags:
+            stakeholders.append(tag)
+
+    return stakeholders
+
+
+def _get_search_params(status=None, importance=None, tags=None,
+                       omit_duplicates=True):
+    return dict(status=_as_list(status), importance=_as_list(importance),
+                tags=_as_list(tags), tags_combinator='Any',
+                omit_duplicates=omit_duplicates)
+
+
+def _safe_encode(string):
+    return string.encode('ascii', 'ignore')
+
+
+def _parse_bug_tasks(tasks):
+    bugs = []
+    for bugtask in tasks:
+        bug = bugtask.bug
+
+        if bugtask.assignee:
+            assignee = bugtask.assignee.display_name
+        else:
+            assignee = u'Unassigned'
+
+        if bugtask.milestone:
+            milestone = bugtask.milestone.name
+        else:
+            milestone = u'None'
+
+        stakeholders = _check_for_stakeholder_tag(bugtask.bug)
+        if not stakeholders:
+            stakeholders = 'Not a stakeholder bug'
+        bugs.append((bug.id, bugtask.date_created, bugtask.status,
+                    bugtask.importance, milestone,
+                    _safe_encode(bug.owner.display_name),
+                    _safe_encode(assignee),
+                    _safe_encode(bug.title.capitalize()), stakeholders))
+    return bugs
+
+
+def get_milestone_bugs(lp_milestone, status=None,
+                       importance=None, tags=None):
+    """Return a list of bug information."""
+
+    params = _get_search_params(status, importance, tags)
+    tasks = lp_milestone.searchTasks(**params)
+
+    return _parse_bug_tasks(tasks)
+
+
+def get_project_bugs(lp_project, status=None, importance=None, tags=None):
+    """Return a list of bug information"""
+
+    params = _get_search_params(status, importance, tags)
+    tasks = lp_project.searchTasks(**params)
+
+    return _parse_bug_tasks(tasks)
+
+
+def format_bug_text(bugs):
+    """Return the list of bug information as formatted text."""
+    resolved = []
+
+    for bug in bugs:
+        # Formatting to ensure fixed spacing
+        status = '{0: <13}'.format(bug[2])
+        importance = '{0: <10}'.format(bug[3])
+        milestone = '{0: <10}'.format(bug[4])
+
+        # construct generic lp link
+        lp_link = 'https://launchpad.net/bugs/{}'.format(bug[0])
+        # Tabs import into google docs better than spaces
+        resolved.append(u"{}\n{}\t{}\t{}\t{}\t{}\t{}\n{}\n{}"
+                        "\n".format(bug[7], bug[0], bug[1].date(), status,
+                                    importance, milestone, bug[5], bug[8],
+                                    lp_link))
+    resolved_text = '\n'.join(resolved)
+    return resolved_text
+
+
+def make_notes(lp, milestone, project, revision):
+    """Return formatted release notes."""
+
+    lp_project = lp.projects[project]
+    lp_milestone = lp_project.getMilestone(name=milestone)
+
+    milestone_link = lp_milestone.web_link
+    meeting_date = u'[Add the meeting date here.]'
+    release_date = u'[Add the anticipated release date here.]'
+    features = u'[Add the features changes here.]'
+
+    bugs = get_milestone_bugs(lp_milestone, status=['Fix Committed'])
+    fixed_bugs = format_bug_text(bugs)
+    num_fixed_bugs = len(bugs)
+    all_fixed_bugs = ('https://launchpad.net/{}/+bugs?'
+                      'field.status%3Alist=FIXCOMMITTED'.format(project))
+
+    bugs = get_project_bugs(lp_project, tags=['stable-blocker', 'rc-blocker'])
+    blocker_tag_bugs = format_bug_text(bugs)
+    num_blocker_bugs = len(bugs)
+
+    bugs = get_project_bugs(lp_project, tags=['cdo-qa-blocker'])
+    cdo_blocker_bugs = format_bug_text(bugs)
+    num_cdo_blocker_bugs = len(bugs)
+
+    bugs = get_project_bugs(lp_project, importance=['Critical'])
+    critical_bugs = format_bug_text(bugs)
+    num_critical_bugs = len(bugs)
+
+    stakeholders = (u'<Juju>, <Conjure-up>, <CDO/OIL>, <Docs>, <CPE>,'
+                    '<Openstack>, <Landscape>,  <JAAS/GUI>, <LXD>, <MAAS>')
+    project_ci_link = u'http://qa.jujucharms.com/releases/{}'.format(revision)
+    project_ci_results = u'[Fill in basic information from project CI]'
+    cdo_qa_link = u'http://people.canonical.com/~jog/cdo-qa-test-report/'
+    cdo_qa_results = u'[Fill in compatibility matrix information from CDO QA]'
+    docs = u'[Add the link to the documentation here.]'
+    release_notes = u'[Add the link to release notes here.]'
+
+    text = TEMPLATE.format(
+        project=project,
+        milestone=milestone,
+        milestone_link=milestone_link,
+        stakeholders=stakeholders,
+        meeting_date=meeting_date,
+        release_date=release_date,
+        project_ci_link=project_ci_link,
+        project_ci_results=project_ci_results,
+        cdo_qa_link=cdo_qa_link,
+        cdo_qa_results=cdo_qa_results,
+        features=features,
+        num_fixed_bugs=num_fixed_bugs,
+        fixed_bugs=fixed_bugs,
+        all_fixed_bugs=all_fixed_bugs,
+        num_cdo_blocker_bugs=num_cdo_blocker_bugs,
+        cdo_blocker_bugs=cdo_blocker_bugs,
+        num_blocker_bugs=num_blocker_bugs,
+        blocker_tag_bugs=blocker_tag_bugs,
+        num_critical_bugs=num_critical_bugs,
+        critical_bugs=critical_bugs,
+        docs=docs,
+        release_notes=release_notes)
+
+    # Normalise the whitespace between sections. The text can have
+    # extra whitespae when blank sections are interpolated.
+    text = text.replace('\n\n\n\n', '\n\n\n')
+    return text
+
+
+def save_notes(text, file_name):
+    """Save the notes to the named file or print to stdout."""
+    if file_name is None:
+        sys.stdout.write(text)
+    else:
+        with open(file_name, 'w') as rn:
+            rn.write(text)
+
+
+def parse_args(args=None):
+    parser = ArgumentParser(description='Create release checklist '
+                            'for a milestone')
+    parser.add_argument(
+        '--file-name', help='the name of file to write.', default=None)
+    parser.add_argument('project', help='the launchpad project to examine.')
+    parser.add_argument('milestone', help='the milestone to examine.')
+    parser.add_argument('revisionbuild', help='the CI revision build to '
+                        'examine.')
+    parser.add_argument('-c', '--credentials', default=None,
+                        help='Launchpad credentials file, '
+                             'or use "SYSTEM" to find one.')
+    return parser.parse_args(args)
+
+
+def get_lp(credentials_file):
+    if credentials_file == 'SYSTEM':
+        from jujureports.jobs.utils import get_credentials_file
+        credentials_file = get_credentials_file()
+    lp = Launchpad.login_with(
+        'release-checklist', service_root='https://api.launchpad.net',
+        version='devel', credentials_file=credentials_file)
+    return lp
+
+
+def main(argv):
+    args = parse_args(argv[1:])
+    lp = get_lp(args.credentials)
+    print('Generating notes for {} based on milestone {}, revision {}'.format(
+        args.project, args.milestone, args.revisionbuild))
+    text = make_notes(lp, args.milestone, args.project, args.revisionbuild)
+    save_notes(text, args.file_name)
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv))