Merge lp:~lool/launchpad-work-items-tracker/jira-support into lp:~linaro-automation/launchpad-work-items-tracker/linaro

[Loïc Minier]

This adds support for getting cards data out of JIRA instead of kanbantool.com + papyrs.com; it's only active if you set cards_source = 'jira' in the project's config though (so it can be merged and remain inactive). This is deployed in staging and works there, but with some limitations:
* description is taken as text, but it's a mixture of HTML and of JIRA wiki syntax in cards; in the current version of status.linaro.org, description is empty/broken so this isn't a regression but it looks ugly
* "Roadmap card id" currently refers to e.g. CARD-12 instead of some human-readable text like TCWG2012-GDB-FOR-ANDROID; this is stricted and safer, but uglier; also, this comes out of the Meta section, but Данило and I had discussed getting this out of the blueprint specification URL
* there are some missing features like setting the sponsor; relatively minor

Probably fixing the latter requires some DB changes; I'm not sure what rules we operate on for these though, e.g. how does one deploy the schema update?

In general, I consider this is ready for merging, but we can keep this branch open until the above bits are also fixed.

[lp:~lool/launchpad-work-items-tracker/jira-support]

340. By Loïc Minier on 2012-06-26

No need to remap priority names and component names.

[Milo Casagrande (milo)]

Hello Loic,

thanks for working on this.

On Tue, Jun 26, 2012 at 5:14 PM, Loïc Minier <email address hidden> wrote:
>
> Probably fixing the latter requires some DB changes; I'm not sure what rules we operate on for these though, e.g. how does one deploy the schema update?

WRT this point, I have to collect info too: our knowledge base does not report anything for this situation, and will have to ask Danilo and maybe IS too.

From a rapid PEP8 run, I get these two errors:

collect_jira:5:15: E401 multiple imports on one line
all-projects:77:80: E501 line too long (94 characters)

Apart from those, everything looks good. In case, fix during the merge.
Thanks.

review: Approve

[lp:~lool/launchpad-work-items-tracker/jira-support]

341. By Loïc Minier on 2012-07-02

Split and sort system imports.

342. By Loïc Minier on 2012-07-02

Misc PEP8 fixes.

343. By Loïc Minier on 2012-07-02

Split and sort system imports.

344. By Loïc Minier on 2012-07-02

Fix a bunch of PEP8 issues.

345. By Loïc Minier on 2012-07-02

Drop useless imports.

[Loïc Minier (lool)]

I've now fixed the PEP8 issues you mention and others, thanks! Note that collect_jira was copied from collect_roadmap (kanbantool.com + papyrs.com code) and copied over the syntax problems; now code isn't much repeated, and I also had removed useless imports, but the original syntax layout remained.

One problem with fixing these issues is that it makes the delta with the upstream project larger, but then I don't think we will ever merge back, so...

(I've not fixed all the E501 line too long issues because there were too many of these.)

[lp:~lool/launchpad-work-items-tracker/jira-support]

346. By Loïc Minier on 2012-07-10

Update sponsor information from JIRA.

347. By Loïc Minier on 2012-07-10

Set card Size from JIRA timetracking information if available.

348. By Loïc Minier on 2012-07-10

Wrap card sponsor in unicode().

349. By Loïc Minier on 2012-07-10

PEP8 line too long fixes.

350. By Loïc Minier on 2012-07-10

Acceptance criteria is in the description by design.

351. By Loïc Minier on 2012-07-10

Merge with Linaro branch of workitems tracker which has some independent PEP8
fixes.

[Loïc Minier (lool)]

I've pushed some further changes to complete support for other fields; what's still missing is support for fetching the description, but need to get beautifulsoup deployed on status.linaro.org to get this done; the other changes can wait and require DB changes.

=== modified file 'all-projects'
--- all-projects	2012-01-30 10:03:24 +0000
+++ all-projects	2012-07-10 16:26:24 +0000
@@ -13,12 +13,19 @@
 import subprocess
 import sys
 
+import report_tools
+
 
 def collect(source_dir, db_file, config_file, extra_args):
     return run_collect_script(source_dir, db_file, config_file, extra_args,
                               "collect")
 
 
+def collect_jira(source_dir, db_file, config_file, extra_args):
+    return run_collect_script(source_dir, db_file, config_file, extra_args,
+                              "collect_jira")
+
+
 def collect_roadmap(source_dir, db_file, config_file, extra_args):
     return run_collect_script(source_dir, db_file, config_file, extra_args,
                               "collect_roadmap")
@@ -67,7 +74,8 @@
 
 
 def main():
-    parser = optparse.OptionParser(usage="%prog <database dir> <www root dir> [www root url]")
+    parser = optparse.OptionParser(
+        usage="%prog <database dir> <www root dir> [www root url]")
     parser.add_option("--config-dir", dest="config_dir", default="config")
     parser.add_option("--roadmap-config-file", dest="roadmap_config_file",
                       default="roadmap-config")
@@ -103,6 +111,11 @@
         os.path.join(source_dir, opts.config_dir, "*%s" % valid_config_suffix))
 
     for config_file in filenames:
+        # read roadmap config to find where to get cards from
+        cfg = report_tools.load_config(config_file)
+        # default to kanbantool
+        cards_source = cfg.get('cards_source', 'kanban')
+
         project_name = os.path.basename(config_file)[:-len(valid_config_suffix)]
         project_output_dir = os.path.join(output_dir, project_name)
         db_file = os.path.join(db_dir, "%s.db" % project_name)
@@ -127,26 +140,38 @@
             sys.stderr.write("collect failed for %s" % project_name)
             continue
 
-        extra_collect_roadmap_args = []
-        extra_collect_roadmap_args.extend(["--board", '10721'])
-        if opts.kanban_token_file is not None:
-            with open(opts.kanban_token_file) as token_file:
-                token = token_file.read()
-                extra_collect_roadmap_args.extend(["--kanbantoken", token])
-        else:
-            sys.stderr.write("No Kanbantool API token given to "
-                             "collect_roadmap for %s" % project_name)
-        if opts.papyrs_token_file is not None:
-            with open(opts.papyrs_token_file) as token_file:
-                token = token_file.read()
-                extra_collect_roadmap_args.extend(["--papyrstoken", token])
-        else:
-            sys.stderr.write("No Papyrs API token given to "
-                             "collect_roadmap for %s" % project_name)
+        if cards_source == 'jira':
+            extra_collect_jira_args = []
+            if not collect_jira(source_dir, db_file, opts.roadmap_config_file,
+                                extra_collect_jira_args):
+                sys.stderr.write("collect_jira failed for %s" % project_name)
+                continue
+        elif cards_source == 'kanban':
+            extra_collect_roadmap_args = []
+            extra_collect_roadmap_args.extend(["--board", '10721'])
+            if opts.kanban_token_file is not None:
+                with open(opts.kanban_token_file) as token_file:
+                    token = token_file.read()
+                    extra_collect_roadmap_args.extend(["--kanbantoken", token])
+            else:
+                sys.stderr.write("No Kanbantool API token given to "
+                                 "collect_roadmap for %s" % project_name)
+            if opts.papyrs_token_file is not None:
+                with open(opts.papyrs_token_file) as token_file:
+                    token = token_file.read()
+                    extra_collect_roadmap_args.extend(["--papyrstoken", token])
+            else:
+                sys.stderr.write("No Papyrs API token given to "
+                                 "collect_roadmap for %s" % project_name)
 
-        if not collect_roadmap(source_dir, db_file, opts.roadmap_config_file,
-                               extra_collect_roadmap_args):
-            sys.stderr.write("collect_roadmap failed for %s" % project_name)
+            if not collect_roadmap(source_dir, db_file,
+                                   opts.roadmap_config_file,
+                                   extra_collect_roadmap_args):
+                sys.stderr.write("collect_roadmap failed for %s" %
+                                 project_name)
+        else:
+            sys.stderr.write("Unknown cards source %s" % cards_source)
+            continue
 
         publish_new_db(project_name, project_output_dir, db_file)
         generate_reports(project_output_dir, config_file, db_file,
 
=== modified file 'collect'
--- collect	2012-06-29 14:04:58 +0000
+++ collect	2012-07-10 16:26:24 +0000
@@ -6,15 +6,16 @@
 # Copyright (C) 2010, 2011 Canonical Ltd.
 # License: GPL-3
 
-import urllib
+import logging
+import optparse
+import os
+import pwd
 import re
+import smtplib
 import sys
-import optparse
-import smtplib
-import pwd
-import os
+import urllib
 import urlparse
-import logging
+
 from email.mime.text import MIMEText
 
 from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
@@ -23,8 +24,8 @@
     CollectorStore,
     PersonCache,
     WorkitemParser,
-    bug_wi_states
-)
+    bug_wi_states,
+    )
 from lpworkitems.database import get_store
 from lpworkitems.error_collector import (
     BlueprintURLError,
@@ -71,11 +72,13 @@
     """Get a link to the Launchpad API object on the website."""
     api_link = item.self_link
     parts = urlparse.urlparse(api_link)
-    link = parts.scheme + "://" + parts.netloc.replace("api.", "") + "/" + parts.path.split("/", 2)[2]
+    link = parts.scheme + "://" + parts.netloc.replace("api.", "") + \
+           "/" + parts.path.split("/", 2)[2]
     return link.decode("utf-8")
 
 
 import simplejson
+
 _orig_loads = simplejson.loads
 
 
@@ -103,7 +106,10 @@
     '''
     model_bp = Blueprint.from_launchpad(bp)
     if model_bp.milestone_name not in collector.valid_milestone_names():
-        data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % model_bp.milestone_name, True)
+        data_error(
+            web_link(bp),
+            'milestone "%s" is unknown/invalid' % model_bp.milestone_name,
+            True)
     model_bp = collector.store_blueprint(model_bp)
     if model_bp:
         dbg('lp_import_blueprint: added blueprint: %s' % bp.name)
@@ -123,7 +129,10 @@
     model_group = BlueprintGroup.from_launchpad(bp)
     model_group.area = area
     if model_group.milestone_name not in collector.valid_milestone_names():
-        data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % model_group.milestone, True)
+        data_error(
+            web_link(bp),
+            'milestone "%s" is unknown/invalid' % model_group.milestone,
+            True)
 
     model_group = collector.store_blueprint_group(model_group)
     if model_group is None:
@@ -154,7 +163,8 @@
     collector.store_meta(key, value, bp_name)
 
 
-def parse_complexity_item(collector, line, bp_name, bp_url, def_milestone, def_assignee):
+def parse_complexity_item(collector, line, bp_name, bp_url, def_milestone,
+                          def_assignee):
     line = line.strip()
     # remove special characters people tend to type
     line = re.sub('[^\w -.]', '', line)
@@ -183,7 +193,9 @@
         dbg('\tComplexity: %s MS: %s Who: %s' % (num, milestone, assignee))
         collector.store_complexity(assignee, num, milestone, bp_name)
     except ValueError:
-        data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError))
+        data_error(bp_url,
+            "\tComplexity line '%s' could not be parsed %s" %
+                (line, ValueError))
 
 
 def milestone_extract(text, valid_milestones):
@@ -196,7 +208,7 @@
     return None
 
 
-def lp_import_blueprint_workitems(collector, bp, distro_release, people_cache=None, projects=None):
+def lp_import_blueprint_workitems(collector, bp, distro_release,
     '''Collect work items from a Launchpad blueprint.
 
     This includes work items from the whiteboard as well as linked bugs.
@@ -212,17 +224,19 @@
 
     model_bp = collector.store.find(
         Blueprint, Blueprint.name == bp.name).one()
-    assert model_bp is not None, "Asked to process workitems of %s when it is not in the db" % bp.name
+    assert model_bp is not None, \
+        "Asked to process workitems of %s when it is not in the db" % bp.name
 
-    dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % (
+    dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s,' \
+        ' spec assignee: %s, spec implementation: %s)' % (
         bp.name, model_bp.milestone_name, model_bp.assignee_name,
         model_bp.implementation))
 
     valid_milestones = collector.valid_milestone_names()
     global error_collector
     parser = WorkitemParser(
-        model_bp, model_bp.milestone_name, collector.lp, people_cache=people_cache,
-        error_collector=error_collector)
+        model_bp, model_bp.milestone_name, collector.lp,
+        people_cache=people_cache, error_collector=error_collector)
 
     # Get work items from both the whiteboard and the new workitems_text
     # property. Once the migration is completed and nobody's using the
@@ -239,16 +253,19 @@
                 m = work_items_re.search(l)
                 if m:
                     in_workitems_block = True
-                    dbg('lp_import_blueprint_workitems(): starting work items block at ' + l)
+                    dbg('lp_import_blueprint_workitems():'
+                        ' starting work items block at ' + l)
                     milestone = milestone_extract(m.group(1), valid_milestones)
                     dbg('  ... setting milestone to ' + str(milestone))
-                    parser.milestone_name = milestone or parser.blueprint.milestone_name
+                    parser.milestone_name = \
+                        milestone or parser.blueprint.milestone_name
                 continue
 
             if in_workitems_block:
                 dbg("\tworkitem (raw): '%s'" % (l.strip()))
                 if not l.strip():
-                    dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l)
+                    dbg('lp_import_blueprint_workitems():'
+                        ' closing work items block with line: ' + l)
                     in_workitems_block = False
                     parser.milestone_name = parser.blueprint.milestone_name
                 workitem = parser.parse_blueprint_workitem(l)
 
=== added file 'collect_jira'
--- collect_jira	1970-01-01 00:00:00 +0000
+++ collect_jira	2012-07-10 16:26:24 +0000
@@ -0,0 +1,229 @@
+#!/usr/bin/python
+#
+# Pull items from cards.linaro.org and put them into a database.
+
+import logging
+import optparse
+import os
+import simplejson
+import sys
+import urllib2
+
+import jira
+from lpworkitems.collect_roadmap import (
+    CollectorStore,
+    )
+from lpworkitems.database import get_store
+from lpworkitems.error_collector import (
+    ErrorCollector,
+    StderrErrorCollector,
+    )
+from lpworkitems.models_roadmap import (
+    Lane,
+    Card,
+    )
+import report_tools
+
+
+# An ErrorCollector to collect the data errors for later reporting
+error_collector = None
+
+
+logger = logging.getLogger("linarojira")
+
+JIRA_API_URL = 'http://cards.linaro.org/rest/api/2'
+JIRA_PROJECT_KEY = 'CARD'
+JIRA_ISSUE_BY_KEY_URL = 'http://cards.linaro.org/browse/%s'
+
+
+def dbg(msg):
+    '''Print out debugging message if debugging is enabled.'''
+    logger.debug(msg)
+
+
+def get_json_data(url):
+    data = None
+    try:
+        data = simplejson.load(urllib2.urlopen(url))
+    except urllib2.HTTPError, e:
+        print "HTTP error for url '%s': %d" % (url, e.code)
+    except urllib2.URLError, e:
+        print "Network error for url '%s': %s" % (url, e.reason.args[1])
+    except ValueError, e:
+        print "Data error for url '%s': %s" % (url, e.args[0])
+
+    return data
+
+
+def jira_import(collector, cfg, opts):
+    '''Collect roadmap items from JIRA into DB.'''
+
+    # import JIRA versions as database Lanes
+    result = jira.do_request(opts, 'project/%s/versions' % JIRA_PROJECT_KEY)
+    for version in result:
+        dbg('Adding lane (name = %s, id = %s)' %
+            (version['name'], version['id']))
+        model_lane = Lane(unicode(version['name']), int(version['id']))
+        if model_lane.name == cfg['current_lane']:
+            model_lane.is_current = True
+        else:
+            model_lane.is_current = False
+        collector.store_lane(model_lane)
+
+    # find id of "Sponsor" custom field in JIRA
+    result = jira.do_request(opts, 'field')
+    sponsor_fields = [field for field in result if field['name'] == 'Sponsor']
+    assert len(sponsor_fields) == 1, 'Not a single Sponsor field'
+    sponsor_field_id = sponsor_fields[0]['id']
+
+    # import JIRA issues as database Cards
+    result = jira.do_request(
+        opts, 'search', jql='project = %s' % JIRA_PROJECT_KEY,
+        fields=['summary', 'fixVersions', 'status', 'components',
+                'priority', 'description', 'timetracking', sponsor_field_id])
+    for issue in result['issues']:
+        fields = issue['fields']
+        name = unicode(fields['summary'])
+        card_id = int(issue['id'])
+        key = unicode(issue['key'])
+        fixVersions = fields['fixVersions']
+        if len(fixVersions) == 0:
+            dbg('Skipping card without lane (name = %s, key = %s)' %
+                (name, key))
+            continue
+        # JIRA allows listing multiple versions in fixVersions
+        assert len(fixVersions) == 1
+        lane_id = int(fixVersions[0]['id'])
+
+        dbg('Adding card (name = %s, id = %s, lane_id = %s, key = %s)' %
+                (name, card_id, lane_id, key))
+        model_card = Card(name, card_id, lane_id, key)
+        model_card.status = unicode(fields['status']['name'])
+        components = fields['components']
+        if len(components) == 0:
+            dbg('Skipping card without component (name = %s, key = %s)' %
+                (name, key))
+        # JIRA allows listing multiple components
+        assert len(components) == 1
+        model_card.team = unicode(components[0]['name'])
+        model_card.priority = unicode(fields['priority']['name'])
+        size_fields = []
+        timetracking = fields['timetracking']
+        if 'originalEstimate' in timetracking:
+            size_fields += [
+                'original estimate: %s' % timetracking['originalEstimate']]
+        if 'remainingEstimate' in timetracking:
+            size_fields += [
+                'remaining estimate: %s' % timetracking['remainingEstimate']]
+        model_card.size = unicode(', '.join(size_fields))
+        model_card.sponsor = u''
+        # None if no sponsor is selected
+        if fields[sponsor_field_id] is not None:
+            sponsors = [s['value'] for s in fields[sponsor_field_id]]
+            model_card.sponsor = unicode(', '.join(sorted(sponsors)))
+        model_card.url = JIRA_ISSUE_BY_KEY_URL % key
+        # XXX need to either download the HTML version or convert this to HTML
+        model_card.description = unicode(fields['description'])
+        # acceptance criteria is in the description
+        model_card.acceptance_criteria = u''
+        collector.store_card(model_card)
+    return
+
+########################################################################
+#
+# Program operations and main
+#
+########################################################################
+
+
+def parse_argv():
+    '''Parse CLI arguments.
+
+    Return (options, args) tuple.
+    '''
+    optparser = optparse.OptionParser()
+    optparser.add_option('-d', '--database',
+        help='Path to database', dest='database', metavar='PATH')
+    optparser.add_option('-c', '--config',
+        help='Path to configuration file', dest='config', metavar='PATH')
+    optparser.add_option('--debug', action='store_true', default=False,
+        help='Enable debugging output in parsing routines')
+    optparser.add_option('--mail', action='store_true', default=False,
+        help='Send data errors as email (according to "error_config" map in '
+            'config file) instead of printing to stderr', dest='mail')
+    optparser.add_option('--jira-username', default='robot',
+        help='JIRA username for authentication', dest='jira_username')
+    optparser.add_option('--jira-password', default='cuf4moh2',
+        help='JIRA password for authentication', dest='jira_password')
+
+    (opts, args) = optparser.parse_args()
+
+    if not opts.database:
+        optparser.error('No database given')
+    if not opts.config:
+        optparser.error('No config given')
+
+    return opts, args
+
+
+def setup_logging(debug):
+    ch = logging.StreamHandler()
+    ch.setLevel(logging.INFO)
+    formatter = logging.Formatter("%(message)s")
+    ch.setFormatter(formatter)
+    logger.setLevel(logging.INFO)
+    logger.addHandler(ch)
+    if debug:
+        ch.setLevel(logging.DEBUG)
+        formatter = logging.Formatter(
+            "%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+        ch.setFormatter(formatter)
+        logger.setLevel(logging.DEBUG)
+
+
+def update_todays_blueprint_daily_count_per_state(collector):
+    """Clear today's entries and create them again to reflect the current
+    state of blueprints."""
+    collector.clear_todays_blueprint_daily_count_per_state()
+    collector.store_roadmap_bp_count_per_state()
+
+
+def main():
+    report_tools.fix_stdouterr()
+
+    (opts, args) = parse_argv()
+    opts.jira_api_url = JIRA_API_URL
+
+    setup_logging(opts.debug)
+
+    global error_collector
+    if opts.mail:
+        error_collector = ErrorCollector()
+    else:
+        error_collector = StderrErrorCollector()
+
+    cfg = report_tools.load_config(opts.config)
+
+    lock_path = opts.database + ".lock"
+    lock_f = open(lock_path, "wb")
+    if report_tools.lock_file(lock_f) is None:
+        print "Another instance is already running"
+        sys.exit(0)
+
+    store = get_store(opts.database)
+    collector = CollectorStore(store, '', error_collector)
+
+    collector.clear_lanes()
+    collector.clear_cards()
+
+    jira_import(collector, cfg, opts)
+
+    update_todays_blueprint_daily_count_per_state(collector)
+
+    store.commit()
+
+    os.unlink(lock_path)
+
+
+if __name__ == '__main__':
+    main()
 
=== modified file 'collect_roadmap'
--- collect_roadmap	2012-01-30 12:08:06 +0000
+++ collect_roadmap	2012-07-10 16:26:24 +0000
@@ -2,10 +2,12 @@
 #
 # Pull items from the Linaro roadmap in Kanbantool and put them into a database.
 
-import urllib2, re, sys, optparse, smtplib, pwd, os
+import logging
+import optparse
+import os
 import simplejson
-import logging
-from email.mime.text import MIMEText
+import sys
+import urllib2
 
 from lpworkitems.collect_roadmap import (
     CollectorStore,
 
=== modified file 'jira.py'
--- jira.py	2012-05-14 11:39:09 +0000
+++ jira.py	2012-07-10 16:26:24 +0000
@@ -6,6 +6,7 @@
 import simplejson
 import urllib2
 
+
 def do_request(opts, relpathname, **kwargs):
     request = urllib2.Request('%s/%s' % (opts.jira_api_url, relpathname))
     if opts.jira_username and opts.jira_password:
@@ -20,6 +21,7 @@
     response_data = urllib2.urlopen(request, request_data)
     return simplejson.load(response_data)
 
+
 def main():
     parser = optparse.OptionParser(usage="%prog")
     parser.add_option("--jira-api-url", dest="jira_api_url",
@@ -31,7 +33,8 @@
     opts, args = parser.parse_args()
 
     # simple search
-    print do_request(opts, 'search', maxResults=1, jql='project = CARD', fields=['summary', 'status'])
+    print do_request(opts, 'search', maxResults=1, jql='project = CARD',
+                     fields=['summary', 'status'])
 
     # information about a project
     #print do_request(opts, 'project/CARD')
@@ -50,4 +53,3 @@
 
 if __name__ == "__main__":
     main()
-
 
=== modified file 'utils.py'
--- utils.py	2011-09-12 14:41:07 +0000
+++ utils.py	2012-07-10 16:26:24 +0000
@@ -4,4 +4,3 @@
     if isinstance(attr, unicode):
         return attr
     return attr.decode("utf-8")
-