Merge lp:~jib/launchpad-work-items-tracker/complexity-points into lp:launchpad-work-items-tracker

[Jos Boumans]

This adds supports for more meta information in the whiteboard, most notably complexity points and notes. This requires a database change which will be done automatically by the 'collect' script.

These are both used by the server team to do capacity planning and to generate their roadmaps.

Existing functionality should remain unchanged and users of the HTML version of the WI tracker will now see 2 extra columns in the overview.

[lp:~jib/launchpad-work-items-tracker/complexity-points]

190. By Jos Boumans <jib@lucid-server> on 2010-07-02

* moin import needs to be aware of the notes column as well
* make sure we set the db version to 4 on new creation

191. By Jos Boumans <jib@lucid-server> on 2010-07-02

* fix typo

[Martin Pitt (pitti)]

Thanks Jos.

Some issues (in the interest of "teach a man to fish.." I'll mention them here):

 - Please use data_error() for real errors in the BP, not just dbg (in the except ValueError)
 - data_error() takes the spec URL as the first argument, you have to pass that down
 - Please follow PEP 8 formatting, i. e. f(x), not f( x)
 - as for the list.reverse() thing, that's expected. reverse() is applied inline, so you have to call it on a variable.

  - as for the default setting:

   + for i in range(len(list), len(defs)):
   + list.append(defs[i])

   How about something like

    list += defs[len(list):]

   ? (untested)

  - I don't quite understand the concept of "complexity". Why can't we just add a new column "complexity" to the spec table?

  - html-report defines a new PrettyPrinter pp, but does not use it anywhere

  - Why a new "notes" column? specifications can already set a "Status:" block in the whiteboard, which seems to be pretty much the same? I wouldn't like to have redundancy like that

Thanks!

review: Needs Fixing

[lp:~jib/launchpad-work-items-tracker/complexity-points]

192. By Jos Boumans <jib@lucid-server> on 2010-07-02

* apply style notes as suggested by pitti

[Martin Pitt (pitti)]

Thanks, so r192 fixes (most of) the PEP-8, data_error(), and the PrettyPrinter thing.

That leaves the question about the "complexity" column, and why we need notes in the first place.

You might have answered that in a previous mail, but for some reason I don't have it, and apparently that answer didn't make it here for some reason. Can you please copy your answer here again? Sorry for the trouble.

[lp:~jib/launchpad-work-items-tracker/complexity-points]

193. By Jos Boumans <jib@lucid-server> on 2010-07-08

* rename 'notes' to 'roadmap_notes' and educate the parser to
  use 'Roadmap Notes:' instead of 'Notes:'
* Don't show notes in the html report

=== modified file 'collect'
--- collect	2010-06-18 14:30:21 +0000
+++ collect	2010-07-08 00:41:40 +0000
@@ -12,6 +12,7 @@
 import report_tools
 
 debug = False
+#debug = True
 
 # if this is None, data errors go to stderr; if this is a list of (spec_name,
 # msg) tuples, data errors get collected here and mailed out at the end
@@ -112,6 +113,7 @@
         'description': '<div class="top-portlet">\s*<p>(.*?)</p>',
         'status': '(?:<p>|^)[Ss]tatus:\s*<br />(.*?)</p>',
         'details_url': 'href="([^"]*)">Read the full specification</a>',
+        'roadmap_notes': '(?:<p>|^)[Rr]oadmap\s+[Nn]otes:\s*<br />(.*?)</p>',
     }
 
     data = {}
@@ -148,11 +150,11 @@
 
     dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (name, str(data)))
 
-    db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?)',
+    db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)',
             (name, url, data['priority'], data['implementation'],
             data['assignee'], data['status'], data['milestone'],
             data['definition'], data['drafter'], data['approver'],
-            data['details_url']))
+            data['details_url'], data['roadmap_notes']))
 
     return True
 
@@ -183,6 +185,52 @@
                 (key, value, bp_name))
     db.commit()
 
+def parse_complexity_item( lp, db, line, bp_name, bp_url, def_milestone, def_assignee):
+    
+    ### cargo culted from parse_blueprint_workitem
+    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
+            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
+            '<a href="/', '<a href="https://launchpad.net/').strip()
+            
+    ### remove special characters people tend to type            
+    line = re.sub('[^\w -.]', '', line)
+                
+    if not line:
+        return
+
+    dbg("\tParsing complexity line '%s'" % line)
+
+    num         = None
+    milestone   = None
+    assignee    = None
+
+    try:
+        ### line.split().reverse() throws a typeerror exception
+        list = line.split()
+        list.reverse()
+
+        ### we may not have any values in the list, so append our
+        ### default values in the right order so they can be mapped
+        ### if we *dont* have enough values, an exception occurs
+        defs = [ None, def_milestone, def_assignee ]
+
+        for i in range(len(list), len(defs)):
+            list.append(defs[i])
+            
+        (num, milestone, assignee) = list
+    
+        if not num:
+            data_error(bp_url, "No complexity points defined for %s" % line)
+
+        dbg("\tComplexity: %s MS: %s Who: %s" % (num, milestone, assignee))
+
+        cur = db.cursor()
+        cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))',
+                (assignee, num, milestone, bp_name))
+        db.commit()
+
+    except ValueError:
+        data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError))
 
 def parse_blueprint_workitem(line, default_assignee, milestone,
         blueprint_url, launchpad, allow_inprogress, result_list):
@@ -326,10 +374,12 @@
     bugnum_re = re.compile('<a href="https://bugs\..*launchpad\.net/bugs/([0-9]+)" class="sprite bug">')
     work_items_re = re.compile('(<p>|^)work items(.*)\s*:\s*<br />', re.I)
     meta_re = re.compile( '(<p>|^)Meta.*?:<br />', re.I )
+    complexity_re = re.compile( '(<p>|^)Complexity.*?:<br />', re.I )
 
     in_workitems_block = False
     in_linked_bugs_block = False
     in_meta_block = False
+    in_complexity_block = False
     found_linked_bugs = False
     work_items = []
     found_wb_workitems = False
@@ -347,7 +397,11 @@
 
     for l in contents.splitlines():
 
-        if not in_workitems_block and not in_linked_bugs_block and not in_meta_block:
+        if not in_workitems_block and not in_linked_bugs_block \
+            and not in_meta_block and not in_complexity_block:
+
+            #dbg( 'processing: %s' % l )
+            
             m = work_items_re.search(l)
             if m:
                 in_workitems_block = True
@@ -361,6 +415,8 @@
                 found_linked_bugs = True
             if meta_re.search(l):
                 in_meta_block = True
+            if complexity_re.search(l):
+                in_complexity_block = True                
             continue
 
         if in_workitems_block:
@@ -394,6 +450,16 @@
                 in_meta_block = False
                 continue
 
+        if in_complexity_block:
+            dbg("\tcomplexity block line (raw): '%s'" % (l.strip()))
+            
+            parse_complexity_item(lp, db, l, bp_name, bp_url, spec_milestone, spec_assignee)
+            
+            ### done with the meta information?
+            if '</p>' in l:
+                in_complexity_block = False
+                continue
+
         if found_wb_workitems and '</div>' in l:
             break
 
@@ -465,7 +531,7 @@
     lp_import_teams(lp, db, cfg)
 
     for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems():
-        dbg('lp_import(): downloading ' + bp)
+        dbg('lp_import(): downloading %s from %s' % (bp, url))
         contents = urllib.urlopen(url).read().decode('UTF-8')
         if lp_import_blueprint(db, bp, url, contents):
             lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release'))
@@ -598,7 +664,7 @@
                 db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))',
                         (desc, name, state, assignee, milestone))
 
-            db.cursor().execute("INSERT INTO specs VALUES (?, ?, 'Medium', 'Unknown', NULL, NULL, ?, NULL, NULL, NULL, NULL, NULL)",
+            db.cursor().execute("INSERT INTO specs VALUES (?, ?, 'Medium', 'Unknown', NULL, NULL, ?, NULL, NULL, NULL, NULL, NULL, NULL)",
                     (name, spec_url, status))
 
 ########################################################################
@@ -621,7 +687,7 @@
         cur.execute('''CREATE TABLE version (
             db_layout_ref INT NOT NULL
             )''')
-        cur.execute('''INSERT INTO version VALUES (3)''')
+        cur.execute('''INSERT INTO version VALUES (4)''')
 
         cur.execute('''CREATE TABLE specs (
             name VARCHAR(255) PRIMARY KEY,
@@ -635,7 +701,8 @@
             definition CHAR(30),
             drafter CHAR(50),
             approver CHAR(50),
-            details_url VARCHAR(1000)
+            details_url VARCHAR(1000),
+            roadmap_notes VARCHAR(5000)
             )''')
 
         cur.execute('''CREATE TABLE work_items (
@@ -663,6 +730,14 @@
             spec VARCHAR(255) REFERENCES specs(name),
             date TIMESTAMP NOT NULL
             )''')
+            
+        cur.execute('''CREATE TABLE complexity (
+            assignee VARCHAR(255),
+            points VARCHAR(255),
+            milestone CHAR(50) REFERENCES milestones(name),
+            spec VARCHAR(255) REFERENCES specs(name),
+            date TIMESTAMP NOT NULL
+            )''')            
 
         db.commit()
     else:
@@ -694,6 +769,21 @@
             db.commit()
             dbg('DB upgrade finished')
             ver = 3
+        
+        if ver == 3:
+            dbg('Upgrading DB to layout version 4')
+            cur.execute('''CREATE TABLE complexity (
+                assignee VARCHAR(255),
+                points VARCHAR(255),
+                milestone CHAR(50) REFERENCES milestones(name),
+                spec VARCHAR(255) REFERENCES specs(name),
+                date TIMESTAMP NOT NULL
+                )''')     
+            cur.execute('ALTER TABLE specs ADD COLUMN roadmap_notes VARCHAR(5000)')
+            cur.execute('UPDATE version SET db_layout_ref = 4')
+            db.commit()
+            dbg('DB upgrade finished')
+            ver = 4
 
 
     return db
@@ -791,6 +881,7 @@
     cur.execute('DELETE FROM work_items WHERE date = date(CURRENT_TIMESTAMP)')
     cur.execute('DELETE FROM specs')
     cur.execute('DELETE FROM meta')
+    cur.execute('DELETE FROM complexity')    
     if opts.refresh:
         cur.execute('DELETE FROM milestones')
         cur.execute('DELETE FROM teams')

=== modified file 'html-report'
--- html-report	2010-05-18 08:31:00 +0000
+++ html-report	2010-07-08 00:41:40 +0000
@@ -7,6 +7,10 @@
 from report_tools import escape_url, escape_html
 import report_tools
 
+
+# import pprint
+# pp = pprint.PrettyPrinter(indent=4)
+
 def format_priority(priority):
     prio_colors = {
             'Undefined': 'gray',
@@ -21,13 +25,13 @@
     if col:
         return '<span style="color: %s">%s</span>' % (col, p)
     return p
-
+    
 def spec_completion(db, team, milestone):
     print '''<h2>Status by specification</h2>
 <p>(Click header to sort)</p>
 <table id="byspecification">
  <thead>
-  <tr><th>Specification</th> <th>todo</th><th>postponed</th><th>done</th> <th>Completion</th> <th>Priority</th> <th>Status</th></tr>
+  <tr><th>Specification</th><th>Complexity</th> <th>todo</th><th>postponed</th><th>done</th> <th>Completion</th> <th>Priority</th> <th>Status</th></tr>
  </thead>
 '''
     data = report_tools.blueprint_completion(db, team, milestone)
@@ -40,9 +44,10 @@
     completion.sort(key=lambda k: k[1]*100+report_tools.priority_value(data[k[0]]['priority']), reverse=True)
 
     for (bp, percent) in completion:
-        print '  <tr><td>%s</td> <td>%i</td><td>%i</td><td>%i</td> <td>%i%%<br/>' \
+        print '  <tr><td>%s</td> <td>%s</td><td>%i</td><td>%i</td><td>%i</td> <td>%i%%<br/>' \
             '<span style="font-size: 70%%">%s</span></td> <td>%s</td> <td>%s</td></tr>' % (
             '<a href="%s">%s</a>' % (data[bp]['url'], escape_html(bp)),
+                data[bp]['complexity'],
                 data[bp]['todo'], data[bp]['postponed'], data[bp]['done'],
                 percent,
                 data[bp]['implementation'],
@@ -57,7 +62,7 @@
 <p>(Click header to sort)</p>
 <table id="byassignee">
  <thead>
-  <tr><th>Assignee</th> <th>todo</th><th>postponed</th><th>done</th>  <th>Completion</th></tr>
+  <tr><th>Assignee</th> <th>Complexity</th><th>todo</th><th>postponed</th><th>done</th>  <th>Completion</th></tr>
  </thead>
 '''
     data = report_tools.assignee_completion(db, team, milestone)
@@ -73,8 +78,8 @@
     for (a, percent) in completion:
         a_html = escape_url(a or team or 'nobody')
         url = '%s/~%s/+specs?role=assignee' % (report_tools.blueprints_base_url, a_html)
-        print '  <tr><td><a href="%s">%s</a></td> <td>%i</td><td>%i</td><td>%i</td> <td>%i%%</td></tr>' % (
-                url, a_html, len(data[a]['todo']), len(data[a]['postponed']),
+        print '  <tr><td><a href="%s">%s</a></td> <td>%s</td><td>%i</td><td>%i</td><td>%i</td> <td>%i%%</td></tr>' % (
+                url, a_html, data[a]['complexity'], len(data[a]['todo']), len(data[a]['postponed']),
                 len(data[a]['done']), percent)
     print '</table>'
 

=== modified file 'report_tools.py'
--- report_tools.py	2010-04-06 13:48:48 +0000
+++ report_tools.py	2010-07-08 00:41:40 +0000
@@ -139,7 +139,7 @@
         if team:
             # include both direct team assignments, as well as assignmnets to
             # members of that team
-            cur.execute('SELECT s.name, COUNT(DISTINCT w.description), s.status, s.priority, s.implementation, s.url '
+            cur.execute('SELECT s.name, COUNT(DISTINCT w.description), s.status, s.priority, s.implementation, s.url, s.roadmap_notes '
                     'FROM work_items w, specs s ON w.spec = s.name '
                     '   LEFT JOIN teams ON (s.assignee = teams.name OR w.assignee = teams.name)'
                     'WHERE w.status = ? AND w.date = ? AND'
@@ -147,20 +147,26 @@
                     'GROUP BY w.spec' % ms_sql,
                     (s, last_date, team, team, team))
         else:
-            cur.execute('SELECT s.name, COUNT(DISTINCT w.description), s.status, s.priority, s.implementation, s.url '
+            cur.execute('SELECT s.name, COUNT(DISTINCT w.description), s.status, s.priority, s.implementation, s.url, s.roadmap_notes '
                     'FROM work_items w, specs s '
                     'ON w.spec = s.name '
                     'WHERE w.status = ? AND w.date = ? %s '
                     'GROUP BY w.spec' % ms_sql,
                     (s, last_date))
 
-        for (bp, num, status, priority, impl, url) in cur:
+        for (bp, num, status, priority, impl, url, roadmap_notes) in cur:
             info = data.setdefault(bp, {'todo': 0, 'done': 0, 'postponed': 0})
             info[s] = num
             info['status'] = status or ''
             info['priority'] = priority
             info['implementation'] = impl
             info['url'] = url
+            info['roadmap_notes'] = roadmap_notes
+            
+            cur2 = db.cursor();
+            cur2.execute('SELECT SUM(points) FROM complexity as w WHERE spec = ? AND date = ? %s' % ms_sql, (bp, last_date))
+            for points in cur2:
+                info['complexity'] = points[0] 
 
     return data
 
@@ -258,6 +264,11 @@
             info = data.setdefault(a, {'todo': [], 'done': [], 'postponed': []})
             info[s].append([bp, description, priority, url])
 
+            cur2 = db.cursor();
+            cur2.execute('SELECT SUM(points) FROM complexity as w WHERE assignee = ? AND date = ? %s' % ms_sql, (a, last_date))
+            for points in cur2:
+                info['complexity'] = points[0]
+
     return data
 
 def spec_information( db, config, team=None, milestone=None ):