Merge lp:~mabac/launchpad-work-items-tracker/linaro-roadmap-tracker into lp:~linaro-automation/launchpad-work-items-tracker/linaro

[Mattias Backman]

Hi,

This branch adds harvesting of the Linaro roadmap from KanbanTool. Everything is still very rough.

The "lanes" (2011Q3, 2011Q4, ... 2013) and "statuses" (Ready, Planning) are represented as workflow_stages in the data from KanbanTool and each workflow_stage has a reference to it's parent workflow_stage. We start by putting all the workflow_stages in a tree to be able to separate lanes from statuses.

The top workflow_stage seems to have no useful data and is discarded. The first level of workflow_stages below the root are the lanes which represent the quarterly planning and we create a Lane db item for each of the lanes found. The next level of workflow_stages represent the sub-statuses of each lane and we only use these to be able to map Kanban cards to the corresponding lane.

Each KanbanTool Task represents a Linaro roadmap card. A card is responsible for tracking it's own status which can be "Ready" or "Planning" according to the Lane information, or something like "Blocked" or "In progress" which will be determined by the linked blueprints.

We need to decide if it's ok to assume that interesting cards always will be two levels deep (lane + status) or if we will also display cards from for instance the "Maybe" lane which does not have sub workflow_stages.

To try it out you would do:
./collect_roadmap --database roadmapdb --config myconfig --board 9977 --token ABCDABCDABCD

where 9977 is the id of the sandbox Kanban board. The token need to be generated by an account with access to the board.

To generate the html page you would do:
./html-report -d roadmapdb --report-type roadmap > test.html

Thanks,

Mattias

[James Westby (james-w)]

Hi,

Thanks for this, great to see it moving forward. Overall it looks good.

52 +def unicode_or_None(attr):
53 + if attr is None:
54 + return attr
55 + if isinstance(attr, unicode):
56 + return attr
57 + return attr.decode("utf-8")

It would be great to not duplicate this further (apologies for the duplication
that is currently there.)

If it's used could you put it in a module or something?

39 +def data_error(spec_url, msg, warning=False):

Deleting these sorts of unused things would be good. It would be better
to do them in a better fashion if they are needed in this script.

451 +def lanes(db):
452 + cur = db.cursor()
453 + cur.execute('SELECT name, lane_id from lane')
454 + return [(i[0], i[1]) for i in cur]

I'd really like to re-use the ORM classes here, rather than hand-writing
SQL. Perhaps we should be doing the work now to enable that?

The issue is that direct SQL access via the sqlite API interferes with
Storm's caching. So we would need to change all the sql to do through
Storm's SQL interface.

It's a lot of code to change, but not difficult.

Perhaps we could even add some sort of compatibility layer so that less
code has to change?

It doesn't have to stop this code from landing, but maybe it should
be the next thing, as it will make later work easier.

Also, do we want to add the code to actually generate the roadmap.html
on status.linaro.org?

Thanks,

James

review: Approve

[Mattias Backman (mabac)]

On Wed, Sep 7, 2011 at 11:09 PM, James Westby <email address hidden> wrote:
> Review: Approve
> Hi,
>
> Thanks for this, great to see it moving forward. Overall it looks good.
>
> 52      +def unicode_or_None(attr):
> 53      + if attr is None:
> 54      + return attr
> 55      + if isinstance(attr, unicode):
> 56      + return attr
> 57      + return attr.decode("utf-8")
>
> It would be great to not duplicate this further (apologies for the duplication
> that is currently there.)
>
> If it's used could you put it in a module or something?
>
> 39      +def data_error(spec_url, msg, warning=False):
>
> Deleting these sorts of unused things would be good. It would be better
> to do them in a better fashion if they are needed in this script.

Sure, I'll fix the above cleanups.

>
> 451     +def lanes(db):
> 452     + cur = db.cursor()
> 453     + cur.execute('SELECT name, lane_id from lane')
> 454     + return [(i[0], i[1]) for i in cur]
>
> I'd really like to re-use the ORM classes here, rather than hand-writing
> SQL. Perhaps we should be doing the work now to enable that?

That would be good. I spent a lot of time trying to find examples of
that, but now I understand why I didn't find it.

>
> The issue is that direct SQL access via the sqlite API interferes with
> Storm's caching. So we would need to change all the sql to do through
> Storm's SQL interface.
>
> It's a lot of code to change, but not difficult.
>
> Perhaps we could even add some sort of compatibility layer so that less
> code has to change?

If we need all explicit sql queries to go away before we can use the
ORM classes, then that might be a good idea. I've looked over the sql
queries and there are a lot of them. On the other hand, if we get rid
of all the hand-written sql in one go, then we won't be stuck
maintaining the left-overs. I'll try to get an idea of how much work
it would be.

>
> It doesn't have to stop this code from landing, but maybe it should
> be the next thing, as it will make later work easier.
>
> Also, do we want to add the code to actually generate the roadmap.html
> on status.linaro.org?

Right, I think I've found how to do that now.

>
> Thanks,
>
> James
>
> --
> https://code.launchpad.net/~mabac/launchpad-work-items-tracker/linaro-roadmap-tracker/+merge/74441
> You are the owner of lp:~mabac/launchpad-work-items-tracker/linaro-roadmap-tracker.
>

[James Westby (james-w)]

On Thu, 08 Sep 2011 10:04:11 -0000, Mattias Backman <email address hidden> wrote:
> On Wed, Sep 7, 2011 at 11:09 PM, James Westby <email address hidden> wrote:
> If we need all explicit sql queries to go away before we can use the
> ORM classes, then that might be a good idea. I've looked over the sql
> queries and there are a lot of them. On the other hand, if we get rid
> of all the hand-written sql in one go, then we won't be stuck
> maintaining the left-overs. I'll try to get an idea of how much work
> it would be.

Nope, we just need to make the queries go through Storm's Store object.

store.execute(SQL) rather that cur = db.cursor(); cur.execute(SQL)

There is also a different in how to iterate over result sets, with storm
it is

  result = store.execute(SQL)
  for row in result:

with sqlite directly it is

  cur.execute(SQL)
  for row in cur:

So either making these translations, and passing around a Store object
rather than the sqlite db is what is needed.

Apologies for not being clear before. Getting rid of all the hand
written SQL would be good, but not a requirement to use the ORM for new
code.

Thanks,

James

[Mattias Backman (mabac)]

> store.execute(SQL) rather that cur = db.cursor(); cur.execute(SQL)
>
> There is also a different in how to iterate over result sets, with storm
> it is
>
>  result = store.execute(SQL)
>  for row in result:
>
> with sqlite directly it is
>
>  cur.execute(SQL)
>  for row in cur:

That sounds manageable. I'll have a go at it.

[Mattias Backman (mabac)]

Oops. This broke status.linaro.org when I merged it. It's reverted now so the next update should fix the pages, I hope.

I got a lot of these:

Traceback (most recent call last):
  File "./html-report", line 551, in <module>
    method(store, opts)
  File "./html-report", line 356, in burndown
    data = self.template_data(db, opts)
  File "./html-report", line 329, in template_data
    data.update(report_tools.days_left(db, milestone=opts.milestone))
  File "/srv/status.linaro.org/work-items-tracker/report_tools.py", line 657, in days_left
    due_date_str = milestone_due_date(store, milestone=milestone)
  File "/srv/status.linaro.org/work-items-tracker/report_tools.py", line 593, in milestone_due_date
    end_date = result.get_one()[0]
TypeError: 'NoneType' object is unsubscriptable

and this wasn't a good idea either:
        for d in result.description:
            spec_dict[ d[0] ] = spec_row[ d[0] ]
perhaps this should be result.values('description') or something similar.

review: Needs Fixing

[Mattias Backman (mabac)]

> Oops. This broke status.linaro.org when I merged it. It's reverted now so the
> next update should fix the pages, I hope.
>
> I got a lot of these:
>
> Traceback (most recent call last):
> File "./html-report", line 551, in <module>
> method(store, opts)
> File "./html-report", line 356, in burndown
> data = self.template_data(db, opts)
> File "./html-report", line 329, in template_data
> data.update(report_tools.days_left(db, milestone=opts.milestone))
> File "/srv/status.linaro.org/work-items-tracker/report_tools.py", line 657,
> in days_left
> due_date_str = milestone_due_date(store, milestone=milestone)
> File "/srv/status.linaro.org/work-items-tracker/report_tools.py", line 593,
> in milestone_due_date
> end_date = result.get_one()[0]
> TypeError: 'NoneType' object is unsubscriptable

Strange. After changing to store.execute() I have to change
        result = store.execute('SELECT due_date FROM milestones WHERE name = ?', (milestone,))
to
        result = store.execute('SELECT due_date FROM milestones WHERE name = ?', (unicode(milestone), ))

I wonder why this worked with cur.execute().

[Mattias Backman (mabac)]

This didn't break anything since I didn't try to merge it yet. I commented on the wrong mp.

review: Abstain

[lp:~mabac/launchpad-work-items-tracker/linaro-roadmap-tracker]

298. By Mattias Backman on 2011-09-09

Merge change to use Storm for sql queries.

299. By Mattias Backman on 2011-09-12

Add code for generating pages.

300. By Mattias Backman on 2011-09-12

Clean up and move unicode_or_None to utils.py.

301. By Mattias Backman on 2011-09-12

Remove send_error_mails. Not sure if I should use it.

[Mattias Backman (mabac)]

Please ignore my latest comments on this mp, they where intended for the storm changes in another branch.

This should now create a roadmap.html and one roadmap-<lane>.html for each lane.

[James Westby (james-w)]

246 +for lane in report_tools.lanes(store):

I don't think we need to query again there, we can just use the "lanes"
variable that was used in the previous call.

This looks good, assuming that it will cope with there being no
lanes in the db for a while. (The pages aren't linked, so it's fine
if they are empty or missing or whatever, but it can't crash if there
are no lanes.)

Thanks,

James

review: Approve

[Mattias Backman (mabac)]

On Mon, Sep 12, 2011 at 9:29 PM, James Westby <email address hidden> wrote:
> Review: Approve
>
> 246     +for lane in report_tools.lanes(store):
>
> I don't think we need to query again there, we can just use the "lanes"
> variable that was used in the previous call.

Fixed. It was the first call that could go away.

>
> This looks good, assuming that it will cope with there being no
> lanes in the db for a while. (The pages aren't linked, so it's fine
> if they are empty or missing or whatever, but it can't crash if there
> are no lanes.)

I think it'll be fine, as long as the new tables are in the db. If the
Lanes table is empty, the roadmap.html will contain only the heading
and the other pages will not be created.

[James Westby (james-w)]

On Tue, 13 Sep 2011 07:13:23 -0000, Mattias Backman <email address hidden> wrote:
> On Mon, Sep 12, 2011 at 9:29 PM, James Westby <email address hidden> wrote:
> > Review: Approve
> >
> > 246     +for lane in report_tools.lanes(store):
> >
> > I don't think we need to query again there, we can just use the "lanes"
> > variable that was used in the previous call.
>
> Fixed. It was the first call that could go away.
>
> >
> > This looks good, assuming that it will cope with there being no
> > lanes in the db for a while. (The pages aren't linked, so it's fine
> > if they are empty or missing or whatever, but it can't crash if there
> > are no lanes.)
>
> I think it'll be fine, as long as the new tables are in the db. If the
> Lanes table is empty, the roadmap.html will contain only the heading
> and the other pages will not be created.

Yep, you can assume that the tables are present given that you coded the
migration, so this should be good to land whenever you like.

Thanks,

James

[lp:~mabac/launchpad-work-items-tracker/linaro-roadmap-tracker]

302. By Mattias Backman on 2011-09-16

Move roadmap generation to happen after all the other pages are generated.

[Mattias Backman (mabac)]

I moved the roadmap generation to happen last so it won't break the rest of status.linaro.org, just in case.

The Storm changes have to land before this branch so we don't mix cursor.execute() with store.execute().

[James Westby (james-w)]

On Fri, 16 Sep 2011 09:58:27 -0000, Mattias Backman <email address hidden> wrote:
> I moved the roadmap generation to happen last so it won't break the rest of status.linaro.org, just in case.

That shouldn't be needed. The actual page is generated in a subprocess,
so the vast majority of the code won't crash the entire thing. Not a
problem to have this though for that small bit of code that runs the
subprocess.

> The Storm changes have to land before this branch so we don't mix cursor.execute() with store.execute().

Ok,

This is good to land whenever you are comfortable doing so.

Thanks,

James

=== modified file 'collect'
--- collect	2011-09-06 13:07:42 +0000
+++ collect	2011-09-16 09:57:23 +0000
@@ -23,6 +23,7 @@
     TeamParticipation,
     Workitem,
     )
+from utils import unicode_or_None
 import report_tools
 
 
@@ -59,14 +60,6 @@
     return link.decode("utf-8")
 
 
-def unicode_or_None(attr):
-    if attr is None:
-        return attr
-    if isinstance(attr, unicode):
-        return attr
-    return attr.decode("utf-8")
-
-
 import simplejson
 _orig_loads = simplejson.loads
 def loads(something):

=== added file 'collect_roadmap'
--- collect_roadmap	1970-01-01 00:00:00 +0000
+++ collect_roadmap	2011-09-16 09:57:23 +0000
@@ -0,0 +1,200 @@
+#!/usr/bin/python
+#
+# Pull items from the Linaro roadmap in Kanbantool and put them into a database.
+
+import urllib2, re, sys, optparse, smtplib, pwd, os
+import simplejson
+import logging
+from email.mime.text import MIMEText
+
+from lpworkitems.collect_roadmap import CollectorStore
+from lpworkitems.database import get_store
+from lpworkitems.error_collector import (
+    ErrorCollector,
+    StderrErrorCollector,
+    )
+from lpworkitems.models_roadmap import (
+    Lane,
+    Card,
+    )
+from utils import unicode_or_None
+import report_tools
+
+
+# An ErrorCollector to collect the data errors for later reporting
+error_collector = None
+
+
+logger = logging.getLogger("linaroroadmap")
+
+
+def dbg(msg):
+    '''Print out debugging message if debugging is enabled.'''
+    logger.debug(msg)
+
+
+def get_kanban_url(item_url, api_token):
+    base_url = 'https://linaro.kanbantool.com/api/v1'
+    return "%s/%s?api_token=%s" % (base_url, item_url, api_token)
+
+
+def get_kanban_data(url):
+    data = None
+    try:
+        data = simplejson.load(urllib2.urlopen(url))
+    except urllib2.HTTPError, e:
+        print "HTTP error: %d" % e.code
+    except urllib2.URLError, e:
+        print "Network error: %s" % e.reason.args[1]
+
+    return data
+
+
+def kanban_import_lanes(collector, workflow_stages):
+    nodes = {}
+    root_node_id = None
+
+    # Iterate over all workflow_stages which may be in any order.
+    for workflow_stage in workflow_stages:
+        parent_id = workflow_stage['parent_id']
+        if parent_id is None:
+            assert root_node_id is None, 'We have already found the root node.'
+            root_node_id = workflow_stage['id']
+        else:
+            if parent_id not in nodes:
+                nodes[parent_id] = []
+            # Add child workflow_stage
+            nodes[parent_id].append(workflow_stage)
+
+    statuses = []
+    for node in nodes[root_node_id]:
+        assert node['parent_id'] == root_node_id
+        model_lane = Lane(unicode_or_None(node['name']),
+                          node['id'])
+        collector.store_lane(model_lane)
+        node_id = node['id']
+        if node_id in nodes:
+            statuses.extend(nodes[node_id])
+    return statuses
+
+
+def kanban_import_cards(collector, tasks, status_list):
+
+    for task in tasks:
+        status_id = task['task']['workflow_stage_id']
+        assert status_id is not None
+        task_status = None
+        for status in status_list:
+            if status['id'] == status_id:
+                task_status = status
+                break
+        if task_status is not None:
+            lane_id = task_status['parent_id']
+            assert lane_id is not None
+            model_card = Card(unicode_or_None(task['task']['name']),
+                              task['task']['id'], lane_id)
+            model_card.status = unicode_or_None(task_status['name'])
+            collector.store_card(model_card)
+        else:
+            print "Task '%s' does not have a status." % task['task']['name']
+
+
+def kanban_import(collector, cfg, board_id, api_token):
+    '''Collect roadmap items from KanbanTool into DB.'''
+    board_url = get_kanban_url('boards/%s.json' % board_id, api_token)
+    board = get_kanban_data(board_url)
+    status_list = kanban_import_lanes(collector,
+                                      board['board']['workflow_stages'])
+
+    tasks_url = get_kanban_url('boards/%s/tasks.json' % board_id, api_token)
+    tasks = get_kanban_data(tasks_url)
+    kanban_import_cards(collector, tasks, status_list)
+
+
+########################################################################
+#
+# Program operations and main
+#
+########################################################################
+
+def parse_argv():
+    '''Parse CLI arguments.
+
+    Return (options, args) tuple.
+    '''
+    optparser = optparse.OptionParser()
+    optparser.add_option('-d', '--database',
+        help='Path to database', dest='database', metavar='PATH')
+    optparser.add_option('-c', '--config',
+        help='Path to configuration file', dest='config', metavar='PATH')
+    optparser.add_option('--debug', action='store_true', default=False,
+        help='Enable debugging output in parsing routines')
+    optparser.add_option('--mail', action='store_true', default=False,
+        help='Send data errors as email (according to "error_config" map in '
+            'config file) instead of printing to stderr', dest='mail')
+    optparser.add_option('--board',
+        help='Board id at Kanbantool', dest='board')
+    optparser.add_option('--token',
+        help='API token for authentication', dest='api_token')
+
+    (opts, args) = optparser.parse_args()
+
+    if not opts.database:
+        optparser.error('No database given')
+    if not opts.config:
+        optparser.error('No config given')
+
+    return opts, args
+
+
+def setup_logging(debug):
+    ch = logging.StreamHandler()
+    ch.setLevel(logging.INFO)
+    formatter = logging.Formatter("%(message)s")
+    ch.setFormatter(formatter)
+    logger.setLevel(logging.INFO)
+    logger.addHandler(ch)
+    if debug:
+        ch.setLevel(logging.DEBUG)
+        formatter = logging.Formatter(
+            "%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+        ch.setFormatter(formatter)
+        logger.setLevel(logging.DEBUG)
+
+
+def main():
+    report_tools.fix_stdouterr()
+
+    (opts, args) = parse_argv()
+
+    setup_logging(opts.debug)
+
+    global error_collector
+    if opts.mail:
+        error_collector = ErrorCollector()
+    else:
+        error_collector = StderrErrorCollector()
+
+    cfg = report_tools.load_config(opts.config)
+
+    lock_path = opts.database + ".collect_lock"
+    lock_f = open(lock_path, "wb")
+    if report_tools.lock_file(lock_f) is None:
+        print "Another instance is already running"
+        sys.exit(0)
+
+    store = get_store(opts.database)
+    collector = CollectorStore(store, '', error_collector)
+
+    collector.clear_lanes()
+    collector.clear_cards()
+
+    kanban_import(collector, cfg, opts.board, opts.api_token)
+
+    store.commit()
+
+    os.unlink(lock_path)
+
+
+if __name__ == '__main__':
+    main()

=== modified file 'generate-all'
--- generate-all	2011-09-12 19:06:34 +0000
+++ generate-all	2011-09-16 09:57:23 +0000
@@ -152,6 +152,15 @@
 basename = os.path.join(opts.output_dir, 'index')
 report_tools.status_overview(my_path, opts.database, basename, opts.config, root=opts.root)
 
+# roadmap overview
+basename = os.path.join(opts.output_dir, 'roadmap')
+report_tools.roadmap_overview(my_path, opts.database, basename, opts.config, root=opts.root)
+
+# roadmap lanes
+for lane in report_tools.lanes(store):
+    basename = os.path.join(opts.output_dir, 'roadmap-' + lane.name)
+    report_tools.roadmap_pages(my_path, opts.database, basename, opts.config, lane, root=opts.root)
+
 
 def copy_files(source_dir):
     for filename in os.listdir(source_dir):

=== modified file 'html-report'
--- html-report	2011-09-12 19:06:34 +0000
+++ html-report	2011-09-16 09:57:23 +0000
@@ -449,6 +449,42 @@
         print report_tools.fill_template(
             "workitem_list.html", data, theme=opts.theme)
 
+    def roadmap(self, store, opts):
+        if not opts.title:
+            title = 'Linaro Roadmap'
+        else:
+            title = opts.title
+
+        data = self.template_data(store, opts)
+        lanes = report_tools.lanes(store)
+            
+        data.update(dict(lanes=lanes))
+        data.update(dict(page_title=title))
+        data.update(dict(page_type="roadmap"))
+        print report_tools.fill_template(
+            "roadmap.html", data, theme=opts.theme)
+
+
+    def roadmap_page(self, store, opts):
+        if opts.lane is None:
+            print "<h1>Error, no lane specified.</h1>"
+        if not opts.title:
+            title = opts.lane
+        else:
+            title = opts.title
+
+        data = self.template_data(store, opts)
+        lane = report_tools.lane(store, opts.lane).one()
+        statuses = []
+        for status, cards in report_tools.statuses(store, lane):
+            statuses.append(dict(name=status, cards=cards))
+            
+        data.update(dict(statuses=statuses))
+        data.update(dict(page_type="roadmap_lane"))
+        data.update(dict(lane=title))
+        print report_tools.fill_template(
+            "roadmap_lane.html", data, theme=opts.theme)
+
 
 class WorkitemsOnDate(object):
 
@@ -527,6 +563,8 @@
             help="Include all milestones targetted to this date.")
     optparser.add_option('--theme', dest="theme",
             help="The theme to use.", default="linaro")
+    optparser.add_option('--lane',
+        help='Roadmap lane', dest='lane')
 
     (opts, args) = optparser.parse_args()
     if not opts.database:

=== added file 'lpworkitems/collect_roadmap.py'
--- lpworkitems/collect_roadmap.py	1970-01-01 00:00:00 +0000
+++ lpworkitems/collect_roadmap.py	2011-09-16 09:57:23 +0000
@@ -0,0 +1,24 @@
+from lpworkitems import models_roadmap
+
+
+class CollectorStore(object):
+
+    def __init__(self, store, base_url, error_collector):
+        self.store = store
+        self.base_url = base_url
+        self.error_collector = error_collector
+
+    def _clear_all(self, *find_args):
+        self.store.find(*find_args).remove()
+
+    def clear_lanes(self):
+        self._clear_all(models_roadmap.Lane)
+
+    def clear_cards(self):
+        self._clear_all(models_roadmap.Card)
+
+    def store_lane(self, lane):
+        self.store.add(lane)
+
+    def store_card(self, card):
+        self.store.add(card)

=== modified file 'lpworkitems/database.py'
--- lpworkitems/database.py	2011-06-24 19:09:26 +0000
+++ lpworkitems/database.py	2011-09-16 09:57:23 +0000
@@ -7,7 +7,7 @@
     store.execute('''CREATE TABLE version (
         db_layout_ref INT NOT NULL
         )''')
-    store.execute('''INSERT INTO version VALUES (10)''')
+    store.execute('''INSERT INTO version VALUES (11)''')
 
     store.execute('''CREATE TABLE specs (
         name VARCHAR(255) PRIMARY KEY,
@@ -90,6 +90,19 @@
         display_name VARCHAR(50)
         )''')
 
+    store.execute('''CREATE TABLE lane (
+        name VARCHAR(200) NOT NULL,
+        lane_id NOT NULL,
+        cards REFERENCES card(card_id)
+        )''')
+
+    store.execute('''CREATE TABLE card (
+        name VARCHAR(200) NOT NULL,
+        card_id NOT NULL,
+        status VARCHAR(50),
+        lane_id REFERENCES lane(lane_id)
+        )''')
+
 
 def upgrade_if_needed(store):
     # upgrade DB layout
@@ -177,6 +190,21 @@
             )''')
         store.execute('UPDATE version SET db_layout_ref = 10')
         ver = 10
+    if ver == 10:
+        store.execute('''CREATE TABLE lane (
+            name VARCHAR(200) NOT NULL,
+            lane_id NOT NULL,
+            cards REFERENCES card(card_id)
+            )''')
+
+        store.execute('''CREATE TABLE card (
+            name VARCHAR(200) NOT NULL,
+            card_id NOT NULL,
+            status VARCHAR(50),
+            lane_id REFERENCES lane(lane_id)
+            )''')
+        store.execute('UPDATE version SET db_layout_ref = 11')
+        ver = 11
 
 
 def get_store(dbpath):
@@ -205,5 +233,6 @@
     store.execute('''CREATE INDEX work_items_date_idx ON work_items (date)''')
     store.execute('''CREATE INDEX work_items_status_idx ON work_items (status)''')
 
+
 def create_v6_indexes(store):
     store.execute('''CREATE INDEX work_items_assignee_milestone_idx on work_items(assignee,milestone)''')

=== modified file 'lpworkitems/models.py'
--- lpworkitems/models.py	2011-03-24 15:37:28 +0000
+++ lpworkitems/models.py	2011-09-16 09:57:23 +0000
@@ -1,17 +1,10 @@
 import datetime
 import re
+from utils import unicode_or_None
 
 from storm.locals import Date, Reference, ReferenceSet, Unicode
 
 
-def unicode_or_None(attr):
-    if attr is None:
-        return attr
-    if isinstance(attr, unicode):
-        return attr
-    return attr.decode("utf-8")
-
-
 def fill_blueprint_info_from_launchpad(model_bp, lp_bp):
     model_bp.name = unicode_or_None(lp_bp.name)
     model_bp.url = unicode_or_None(lp_bp.web_link)

=== added file 'lpworkitems/models_roadmap.py'
--- lpworkitems/models_roadmap.py	1970-01-01 00:00:00 +0000
+++ lpworkitems/models_roadmap.py	2011-09-16 09:57:23 +0000
@@ -0,0 +1,33 @@
+import datetime
+import re
+from utils import unicode_or_None
+
+from storm.locals import Date, Reference, ReferenceSet, Unicode, Int
+
+
+class Card(object):
+
+    __storm_table__ = "card"
+
+    name = Unicode()
+    status = Unicode()
+    card_id = Int(primary=True)
+    lane_id = Int()
+
+    def __init__(self, name, card_id, lane_id):
+        self.lane_id = lane_id
+        self.card_id = card_id
+        self.name = name
+
+
+class Lane(object):
+
+    __storm_table__ = "lane"
+
+    name = Unicode()
+    lane_id = Int(primary=True)
+    cards = ReferenceSet(lane_id, Card.lane_id)
+
+    def __init__(self, name, lane_id):
+        self.lane_id = lane_id
+        self.name = name

=== modified file 'report_tools.py'
--- report_tools.py	2011-09-12 19:06:34 +0000
+++ report_tools.py	2011-09-16 09:57:23 +0000
@@ -9,6 +9,10 @@
 from cgi import escape
 import errno
 import fcntl
+from lpworkitems.models_roadmap import (
+    Lane,
+    Card,
+)
 
 valid_states = [u'inprogress', u'blocked', u'todo', u'done', u'postponed']
 state_labels = [u'In Progress', u'Blocked', u'Todo', u'Done', u'Postponed']
@@ -180,6 +184,39 @@
         fh.close()
 
 
+def roadmap_overview(my_path, database, basename, config, root=None):
+    cfg = load_config(config)
+    fh = open(basename + '.html', 'w')
+    try:
+        args = [os.path.join(my_path, 'html-report'), '-d', database]
+        args += ['--report-type', 'roadmap']
+        if root:
+            args += ['--root', root]
+        report_args(args, theme=get_theme(cfg))
+        proc = Popen(args, stdout=fh)
+        print basename + '.html'
+        proc.wait()
+    finally:
+        fh.close()
+
+
+def roadmap_pages(my_path, database, basename, config, lane, root=None):
+    cfg = load_config(config)
+    fh = open(basename + '.html', 'w')
+    try:
+        args = [os.path.join(my_path, 'html-report'), '-d', database]
+        args += ['--report-type', 'roadmap_page']
+        args += ['--lane', lane.name]
+        if root:
+            args += ['--root', root]
+        report_args(args, theme=get_theme(cfg))
+        proc = Popen(args, stdout=fh)
+        print basename + '.html'
+        proc.wait()
+    finally:
+        fh.close()
+
+
 def run_reports(my_path, database, basename, config, milestone=None, team=None,
         user=None, trend_starts=None, trend_override=None, burnup=False, root=None, date=None):
 
@@ -886,6 +923,7 @@
     return rv
 
 
+<<<<<<< TREE
 def subteams(db, team):
     cur = db.cursor()
     cur.execute('SELECT name from team_structure where team = ?', (team,))
@@ -899,6 +937,41 @@
 
 
 def milestone_list(db):
+=======
+def lanes(store):
+    return store.find(Lane)
+
+
+def lane(store, lane):
+    return store.find(Lane, Lane.name == unicode(lane))
+
+
+def lane_cards(store, lane):
+    return store.find(Card, Card.lane_id == lane.lane_id)
+
+
+def statuses(store, lane):
+    result = []
+    for status in store.find(Card.status,
+                             Card.lane_id == lane.lane_id).config(distinct=True):
+        result.append((status, store.find(Card,
+                                          Card.lane_id == lane.lane_id,
+                                          Card.status == status)))
+    return result
+
+
+def subteams(store, team):
+    result = store.execute('SELECT name from team_structure where team = ?', (team,))
+    return [i[0] for i in result]
+
+
+def all_teams(store):
+    result = store.execute('SELECT DISTINCT team from teams where team is not NULL')
+    return [i[0] for i in result]
+
+
+def milestone_list(store):
+>>>>>>> MERGE-SOURCE
         milestones = {}
 
         cur = db.cursor()

=== added file 'utils.py'
--- utils.py	1970-01-01 00:00:00 +0000
+++ utils.py	2011-09-16 09:57:23 +0000
@@ -0,0 +1,7 @@
+def unicode_or_None(attr):
+    if attr is None:
+        return attr
+    if isinstance(attr, unicode):
+        return attr
+    return attr.decode("utf-8")
+