Merge lp:~james-w/launchpad-work-items-tracker/blueprints-api into lp:launchpad-work-items-tracker

[James Westby]

Hi,

This makes use of the newly exposed blueprints API on Launchpad, reducing
the amount of screen scraping, round trips, and data transferred enormously.

The only regression that I think this will cause is that when someone writes
"bug 12345" in the whiteboard, it will no longer be a link to that bug
when viewed on the workitems page, but I think we can live with that for
now. If you disagree let me know and I will code it.

Apologies that the diff is a little large and not particularly clear, but
I couldn't think of a good way to migrate a bit at a time.

The change is now rolled out on the Launchpad production instance, so we
can make use of this.

Note that you may still get AttributeError: ... 'bugs' due to caching of the WADL.
If you delete ~/.launchpadlib/api.launchpad.net/cache/api.launchpad.net,devel,-application,vnd.sun.*
and retry it should work.

Thanks,

James

[Clint Byrum (clint-fewbar)]

Upon running the code as-is, I received a failure rather quickly:

clint@clint-MacBookPro:~/src/wi/bzr/trunk$ ./collect -d ../natty.db -c ../natty.cfg --debug
lp_import_milestones(): milestone table already filled
lp_import_teams(): teams table already filled
lp_import(): downloading cloud-server-n-automated-testing from https://api.launchpad.net/devel/ubuntu/+spec/cloud-server-n-automated-testing
lp_import_blueprint(cloud-server-n-automated-testing): finished parsing; data: {'status': 'Session for discussion about automated testing of Ubuntu Server; automated ISO testing was implemented for Maverick using libvirt/kvm + Hudson (see http://launchpad.net/ubuntu-server-iso-testing). This approach could be applied in other variants and for other aspects of server testing', 'definition': 'Approved', 'implementation': 'Started', 'milestone': 'natty-alpha-3', 'approver': 'robbie.w', 'details_url': None, 'priority': 'Essential', 'assignee': 'james-page', 'roadmap_notes': None, 'drafter': 'james-page'}
lp_import_blueprint_workitems(): processing cloud-server-n-automated-testing (spec milestone: natty-alpha-3, spec assignee: james-page, spec implementation: Started)
lp_import_blueprint_workitems(): starting work items block at Work items for natty-alpha-2:
  ... setting milestone to natty-alpha-2
 workitem (raw): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
 workitem (clean): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
 workitem (raw): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
 workitem (clean): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
 workitem (raw): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
 workitem (clean): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
 workitem (raw): '[james-page] Package ubuntu-server-iso-testing and locate in PPA: INPROGRESS'
 workitem (clean): '[james-page] Package ubuntu-server-iso-testing and locate in PPA: INPROGRESS'
 workitem (raw): ''
lp_import_blueprint_workitems(): closing work items block with line:
lp_import_blueprint_workitems(): starting work items block at Work items for natty-alpha-3:
  ... setting milestone to natty-alpha-3
 workitem (raw): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
 workitem (clean): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
 workitem (raw): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
 workitem (clean): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
 workitem (raw): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
 workitem (clean): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
https://launchpad.net/ubuntu/+spec/cloud-server-n-automated-testing
  [WARNING] assignee "cr2" is not a valid Launchpad account
 workitem (raw): '[cr2] Checkbox plugin to download tests from couchdb to ex...

review: Resubmit

[Jamie Bennett (jamiebennett)]

On testing the code I was prompted to log in using open id (from the command-line). As this was tested via ssh to a development box 'Links' was launched as the web browser to gather my open id information. From there you cannot log in (Launchpad/Links issue). This did not happen on previous versions.

review: Needs Information

[James Westby (james-w)]

The change is now rolled out on the Launchpad production instance, so we
can make use of this.

Note that you may still get AttributeError: ... 'bugs' due to caching of the WADL.
If you delete ~/.launchpadlib/api.launchpad.net/cache/api.launchpad.net,devel,-application,vnd.sun.*
and retry it should work.

[James Westby (james-w)]

Jamie, I don't see how that could have been caused by this change, as
the API was still used for some things in the old code. I suspect it is
just coincidence that you are seeing this for the first time now.

Thanks,

James

[James Westby (james-w)]

Jamie, I've worked out why you saw that behaviour now, it's because
I changed from "edge" to "production" as edge is going away, so this
will indeed affect all users.

The other problem is that people.canonical.com doesn't have a new enough
version of launchpadlib to know about the "version" argument to
Launchpad.login_with, which is required here as blueprints are only exposed
on "devel".

Thanks,

James

[lp:~james-w/launchpad-work-items-tracker/blueprints-api]

253. By Martin Pitt on 2010-12-10

revert extra-projects merge for now; this is creating a conceptual conflict on milestones, and should rather be handled in separate configurations for linaro

254. By Martin Pitt on 2010-12-10

consider fixed bugs in bug list based reports as well

255. By Martin Pitt on 2010-12-17

add "Unknown" bug state

256. By Martin Pitt on 2011-01-03

drop Unknown bug state, does not exist any more

257. By Martin Pitt on 2011-01-03

better fix for "Unknown" task status

258. By Martin Pitt on 2011-01-03

finally fix unknown bug states

[Martin Pitt (pitti)]

lillypilly (aka people.u.c.) got upgraded to lucid over the holidays, so we have a recent enough launchpadlib available now.

This looks a lot better than the screenscraping, thanks for porting this! Surprising that they added API for blueprints, I thought they were doomed to go away entirely.

review: Approve

[Martin Pitt (pitti)]

This breaks the --pattern option, which is quite handy for debugging. I'll fix this when merging.

[Martin Pitt (pitti)]

Unfortunately this exposes another bug in launchpadlib for building the cache file names:

IOError: [Errno 36] File name too long: '/home/martin/.launchpadlib/api.launchpad.net/cache/api.launchpad.net,devel,ubuntu,+spec,packageselection-desktop-n-bringing-desktop-and-netbook-image-closer,bugs-applic,89e26dc98142f3393a4550f041f7b150'

In order to not lose my modifications, I pushed the merge (and fix) to lp:~work-items-tracker-hackers/launchpad-work-items-tracker/blueprints-api

[lp:~james-w/launchpad-work-items-tracker/blueprints-api]

259. By Martin Pitt on 2011-01-11

html-report: Add textual stats over time

260. By Martin Pitt on 2011-01-11

html-report: sort dates

261. By Martin Pitt on 2011-01-11

fix time-stats todo column

262. By Martin Pitt on 2011-01-11

time-stats: not all states exist always

263. By Martin Pitt on 2011-01-29

collect: add "Expired" bug state

264. By James Westby on 2011-02-07

Merge blueprints API branch.

[Martin Pitt (pitti)]

This is all merged into lp:~work-items-tracker-hackers/launchpad-work-items-tracker/blueprints-api now. That branch collects up all of your recent work, many thanks! It works fine locally, and it seems the "too long file name" problem was just transient. Please see https://code.launchpad.net/~work-items-tracker-hackers/launchpad-work-items-tracker/blueprints-api/+merge/49047 for the final merge request into trunk, which is currently blocked by an RT.

review: Approve

=== modified file 'collect'
--- collect	2011-01-29 09:08:54 +0000
+++ collect	2011-02-07 20:14:16 +0000
@@ -3,11 +3,11 @@
 # Pull work items from various sources (blueprint whiteboards, linked blueprint
 # bugs, wiki pages) and put them into a database.
 
-import urllib, re, sys, optparse, smtplib, pwd, os
+import urllib, re, sys, optparse, smtplib, pwd, os, urlparse
 from email.mime.text import MIMEText
 import sqlite3 as dbapi2
 
-from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
+from launchpadlib.launchpad import Launchpad
 
 import report_tools
 
@@ -59,51 +59,20 @@
         dbg('Queueing data error: ' + s)
 
 
+def web_link(item):
+    """Get a link to the Launchpad API object on the website."""
+    api_link = item.self_link
+    parts = urlparse.urlparse(api_link)
+    return parts.scheme + "://" + parts.netloc.replace("api.", "") + "/" + parts.path.split("/", 2)[2]
+
+
 ########################################################################
 #
 # Functions for parsing Launchpad data
 #
 ########################################################################
 
-def lp_blueprints_from_list(url, name_pattern = None):
-    '''Return blueprints from a LP list view.
-
-    This can optionally specify a name pattern (which is mostly useful for
-    testing and debugging, to only parse a small subset of blueprints)
-
-    Return a dictionary name -> (url, milestone).
-    '''
-    blueprint_name_filter = re.compile('href="(/[a-zA-Z0-9-]+/\+spec/%s[^"]+)"(?!.*class.*sprite)' %
-            (name_pattern or '.'))
-    milestone_re = re.compile('href=".*/[a-zA-Z0-9-]+/\+milestone/([^"]+)"')
-
-    result = {}
-    scan_tr_end = False
-    milestone = None
-    bp = None
-    dbg("lp_blueprints_from_list(): Loading '%s'" % url)
-    for l in urllib.urlopen(url):
-        if scan_tr_end:
-            m = milestone_re.search(l)
-            if m:
-                milestone = m.group(1)
-                dbg('lp_blueprints_from_list(): current spec milestone: ' + milestone)
-            if '</tr>' in l:
-                scan_tr_end = False
-                if bp:
-                    result[bp.split('/')[-1]] = (bp, milestone)
-                bp = None
-                milestone = None
-        else:
-            m = blueprint_name_filter.search(l)
-            if m:
-                bp = report_tools.blueprints_base_url + m.group(1)
-                dbg('lp_blueprints_from_list(): found BP: ' + bp)
-                scan_tr_end = True
-
-    return result
-
-def lp_import_blueprint(db, name, url, contents):
+def lp_import_blueprint(db, bp):
     '''Import details of a blueprint into the specs table.
 
     If a blueprint does not have a status, use the description.
@@ -117,56 +86,50 @@
     cur.execute('SELECT name FROM milestones')
     valid_milestones = [m[0] for m in cur] + [None]
 
-    queries = {
-        'priority': '<dt>Priority:</dt>\s*<dd>\s*([^\n]*)',
-        'definition': '<dt>Definition:</dt>\s*<dd>\s*([^\n]*)',
-        'implementation': '<dt>Implementation:</dt>\s*<dd>\s*([^\n]*)',
-        'approver': '<dt>Approver:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'drafter': '<dt>Drafter:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'assignee': '<dt>Assignee:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'milestone': '<dt>Milestone target:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/\w+/\+milestone/([a-zA-Z0-9_\.-]+)"',
-        'description': '<div class="top-portlet">\s*<p>(.*?)</p>',
-        'status': '(?:<p>|^)[Ss]tatus:\s*<br />(.*?)</p>',
-        'details_url': 'href="([^"]*)">Read the full specification</a>',
-        'roadmap_notes': '(?:<p>|^)[Rr]oadmap\s+[Nn]otes:\s*<br />(.*?)</p>',
-    }
+    def get_whiteboard_section(heading):
+        if bp.whiteboard is None:
+            return None
+        regex = '^' + heading + ':\s*\n(.*?)\n\n'
+        m = re.search(regex, bp.whiteboard, re.S | re.I | re.M)
+        if m is not None:
+            section = m.group(1).strip()
+        else:
+            section = None
+        return section
 
     data = {}
-    for key, r in queries.iteritems():
-        m = re.search(r, contents, re.S)
-        if not m:
-            dbg('lp_import_blueprint(%s): did not find a match for key %s' % (name, key))
-            if key == 'description':
-                data[key] = '(no description)'
-            else:
-                data[key] = None
-        else:
-            data[key] = m.group(1).strip()
+    data['priority'] = bp.priority
+    data['definition'] = bp.definition_status
+    data['implementation'] = bp.implementation_status
+    data['approver'] = bp.approver and bp.approver.name or None
+    data['drafter'] = bp.drafter and bp.drafter.name or None
+    data['assignee'] = bp.assignee and bp.assignee.name or None
+    data['milestone'] = bp.milestone and bp.milestone.name or None
+    data['description'] = bp.summary or "(no description)"
+    data['status'] = get_whiteboard_section('Status')
+    data['details_url'] = bp.specification_url
+    data['roadmap_notes'] = get_whiteboard_section('Roadmap\s+Notes')
 
     # ignore "later" milestone"
     if data['milestone'] == 'later':
-        dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % name)
+        dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % bp.name)
         return False
 
     if data['milestone'] not in valid_milestones:
-        data_error(url, 'milestone "%s" is unknown/invalid' % data['milestone'], True)
+        data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % data['milestone'], True)
 
     # if we have an explicit status: in the whiteboard, use that; otherwise use
     # description as status text
     if data['status']:
-        data['status'] = data['status'].replace('<br />',
-                '\n').replace('</div>', '').replace('&nbsp;', ' ').replace(
-                '<wbr></wbr>', '').strip()
+        data['status'] = data['status'].strip()
     else:
-        data['status'] = data['description'].replace('<br />',
-                '\n').replace('</div>', '').replace('&nbsp;', ' ').replace(
-                '<wbr></wbr>', '').strip()
+        data['status'] = data['description'].strip()
     del data['description']
 
-    dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (name, str(data)))
+    dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (bp.name, str(data)))
 
     db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)',
-            (name, url, data['priority'], data['implementation'],
+            (bp.name, web_link(bp), data['priority'], data['implementation'],
             data['assignee'], data['status'], data['milestone'],
             data['definition'], data['drafter'], data['approver'],
             data['details_url'], data['roadmap_notes']))
@@ -178,10 +141,7 @@
 
     '''
 
-    ### cargo culted from parse_blueprint_workitem
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
+    line = line.strip()
     if not line:
         return
 
@@ -201,14 +161,9 @@
     db.commit()
 
 def parse_complexity_item(lp, db, line, bp_name, bp_url, def_milestone, def_assignee):
-    
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
-            
-    # remove special characters people tend to type            
+    line = line.strip()
+    # remove special characters people tend to type
     line = re.sub('[^\w -.]', '', line)
-                
     if not line:
         return
 
@@ -219,17 +174,15 @@
     assignee    = None
 
     try:
-        list = line.split()
-        list.reverse()
+        complexity_list = line.split()
+        complexity_list.reverse()
 
         # we may not have any values in the list, so append our
         # default values in the right order so they can be mapped
         defs = [None, def_milestone, def_assignee]
-        for i in range(len(list), len(defs)):
-            list.append(defs[i])
-            
-        (num, milestone, assignee) = list
-    
+        for i in range(len(complexity_list), len(defs)):
+            complexity_list.append(defs[i])
+        (num, milestone, assignee) = complexity_list
         if not num:
             data_error(bp_url, 'No complexity points defined for %s' % line)
 
@@ -238,17 +191,17 @@
         cur = db.cursor()
         cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))',
                 (assignee, num, milestone, bp_name))
-
     except ValueError:
         data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError))
 
+
 def parse_blueprint_workitem(line, default_assignee, milestone,
         blueprint_url, launchpad, result_list):
     '''Parse a work item line of a blueprint whiteboard.'''
 
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
+    # FIXME: we lose bug linking etc. that is done by the tales
+    # formatters in LP here.
+    line = line.strip()
     if not line:
         return
     dbg("\tworkitem (clean): '%s'" % (line))
@@ -291,13 +244,10 @@
 
     result_list.append((desc, state, assignee, milestone))
 
-def follow_blueprint_buglink(bugnum, default_assignee, blueprint_name,
+def follow_blueprint_buglink(bug, default_assignee, blueprint_name,
         launchpad, release, default_milestone, result_list):
     '''Query launchpad for the information on a linked bug'''
 
-    bugnum = int(bugnum)
-    dbg('follow_blueprint_buglink(): processing bug %i (default milestone: %s)' % (bugnum, default_milestone))
-    bug = launchpad.bugs[bugnum]
     for task in bug.bug_tasks:
         if task.status == 'Unknown':
             # this can happen for upstream tasks
@@ -338,7 +288,7 @@
             if better_task:
                 continue
 
-        desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bugnum, bugnum) + bug.title
+        desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bug.id, bug.id) + bug.title
         if rtype != 'distribution':
             desc += ' (%s)' % target.name
 
@@ -363,32 +313,27 @@
             return word
     return None
 
-def lp_import_blueprint_workitems(lp, db, bp_name, bp_url, contents, release):
+def lp_import_blueprint_workitems(lp, db, bp, release):
     '''Collect work items from a Launchpad blueprint.
 
     This includes work items from the whiteboard as well as linked bugs.
     '''
-    linked_bugs_re = re.compile('<div id="bug_links".*>', re.I)
-    bugnum_re = re.compile('<a href="https://bugs\..*launchpad\.net/bugs/([0-9]+)" class="sprite bug">')
-    work_items_re = re.compile('(<p>|^)work items(.*)\s*:\s*<br />', re.I)
-    meta_re = re.compile( '(<p>|^)Meta.*?:<br />', re.I )
-    complexity_re = re.compile( '(<p>|^)Complexity.*?:<br />', re.I )
+    work_items_re = re.compile('^work items(.*)\s*:\s*$', re.I)
+    meta_re = re.compile('^Meta.*?:$', re.I)
+    complexity_re = re.compile('^Complexity.*?:$', re.I)
 
     in_workitems_block = False
-    in_linked_bugs_block = False
     in_meta_block = False
     in_complexity_block = False
-    found_linked_bugs = False
     work_items = []
-    found_wb_workitems = False
     milestone = None
 
     cur = db.cursor()
-    cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp_name,))
+    cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp.name,))
     (spec_assignee, spec_milestone, spec_implementation) = cur.fetchone()
 
     dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % (
-        bp_name, spec_milestone, spec_assignee, spec_implementation))
+        bp.name, spec_milestone, spec_assignee, spec_implementation))
 
     cur.execute('SELECT team FROM teams WHERE name = ?', (spec_assignee,))
     assignee_teams = [t[0] for t in cur]
@@ -396,69 +341,51 @@
     cur.execute('SELECT name FROM milestones')
     valid_milestones = [m[0] for m in cur]
 
-    for l in contents.splitlines():
-
-        if not in_workitems_block and not in_linked_bugs_block \
-                and not in_meta_block and not in_complexity_block:
-            m = work_items_re.search(l)
-            if m:
-                in_workitems_block = True
-                found_wb_workitems = True
-                dbg('lp_import_blueprint_workitems(): starting work items block at ' + l)
-                if not milestone:
-                    milestone = milestone_extract(m.group(2), valid_milestones)
-                    dbg('  ... setting milestone to ' + str(milestone))
-            if linked_bugs_re.search(l):
-                in_linked_bugs_block = True
-                found_linked_bugs = True
-            if meta_re.search(l):
-                in_meta_block = True
-            if complexity_re.search(l):
-                in_complexity_block = True                
-            continue
-
-        if in_workitems_block:
-            dbg("\tworkitem (raw): '%s'" % (l.strip()))
-            parse_blueprint_workitem(l, spec_assignee, milestone or
-                    spec_milestone, bp_url, lp, work_items)
-
-            if '</p>' in l:
-                dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l)
-                in_workitems_block = False
-                milestone = None
-
-        if in_linked_bugs_block:
-            dbg("\tbug block line (raw): '%s'" % (l.strip()))
-            m = bugnum_re.search(l)
-            if m and lp:
-                follow_blueprint_buglink(m.group(1), spec_assignee,
-                        bp_name, lp, release,
-                        milestone or spec_milestone, work_items)
-
-            if '</div>' in l:
-                in_linked_bugs_block = False
-                continue
-
-        if in_meta_block:
-            dbg("\tmeta line (raw): '%s'" % (l.strip()))
-            parse_meta_item( lp, db, l, bp_name )
-
-            # done with the meta information?
-            if '</p>' in l:
-                in_meta_block = False
-                continue
-
-        if in_complexity_block:
-            dbg("\tcomplexity block line (raw): '%s'" % (l.strip()))
-            parse_complexity_item(lp, db, l, bp_name, bp_url, spec_milestone, spec_assignee)
-            
-            # done with the meta information?
-            if '</p>' in l:
-                in_complexity_block = False
-                continue
-
-        if found_wb_workitems and '</div>' in l:
-            break
+    if bp.whiteboard:
+        for l in bp.whiteboard.splitlines():
+
+            if (not in_workitems_block
+                and not in_meta_block and not in_complexity_block):
+                m = work_items_re.search(l)
+                if m:
+                    in_workitems_block = True
+                    dbg('lp_import_blueprint_workitems(): starting work items block at ' + l)
+                    if not milestone:
+                        milestone = milestone_extract(m.group(1), valid_milestones)
+                        dbg('  ... setting milestone to ' + str(milestone))
+                if meta_re.search(l):
+                    in_meta_block = True
+                if complexity_re.search(l):
+                    in_complexity_block = True
+                continue
+
+            if in_workitems_block:
+                dbg("\tworkitem (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l)
+                    in_workitems_block = False
+                    milestone = None
+                parse_blueprint_workitem(l, spec_assignee, milestone or
+                        spec_milestone, web_link(bp), lp, work_items)
+
+            if in_meta_block:
+                dbg("\tmeta line (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    in_meta_block = False
+                    continue
+                parse_meta_item(lp, db, l, bp.name)
+
+            if in_complexity_block:
+                dbg("\tcomplexity block line (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    in_complexity_block = False
+                    continue
+                parse_complexity_item(lp, db, l, bp.name, web_link(bp), spec_milestone, spec_assignee)
+
+    if lp:
+        for bug in bp.bugs:
+            follow_blueprint_buglink(bug, spec_assignee, bp.name, lp, release,
+                    spec_milestone, work_items)
 
     if not work_items:
         #data_error(bp_url, 'no work items defined', True)
@@ -474,7 +401,8 @@
 
     for (desc, status, assignee, milestone) in work_items:
         db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))',
-                (desc, bp_name, status, assignee, milestone))
+                (desc, bp.name, status, assignee, milestone))
+
 
 def lp_import_milestones(lp_project, db):
     '''Import milestones from Launchpad into DB.
@@ -550,29 +478,41 @@
         db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))',
                 (title, name, state, assignee, milestone))
 
-def lp_import(db, cfg, name_pattern = None):
+def lp_import(db, cfg, name_pattern=None):
     '''Collect blueprint work items and status from Launchpad into DB.'''
 
-    lp = Launchpad.login_with('ubuntu-work-items', service_root=EDGE_SERVICE_ROOT)
+    lp = Launchpad.login_with('ubuntu-work-items', service_root="production", version="devel")
+    projects = []
 
     if 'release' in cfg:
         lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release'])
-        url = '%s/ubuntu/%s/+specs?batch=300' % (
-                report_tools.blueprints_base_url, cfg['release'])
+        projects.append(lp_project)
     else:
         assert 'project' in cfg, 'Configuration needs to specify project or release'
         lp_project = lp.projects[cfg['project']]
-        url = '%s/%s/+specs?batch=300' % (
-                report_tools.blueprints_base_url, cfg['project'])
+        projects.append(lp_project)
 
     lp_import_milestones(lp_project, db)
     lp_import_teams(lp, db, cfg)
 
-    for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems():
-        dbg('lp_import(): downloading %s from %s' % (bp, url))
-        contents = urllib.urlopen(url).read().decode('UTF-8')
-        if lp_import_blueprint(db, bp, url, contents):
-            lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release'))
+    extra_projects = cfg.get('extra_projects', [])
+    for extra_project_name in extra_projects:
+        extra_project = lp.projects[extra_project_name]
+        lp_import_milestones(extra_project, db)
+        projects.append(extra_project)
+
+    if name_pattern:
+        name_filter = re.compile(name_pattern)
+
+    for project in projects:
+        # XXX: should this be valid_ or all_specifications?
+        for bp in project.valid_specifications:
+            if name_pattern:
+                if not name_filter.search(bp.name):
+                    continue
+            dbg('lp_import(): downloading %s from %s' % (bp.name, bp.self_link))
+            if lp_import_blueprint(db, bp):
+                lp_import_blueprint_workitems(lp, db, bp, cfg.get('release'))
 
     lp_import_bug_workitems(lp_project, db, cfg)