Merge lp:~james-w/launchpad-work-items-tracker/blueprints-api into lp:launchpad-work-items-tracker

[James Westby]

Hi,

This makes use of the newly exposed blueprints API on Launchpad, reducing
the amount of screen scraping, round trips, and data transferred enormously.

The only regression that I think this will cause is that when someone writes
"bug 12345" in the whiteboard, it will no longer be a link to that bug
when viewed on the workitems page, but I think we can live with that for
now. If you disagree let me know and I will code it.

Apologies that the diff is a little large and not particularly clear, but
I couldn't think of a good way to migrate a bit at a time.

Thanks,

James

[Clint Byrum (clint-fewbar)]

Upon running the code as-is, I received a failure rather quickly:

clint@clint-MacBookPro:~/src/wi/bzr/trunk$ ./collect -d ../natty.db -c ../natty.cfg --debug
lp_import_milestones(): milestone table already filled
lp_import_teams(): teams table already filled
lp_import(): downloading cloud-server-n-automated-testing from https://api.launchpad.net/devel/ubuntu/+spec/cloud-server-n-automated-testing
lp_import_blueprint(cloud-server-n-automated-testing): finished parsing; data: {'status': 'Session for discussion about automated testing of Ubuntu Server; automated ISO testing was implemented for Maverick using libvirt/kvm + Hudson (see http://launchpad.net/ubuntu-server-iso-testing). This approach could be applied in other variants and for other aspects of server testing', 'definition': 'Approved', 'implementation': 'Started', 'milestone': 'natty-alpha-3', 'approver': 'robbie.w', 'details_url': None, 'priority': 'Essential', 'assignee': 'james-page', 'roadmap_notes': None, 'drafter': 'james-page'}
lp_import_blueprint_workitems(): processing cloud-server-n-automated-testing (spec milestone: natty-alpha-3, spec assignee: james-page, spec implementation: Started)
lp_import_blueprint_workitems(): starting work items block at Work items for natty-alpha-2:
  ... setting milestone to natty-alpha-2
 workitem (raw): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
 workitem (clean): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
 workitem (raw): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
 workitem (clean): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
 workitem (raw): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
 workitem (clean): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
 workitem (raw): '[james-page] Package ubuntu-server-iso-testing and locate in PPA: INPROGRESS'
 workitem (clean): '[james-page] Package ubuntu-server-iso-testing and locate in PPA: INPROGRESS'
 workitem (raw): ''
lp_import_blueprint_workitems(): closing work items block with line:
lp_import_blueprint_workitems(): starting work items block at Work items for natty-alpha-3:
  ... setting milestone to natty-alpha-3
 workitem (raw): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
 workitem (clean): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
 workitem (raw): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
 workitem (clean): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
 workitem (raw): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
 workitem (clean): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
https://launchpad.net/ubuntu/+spec/cloud-server-n-automated-testing
  [WARNING] assignee "cr2" is not a valid Launchpad account
 workitem (raw): '[cr2] Checkbox plugin to download tests from couchdb to ex...

review: Resubmit

[lp:~james-w/launchpad-work-items-tracker/blueprints-api]

250. By Clint Byrum on 2010-12-08

merging extra project support from James Westby

251. By Martin Pitt on 2010-12-09

fix previous commit to not crash

[Jamie Bennett (jamiebennett)]

On testing the code I was prompted to log in using open id (from the command-line). As this was tested via ssh to a development box 'Links' was launched as the web browser to gather my open id information. From there you cannot log in (Launchpad/Links issue). This did not happen on previous versions.

review: Needs Information

[lp:~james-w/launchpad-work-items-tracker/blueprints-api]

252. By Martin Pitt on 2010-12-09

fix previous commit properly, thanks James

253. By Martin Pitt on 2010-12-10

revert extra-projects merge for now; this is creating a conceptual conflict on milestones, and should rather be handled in separate configurations for linaro

254. By Martin Pitt on 2010-12-10

consider fixed bugs in bug list based reports as well

255. By Martin Pitt on 2010-12-17

add "Unknown" bug state

256. By Martin Pitt on 2011-01-03

drop Unknown bug state, does not exist any more

257. By Martin Pitt on 2011-01-03

better fix for "Unknown" task status

258. By Martin Pitt on 2011-01-03

finally fix unknown bug states

259. By Martin Pitt on 2011-01-11

html-report: Add textual stats over time

260. By Martin Pitt on 2011-01-11

html-report: sort dates

261. By Martin Pitt on 2011-01-11

fix time-stats todo column

262. By Martin Pitt on 2011-01-11

time-stats: not all states exist always

263. By Martin Pitt on 2011-01-29

collect: add "Expired" bug state

264. By James Westby on 2011-02-07

Merge blueprints API branch.

Unmerged revisions

=== modified file 'collect'
--- collect	2010-12-09 13:58:55 +0000
+++ collect	2010-12-09 15:54:09 +0000
@@ -3,7 +3,7 @@
 # Pull work items from various sources (blueprint whiteboards, linked blueprint
 # bugs, wiki pages) and put them into a database.
 
-import urllib, re, sys, optparse, smtplib, pwd, os
+import urllib, re, sys, optparse, smtplib, pwd, os, urlparse
 from email.mime.text import MIMEText
 import sqlite3 as dbapi2
 
@@ -54,51 +54,20 @@
         dbg('Queueing data error: ' + s)
 
 
+def web_link(item):
+    """Get a link to the Launchpad API object on the website."""
+    api_link = item.self_link
+    parts = urlparse.urlparse(api_link)
+    return parts.scheme + "://" + parts.netloc.replace("api.", "") + "/" + parts.path.split("/", 2)[2]
+
+
 ########################################################################
 #
 # Functions for parsing Launchpad data
 #
 ########################################################################
 
-def lp_blueprints_from_list(url, name_pattern = None):
-    '''Return blueprints from a LP list view.
-
-    This can optionally specify a name pattern (which is mostly useful for
-    testing and debugging, to only parse a small subset of blueprints)
-
-    Return a dictionary name -> (url, milestone).
-    '''
-    blueprint_name_filter = re.compile('href="(/[a-zA-Z0-9-]+/\+spec/%s[^"]+)"(?!.*class.*sprite)' %
-            (name_pattern or '.'))
-    milestone_re = re.compile('href=".*/[a-zA-Z0-9-]+/\+milestone/([^"]+)"')
-
-    result = {}
-    scan_tr_end = False
-    milestone = None
-    bp = None
-    dbg("lp_blueprints_from_list(): Loading '%s'" % url)
-    for l in urllib.urlopen(url):
-        if scan_tr_end:
-            m = milestone_re.search(l)
-            if m:
-                milestone = m.group(1)
-                dbg('lp_blueprints_from_list(): current spec milestone: ' + milestone)
-            if '</tr>' in l:
-                scan_tr_end = False
-                if bp:
-                    result[bp.split('/')[-1]] = (bp, milestone)
-                bp = None
-                milestone = None
-        else:
-            m = blueprint_name_filter.search(l)
-            if m:
-                bp = report_tools.blueprints_base_url + m.group(1)
-                dbg('lp_blueprints_from_list(): found BP: ' + bp)
-                scan_tr_end = True
-
-    return result
-
-def lp_import_blueprint(db, name, url, contents):
+def lp_import_blueprint(db, bp):
     '''Import details of a blueprint into the specs table.
 
     If a blueprint does not have a status, use the description.
@@ -112,56 +81,50 @@
     cur.execute('SELECT name FROM milestones')
     valid_milestones = [m[0] for m in cur] + [None]
 
-    queries = {
-        'priority': '<dt>Priority:</dt>\s*<dd>\s*([^\n]*)',
-        'definition': '<dt>Definition:</dt>\s*<dd>\s*([^\n]*)',
-        'implementation': '<dt>Implementation:</dt>\s*<dd>\s*([^\n]*)',
-        'approver': '<dt>Approver:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'drafter': '<dt>Drafter:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'assignee': '<dt>Assignee:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ',
-        'milestone': '<dt>Milestone target:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/\w+/\+milestone/([a-zA-Z0-9_\.-]+)"',
-        'description': '<div class="top-portlet">\s*<p>(.*?)</p>',
-        'status': '(?:<p>|^)[Ss]tatus:\s*<br />(.*?)</p>',
-        'details_url': 'href="([^"]*)">Read the full specification</a>',
-        'roadmap_notes': '(?:<p>|^)[Rr]oadmap\s+[Nn]otes:\s*<br />(.*?)</p>',
-    }
+    def get_whiteboard_section(heading):
+        if bp.whiteboard is None:
+            return None
+        regex = '^' + heading + ':\s*\n(.*?)\n\n'
+        m = re.search(regex, bp.whiteboard, re.S | re.I | re.M)
+        if m is not None:
+            section = m.group(1).strip()
+        else:
+            section = None
+        return section
 
     data = {}
-    for key, r in queries.iteritems():
-        m = re.search(r, contents, re.S)
-        if not m:
-            dbg('lp_import_blueprint(%s): did not find a match for key %s' % (name, key))
-            if key == 'description':
-                data[key] = '(no description)'
-            else:
-                data[key] = None
-        else:
-            data[key] = m.group(1).strip()
+    data['priority'] = bp.priority
+    data['definition'] = bp.definition_status
+    data['implementation'] = bp.implementation_status
+    data['approver'] = bp.approver and bp.approver.name or None
+    data['drafter'] = bp.drafter and bp.drafter.name or None
+    data['assignee'] = bp.assignee and bp.assignee.name or None
+    data['milestone'] = bp.milestone and bp.milestone.name or None
+    data['description'] = bp.summary or "(no description)"
+    data['status'] = get_whiteboard_section('Status')
+    data['details_url'] = bp.specification_url
+    data['roadmap_notes'] = get_whiteboard_section('Roadmap\s+Notes')
 
     # ignore "later" milestone"
     if data['milestone'] == 'later':
-        dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % name)
+        dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % bp.name)
         return False
 
     if data['milestone'] not in valid_milestones:
-        data_error(url, 'milestone "%s" is unknown/invalid' % data['milestone'], True)
+        data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % data['milestone'], True)
 
     # if we have an explicit status: in the whiteboard, use that; otherwise use
     # description as status text
     if data['status']:
-        data['status'] = data['status'].replace('<br />',
-                '\n').replace('</div>', '').replace('&nbsp;', ' ').replace(
-                '<wbr></wbr>', '').strip()
+        data['status'] = data['status'].strip()
     else:
-        data['status'] = data['description'].replace('<br />',
-                '\n').replace('</div>', '').replace('&nbsp;', ' ').replace(
-                '<wbr></wbr>', '').strip()
+        data['status'] = data['description'].strip()
     del data['description']
 
-    dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (name, str(data)))
+    dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (bp.name, str(data)))
 
     db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)',
-            (name, url, data['priority'], data['implementation'],
+            (bp.name, web_link(bp), data['priority'], data['implementation'],
             data['assignee'], data['status'], data['milestone'],
             data['definition'], data['drafter'], data['approver'],
             data['details_url'], data['roadmap_notes']))
@@ -173,10 +136,7 @@
 
     '''
 
-    ### cargo culted from parse_blueprint_workitem
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
+    line = line.strip()
     if not line:
         return
 
@@ -196,14 +156,9 @@
     db.commit()
 
 def parse_complexity_item(lp, db, line, bp_name, bp_url, def_milestone, def_assignee):
-    
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
-            
-    # remove special characters people tend to type            
+    line = line.strip()
+    # remove special characters people tend to type
     line = re.sub('[^\w -.]', '', line)
-                
     if not line:
         return
 
@@ -214,17 +169,15 @@
     assignee    = None
 
     try:
-        list = line.split()
-        list.reverse()
+        complexity_list = line.split()
+        complexity_list.reverse()
 
         # we may not have any values in the list, so append our
         # default values in the right order so they can be mapped
         defs = [None, def_milestone, def_assignee]
-        for i in range(len(list), len(defs)):
-            list.append(defs[i])
-            
-        (num, milestone, assignee) = list
-    
+        for i in range(len(complexity_list), len(defs)):
+            complexity_list.append(defs[i])
+        (num, milestone, assignee) = complexity_list
         if not num:
             data_error(bp_url, 'No complexity points defined for %s' % line)
 
@@ -233,17 +186,17 @@
         cur = db.cursor()
         cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))',
                 (assignee, num, milestone, bp_name))
-
     except ValueError:
         data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError))
 
+
 def parse_blueprint_workitem(line, default_assignee, milestone,
         blueprint_url, launchpad, result_list):
     '''Parse a work item line of a blueprint whiteboard.'''
 
-    line = line.replace('<br />', '').replace('</div>', '').replace('</p>',
-            '').replace('<wbr></wbr>', '').replace('&nbsp;', ' ').replace(
-            '<a href="/', '<a href="https://launchpad.net/').strip()
+    # FIXME: we lose bug linking etc. that is done by the tales
+    # formatters in LP here.
+    line = line.strip()
     if not line:
         return
     dbg("\tworkitem (clean): '%s'" % (line))
@@ -286,13 +239,10 @@
 
     result_list.append((desc, state, assignee, milestone))
 
-def follow_blueprint_buglink(bugnum, default_assignee, blueprint_name,
+def follow_blueprint_buglink(bug, default_assignee, blueprint_name,
         launchpad, release, default_milestone, result_list):
     '''Query launchpad for the information on a linked bug'''
 
-    bugnum = int(bugnum)
-    dbg('follow_blueprint_buglink(): processing bug %i (default milestone: %s)' % (bugnum, default_milestone))
-    bug = launchpad.bugs[bugnum]
     for task in bug.bug_tasks:
         state = bug_wi_states.get(task.status, 'todo')
         if state is None:
@@ -329,7 +279,7 @@
             if better_task:
                 continue
 
-        desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bugnum, bugnum) + bug.title
+        desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bug.id, bug.id) + bug.title
         if rtype != 'distribution':
             desc += ' (%s)' % target.name
 
@@ -354,32 +304,27 @@
             return word
     return None
 
-def lp_import_blueprint_workitems(lp, db, bp_name, bp_url, contents, release):
+def lp_import_blueprint_workitems(lp, db, bp, release):
     '''Collect work items from a Launchpad blueprint.
 
     This includes work items from the whiteboard as well as linked bugs.
     '''
-    linked_bugs_re = re.compile('<div id="bug_links".*>', re.I)
-    bugnum_re = re.compile('<a href="https://bugs\..*launchpad\.net/bugs/([0-9]+)" class="sprite bug">')
-    work_items_re = re.compile('(<p>|^)work items(.*)\s*:\s*<br />', re.I)
-    meta_re = re.compile( '(<p>|^)Meta.*?:<br />', re.I )
-    complexity_re = re.compile( '(<p>|^)Complexity.*?:<br />', re.I )
+    work_items_re = re.compile('^work items(.*)\s*:\s*$', re.I)
+    meta_re = re.compile('^Meta.*?:$', re.I)
+    complexity_re = re.compile('^Complexity.*?:$', re.I)
 
     in_workitems_block = False
-    in_linked_bugs_block = False
     in_meta_block = False
     in_complexity_block = False
-    found_linked_bugs = False
     work_items = []
-    found_wb_workitems = False
     milestone = None
 
     cur = db.cursor()
-    cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp_name,))
+    cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp.name,))
     (spec_assignee, spec_milestone, spec_implementation) = cur.fetchone()
 
     dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % (
-        bp_name, spec_milestone, spec_assignee, spec_implementation))
+        bp.name, spec_milestone, spec_assignee, spec_implementation))
 
     cur.execute('SELECT team FROM teams WHERE name = ?', (spec_assignee,))
     assignee_teams = [t[0] for t in cur]
@@ -387,69 +332,51 @@
     cur.execute('SELECT name FROM milestones')
     valid_milestones = [m[0] for m in cur]
 
-    for l in contents.splitlines():
-
-        if not in_workitems_block and not in_linked_bugs_block \
-                and not in_meta_block and not in_complexity_block:
-            m = work_items_re.search(l)
-            if m:
-                in_workitems_block = True
-                found_wb_workitems = True
-                dbg('lp_import_blueprint_workitems(): starting work items block at ' + l)
-                if not milestone:
-                    milestone = milestone_extract(m.group(2), valid_milestones)
-                    dbg('  ... setting milestone to ' + str(milestone))
-            if linked_bugs_re.search(l):
-                in_linked_bugs_block = True
-                found_linked_bugs = True
-            if meta_re.search(l):
-                in_meta_block = True
-            if complexity_re.search(l):
-                in_complexity_block = True                
-            continue
-
-        if in_workitems_block:
-            dbg("\tworkitem (raw): '%s'" % (l.strip()))
-            parse_blueprint_workitem(l, spec_assignee, milestone or
-                    spec_milestone, bp_url, lp, work_items)
-
-            if '</p>' in l:
-                dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l)
-                in_workitems_block = False
-                milestone = None
-
-        if in_linked_bugs_block:
-            dbg("\tbug block line (raw): '%s'" % (l.strip()))
-            m = bugnum_re.search(l)
-            if m and lp:
-                follow_blueprint_buglink(m.group(1), spec_assignee,
-                        bp_name, lp, release,
-                        milestone or spec_milestone, work_items)
-
-            if '</div>' in l:
-                in_linked_bugs_block = False
-                continue
-
-        if in_meta_block:
-            dbg("\tmeta line (raw): '%s'" % (l.strip()))
-            parse_meta_item( lp, db, l, bp_name )
-
-            # done with the meta information?
-            if '</p>' in l:
-                in_meta_block = False
-                continue
-
-        if in_complexity_block:
-            dbg("\tcomplexity block line (raw): '%s'" % (l.strip()))
-            parse_complexity_item(lp, db, l, bp_name, bp_url, spec_milestone, spec_assignee)
-            
-            # done with the meta information?
-            if '</p>' in l:
-                in_complexity_block = False
-                continue
-
-        if found_wb_workitems and '</div>' in l:
-            break
+    if bp.whiteboard:
+        for l in bp.whiteboard.splitlines():
+
+            if (not in_workitems_block
+                and not in_meta_block and not in_complexity_block):
+                m = work_items_re.search(l)
+                if m:
+                    in_workitems_block = True
+                    dbg('lp_import_blueprint_workitems(): starting work items block at ' + l)
+                    if not milestone:
+                        milestone = milestone_extract(m.group(1), valid_milestones)
+                        dbg('  ... setting milestone to ' + str(milestone))
+                if meta_re.search(l):
+                    in_meta_block = True
+                if complexity_re.search(l):
+                    in_complexity_block = True
+                continue
+
+            if in_workitems_block:
+                dbg("\tworkitem (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l)
+                    in_workitems_block = False
+                    milestone = None
+                parse_blueprint_workitem(l, spec_assignee, milestone or
+                        spec_milestone, web_link(bp), lp, work_items)
+
+            if in_meta_block:
+                dbg("\tmeta line (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    in_meta_block = False
+                    continue
+                parse_meta_item(lp, db, l, bp.name)
+
+            if in_complexity_block:
+                dbg("\tcomplexity block line (raw): '%s'" % (l.strip()))
+                if not l.strip():
+                    in_complexity_block = False
+                    continue
+                parse_complexity_item(lp, db, l, bp.name, web_link(bp), spec_milestone, spec_assignee)
+
+    if lp:
+        for bug in bp.bugs:
+            follow_blueprint_buglink(bug, spec_assignee, bp.name, lp, release,
+                    spec_milestone, work_items)
 
     if not work_items:
         #data_error(bp_url, 'no work items defined', True)
@@ -465,7 +392,8 @@
 
     for (desc, status, assignee, milestone) in work_items:
         db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))',
-                (desc, bp_name, status, assignee, milestone))
+                (desc, bp.name, status, assignee, milestone))
+
 
 def lp_import_milestones(lp_project, db):
     '''Import milestones from Launchpad into DB.
@@ -544,40 +472,33 @@
 def lp_import(db, cfg, name_pattern = None):
     '''Collect blueprint work items and status from Launchpad into DB.'''
 
-    lp = Launchpad.login_with('ubuntu-work-items', service_root=EDGE_SERVICE_ROOT)
-
-    urls = []
-
-    def import_project(project):
-        lp_import_milestones(project, db)
-        url = '%s/%s/+specs?batch=300' % (
-                report_tools.blueprints_base_url, project.name)
-        urls.append(url)
+    lp = Launchpad.login_with('ubuntu-work-items', service_root="production", version="devel")
+
+    projects = []
 
     if 'release' in cfg:
         lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release'])
-        url = '%s/ubuntu/%s/+specs?batch=300' % (
-                report_tools.blueprints_base_url, cfg['release'])
-        urls.append(url)
+        projects.append(lp_project)
     else:
         assert 'project' in cfg, 'Configuration needs to specify project or release'
         lp_project = lp.projects[cfg['project']]
-        import_project(lp_project)
+        projects.append(lp_project)
+
+    lp_import_milestones(lp_project, db)
+    lp_import_teams(lp, db, cfg)
 
     extra_projects = cfg.get('extra_projects', [])
     for extra_project_name in extra_projects:
         extra_project = lp.projects[extra_project_name]
-        import_project(extra_project)
-
-    lp_import_milestones(lp_project, db)
-    lp_import_teams(lp, db, cfg)
-
-    for url in urls:
-        for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems():
-            dbg('lp_import(): downloading %s from %s' % (bp, url))
-            contents = urllib.urlopen(url).read().decode('UTF-8')
-            if lp_import_blueprint(db, bp, url, contents):
-                lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release'))
+        lp_import_milestones(extra_project, db)
+        projects.append(extra_project)
+
+    for project in projects:
+    # XXX: should this be valid_ or all_specifications?
+        for bp in lp_project.valid_specifications:
+            dbg('lp_import(): downloading %s from %s' % (bp.name, bp.self_link))
+            if lp_import_blueprint(db, bp):
+                lp_import_blueprint_workitems(lp, db, bp, cfg.get('release'))
     lp_import_bug_workitems(lp_project, db, cfg)
 
 ########################################################################