Merge lp:~james-w/launchpad-work-items-tracker/blueprints-api into lp:launchpad-work-items-tracker
- blueprints-api
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~james-w/launchpad-work-items-tracker/blueprints-api |
Merge into: | lp:launchpad-work-items-tracker |
Diff against target: |
476 lines (+120/-199) 1 file modified
collect (+120/-199) |
To merge this branch: | bzr merge lp:~james-w/launchpad-work-items-tracker/blueprints-api |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jamie Bennett (community) | Needs Information | ||
Clint Byrum (community) | Needs Resubmitting | ||
Review via email: mp+43136@code.launchpad.net |
This proposal has been superseded by a proposal from 2010-12-09.
Commit message
Description of the change
Hi,
This makes use of the newly exposed blueprints API on Launchpad, reducing
the amount of screen scraping, round trips, and data transferred enormously.
The only regression that I think this will cause is that when someone writes
"bug 12345" in the whiteboard, it will no longer be a link to that bug
when viewed on the workitems page, but I think we can live with that for
now. If you disagree let me know and I will code it.
Apologies that the diff is a little large and not particularly clear, but
I couldn't think of a good way to migrate a bit at a time.
Thanks,
James
- 250. By Clint Byrum
-
merging extra project support from James Westby
- 251. By Martin Pitt
-
fix previous commit to not crash
Jamie Bennett (jamiebennett) wrote : | # |
On testing the code I was prompted to log in using open id (from the command-line). As this was tested via ssh to a development box 'Links' was launched as the web browser to gather my open id information. From there you cannot log in (Launchpad/Links issue). This did not happen on previous versions.
- 252. By Martin Pitt
-
fix previous commit properly, thanks James
- 253. By Martin Pitt
-
revert extra-projects merge for now; this is creating a conceptual conflict on milestones, and should rather be handled in separate configurations for linaro
- 254. By Martin Pitt
-
consider fixed bugs in bug list based reports as well
- 255. By Martin Pitt
-
add "Unknown" bug state
- 256. By Martin Pitt
-
drop Unknown bug state, does not exist any more
- 257. By Martin Pitt
-
better fix for "Unknown" task status
- 258. By Martin Pitt
-
finally fix unknown bug states
- 259. By Martin Pitt
-
html-report: Add textual stats over time
- 260. By Martin Pitt
-
html-report: sort dates
- 261. By Martin Pitt
-
fix time-stats todo column
- 262. By Martin Pitt
-
time-stats: not all states exist always
- 263. By Martin Pitt
-
collect: add "Expired" bug state
- 264. By James Westby
-
Merge blueprints API branch.
Unmerged revisions
Preview Diff
1 | === modified file 'collect' |
2 | --- collect 2010-12-09 13:58:55 +0000 |
3 | +++ collect 2010-12-09 15:54:09 +0000 |
4 | @@ -3,7 +3,7 @@ |
5 | # Pull work items from various sources (blueprint whiteboards, linked blueprint |
6 | # bugs, wiki pages) and put them into a database. |
7 | |
8 | -import urllib, re, sys, optparse, smtplib, pwd, os |
9 | +import urllib, re, sys, optparse, smtplib, pwd, os, urlparse |
10 | from email.mime.text import MIMEText |
11 | import sqlite3 as dbapi2 |
12 | |
13 | @@ -54,51 +54,20 @@ |
14 | dbg('Queueing data error: ' + s) |
15 | |
16 | |
17 | +def web_link(item): |
18 | + """Get a link to the Launchpad API object on the website.""" |
19 | + api_link = item.self_link |
20 | + parts = urlparse.urlparse(api_link) |
21 | + return parts.scheme + "://" + parts.netloc.replace("api.", "") + "/" + parts.path.split("/", 2)[2] |
22 | + |
23 | + |
24 | ######################################################################## |
25 | # |
26 | # Functions for parsing Launchpad data |
27 | # |
28 | ######################################################################## |
29 | |
30 | -def lp_blueprints_from_list(url, name_pattern = None): |
31 | - '''Return blueprints from a LP list view. |
32 | - |
33 | - This can optionally specify a name pattern (which is mostly useful for |
34 | - testing and debugging, to only parse a small subset of blueprints) |
35 | - |
36 | - Return a dictionary name -> (url, milestone). |
37 | - ''' |
38 | - blueprint_name_filter = re.compile('href="(/[a-zA-Z0-9-]+/\+spec/%s[^"]+)"(?!.*class.*sprite)' % |
39 | - (name_pattern or '.')) |
40 | - milestone_re = re.compile('href=".*/[a-zA-Z0-9-]+/\+milestone/([^"]+)"') |
41 | - |
42 | - result = {} |
43 | - scan_tr_end = False |
44 | - milestone = None |
45 | - bp = None |
46 | - dbg("lp_blueprints_from_list(): Loading '%s'" % url) |
47 | - for l in urllib.urlopen(url): |
48 | - if scan_tr_end: |
49 | - m = milestone_re.search(l) |
50 | - if m: |
51 | - milestone = m.group(1) |
52 | - dbg('lp_blueprints_from_list(): current spec milestone: ' + milestone) |
53 | - if '</tr>' in l: |
54 | - scan_tr_end = False |
55 | - if bp: |
56 | - result[bp.split('/')[-1]] = (bp, milestone) |
57 | - bp = None |
58 | - milestone = None |
59 | - else: |
60 | - m = blueprint_name_filter.search(l) |
61 | - if m: |
62 | - bp = report_tools.blueprints_base_url + m.group(1) |
63 | - dbg('lp_blueprints_from_list(): found BP: ' + bp) |
64 | - scan_tr_end = True |
65 | - |
66 | - return result |
67 | - |
68 | -def lp_import_blueprint(db, name, url, contents): |
69 | +def lp_import_blueprint(db, bp): |
70 | '''Import details of a blueprint into the specs table. |
71 | |
72 | If a blueprint does not have a status, use the description. |
73 | @@ -112,56 +81,50 @@ |
74 | cur.execute('SELECT name FROM milestones') |
75 | valid_milestones = [m[0] for m in cur] + [None] |
76 | |
77 | - queries = { |
78 | - 'priority': '<dt>Priority:</dt>\s*<dd>\s*([^\n]*)', |
79 | - 'definition': '<dt>Definition:</dt>\s*<dd>\s*([^\n]*)', |
80 | - 'implementation': '<dt>Implementation:</dt>\s*<dd>\s*([^\n]*)', |
81 | - 'approver': '<dt>Approver:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', |
82 | - 'drafter': '<dt>Drafter:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', |
83 | - 'assignee': '<dt>Assignee:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', |
84 | - 'milestone': '<dt>Milestone target:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/\w+/\+milestone/([a-zA-Z0-9_\.-]+)"', |
85 | - 'description': '<div class="top-portlet">\s*<p>(.*?)</p>', |
86 | - 'status': '(?:<p>|^)[Ss]tatus:\s*<br />(.*?)</p>', |
87 | - 'details_url': 'href="([^"]*)">Read the full specification</a>', |
88 | - 'roadmap_notes': '(?:<p>|^)[Rr]oadmap\s+[Nn]otes:\s*<br />(.*?)</p>', |
89 | - } |
90 | + def get_whiteboard_section(heading): |
91 | + if bp.whiteboard is None: |
92 | + return None |
93 | + regex = '^' + heading + ':\s*\n(.*?)\n\n' |
94 | + m = re.search(regex, bp.whiteboard, re.S | re.I | re.M) |
95 | + if m is not None: |
96 | + section = m.group(1).strip() |
97 | + else: |
98 | + section = None |
99 | + return section |
100 | |
101 | data = {} |
102 | - for key, r in queries.iteritems(): |
103 | - m = re.search(r, contents, re.S) |
104 | - if not m: |
105 | - dbg('lp_import_blueprint(%s): did not find a match for key %s' % (name, key)) |
106 | - if key == 'description': |
107 | - data[key] = '(no description)' |
108 | - else: |
109 | - data[key] = None |
110 | - else: |
111 | - data[key] = m.group(1).strip() |
112 | + data['priority'] = bp.priority |
113 | + data['definition'] = bp.definition_status |
114 | + data['implementation'] = bp.implementation_status |
115 | + data['approver'] = bp.approver and bp.approver.name or None |
116 | + data['drafter'] = bp.drafter and bp.drafter.name or None |
117 | + data['assignee'] = bp.assignee and bp.assignee.name or None |
118 | + data['milestone'] = bp.milestone and bp.milestone.name or None |
119 | + data['description'] = bp.summary or "(no description)" |
120 | + data['status'] = get_whiteboard_section('Status') |
121 | + data['details_url'] = bp.specification_url |
122 | + data['roadmap_notes'] = get_whiteboard_section('Roadmap\s+Notes') |
123 | |
124 | # ignore "later" milestone" |
125 | if data['milestone'] == 'later': |
126 | - dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % name) |
127 | + dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % bp.name) |
128 | return False |
129 | |
130 | if data['milestone'] not in valid_milestones: |
131 | - data_error(url, 'milestone "%s" is unknown/invalid' % data['milestone'], True) |
132 | + data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % data['milestone'], True) |
133 | |
134 | # if we have an explicit status: in the whiteboard, use that; otherwise use |
135 | # description as status text |
136 | if data['status']: |
137 | - data['status'] = data['status'].replace('<br />', |
138 | - '\n').replace('</div>', '').replace(' ', ' ').replace( |
139 | - '<wbr></wbr>', '').strip() |
140 | + data['status'] = data['status'].strip() |
141 | else: |
142 | - data['status'] = data['description'].replace('<br />', |
143 | - '\n').replace('</div>', '').replace(' ', ' ').replace( |
144 | - '<wbr></wbr>', '').strip() |
145 | + data['status'] = data['description'].strip() |
146 | del data['description'] |
147 | |
148 | - dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (name, str(data))) |
149 | + dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (bp.name, str(data))) |
150 | |
151 | db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)', |
152 | - (name, url, data['priority'], data['implementation'], |
153 | + (bp.name, web_link(bp), data['priority'], data['implementation'], |
154 | data['assignee'], data['status'], data['milestone'], |
155 | data['definition'], data['drafter'], data['approver'], |
156 | data['details_url'], data['roadmap_notes'])) |
157 | @@ -173,10 +136,7 @@ |
158 | |
159 | ''' |
160 | |
161 | - ### cargo culted from parse_blueprint_workitem |
162 | - line = line.replace('<br />', '').replace('</div>', '').replace('</p>', |
163 | - '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( |
164 | - '<a href="/', '<a href="https://launchpad.net/').strip() |
165 | + line = line.strip() |
166 | if not line: |
167 | return |
168 | |
169 | @@ -196,14 +156,9 @@ |
170 | db.commit() |
171 | |
172 | def parse_complexity_item(lp, db, line, bp_name, bp_url, def_milestone, def_assignee): |
173 | - |
174 | - line = line.replace('<br />', '').replace('</div>', '').replace('</p>', |
175 | - '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( |
176 | - '<a href="/', '<a href="https://launchpad.net/').strip() |
177 | - |
178 | - # remove special characters people tend to type |
179 | + line = line.strip() |
180 | + # remove special characters people tend to type |
181 | line = re.sub('[^\w -.]', '', line) |
182 | - |
183 | if not line: |
184 | return |
185 | |
186 | @@ -214,17 +169,15 @@ |
187 | assignee = None |
188 | |
189 | try: |
190 | - list = line.split() |
191 | - list.reverse() |
192 | + complexity_list = line.split() |
193 | + complexity_list.reverse() |
194 | |
195 | # we may not have any values in the list, so append our |
196 | # default values in the right order so they can be mapped |
197 | defs = [None, def_milestone, def_assignee] |
198 | - for i in range(len(list), len(defs)): |
199 | - list.append(defs[i]) |
200 | - |
201 | - (num, milestone, assignee) = list |
202 | - |
203 | + for i in range(len(complexity_list), len(defs)): |
204 | + complexity_list.append(defs[i]) |
205 | + (num, milestone, assignee) = complexity_list |
206 | if not num: |
207 | data_error(bp_url, 'No complexity points defined for %s' % line) |
208 | |
209 | @@ -233,17 +186,17 @@ |
210 | cur = db.cursor() |
211 | cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))', |
212 | (assignee, num, milestone, bp_name)) |
213 | - |
214 | except ValueError: |
215 | data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError)) |
216 | |
217 | + |
218 | def parse_blueprint_workitem(line, default_assignee, milestone, |
219 | blueprint_url, launchpad, result_list): |
220 | '''Parse a work item line of a blueprint whiteboard.''' |
221 | |
222 | - line = line.replace('<br />', '').replace('</div>', '').replace('</p>', |
223 | - '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( |
224 | - '<a href="/', '<a href="https://launchpad.net/').strip() |
225 | + # FIXME: we lose bug linking etc. that is done by the tales |
226 | + # formatters in LP here. |
227 | + line = line.strip() |
228 | if not line: |
229 | return |
230 | dbg("\tworkitem (clean): '%s'" % (line)) |
231 | @@ -286,13 +239,10 @@ |
232 | |
233 | result_list.append((desc, state, assignee, milestone)) |
234 | |
235 | -def follow_blueprint_buglink(bugnum, default_assignee, blueprint_name, |
236 | +def follow_blueprint_buglink(bug, default_assignee, blueprint_name, |
237 | launchpad, release, default_milestone, result_list): |
238 | '''Query launchpad for the information on a linked bug''' |
239 | |
240 | - bugnum = int(bugnum) |
241 | - dbg('follow_blueprint_buglink(): processing bug %i (default milestone: %s)' % (bugnum, default_milestone)) |
242 | - bug = launchpad.bugs[bugnum] |
243 | for task in bug.bug_tasks: |
244 | state = bug_wi_states.get(task.status, 'todo') |
245 | if state is None: |
246 | @@ -329,7 +279,7 @@ |
247 | if better_task: |
248 | continue |
249 | |
250 | - desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bugnum, bugnum) + bug.title |
251 | + desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bug.id, bug.id) + bug.title |
252 | if rtype != 'distribution': |
253 | desc += ' (%s)' % target.name |
254 | |
255 | @@ -354,32 +304,27 @@ |
256 | return word |
257 | return None |
258 | |
259 | -def lp_import_blueprint_workitems(lp, db, bp_name, bp_url, contents, release): |
260 | +def lp_import_blueprint_workitems(lp, db, bp, release): |
261 | '''Collect work items from a Launchpad blueprint. |
262 | |
263 | This includes work items from the whiteboard as well as linked bugs. |
264 | ''' |
265 | - linked_bugs_re = re.compile('<div id="bug_links".*>', re.I) |
266 | - bugnum_re = re.compile('<a href="https://bugs\..*launchpad\.net/bugs/([0-9]+)" class="sprite bug">') |
267 | - work_items_re = re.compile('(<p>|^)work items(.*)\s*:\s*<br />', re.I) |
268 | - meta_re = re.compile( '(<p>|^)Meta.*?:<br />', re.I ) |
269 | - complexity_re = re.compile( '(<p>|^)Complexity.*?:<br />', re.I ) |
270 | + work_items_re = re.compile('^work items(.*)\s*:\s*$', re.I) |
271 | + meta_re = re.compile('^Meta.*?:$', re.I) |
272 | + complexity_re = re.compile('^Complexity.*?:$', re.I) |
273 | |
274 | in_workitems_block = False |
275 | - in_linked_bugs_block = False |
276 | in_meta_block = False |
277 | in_complexity_block = False |
278 | - found_linked_bugs = False |
279 | work_items = [] |
280 | - found_wb_workitems = False |
281 | milestone = None |
282 | |
283 | cur = db.cursor() |
284 | - cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp_name,)) |
285 | + cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp.name,)) |
286 | (spec_assignee, spec_milestone, spec_implementation) = cur.fetchone() |
287 | |
288 | dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % ( |
289 | - bp_name, spec_milestone, spec_assignee, spec_implementation)) |
290 | + bp.name, spec_milestone, spec_assignee, spec_implementation)) |
291 | |
292 | cur.execute('SELECT team FROM teams WHERE name = ?', (spec_assignee,)) |
293 | assignee_teams = [t[0] for t in cur] |
294 | @@ -387,69 +332,51 @@ |
295 | cur.execute('SELECT name FROM milestones') |
296 | valid_milestones = [m[0] for m in cur] |
297 | |
298 | - for l in contents.splitlines(): |
299 | - |
300 | - if not in_workitems_block and not in_linked_bugs_block \ |
301 | - and not in_meta_block and not in_complexity_block: |
302 | - m = work_items_re.search(l) |
303 | - if m: |
304 | - in_workitems_block = True |
305 | - found_wb_workitems = True |
306 | - dbg('lp_import_blueprint_workitems(): starting work items block at ' + l) |
307 | - if not milestone: |
308 | - milestone = milestone_extract(m.group(2), valid_milestones) |
309 | - dbg(' ... setting milestone to ' + str(milestone)) |
310 | - if linked_bugs_re.search(l): |
311 | - in_linked_bugs_block = True |
312 | - found_linked_bugs = True |
313 | - if meta_re.search(l): |
314 | - in_meta_block = True |
315 | - if complexity_re.search(l): |
316 | - in_complexity_block = True |
317 | - continue |
318 | - |
319 | - if in_workitems_block: |
320 | - dbg("\tworkitem (raw): '%s'" % (l.strip())) |
321 | - parse_blueprint_workitem(l, spec_assignee, milestone or |
322 | - spec_milestone, bp_url, lp, work_items) |
323 | - |
324 | - if '</p>' in l: |
325 | - dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l) |
326 | - in_workitems_block = False |
327 | - milestone = None |
328 | - |
329 | - if in_linked_bugs_block: |
330 | - dbg("\tbug block line (raw): '%s'" % (l.strip())) |
331 | - m = bugnum_re.search(l) |
332 | - if m and lp: |
333 | - follow_blueprint_buglink(m.group(1), spec_assignee, |
334 | - bp_name, lp, release, |
335 | - milestone or spec_milestone, work_items) |
336 | - |
337 | - if '</div>' in l: |
338 | - in_linked_bugs_block = False |
339 | - continue |
340 | - |
341 | - if in_meta_block: |
342 | - dbg("\tmeta line (raw): '%s'" % (l.strip())) |
343 | - parse_meta_item( lp, db, l, bp_name ) |
344 | - |
345 | - # done with the meta information? |
346 | - if '</p>' in l: |
347 | - in_meta_block = False |
348 | - continue |
349 | - |
350 | - if in_complexity_block: |
351 | - dbg("\tcomplexity block line (raw): '%s'" % (l.strip())) |
352 | - parse_complexity_item(lp, db, l, bp_name, bp_url, spec_milestone, spec_assignee) |
353 | - |
354 | - # done with the meta information? |
355 | - if '</p>' in l: |
356 | - in_complexity_block = False |
357 | - continue |
358 | - |
359 | - if found_wb_workitems and '</div>' in l: |
360 | - break |
361 | + if bp.whiteboard: |
362 | + for l in bp.whiteboard.splitlines(): |
363 | + |
364 | + if (not in_workitems_block |
365 | + and not in_meta_block and not in_complexity_block): |
366 | + m = work_items_re.search(l) |
367 | + if m: |
368 | + in_workitems_block = True |
369 | + dbg('lp_import_blueprint_workitems(): starting work items block at ' + l) |
370 | + if not milestone: |
371 | + milestone = milestone_extract(m.group(1), valid_milestones) |
372 | + dbg(' ... setting milestone to ' + str(milestone)) |
373 | + if meta_re.search(l): |
374 | + in_meta_block = True |
375 | + if complexity_re.search(l): |
376 | + in_complexity_block = True |
377 | + continue |
378 | + |
379 | + if in_workitems_block: |
380 | + dbg("\tworkitem (raw): '%s'" % (l.strip())) |
381 | + if not l.strip(): |
382 | + dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l) |
383 | + in_workitems_block = False |
384 | + milestone = None |
385 | + parse_blueprint_workitem(l, spec_assignee, milestone or |
386 | + spec_milestone, web_link(bp), lp, work_items) |
387 | + |
388 | + if in_meta_block: |
389 | + dbg("\tmeta line (raw): '%s'" % (l.strip())) |
390 | + if not l.strip(): |
391 | + in_meta_block = False |
392 | + continue |
393 | + parse_meta_item(lp, db, l, bp.name) |
394 | + |
395 | + if in_complexity_block: |
396 | + dbg("\tcomplexity block line (raw): '%s'" % (l.strip())) |
397 | + if not l.strip(): |
398 | + in_complexity_block = False |
399 | + continue |
400 | + parse_complexity_item(lp, db, l, bp.name, web_link(bp), spec_milestone, spec_assignee) |
401 | + |
402 | + if lp: |
403 | + for bug in bp.bugs: |
404 | + follow_blueprint_buglink(bug, spec_assignee, bp.name, lp, release, |
405 | + spec_milestone, work_items) |
406 | |
407 | if not work_items: |
408 | #data_error(bp_url, 'no work items defined', True) |
409 | @@ -465,7 +392,8 @@ |
410 | |
411 | for (desc, status, assignee, milestone) in work_items: |
412 | db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))', |
413 | - (desc, bp_name, status, assignee, milestone)) |
414 | + (desc, bp.name, status, assignee, milestone)) |
415 | + |
416 | |
417 | def lp_import_milestones(lp_project, db): |
418 | '''Import milestones from Launchpad into DB. |
419 | @@ -544,40 +472,33 @@ |
420 | def lp_import(db, cfg, name_pattern = None): |
421 | '''Collect blueprint work items and status from Launchpad into DB.''' |
422 | |
423 | - lp = Launchpad.login_with('ubuntu-work-items', service_root=EDGE_SERVICE_ROOT) |
424 | - |
425 | - urls = [] |
426 | - |
427 | - def import_project(project): |
428 | - lp_import_milestones(project, db) |
429 | - url = '%s/%s/+specs?batch=300' % ( |
430 | - report_tools.blueprints_base_url, project.name) |
431 | - urls.append(url) |
432 | + lp = Launchpad.login_with('ubuntu-work-items', service_root="production", version="devel") |
433 | + |
434 | + projects = [] |
435 | |
436 | if 'release' in cfg: |
437 | lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release']) |
438 | - url = '%s/ubuntu/%s/+specs?batch=300' % ( |
439 | - report_tools.blueprints_base_url, cfg['release']) |
440 | - urls.append(url) |
441 | + projects.append(lp_project) |
442 | else: |
443 | assert 'project' in cfg, 'Configuration needs to specify project or release' |
444 | lp_project = lp.projects[cfg['project']] |
445 | - import_project(lp_project) |
446 | + projects.append(lp_project) |
447 | + |
448 | + lp_import_milestones(lp_project, db) |
449 | + lp_import_teams(lp, db, cfg) |
450 | |
451 | extra_projects = cfg.get('extra_projects', []) |
452 | for extra_project_name in extra_projects: |
453 | extra_project = lp.projects[extra_project_name] |
454 | - import_project(extra_project) |
455 | - |
456 | - lp_import_milestones(lp_project, db) |
457 | - lp_import_teams(lp, db, cfg) |
458 | - |
459 | - for url in urls: |
460 | - for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems(): |
461 | - dbg('lp_import(): downloading %s from %s' % (bp, url)) |
462 | - contents = urllib.urlopen(url).read().decode('UTF-8') |
463 | - if lp_import_blueprint(db, bp, url, contents): |
464 | - lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release')) |
465 | + lp_import_milestones(extra_project, db) |
466 | + projects.append(extra_project) |
467 | + |
468 | + for project in projects: |
469 | + # XXX: should this be valid_ or all_specifications? |
470 | + for bp in lp_project.valid_specifications: |
471 | + dbg('lp_import(): downloading %s from %s' % (bp.name, bp.self_link)) |
472 | + if lp_import_blueprint(db, bp): |
473 | + lp_import_blueprint_workitems(lp, db, bp, cfg.get('release')) |
474 | lp_import_bug_workitems(lp_project, db, cfg) |
475 | |
476 | ######################################################################## |
Upon running the code as-is, I received a failure rather quickly:
clint@clint- MacBookPro: ~/src/wi/ bzr/trunk$ ./collect -d ../natty.db -c ../natty.cfg --debug milestones( ): milestone table already filled n-automated- testing from https:/ /api.launchpad. net/devel/ ubuntu/ +spec/cloud- server- n-automated- testing blueprint( cloud-server- n-automated- testing) : finished parsing; data: {'status': 'Session for discussion about automated testing of Ubuntu Server; automated ISO testing was implemented for Maverick using libvirt/kvm + Hudson (see http:// launchpad. net/ubuntu- server- iso-testing). This approach could be applied in other variants and for other aspects of server testing', 'definition': 'Approved', 'implementation': 'Started', 'milestone': 'natty-alpha-3', 'approver': 'robbie.w', 'details_url': None, 'priority': 'Essential', 'assignee': 'james-page', 'roadmap_notes': None, 'drafter': 'james-page'} blueprint_ workitems( ): processing cloud-server- n-automated- testing (spec milestone: natty-alpha-3, spec assignee: james-page, spec implementation: Started) blueprint_ workitems( ): starting work items block at Work items for natty-alpha-2: server- iso-testing and locate in PPA: INPROGRESS' server- iso-testing and locate in PPA: INPROGRESS' blueprint_ workitems( ): closing work items block with line: blueprint_ workitems( ): starting work items block at Work items for natty-alpha-3: /launchpad. net/ubuntu/ +spec/cloud- server- n-automated- testing
lp_import_
lp_import_teams(): teams table already filled
lp_import(): downloading cloud-server-
lp_import_
lp_import_
lp_import_
... setting milestone to natty-alpha-2
workitem (raw): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
workitem (clean): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
workitem (raw): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
workitem (clean): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
workitem (raw): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
workitem (clean): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
workitem (raw): '[james-page] Package ubuntu-
workitem (clean): '[james-page] Package ubuntu-
workitem (raw): ''
lp_import_
lp_import_
... setting milestone to natty-alpha-3
workitem (raw): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
workitem (clean): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
workitem (raw): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
workitem (clean): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
workitem (raw): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
workitem (clean): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
https:/
[WARNING] assignee "cr2" is not a valid Launchpad account
workitem (raw): '[cr2] Checkbox plugin to download tests from couchdb to ex...