Merge lp:~james-w/launchpad-work-items-tracker/blueprints-api into lp:launchpad-work-items-tracker
- blueprints-api
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~james-w/launchpad-work-items-tracker/blueprints-api |
Merge into: | lp:launchpad-work-items-tracker |
Diff against target: |
476 lines (+120/-199) 1 file modified
collect (+120/-199) |
To merge this branch: | bzr merge lp:~james-w/launchpad-work-items-tracker/blueprints-api |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jamie Bennett (community) | Needs Information | ||
Clint Byrum (community) | Needs Resubmitting | ||
Review via email: mp+43136@code.launchpad.net |
This proposal has been superseded by a proposal from 2010-12-09.
Commit message
Description of the change
Hi,
This makes use of the newly exposed blueprints API on Launchpad, reducing
the amount of screen scraping, round trips, and data transferred enormously.
The only regression that I think this will cause is that when someone writes
"bug 12345" in the whiteboard, it will no longer be a link to that bug
when viewed on the workitems page, but I think we can live with that for
now. If you disagree let me know and I will code it.
Apologies that the diff is a little large and not particularly clear, but
I couldn't think of a good way to migrate a bit at a time.
Thanks,
James
- 250. By Clint Byrum
-
merging extra project support from James Westby
- 251. By Martin Pitt
-
fix previous commit to not crash
Jamie Bennett (jamiebennett) wrote : | # |
On testing the code I was prompted to log in using open id (from the command-line). As this was tested via ssh to a development box 'Links' was launched as the web browser to gather my open id information. From there you cannot log in (Launchpad/Links issue). This did not happen on previous versions.
- 252. By Martin Pitt
-
fix previous commit properly, thanks James
- 253. By Martin Pitt
-
revert extra-projects merge for now; this is creating a conceptual conflict on milestones, and should rather be handled in separate configurations for linaro
- 254. By Martin Pitt
-
consider fixed bugs in bug list based reports as well
- 255. By Martin Pitt
-
add "Unknown" bug state
- 256. By Martin Pitt
-
drop Unknown bug state, does not exist any more
- 257. By Martin Pitt
-
better fix for "Unknown" task status
- 258. By Martin Pitt
-
finally fix unknown bug states
- 259. By Martin Pitt
-
html-report: Add textual stats over time
- 260. By Martin Pitt
-
html-report: sort dates
- 261. By Martin Pitt
-
fix time-stats todo column
- 262. By Martin Pitt
-
time-stats: not all states exist always
- 263. By Martin Pitt
-
collect: add "Expired" bug state
- 264. By James Westby
-
Merge blueprints API branch.
Unmerged revisions
Preview Diff
1 | === modified file 'collect' | |||
2 | --- collect 2010-12-09 13:58:55 +0000 | |||
3 | +++ collect 2010-12-09 15:54:09 +0000 | |||
4 | @@ -3,7 +3,7 @@ | |||
5 | 3 | # Pull work items from various sources (blueprint whiteboards, linked blueprint | 3 | # Pull work items from various sources (blueprint whiteboards, linked blueprint |
6 | 4 | # bugs, wiki pages) and put them into a database. | 4 | # bugs, wiki pages) and put them into a database. |
7 | 5 | 5 | ||
9 | 6 | import urllib, re, sys, optparse, smtplib, pwd, os | 6 | import urllib, re, sys, optparse, smtplib, pwd, os, urlparse |
10 | 7 | from email.mime.text import MIMEText | 7 | from email.mime.text import MIMEText |
11 | 8 | import sqlite3 as dbapi2 | 8 | import sqlite3 as dbapi2 |
12 | 9 | 9 | ||
13 | @@ -54,51 +54,20 @@ | |||
14 | 54 | dbg('Queueing data error: ' + s) | 54 | dbg('Queueing data error: ' + s) |
15 | 55 | 55 | ||
16 | 56 | 56 | ||
17 | 57 | def web_link(item): | ||
18 | 58 | """Get a link to the Launchpad API object on the website.""" | ||
19 | 59 | api_link = item.self_link | ||
20 | 60 | parts = urlparse.urlparse(api_link) | ||
21 | 61 | return parts.scheme + "://" + parts.netloc.replace("api.", "") + "/" + parts.path.split("/", 2)[2] | ||
22 | 62 | |||
23 | 63 | |||
24 | 57 | ######################################################################## | 64 | ######################################################################## |
25 | 58 | # | 65 | # |
26 | 59 | # Functions for parsing Launchpad data | 66 | # Functions for parsing Launchpad data |
27 | 60 | # | 67 | # |
28 | 61 | ######################################################################## | 68 | ######################################################################## |
29 | 62 | 69 | ||
69 | 63 | def lp_blueprints_from_list(url, name_pattern = None): | 70 | def lp_import_blueprint(db, bp): |
31 | 64 | '''Return blueprints from a LP list view. | ||
32 | 65 | |||
33 | 66 | This can optionally specify a name pattern (which is mostly useful for | ||
34 | 67 | testing and debugging, to only parse a small subset of blueprints) | ||
35 | 68 | |||
36 | 69 | Return a dictionary name -> (url, milestone). | ||
37 | 70 | ''' | ||
38 | 71 | blueprint_name_filter = re.compile('href="(/[a-zA-Z0-9-]+/\+spec/%s[^"]+)"(?!.*class.*sprite)' % | ||
39 | 72 | (name_pattern or '.')) | ||
40 | 73 | milestone_re = re.compile('href=".*/[a-zA-Z0-9-]+/\+milestone/([^"]+)"') | ||
41 | 74 | |||
42 | 75 | result = {} | ||
43 | 76 | scan_tr_end = False | ||
44 | 77 | milestone = None | ||
45 | 78 | bp = None | ||
46 | 79 | dbg("lp_blueprints_from_list(): Loading '%s'" % url) | ||
47 | 80 | for l in urllib.urlopen(url): | ||
48 | 81 | if scan_tr_end: | ||
49 | 82 | m = milestone_re.search(l) | ||
50 | 83 | if m: | ||
51 | 84 | milestone = m.group(1) | ||
52 | 85 | dbg('lp_blueprints_from_list(): current spec milestone: ' + milestone) | ||
53 | 86 | if '</tr>' in l: | ||
54 | 87 | scan_tr_end = False | ||
55 | 88 | if bp: | ||
56 | 89 | result[bp.split('/')[-1]] = (bp, milestone) | ||
57 | 90 | bp = None | ||
58 | 91 | milestone = None | ||
59 | 92 | else: | ||
60 | 93 | m = blueprint_name_filter.search(l) | ||
61 | 94 | if m: | ||
62 | 95 | bp = report_tools.blueprints_base_url + m.group(1) | ||
63 | 96 | dbg('lp_blueprints_from_list(): found BP: ' + bp) | ||
64 | 97 | scan_tr_end = True | ||
65 | 98 | |||
66 | 99 | return result | ||
67 | 100 | |||
68 | 101 | def lp_import_blueprint(db, name, url, contents): | ||
70 | 102 | '''Import details of a blueprint into the specs table. | 71 | '''Import details of a blueprint into the specs table. |
71 | 103 | 72 | ||
72 | 104 | If a blueprint does not have a status, use the description. | 73 | If a blueprint does not have a status, use the description. |
73 | @@ -112,56 +81,50 @@ | |||
74 | 112 | cur.execute('SELECT name FROM milestones') | 81 | cur.execute('SELECT name FROM milestones') |
75 | 113 | valid_milestones = [m[0] for m in cur] + [None] | 82 | valid_milestones = [m[0] for m in cur] + [None] |
76 | 114 | 83 | ||
90 | 115 | queries = { | 84 | def get_whiteboard_section(heading): |
91 | 116 | 'priority': '<dt>Priority:</dt>\s*<dd>\s*([^\n]*)', | 85 | if bp.whiteboard is None: |
92 | 117 | 'definition': '<dt>Definition:</dt>\s*<dd>\s*([^\n]*)', | 86 | return None |
93 | 118 | 'implementation': '<dt>Implementation:</dt>\s*<dd>\s*([^\n]*)', | 87 | regex = '^' + heading + ':\s*\n(.*?)\n\n' |
94 | 119 | 'approver': '<dt>Approver:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', | 88 | m = re.search(regex, bp.whiteboard, re.S | re.I | re.M) |
95 | 120 | 'drafter': '<dt>Drafter:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', | 89 | if m is not None: |
96 | 121 | 'assignee': '<dt>Assignee:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/~([a-zA-Z0-9_-]+)" ', | 90 | section = m.group(1).strip() |
97 | 122 | 'milestone': '<dt>Milestone target:</dt>\s*<dd>\s*<a href="https://.*?launchpad.net/\w+/\+milestone/([a-zA-Z0-9_\.-]+)"', | 91 | else: |
98 | 123 | 'description': '<div class="top-portlet">\s*<p>(.*?)</p>', | 92 | section = None |
99 | 124 | 'status': '(?:<p>|^)[Ss]tatus:\s*<br />(.*?)</p>', | 93 | return section |
87 | 125 | 'details_url': 'href="([^"]*)">Read the full specification</a>', | ||
88 | 126 | 'roadmap_notes': '(?:<p>|^)[Rr]oadmap\s+[Nn]otes:\s*<br />(.*?)</p>', | ||
89 | 127 | } | ||
100 | 128 | 94 | ||
101 | 129 | data = {} | 95 | data = {} |
112 | 130 | for key, r in queries.iteritems(): | 96 | data['priority'] = bp.priority |
113 | 131 | m = re.search(r, contents, re.S) | 97 | data['definition'] = bp.definition_status |
114 | 132 | if not m: | 98 | data['implementation'] = bp.implementation_status |
115 | 133 | dbg('lp_import_blueprint(%s): did not find a match for key %s' % (name, key)) | 99 | data['approver'] = bp.approver and bp.approver.name or None |
116 | 134 | if key == 'description': | 100 | data['drafter'] = bp.drafter and bp.drafter.name or None |
117 | 135 | data[key] = '(no description)' | 101 | data['assignee'] = bp.assignee and bp.assignee.name or None |
118 | 136 | else: | 102 | data['milestone'] = bp.milestone and bp.milestone.name or None |
119 | 137 | data[key] = None | 103 | data['description'] = bp.summary or "(no description)" |
120 | 138 | else: | 104 | data['status'] = get_whiteboard_section('Status') |
121 | 139 | data[key] = m.group(1).strip() | 105 | data['details_url'] = bp.specification_url |
122 | 106 | data['roadmap_notes'] = get_whiteboard_section('Roadmap\s+Notes') | ||
123 | 140 | 107 | ||
124 | 141 | # ignore "later" milestone" | 108 | # ignore "later" milestone" |
125 | 142 | if data['milestone'] == 'later': | 109 | if data['milestone'] == 'later': |
127 | 143 | dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % name) | 110 | dbg('lp_import_blueprint(%s): spec has "later" milestone, ignoring it' % bp.name) |
128 | 144 | return False | 111 | return False |
129 | 145 | 112 | ||
130 | 146 | if data['milestone'] not in valid_milestones: | 113 | if data['milestone'] not in valid_milestones: |
132 | 147 | data_error(url, 'milestone "%s" is unknown/invalid' % data['milestone'], True) | 114 | data_error(web_link(bp), 'milestone "%s" is unknown/invalid' % data['milestone'], True) |
133 | 148 | 115 | ||
134 | 149 | # if we have an explicit status: in the whiteboard, use that; otherwise use | 116 | # if we have an explicit status: in the whiteboard, use that; otherwise use |
135 | 150 | # description as status text | 117 | # description as status text |
136 | 151 | if data['status']: | 118 | if data['status']: |
140 | 152 | data['status'] = data['status'].replace('<br />', | 119 | data['status'] = data['status'].strip() |
138 | 153 | '\n').replace('</div>', '').replace(' ', ' ').replace( | ||
139 | 154 | '<wbr></wbr>', '').strip() | ||
141 | 155 | else: | 120 | else: |
145 | 156 | data['status'] = data['description'].replace('<br />', | 121 | data['status'] = data['description'].strip() |
143 | 157 | '\n').replace('</div>', '').replace(' ', ' ').replace( | ||
144 | 158 | '<wbr></wbr>', '').strip() | ||
146 | 159 | del data['description'] | 122 | del data['description'] |
147 | 160 | 123 | ||
149 | 161 | dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (name, str(data))) | 124 | dbg('lp_import_blueprint(%s): finished parsing; data: %s' % (bp.name, str(data))) |
150 | 162 | 125 | ||
151 | 163 | db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)', | 126 | db.cursor().execute('INSERT INTO specs VALUES (?, ?, ?, ?, ?, NULL, ?, ?, ?, ?, ?, ?,?)', |
153 | 164 | (name, url, data['priority'], data['implementation'], | 127 | (bp.name, web_link(bp), data['priority'], data['implementation'], |
154 | 165 | data['assignee'], data['status'], data['milestone'], | 128 | data['assignee'], data['status'], data['milestone'], |
155 | 166 | data['definition'], data['drafter'], data['approver'], | 129 | data['definition'], data['drafter'], data['approver'], |
156 | 167 | data['details_url'], data['roadmap_notes'])) | 130 | data['details_url'], data['roadmap_notes'])) |
157 | @@ -173,10 +136,7 @@ | |||
158 | 173 | 136 | ||
159 | 174 | ''' | 137 | ''' |
160 | 175 | 138 | ||
165 | 176 | ### cargo culted from parse_blueprint_workitem | 139 | line = line.strip() |
162 | 177 | line = line.replace('<br />', '').replace('</div>', '').replace('</p>', | ||
163 | 178 | '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( | ||
164 | 179 | '<a href="/', '<a href="https://launchpad.net/').strip() | ||
166 | 180 | if not line: | 140 | if not line: |
167 | 181 | return | 141 | return |
168 | 182 | 142 | ||
169 | @@ -196,14 +156,9 @@ | |||
170 | 196 | db.commit() | 156 | db.commit() |
171 | 197 | 157 | ||
172 | 198 | def parse_complexity_item(lp, db, line, bp_name, bp_url, def_milestone, def_assignee): | 158 | def parse_complexity_item(lp, db, line, bp_name, bp_url, def_milestone, def_assignee): |
179 | 199 | 159 | line = line.strip() | |
180 | 200 | line = line.replace('<br />', '').replace('</div>', '').replace('</p>', | 160 | # remove special characters people tend to type |
175 | 201 | '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( | ||
176 | 202 | '<a href="/', '<a href="https://launchpad.net/').strip() | ||
177 | 203 | |||
178 | 204 | # remove special characters people tend to type | ||
181 | 205 | line = re.sub('[^\w -.]', '', line) | 161 | line = re.sub('[^\w -.]', '', line) |
182 | 206 | |||
183 | 207 | if not line: | 162 | if not line: |
184 | 208 | return | 163 | return |
185 | 209 | 164 | ||
186 | @@ -214,17 +169,15 @@ | |||
187 | 214 | assignee = None | 169 | assignee = None |
188 | 215 | 170 | ||
189 | 216 | try: | 171 | try: |
192 | 217 | list = line.split() | 172 | complexity_list = line.split() |
193 | 218 | list.reverse() | 173 | complexity_list.reverse() |
194 | 219 | 174 | ||
195 | 220 | # we may not have any values in the list, so append our | 175 | # we may not have any values in the list, so append our |
196 | 221 | # default values in the right order so they can be mapped | 176 | # default values in the right order so they can be mapped |
197 | 222 | defs = [None, def_milestone, def_assignee] | 177 | defs = [None, def_milestone, def_assignee] |
203 | 223 | for i in range(len(list), len(defs)): | 178 | for i in range(len(complexity_list), len(defs)): |
204 | 224 | list.append(defs[i]) | 179 | complexity_list.append(defs[i]) |
205 | 225 | 180 | (num, milestone, assignee) = complexity_list | |
201 | 226 | (num, milestone, assignee) = list | ||
202 | 227 | |||
206 | 228 | if not num: | 181 | if not num: |
207 | 229 | data_error(bp_url, 'No complexity points defined for %s' % line) | 182 | data_error(bp_url, 'No complexity points defined for %s' % line) |
208 | 230 | 183 | ||
209 | @@ -233,17 +186,17 @@ | |||
210 | 233 | cur = db.cursor() | 186 | cur = db.cursor() |
211 | 234 | cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))', | 187 | cur.execute('INSERT INTO complexity VALUES(?,?,?,?,date(CURRENT_TIMESTAMP))', |
212 | 235 | (assignee, num, milestone, bp_name)) | 188 | (assignee, num, milestone, bp_name)) |
213 | 236 | |||
214 | 237 | except ValueError: | 189 | except ValueError: |
215 | 238 | data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError)) | 190 | data_error(bp_url, "\tComplexity line '%s' could not be parsed %s" % (line, ValueError)) |
216 | 239 | 191 | ||
217 | 192 | |||
218 | 240 | def parse_blueprint_workitem(line, default_assignee, milestone, | 193 | def parse_blueprint_workitem(line, default_assignee, milestone, |
219 | 241 | blueprint_url, launchpad, result_list): | 194 | blueprint_url, launchpad, result_list): |
220 | 242 | '''Parse a work item line of a blueprint whiteboard.''' | 195 | '''Parse a work item line of a blueprint whiteboard.''' |
221 | 243 | 196 | ||
225 | 244 | line = line.replace('<br />', '').replace('</div>', '').replace('</p>', | 197 | # FIXME: we lose bug linking etc. that is done by the tales |
226 | 245 | '').replace('<wbr></wbr>', '').replace(' ', ' ').replace( | 198 | # formatters in LP here. |
227 | 246 | '<a href="/', '<a href="https://launchpad.net/').strip() | 199 | line = line.strip() |
228 | 247 | if not line: | 200 | if not line: |
229 | 248 | return | 201 | return |
230 | 249 | dbg("\tworkitem (clean): '%s'" % (line)) | 202 | dbg("\tworkitem (clean): '%s'" % (line)) |
231 | @@ -286,13 +239,10 @@ | |||
232 | 286 | 239 | ||
233 | 287 | result_list.append((desc, state, assignee, milestone)) | 240 | result_list.append((desc, state, assignee, milestone)) |
234 | 288 | 241 | ||
236 | 289 | def follow_blueprint_buglink(bugnum, default_assignee, blueprint_name, | 242 | def follow_blueprint_buglink(bug, default_assignee, blueprint_name, |
237 | 290 | launchpad, release, default_milestone, result_list): | 243 | launchpad, release, default_milestone, result_list): |
238 | 291 | '''Query launchpad for the information on a linked bug''' | 244 | '''Query launchpad for the information on a linked bug''' |
239 | 292 | 245 | ||
240 | 293 | bugnum = int(bugnum) | ||
241 | 294 | dbg('follow_blueprint_buglink(): processing bug %i (default milestone: %s)' % (bugnum, default_milestone)) | ||
242 | 295 | bug = launchpad.bugs[bugnum] | ||
243 | 296 | for task in bug.bug_tasks: | 246 | for task in bug.bug_tasks: |
244 | 297 | state = bug_wi_states.get(task.status, 'todo') | 247 | state = bug_wi_states.get(task.status, 'todo') |
245 | 298 | if state is None: | 248 | if state is None: |
246 | @@ -329,7 +279,7 @@ | |||
247 | 329 | if better_task: | 279 | if better_task: |
248 | 330 | continue | 280 | continue |
249 | 331 | 281 | ||
251 | 332 | desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bugnum, bugnum) + bug.title | 282 | desc = '<a href="https://launchpad.net/bugs/%d">LP: #%d</a>: ' % (bug.id, bug.id) + bug.title |
252 | 333 | if rtype != 'distribution': | 283 | if rtype != 'distribution': |
253 | 334 | desc += ' (%s)' % target.name | 284 | desc += ' (%s)' % target.name |
254 | 335 | 285 | ||
255 | @@ -354,32 +304,27 @@ | |||
256 | 354 | return word | 304 | return word |
257 | 355 | return None | 305 | return None |
258 | 356 | 306 | ||
260 | 357 | def lp_import_blueprint_workitems(lp, db, bp_name, bp_url, contents, release): | 307 | def lp_import_blueprint_workitems(lp, db, bp, release): |
261 | 358 | '''Collect work items from a Launchpad blueprint. | 308 | '''Collect work items from a Launchpad blueprint. |
262 | 359 | 309 | ||
263 | 360 | This includes work items from the whiteboard as well as linked bugs. | 310 | This includes work items from the whiteboard as well as linked bugs. |
264 | 361 | ''' | 311 | ''' |
270 | 362 | linked_bugs_re = re.compile('<div id="bug_links".*>', re.I) | 312 | work_items_re = re.compile('^work items(.*)\s*:\s*$', re.I) |
271 | 363 | bugnum_re = re.compile('<a href="https://bugs\..*launchpad\.net/bugs/([0-9]+)" class="sprite bug">') | 313 | meta_re = re.compile('^Meta.*?:$', re.I) |
272 | 364 | work_items_re = re.compile('(<p>|^)work items(.*)\s*:\s*<br />', re.I) | 314 | complexity_re = re.compile('^Complexity.*?:$', re.I) |
268 | 365 | meta_re = re.compile( '(<p>|^)Meta.*?:<br />', re.I ) | ||
269 | 366 | complexity_re = re.compile( '(<p>|^)Complexity.*?:<br />', re.I ) | ||
273 | 367 | 315 | ||
274 | 368 | in_workitems_block = False | 316 | in_workitems_block = False |
275 | 369 | in_linked_bugs_block = False | ||
276 | 370 | in_meta_block = False | 317 | in_meta_block = False |
277 | 371 | in_complexity_block = False | 318 | in_complexity_block = False |
278 | 372 | found_linked_bugs = False | ||
279 | 373 | work_items = [] | 319 | work_items = [] |
280 | 374 | found_wb_workitems = False | ||
281 | 375 | milestone = None | 320 | milestone = None |
282 | 376 | 321 | ||
283 | 377 | cur = db.cursor() | 322 | cur = db.cursor() |
285 | 378 | cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp_name,)) | 323 | cur.execute('SELECT assignee, milestone, implementation FROM specs WHERE name = ?', (bp.name,)) |
286 | 379 | (spec_assignee, spec_milestone, spec_implementation) = cur.fetchone() | 324 | (spec_assignee, spec_milestone, spec_implementation) = cur.fetchone() |
287 | 380 | 325 | ||
288 | 381 | dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % ( | 326 | dbg('lp_import_blueprint_workitems(): processing %s (spec milestone: %s, spec assignee: %s, spec implementation: %s)' % ( |
290 | 382 | bp_name, spec_milestone, spec_assignee, spec_implementation)) | 327 | bp.name, spec_milestone, spec_assignee, spec_implementation)) |
291 | 383 | 328 | ||
292 | 384 | cur.execute('SELECT team FROM teams WHERE name = ?', (spec_assignee,)) | 329 | cur.execute('SELECT team FROM teams WHERE name = ?', (spec_assignee,)) |
293 | 385 | assignee_teams = [t[0] for t in cur] | 330 | assignee_teams = [t[0] for t in cur] |
294 | @@ -387,69 +332,51 @@ | |||
295 | 387 | cur.execute('SELECT name FROM milestones') | 332 | cur.execute('SELECT name FROM milestones') |
296 | 388 | valid_milestones = [m[0] for m in cur] | 333 | valid_milestones = [m[0] for m in cur] |
297 | 389 | 334 | ||
361 | 390 | for l in contents.splitlines(): | 335 | if bp.whiteboard: |
362 | 391 | 336 | for l in bp.whiteboard.splitlines(): | |
363 | 392 | if not in_workitems_block and not in_linked_bugs_block \ | 337 | |
364 | 393 | and not in_meta_block and not in_complexity_block: | 338 | if (not in_workitems_block |
365 | 394 | m = work_items_re.search(l) | 339 | and not in_meta_block and not in_complexity_block): |
366 | 395 | if m: | 340 | m = work_items_re.search(l) |
367 | 396 | in_workitems_block = True | 341 | if m: |
368 | 397 | found_wb_workitems = True | 342 | in_workitems_block = True |
369 | 398 | dbg('lp_import_blueprint_workitems(): starting work items block at ' + l) | 343 | dbg('lp_import_blueprint_workitems(): starting work items block at ' + l) |
370 | 399 | if not milestone: | 344 | if not milestone: |
371 | 400 | milestone = milestone_extract(m.group(2), valid_milestones) | 345 | milestone = milestone_extract(m.group(1), valid_milestones) |
372 | 401 | dbg(' ... setting milestone to ' + str(milestone)) | 346 | dbg(' ... setting milestone to ' + str(milestone)) |
373 | 402 | if linked_bugs_re.search(l): | 347 | if meta_re.search(l): |
374 | 403 | in_linked_bugs_block = True | 348 | in_meta_block = True |
375 | 404 | found_linked_bugs = True | 349 | if complexity_re.search(l): |
376 | 405 | if meta_re.search(l): | 350 | in_complexity_block = True |
377 | 406 | in_meta_block = True | 351 | continue |
378 | 407 | if complexity_re.search(l): | 352 | |
379 | 408 | in_complexity_block = True | 353 | if in_workitems_block: |
380 | 409 | continue | 354 | dbg("\tworkitem (raw): '%s'" % (l.strip())) |
381 | 410 | 355 | if not l.strip(): | |
382 | 411 | if in_workitems_block: | 356 | dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l) |
383 | 412 | dbg("\tworkitem (raw): '%s'" % (l.strip())) | 357 | in_workitems_block = False |
384 | 413 | parse_blueprint_workitem(l, spec_assignee, milestone or | 358 | milestone = None |
385 | 414 | spec_milestone, bp_url, lp, work_items) | 359 | parse_blueprint_workitem(l, spec_assignee, milestone or |
386 | 415 | 360 | spec_milestone, web_link(bp), lp, work_items) | |
387 | 416 | if '</p>' in l: | 361 | |
388 | 417 | dbg('lp_import_blueprint_workitems(): closing work items block with line: ' + l) | 362 | if in_meta_block: |
389 | 418 | in_workitems_block = False | 363 | dbg("\tmeta line (raw): '%s'" % (l.strip())) |
390 | 419 | milestone = None | 364 | if not l.strip(): |
391 | 420 | 365 | in_meta_block = False | |
392 | 421 | if in_linked_bugs_block: | 366 | continue |
393 | 422 | dbg("\tbug block line (raw): '%s'" % (l.strip())) | 367 | parse_meta_item(lp, db, l, bp.name) |
394 | 423 | m = bugnum_re.search(l) | 368 | |
395 | 424 | if m and lp: | 369 | if in_complexity_block: |
396 | 425 | follow_blueprint_buglink(m.group(1), spec_assignee, | 370 | dbg("\tcomplexity block line (raw): '%s'" % (l.strip())) |
397 | 426 | bp_name, lp, release, | 371 | if not l.strip(): |
398 | 427 | milestone or spec_milestone, work_items) | 372 | in_complexity_block = False |
399 | 428 | 373 | continue | |
400 | 429 | if '</div>' in l: | 374 | parse_complexity_item(lp, db, l, bp.name, web_link(bp), spec_milestone, spec_assignee) |
401 | 430 | in_linked_bugs_block = False | 375 | |
402 | 431 | continue | 376 | if lp: |
403 | 432 | 377 | for bug in bp.bugs: | |
404 | 433 | if in_meta_block: | 378 | follow_blueprint_buglink(bug, spec_assignee, bp.name, lp, release, |
405 | 434 | dbg("\tmeta line (raw): '%s'" % (l.strip())) | 379 | spec_milestone, work_items) |
343 | 435 | parse_meta_item( lp, db, l, bp_name ) | ||
344 | 436 | |||
345 | 437 | # done with the meta information? | ||
346 | 438 | if '</p>' in l: | ||
347 | 439 | in_meta_block = False | ||
348 | 440 | continue | ||
349 | 441 | |||
350 | 442 | if in_complexity_block: | ||
351 | 443 | dbg("\tcomplexity block line (raw): '%s'" % (l.strip())) | ||
352 | 444 | parse_complexity_item(lp, db, l, bp_name, bp_url, spec_milestone, spec_assignee) | ||
353 | 445 | |||
354 | 446 | # done with the meta information? | ||
355 | 447 | if '</p>' in l: | ||
356 | 448 | in_complexity_block = False | ||
357 | 449 | continue | ||
358 | 450 | |||
359 | 451 | if found_wb_workitems and '</div>' in l: | ||
360 | 452 | break | ||
406 | 453 | 380 | ||
407 | 454 | if not work_items: | 381 | if not work_items: |
408 | 455 | #data_error(bp_url, 'no work items defined', True) | 382 | #data_error(bp_url, 'no work items defined', True) |
409 | @@ -465,7 +392,8 @@ | |||
410 | 465 | 392 | ||
411 | 466 | for (desc, status, assignee, milestone) in work_items: | 393 | for (desc, status, assignee, milestone) in work_items: |
412 | 467 | db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))', | 394 | db.cursor().execute('INSERT INTO work_items VALUES (?, ?, ?, ?, ?, date(CURRENT_TIMESTAMP))', |
414 | 468 | (desc, bp_name, status, assignee, milestone)) | 395 | (desc, bp.name, status, assignee, milestone)) |
415 | 396 | |||
416 | 469 | 397 | ||
417 | 470 | def lp_import_milestones(lp_project, db): | 398 | def lp_import_milestones(lp_project, db): |
418 | 471 | '''Import milestones from Launchpad into DB. | 399 | '''Import milestones from Launchpad into DB. |
419 | @@ -544,40 +472,33 @@ | |||
420 | 544 | def lp_import(db, cfg, name_pattern = None): | 472 | def lp_import(db, cfg, name_pattern = None): |
421 | 545 | '''Collect blueprint work items and status from Launchpad into DB.''' | 473 | '''Collect blueprint work items and status from Launchpad into DB.''' |
422 | 546 | 474 | ||
432 | 547 | lp = Launchpad.login_with('ubuntu-work-items', service_root=EDGE_SERVICE_ROOT) | 475 | lp = Launchpad.login_with('ubuntu-work-items', service_root="production", version="devel") |
433 | 548 | 476 | ||
434 | 549 | urls = [] | 477 | projects = [] |
426 | 550 | |||
427 | 551 | def import_project(project): | ||
428 | 552 | lp_import_milestones(project, db) | ||
429 | 553 | url = '%s/%s/+specs?batch=300' % ( | ||
430 | 554 | report_tools.blueprints_base_url, project.name) | ||
431 | 555 | urls.append(url) | ||
435 | 556 | 478 | ||
436 | 557 | if 'release' in cfg: | 479 | if 'release' in cfg: |
437 | 558 | lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release']) | 480 | lp_project = lp.distributions['ubuntu'].getSeries(name_or_version=cfg['release']) |
441 | 559 | url = '%s/ubuntu/%s/+specs?batch=300' % ( | 481 | projects.append(lp_project) |
439 | 560 | report_tools.blueprints_base_url, cfg['release']) | ||
440 | 561 | urls.append(url) | ||
442 | 562 | else: | 482 | else: |
443 | 563 | assert 'project' in cfg, 'Configuration needs to specify project or release' | 483 | assert 'project' in cfg, 'Configuration needs to specify project or release' |
444 | 564 | lp_project = lp.projects[cfg['project']] | 484 | lp_project = lp.projects[cfg['project']] |
446 | 565 | import_project(lp_project) | 485 | projects.append(lp_project) |
447 | 486 | |||
448 | 487 | lp_import_milestones(lp_project, db) | ||
449 | 488 | lp_import_teams(lp, db, cfg) | ||
450 | 566 | 489 | ||
451 | 567 | extra_projects = cfg.get('extra_projects', []) | 490 | extra_projects = cfg.get('extra_projects', []) |
452 | 568 | for extra_project_name in extra_projects: | 491 | for extra_project_name in extra_projects: |
453 | 569 | extra_project = lp.projects[extra_project_name] | 492 | extra_project = lp.projects[extra_project_name] |
465 | 570 | import_project(extra_project) | 493 | lp_import_milestones(extra_project, db) |
466 | 571 | 494 | projects.append(extra_project) | |
467 | 572 | lp_import_milestones(lp_project, db) | 495 | |
468 | 573 | lp_import_teams(lp, db, cfg) | 496 | for project in projects: |
469 | 574 | 497 | # XXX: should this be valid_ or all_specifications? | |
470 | 575 | for url in urls: | 498 | for bp in lp_project.valid_specifications: |
471 | 576 | for (bp, (url, milestone)) in lp_blueprints_from_list(url, name_pattern).iteritems(): | 499 | dbg('lp_import(): downloading %s from %s' % (bp.name, bp.self_link)) |
472 | 577 | dbg('lp_import(): downloading %s from %s' % (bp, url)) | 500 | if lp_import_blueprint(db, bp): |
473 | 578 | contents = urllib.urlopen(url).read().decode('UTF-8') | 501 | lp_import_blueprint_workitems(lp, db, bp, cfg.get('release')) |
463 | 579 | if lp_import_blueprint(db, bp, url, contents): | ||
464 | 580 | lp_import_blueprint_workitems(lp, db, bp, url, contents, cfg.get('release')) | ||
474 | 581 | lp_import_bug_workitems(lp_project, db, cfg) | 502 | lp_import_bug_workitems(lp_project, db, cfg) |
475 | 582 | 503 | ||
476 | 583 | ######################################################################## | 504 | ######################################################################## |
Upon running the code as-is, I received a failure rather quickly:
clint@clint- MacBookPro: ~/src/wi/ bzr/trunk$ ./collect -d ../natty.db -c ../natty.cfg --debug milestones( ): milestone table already filled n-automated- testing from https:/ /api.launchpad. net/devel/ ubuntu/ +spec/cloud- server- n-automated- testing blueprint( cloud-server- n-automated- testing) : finished parsing; data: {'status': 'Session for discussion about automated testing of Ubuntu Server; automated ISO testing was implemented for Maverick using libvirt/kvm + Hudson (see http:// launchpad. net/ubuntu- server- iso-testing). This approach could be applied in other variants and for other aspects of server testing', 'definition': 'Approved', 'implementation': 'Started', 'milestone': 'natty-alpha-3', 'approver': 'robbie.w', 'details_url': None, 'priority': 'Essential', 'assignee': 'james-page', 'roadmap_notes': None, 'drafter': 'james-page'} blueprint_ workitems( ): processing cloud-server- n-automated- testing (spec milestone: natty-alpha-3, spec assignee: james-page, spec implementation: Started) blueprint_ workitems( ): starting work items block at Work items for natty-alpha-2: server- iso-testing and locate in PPA: INPROGRESS' server- iso-testing and locate in PPA: INPROGRESS' blueprint_ workitems( ): closing work items block with line: blueprint_ workitems( ): starting work items block at Work items for natty-alpha-3: /launchpad. net/ubuntu/ +spec/cloud- server- n-automated- testing
lp_import_
lp_import_teams(): teams table already filled
lp_import(): downloading cloud-server-
lp_import_
lp_import_
lp_import_
... setting milestone to natty-alpha-2
workitem (raw): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
workitem (clean): '[hggdh2] Way forward on production deployment of ISO testing: TODO'
workitem (raw): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
workitem (clean): '[james-page] Move Server ISO tests to normal PXE + TFTP instead for broader fit with potential test architectures: DONE'
workitem (raw): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
workitem (clean): '[james-page] Server ISO test - review what the iso overlay looks like and refactor as required: DONE'
workitem (raw): '[james-page] Package ubuntu-
workitem (clean): '[james-page] Package ubuntu-
workitem (raw): ''
lp_import_
lp_import_
... setting milestone to natty-alpha-3
workitem (raw): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
workitem (clean): '[james-page] Automate EC2 testing and increase depth of image testing using unittest/subunit: TODO'
workitem (raw): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
workitem (clean): '[james-page] Server ISO test - fix concurrency in ISO download: TODO'
workitem (raw): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
workitem (clean): '[cr2] Output plugin for checkbox to write to couchdb: TODO'
https:/
[WARNING] assignee "cr2" is not a valid Launchpad account
workitem (raw): '[cr2] Checkbox plugin to download tests from couchdb to ex...