Merge lp:~nigelbabu/summit/stop-screen-scrape into lp:summit

[Nigel Babu]

Stop the screen scape and use the json API instead.

Stop the screen scape and use the json API instead.

[James Westby (james-w)]

Hi,

The assumptions this makes about the API aren't great, but it's
only really as fragile as what was there before, so there's
no reason it shouldn't go in.

Thanks,

James

review: Approve

=== modified file 'summit/schedule/models/meetingmodel.py'
--- summit/schedule/models/meetingmodel.py	2011-05-22 19:14:22 +0000
+++ summit/schedule/models/meetingmodel.py	2011-06-05 14:25:54 +0000
@@ -18,14 +18,13 @@
 from urlparse import urlparse
 from datetime import datetime
 
-from BeautifulSoup import BeautifulSoup
-
 from django.db import models
 from django.core.exceptions import ObjectDoesNotExist
 from django.conf import settings
 from django.core.urlresolvers import reverse
 #from summit.brainstorm.models import Idea
 
+import simplejson as json
 from summit.schedule.fields import NameField
 
 from summit.schedule.models.summitmodel import Summit
@@ -204,19 +203,13 @@
             self.spec_url = spec_url
 
             # fetch to update the title
-            # if only Launchpad had some kind of API which meant we didn't
-            # have to screenscrape... :-/
+            # Now, LP has an API, we're stopping the sceen scrape :)
             try:
-                lpdata = urllib2.urlopen(spec_url)
-                soup = BeautifulSoup(lpdata)
-                title = soup('h1')[0]('span')[0].string.replace(
-                    "&lt;", "<").replace(
-                        "&gt;", ">").replace(
-                            "&apos;", "'").replace(
-                                "&quot;", '"').replace(
-                                    "&amp;", "&").rstrip()
-                if title:
-                    self.title = title[:100]
+                apiurl = spec_url.replace('blueprints.launchpad.net', 'api.launchpad.net/devel') + '?ws.accept=application/json'
+                lpdata = urllib2.urlopen(apiurl)
+                data = json.loads(lpdata)
+                if 'title' in data:
+                    self.title = data['title'][:100]
             except (urllib2.HTTPError, KeyError):
                 pass
         else: