Merge lp:~marco-gallotta/ibid/flight into lp:~ibid-core/ibid/old-trunk-1.6

Proposed by marcog on 2010-01-07
Status: Merged
Approved by: Jonathan Hitchcock on 2010-01-24
Approved revision: not available
Merged at revision: 855
Proposed branch: lp:~marco-gallotta/ibid/flight
Merge into: lp:~ibid-core/ibid/old-trunk-1.6
Diff against target: 362 lines (+315/-6)
2 files modified
ibid/plugins/geography.py (+301/-3)
ibid/utils/__init__.py (+14/-3)
To merge this branch: bzr merge lp:~marco-gallotta/ibid/flight
Reviewer Review Type Date Requested Status
Jonathan Hitchcock Approve on 2010-01-24
Michael Gorven Approve on 2010-01-09
Stefano Rivera 2010-01-07 Approve on 2010-01-08
Review via email: mp+16964@code.launchpad.net

This proposal supersedes a proposal from 2010-01-07.

To post a comment you must log in.
marcog (marco-gallotta) wrote : Posted in a previous version of this proposal

I think the flight plugin is mature enough for some reviews. It's quite possible some gaping bugs are still out there, but it seems to be robust enough.

Stefano Rivera (stefanor) wrote : Posted in a previous version of this proposal

Whoops, conflicts in the diff.

Stefano Rivera (stefanor) wrote :

+from xml.etree import ElementTree

unused

Query: airport for cape town
Response: Airport code must be 3 or 4 characters
^ You should allow "for" as well as "in"

assume lenght 4
^ comment typo

One way flights?

Query: airports in london
Response: Found the following airports: London (YXU/CYXU), Luton (LTN/EGGW), Gatwick (LGW/EGKK), City (LCY/EGLC), Heathrow (LHR/EGLL), Stansted (STN/EGSS), East London (ELS/FAEL) and Groton New London (GON/KGON)

^ names like "city" are going to be a problem here. Does travelocity have an "all london airports" meta-airport? Most flight search engines I've used do.

Query: cheapest flight from cape town to lhr from 15th to 20th
Response: Departing flight: South African Airways Flight 220 departing 8:10pm from Capetown, South Africa (CPT), arriving 6:55am Sat, Jan 16 at (LHR) (flight time 12hrs 45min, Nonstop) costs $1,421 per person
Response: Returning flight: United Flight 8818 operated by Lufthansa / 9248 operated by Lufthansa departing 5:05pm from London, Great Britain (LHR), arriving 11:20am Thu, Jan 21 at (CPT) (flight time 16hrs 15min, 1 Stop) costs $4,124 per person
Response: Full results: http://travel.travelocity.com/flights/AirSearch.do;jsessionid=37871D566EEB62B4EEA856665138C410.p0263?SEQ=1262869510635072010

^ Don't return 3 responses, do it in one.

Yeah results in local currency would also be awesome, but that can happen in phase 2 :)

review: Needs Fixing
lp:~marco-gallotta/ibid/flight updated on 2010-01-07
860. By marcog on 2010-01-07

Fix typos; allow "airport *for* cape town

861. By marcog on 2010-01-07

Condense responses into one

862. By marcog on 2010-01-07

Remove unused imports

marcog (marco-gallotta) wrote :

> +from xml.etree import ElementTree
>
> unused
>
> Query: airport for cape town
> Response: Airport code must be 3 or 4 characters
> ^ You should allow "for" as well as "in"
>
> assume lenght 4
> ^ comment typo

Fixed all the above

> One way flights?

This is going to add more to an already complex query syntax, so I'd prefer to leave this for phase 2.

> Query: airports in london
> Response: Found the following airports: London (YXU/CYXU), Luton (LTN/EGGW),
> Gatwick (LGW/EGKK), City (LCY/EGLC), Heathrow (LHR/EGLL), Stansted (STN/EGSS),
> East London (ELS/FAEL) and Groton New London (GON/KGON)
>
> ^ names like "city" are going to be a problem here. Does travelocity have an
> "all london airports" meta-airport? Most flight search engines I've used do.

I want to grok on how best to handle this. I'm currently considering that if it matches he entire city name or entire airport name, then return that airport only. However, this could cause problems if it matches but isn't the expected result. I'm currently contemplating putting this on hold for phase 2.

Related to this, we should probably find a working method of matching unicode names with ascii queries (like you and I originally attempted).

> Query: cheapest flight from cape town to lhr from 15th to 20th
> Response: Departing flight: South African Airways Flight 220 departing 8:10pm
> from Capetown, South Africa (CPT), arriving 6:55am Sat, Jan 16 at (LHR)
> (flight time 12hrs 45min, Nonstop) costs $1,421 per person
> Response: Returning flight: United Flight 8818 operated
> by Lufthansa / 9248 operated by Lufthansa departing 5:05pm from London, Great
> Britain (LHR), arriving 11:20am Thu, Jan 21 at (CPT) (flight time
> 16hrs 15min, 1 Stop) costs $4,124 per person
> Response: Full results: http://travel.travelocity.com/flights/AirSearch.do;jse
> ssionid=37871D566EEB62B4EEA856665138C410.p0263?SEQ=1262869510635072010
>
> ^ Don't return 3 responses, do it in one.

Done!

> Yeah results in local currency would also be awesome, but that can happen in
> phase 2 :)

Yep, phase 2. Would be nice to use the currency stuff in the lookup plugin.

lp:~marco-gallotta/ibid/flight updated on 2010-01-07
863. By marcog on 2010-01-07

Make use of etree's findtext and path querying

Stefano Rivera (stefanor) wrote :

I'd merge the airport and flight features.

Otherwise I'm happy

review: Approve
lp:~marco-gallotta/ibid/flight updated on 2010-01-08
864. By marcog on 2010-01-08

Merge flight and airport features

865. By marcog on 2010-01-08

Use dict substitutions for responses

marcog (marco-gallotta) wrote :

> I'd merge the airport and flight features.

Done.

Michael Gorven (mgorven) wrote :

> + date = parse(date)

parse() will throw a ValueError if the date is invalid -- you should catch
this and return a suitable error.

 review approve

review: Approve
lp:~marco-gallotta/ibid/flight updated on 2010-01-09
866. By marcog on 2010-01-09

Catch ValueError thrown by parse() when the date is invalid

marcog (marco-gallotta) wrote :

> > + date = parse(date)
>
> parse() will throw a ValueError if the date is invalid -- you should catch
> this and return a suitable error.

Done

Jonathan Hitchcock (vhata) wrote :

Line 76:

  query = [unicode(q) for q in query.lower().split(' ') if q]

That can just be:

  query = [unicode(q) for q in query.lower().split()]

(This note brought to you from the Minor Quibbles Now, Proper Reviews Later Dept.)

Stefano Rivera (stefanor) wrote :

If Vhata is being pedantic, here are some more pedantisms:

> query = [unicode(q) for q in query.lower().split(' ') if q]

There are a few more of those. Also, that probably doesn't need to be unicode()d.

While you are at it, convert all the unicode-touching string to u'string'

> [d for d in etree.getiterator('div') if d.get(u'class') == 'e_content']

that can be etree.findAll('div', attrs={'class': u'e_content'})

> for td in tr.getiterator('td'):
> if td.get(u'class').strip() in ['tfPrice', 'tfPriceOrButton']:

another very similar one

> table = [t for t in etree.getiterator('table') if t.get(u'id') == 'tfGrid'][0]
> trs = [t for t in table.getiterator('tr')]

and again. and 223

> if priority == 'cheapest':

priority.lower()

lp:~marco-gallotta/ibid/flight updated on 2010-01-12
867. By marcog on 2010-01-12

Make strings unicode in places where they weren't; split(' ') if s -> string.split()

marcog (marco-gallotta) wrote :

> > query = [unicode(q) for q in query.lower().split(' ') if q]
>
> There are a few more of those. Also, that probably doesn't need to be
> unicode()d.
>
> While you are at it, convert all the unicode-touching string to u'string'

Done

> > [d for d in etree.getiterator('div') if d.get(u'class') == 'e_content']
>
> that can be etree.findAll('div', attrs={'class': u'e_content'})
>
> > for td in tr.getiterator('td'):
> > if td.get(u'class').strip() in ['tfPrice', 'tfPriceOrButton']:
>
> another very similar one
>
> > table = [t for t in etree.getiterator('table') if t.get(u'id') ==
> 'tfGrid'][0]
> > trs = [t for t in table.getiterator('tr')]
>
> and again. and 223

As we discussed, etree doesn't a findall that takes an attrs arg.

> > if priority == 'cheapest':
>
> priority.lower()

Done

lp:~marco-gallotta/ibid/flight updated on 2010-01-14
868. By marcog on 2010-01-14

Make parsing a little more resiliant

869. By marcog on 2010-01-14

strftime needs an ascii string for pre-2.6

Jonathan Hitchcock (vhata) wrote :

Please merge trunk's plugin-reorg into this branch, and then merge your plugin into the geography plugin (preserving as much history as you can, please).

review: Needs Fixing
marcog (marco-gallotta) wrote :

> Please merge trunk's plugin-reorg into this branch, and then merge your plugin
> into the geography plugin (preserving as much history as you can, please).

Merged

lp:~marco-gallotta/ibid/flight updated on 2010-01-18
870. By marcog on 2010-01-18

Merge trunk and move flight into geography

Jonathan Hitchcock (vhata) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ibid/plugins/geography.py'
2--- ibid/plugins/geography.py 2010-01-18 18:38:39 +0000
3+++ ibid/plugins/geography.py 2010-01-18 22:43:10 +0000
4@@ -1,5 +1,5 @@
5 from math import acos, sin, cos, radians
6-from urllib import quote
7+from urllib import quote, urlencode
8 from urlparse import urljoin
9 import re
10 import logging
11@@ -8,11 +8,15 @@
12 from os import walk
13 from dateutil.parser import parse
14 from dateutil.tz import gettz, tzlocal, tzoffset
15+import csv
16+from sys import maxint
17+
18+from dateutil.parser import parse
19
20 from ibid.plugins import Processor, match
21-from ibid.utils import json_webservice, human_join, format_date
22+from ibid.utils import json_webservice, human_join, format_date, cacheable_download
23 from ibid.utils.html import get_html_parse_tree
24-from ibid.config import Option, DictOption
25+from ibid.config import Option, DictOption, IntOption
26 from ibid.compat import defaultdict
27
28 log = logging.getLogger('plugins.geography')
29@@ -349,4 +353,298 @@
30 def time(self, event, place):
31 self.convert(event, None, None, place)
32
33+help['flight'] = u'Search for flights on travelocity'
34+class Flight:
35+ def __init__(self):
36+ self.flight, self.depart_time, self.depart_ap, self.arrive_time, \
37+ self.arrive_ap, self.duration, self.stops, self.price = \
38+ [], None, None, None, None, None, None, None
39+
40+ def int_price(self):
41+ try:
42+ return int(self.price[1:])
43+ except ValueError:
44+ return maxint
45+
46+ def int_duration(self):
47+ hours, minutes = 0, 0
48+ match = re.search(r'(\d+)hr', self.duration)
49+ if match:
50+ hours = int(match.group(1))
51+ match = re.search(r'(\d+)min', self.duration)
52+ if match:
53+ minutes = int(match.group(1))
54+ return int(hours)*60 + int(minutes)
55+
56+MONTH_SHORT = ('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec')
57+MONTH_LONG = ('January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December')
58+OTHER_STUFF = ('am', 'pm', 'st', 'nd', 'rd', 'th', 'morning', 'afternoon', 'evening', 'anytime')
59+DATE = r'(?:[0-9.:/hT -]|%s)+' % '|'.join(MONTH_SHORT+MONTH_LONG+OTHER_STUFF)
60+
61+class FlightException(Exception):
62+ pass
63+
64+class FlightSearch(Processor):
65+ """airport [in] <name|location|code>
66+ [<cheapest|quickest>] flight from <departure> to <destination> from <depart_date> [anytime|morning|afternoon|evening|<time>] to <return_date> [anytime|morning|afternoon|evening|<time>]"""
67+
68+ feature = 'flight'
69+
70+ airports_url = u'http://openflights.svn.sourceforge.net/viewvc/openflights/openflights/data/airports.dat'
71+ max_results = IntOption('max_results', 'Maximum number of results to list', 5)
72+
73+ airports = {}
74+
75+ def read_airport_data(self):
76+ # File is listed as ISO 8859-1 (Latin-1) encoded on
77+ # http://openflights.org/data.html, but from decoding it appears to
78+ # actually be UTF8
79+ filename = cacheable_download(self.airports_url, u'flight/airports.dat')
80+ reader = csv.reader(open(filename), delimiter=',', quotechar='"')
81+ for row in reader:
82+ self.airports[int(row[0])] = [unicode(r, u'utf-8') for r in row[1:]]
83+
84+ def _airport_search(self, query, search_loc = True):
85+ if not self.airports:
86+ self.read_airport_data()
87+ if search_loc:
88+ ids = self._airport_search(query, False)
89+ if len(ids) == 1:
90+ return ids
91+ query = [q for q in query.lower().split()]
92+ else:
93+ query = [query.lower()]
94+ ids = []
95+ for id, airport in self.airports.items():
96+ if search_loc:
97+ data = (u' '.join(c.lower() for c in airport[:5])).split()
98+ elif len(query[0]) == 3:
99+ data = [airport[3].lower()]
100+ else: # assume length 4 (won't break if not)
101+ data = [airport[4].lower()]
102+ if len(filter(lambda q: q in data, query)) == len(query):
103+ ids.append(id)
104+ return ids
105+
106+ def repr_airport(self, id):
107+ airport = self.airports[id]
108+ code = u''
109+ if airport[3] or airport[4]:
110+ code = u' (%s)' % u'/'.join(filter(lambda c: c, airport[3:5]))
111+ return u'%s%s' % (airport[0], code)
112+
113+ @match(r'^airports?\s+((?:in|for)\s+)?(.+)$')
114+ def airport_search(self, event, search_loc, query):
115+ search_loc = search_loc is not None
116+ if not search_loc and not 3 <= len(query) <= 4:
117+ event.addresponse(u'Airport code must be 3 or 4 characters')
118+ return
119+ ids = self._airport_search(query, search_loc)
120+ if len(ids) == 0:
121+ event.addresponse(u"Sorry, I don't know that airport")
122+ elif len(ids) == 1:
123+ id = ids[0]
124+ airport = self.airports[id]
125+ code = u'unknown code'
126+ if airport[3] and airport[4]:
127+ code = u'codes %s and %s' % (airport[3], airport[4])
128+ elif airport[3]:
129+ code = u'code %s' % airport[3]
130+ elif airport[4]:
131+ code = u'code %s' % airport[4]
132+ event.addresponse(u'%(airport)s in %(city)s, %(country)s has %(code)s', {
133+ u'airport': airport[0],
134+ u'city': airport[1],
135+ u'country': airport[2],
136+ u'code': code,
137+ })
138+ else:
139+ event.addresponse(u'Found the following airports: %s', human_join(self.repr_airport(id) for id in ids)[:480])
140+
141+ def _flight_search(self, event, dpt, to, dep_date, ret_date):
142+ airport_dpt = self._airport_search(dpt)
143+ airport_to = self._airport_search(to)
144+ if len(airport_dpt) == 0:
145+ event.addresponse(u"Sorry, I don't know the airport you want to leave from")
146+ return
147+ if len(airport_to) == 0:
148+ event.addresponse(u"Sorry, I don't know the airport you want to fly to")
149+ return
150+ if len(airport_dpt) > 1:
151+ event.addresponse(u'The following airports match the departure: %s', human_join(self.repr_airport(id) for id in airport_dpt)[:480])
152+ return
153+ if len(airport_to) > 1:
154+ event.addresponse(u'The following airports match the destination: %s', human_join(self.repr_airport(id) for id in airport_to)[:480])
155+ return
156+
157+ dpt = airport_dpt[0]
158+ to = airport_to[0]
159+
160+ def to_travelocity_date(date):
161+ date = date.lower()
162+ time = None
163+ for period in [u'anytime', u'morning', u'afternoon', u'evening']:
164+ if period in date:
165+ time = period.title()
166+ date = date.replace(period, u'')
167+ break
168+ try:
169+ date = parse(date)
170+ except ValueError:
171+ raise FlightException(u"Sorry, I can't understand the date %s" % date)
172+ if time is None:
173+ if date.hour == 0 and date.minute == 0:
174+ time = u'Anytime'
175+ else:
176+ time = date.strftime('%I:00')
177+ if time[0] == u'0':
178+ time = time[1:]
179+ if date.hour < 12:
180+ time += u'am'
181+ else:
182+ time += u'pm'
183+ date = date.strftime('%m/%d/%Y')
184+ return (date, time)
185+
186+ (dep_date, dep_time) = to_travelocity_date(dep_date)
187+ (ret_date, ret_time) = to_travelocity_date(ret_date)
188+
189+ params = {}
190+ params[u'leavingFrom'] = self.airports[dpt][3]
191+ params[u'goingTo'] = self.airports[to][3]
192+ params[u'leavingDate'] = dep_date
193+ params[u'dateLeavingTime'] = dep_time
194+ params[u'returningDate'] = ret_date
195+ params[u'dateReturningTime'] = ret_time
196+ etree = get_html_parse_tree('http://travel.travelocity.com/flights/InitialSearch.do', data=urlencode(params), treetype='etree')
197+ while True:
198+ script = [script for script in etree.getiterator(u'script')][1]
199+ matches = script.text and re.search(r'var finurl = "(.*)"', script.text)
200+ if matches:
201+ url = u'http://travel.travelocity.com/flights/%s' % matches.group(1)
202+ etree = get_html_parse_tree(url, treetype=u'etree')
203+ else:
204+ break
205+
206+ # Handle error
207+ div = [d for d in etree.getiterator(u'div') if d.get(u'class') == u'e_content']
208+ if len(div):
209+ error = div[0].find(u'h3').text
210+ raise FlightException(error)
211+
212+ departing_flights = self._parse_travelocity(etree)
213+ return_url = None
214+ table = [t for t in etree.getiterator(u'table') if t.get(u'id') == u'tfGrid'][0]
215+ for tr in table.getiterator(u'tr'):
216+ for td in tr.getiterator(u'td'):
217+ if td.get(u'class').strip() in [u'tfPrice', u'tfPriceOrButton']:
218+ onclick = td.find(u'div/button').get(u'onclick')
219+ match = re.search(r"location.href='\.\./flights/(.+)'", onclick)
220+ url_page = match.group(1)
221+ match = re.search(r'^(.*?)[^/]*$', url)
222+ url_base = match.group(1)
223+ return_url = url_base + url_page
224+
225+ etree = get_html_parse_tree(return_url, treetype=u'etree')
226+ returning_flights = self._parse_travelocity(etree)
227+
228+ return (departing_flights, returning_flights, url)
229+
230+ def _parse_travelocity(self, etree):
231+ flights = []
232+ table = [t for t in etree.getiterator(u'table') if t.get(u'id') == u'tfGrid'][0]
233+ trs = [t for t in table.getiterator(u'tr')]
234+ tr_index = 1
235+ while tr_index < len(trs):
236+ tds = []
237+ while True:
238+ new_tds = [t for t in trs[tr_index].getiterator(u'td')]
239+ tds.extend(new_tds)
240+ tr_index += 1
241+ if len(filter(lambda t: t.get(u'class').strip() == u'tfAirlineSeatsMR', new_tds)):
242+ break
243+ flight = Flight()
244+ for td in tds:
245+ if td.get(u'class').strip() == u'tfAirline':
246+ anchor = td.find(u'a')
247+ if anchor is not None:
248+ airline = anchor.text.strip()
249+ else:
250+ airline = td.text.split(u'\n')[0].strip()
251+ flight.flight.append(u'%s %s' % (airline, td.findtext(u'div').strip()))
252+ if td.get(u'class').strip() == u'tfDepart' and td.text:
253+ flight.depart_time = td.text.split(u'\n')[0].strip()
254+ flight.depart_ap = u'%s %s' % (td.findtext(u'div').strip(),
255+ td.findtext(u'div/span').strip())
256+ if td.get(u'class').strip() == u'tfArrive' and td.text:
257+ flight.arrive_time = td.text.split(u'\n')[0].strip()
258+ span = td.find(u'span')
259+ if span is not None and span.get(u'class').strip() == u'tfNextDayDate':
260+ flight.arrive_time = u'%s %s' % (flight.arrive_time, span.text.strip()[2:])
261+ span = [s for s in td.find(u'div').getiterator(u'span')][1]
262+ flight.arrive_ap = u'%s %s' % (td.findtext(u'div').strip(),
263+ span.text.strip())
264+ else:
265+ flight.arrive_ap = u'%s %s' % (td.findtext(u'div').strip(),
266+ td.findtext(u'div/span').strip())
267+ if td.get(u'class').strip() == u'tfTime' and td.text:
268+ flight.duration = td.text.strip()
269+ flight.stops = td.findtext(u'span/a').strip()
270+ if td.get(u'class').strip() in [u'tfPrice', u'tfPriceOr'] and td.text:
271+ flight.price = td.text.strip()
272+ flight.flight = human_join(flight.flight)
273+ flights.append(flight)
274+
275+ return flights
276+
277+ @match(r'^(?:(cheapest|quickest)\s+)?flights?\s+from\s+(.+)\s+to\s+(.+)\s+from\s+(%s)\s+to\s+(%s)$' % (DATE, DATE))
278+ def flight_search(self, event, priority, dpt, to, dep_date, ret_date):
279+ try:
280+ flights = self._flight_search(event, dpt, to, dep_date, ret_date)
281+ except FlightException, e:
282+ event.addresponse(unicode(e))
283+ return
284+ if flights is None:
285+ return
286+ if len(flights[0]) == 0:
287+ event.addresponse(u'No matching departure flights found')
288+ return
289+ if len(flights[1]) == 0:
290+ event.addresponse(u'No matching return flights found')
291+ return
292+
293+ cmp = None
294+ if priority is not None:
295+ priority = priority.lower()
296+ if priority == u'cheapest':
297+ cmp = lambda a, b: a.int_price() < b.int_price()
298+ elif priority == u'quickest':
299+ cmp = lambda a, b: a.int_duration() < b.int_duration()
300+ if cmp:
301+ # select best flight based on priority
302+ for i in xrange(2):
303+ flights[i].sort(cmp=cmp)
304+ del flights[i][1:]
305+ response = []
306+ for i, flight_type in zip(xrange(2), [u'Departing', u'Returning']):
307+ if len(flights[i]) > 1:
308+ response.append(u'%s flights:' % flight_type)
309+ for flight in flights[i][:self.max_results]:
310+ leading = u''
311+ if len(flights[i]) == 1:
312+ leading = u'%s flight: ' % flight_type
313+ response.append(u'%(leading)s%(flight)s departing %(depart_time)s from %(depart_airport)s, arriving %(arrive_time)s at %(arrive_airport)s (flight time %(duration)s, %(stops)s) costs %(price)s per person' % {
314+ 'leading': leading,
315+ 'flight': flight.flight,
316+ 'depart_time': flight.depart_time,
317+ 'depart_airport': flight.depart_ap,
318+ 'arrive_time': flight.arrive_time,
319+ 'arrive_airport': flight.arrive_ap,
320+ 'duration': flight.duration,
321+ 'stops': flight.stops,
322+ 'price': flight.price or 'unknown'
323+ })
324+ response.append(u'Full results: %s' % flights[2])
325+ event.addresponse(u'\n'.join(response), conflate=False)
326+
327 # vi: set et sta sw=4 ts=4:
328
329=== modified file 'ibid/utils/__init__.py'
330--- ibid/utils/__init__.py 2010-01-06 23:19:42 +0000
331+++ ibid/utils/__init__.py 2010-01-18 22:43:10 +0000
332@@ -80,9 +80,14 @@
333 req.add_header('User-Agent', 'Ibid/' + (ibid_version() or 'dev'))
334
335 if exists:
336- modified = os.path.getmtime(cachefile)
337- modified = time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(modified))
338- req.add_header("If-Modified-Since", modified)
339+ if os.path.isfile(cachefile + '.etag'):
340+ f = file(cachefile + '.etag', 'r')
341+ req.add_header("If-None-Match", f.readline().strip())
342+ f.close()
343+ else:
344+ modified = os.path.getmtime(cachefile)
345+ modified = time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(modified))
346+ req.add_header("If-Modified-Since", modified)
347
348 try:
349 connection = urllib2.urlopen(req)
350@@ -106,6 +111,12 @@
351 gzipper = GzipFile(fileobj=compressedstream)
352 data = gzipper.read()
353
354+ etag = connection.headers.get('etag')
355+ if etag:
356+ f = file(cachefile + '.etag', 'w')
357+ f.write(etag + '\n')
358+ f.close()
359+
360 outfile = file(cachefile, 'wb')
361 outfile.write(data)
362 outfile.close()

Subscribers

People subscribed via source and target branches