lp:~robru/friends/since_id

Branch merges

Propose for merging

No branches dependent on this one.

Merged into lp:friends at revision 160

Ken VanDine: Approve on 2013-03-08

PS Jenkins bot (community): Approve (continuous-integration) on 2013-03-08

Diff: 505 lines (+250/-34)

6 files modified

friends/protocols/twitter.py (+48/-25)
friends/tests/test_cache.py (+69/-0)
friends/tests/test_identica.py (+12/-3)
friends/tests/test_twitter.py (+44/-5)
friends/utils/base.py (+1/-1)
friends/utils/cache.py (+76/-0)

api

Related source package recipes

No recipes using this branch. (?)

Create packaging recipe

Related snap packages

Related bugs

Bug #1152417: Friends re-publishing and re-notifying about stale tweets.

Critical

Fix Released

Link a bug report

Related blueprints

158. By Robert Bruce Park on 2013-03-08

Start using since_id= on Twitter API requests. (LP: #1152417)

This was accomplished by implementing two new classes, and ended up
simplifying some of the RateLimiter code as a side effect.

The first new class is called JsonCache. It is a subclass of dict,
which attempts to populate it's initial state by reading in a json
text file at a configurable location, and also adds a new "write()"
method that dumps the json back out to the same location. This class
was a generalization of what we were already doing inside the
RateLimiter, so it should not be considered a "new feature" if we are
going to butt heads with today's feature freeze.

The second new class is a subclass of JsonCache, which enforces that:

A) keys may not contain slashes, to avoid it getting polluted with
every search term ever searched for, or every message that's ever been
replied to ever, it only observes the values of the "main" streams,
such as "messages", "mentions" and "private", although those values
are not hardcoded so it's flexible to adapt to new streams in the
future.

B) values must be ints (tweet_ids), and values can only be
incremented. This is so that we can easily just throw every observed
tweet_id into the cache, and it only records the largest (newest) one.

The end result is that we now have two new files located at
~/.cache/friends/twitter_ids.json and
~/.cache/friends/identica_ids.json which track the newest tweet_id
that we have ever seen for each of the streams that we publish to.
These values are then consulted to form the since_id= argument to
several of Twitter's API endpoints, which solves bug #1152417.

As an added bonus, this also greatly reduces our network usage because
we are no longer redownloading duplicate messages over and over, so if
there are no new messages, Twitter is now returning an empty list of
Tweets rather than a large list of stale tweets.

This commit includes full test coverage for all new code.

157. By Robert Bruce Park on 2013-03-04

Bring up DBus interface sooner during startup.

We were having some racy timeout issues where we'd be invoked by dbus,
but then didn't get the dispatcher up in time to respond to dbus
method invocations before dbus timed out on us, because waiting for
the DeeModel to sync was too slow.

Instead, we bring up the dispatcher first, but also acquire the
_publish_lock, which prevents the dispatcher from trying to publish
prematurely (eg, before caches have been populated and and the
DeeModel syncronized). Then once the DeeModel syncs, we release the
lock and allow the dispatcher to function as normal.

Ken & I both tested this branch throughout the entire weekend and not
a single timeout occurred the entire time, so this is a big win.

156. By Robert Bruce Park on 2013-03-01

Fix quantal compatibility.

155. By Robert Bruce Park on 2013-02-28

Persist Twitter rate limiter data (LP: #1118878)

154. By Robert Bruce Park on 2013-02-28

Fix a flaky test.

This one was passing mostly, but had an intermittent failure based on
sort order of an os.listdir() output. Now I force the output to be
sorted and the test passes reliably.

153. By Robert Bruce Park on 2013-02-27

Fetch Flickr avatar for uploads.

152. By Robert Bruce Park on 2013-02-27

Slightly more robust extraction of error messages.

151. By Robert Bruce Park on 2013-02-27

More robust error handling logic.

Previously we were ignoring / incorrectly reporting errors from Flickr
and Twitter, while only Facebook had correct error handling. This
moves Facebook's good error handling code up into the Base class,
slightly generalised, so that it can be used by Flickr and Twitter.

150. By Robert Bruce Park on 2013-02-27

Keep original sized avatars and rescale 100px versions at alternate path.

149. By Robert Bruce Park on 2013-02-26

Fix tests.