Merge lp:~robru/gwibber/twitter into lp:~barry/gwibber/py3

[Robert Bruce Park]

Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit prematurely because it's still the easiest way to see an overall diff ;-)

[lp:~robru/gwibber/twitter]

1474. By Robert Bruce Park on 2012-10-01

Implement Base._unpublish for the sake of deleting messages.

This required adding a new module global to base.py, _seen_ids, which
is a dict mapping message ids to SharedModel row iters. Without this
it would have been necessary to iterate over the entire model every
time you wanted to delete a single row.

1475. By Robert Bruce Park on 2012-10-01

Add a test case for Base._unpublish.

1476. By Robert Bruce Park on 2012-10-01

Fill out supported features in test_dbus.

All tests pass!

1477. By Robert Bruce Park on 2012-10-01

Uncomment line ;-)

[Barry Warsaw (barry)]

On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:

>Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>prematurely because it's still the easiest way to see an overall diff ;-)

The branch is looking pretty good. I'm glad you were able to verify that the
OAuth signatures were working, at least for public tweets. I guess more
research will have to be done to figure out why private replies are broken.

Anyway, here's a review of the branch so far. In generally, it looks pretty
good. There are a few error cases that need to be thought about, a few
additional tests that I think need to be written, and just a few minor style
nits here and there. But overall, it's looking fantastic and the small
problems should be easy to fix. I think this can be landed pretty quickly.

Detailed comments follow.

Do you intend to include the debug_twitter_*.py scripts in the branch? If so,
let's put them in a tools subdirectory (i.e. not in the gwibber Python
package). That way they'll be easier to omit from the Debian package.
Also...

=== added file 'gwibber/debug_twitter_live.py'
--- gwibber/debug_twitter_live.py 1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_live.py 2012-10-02 14:33:58 +0000
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +
> +from gwibber.utils.account import AccountManager
> +from gwibber.utils.model import Model
> +from gi.repository import GObject
> +
> +def refresh (account):
> + print ("############################# Receive ####################################")

This can probably be fit into 79 characters. :)

Also, no space between print and open paren. (PEP 8)

> + account.protocol.user()
> + account.protocol.delete('252823527978311680')

What does this number correspond to? Is it private information?

> + for row in Model:
> + print([col for col in row])
> + print()
> + print ("ROWS: ", len(Model))

extra space

> + return True
> +
> +if __name__ == "__main__":
> +
> + found = False
> + a = AccountManager(None)
> +
> + for account_id, account in a._accounts.items():
> + if account_id.endswith('twitter'):
> + found = True
> + refresh(account)
> + GObject.timeout_add_seconds(300, refresh, account)
> +
> + if not found:
> + print('No Twitter account found in your Ubuntu Online Accounts!')
> + else:
> + GObject.MainLoop().run()

=== added file 'gwibber/debug_twitter_slave.py'
--- gwibber/debug_twitter_slave.py 1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_slave.py 2012-10-02 14:33:58 +0000
> @@ -0,0 +1,23 @@
> +#!/usr/bin/env python3
> +
> +from gi.repository import Dee
> +from gi.repository import GObject
> +
> +
> +class Slave:
> + def __init__(self):

4 space indents (PEP 8).

> + model_name = "com.Gwibber.Streams"
> + print ("Joining model %s" % model_name)

extra space.

> + self.model = Dee.SharedModel.new(model_name)
> + self.model.connect("row-added", self.on_row_added)
> +
> + def on_row_added (self, model, itr):
> + row = self.model.get_row (itr)
> + print (row)
> + print ("ROWS: ", len(self.model))

extra spaces.

> +
> +if __name__ == "__main__":
> +
> + s =...

[Robert Bruce Park (robru)]

On 12-10-02 02:13 PM, Barry Warsaw wrote:
> On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:
>
>> Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>> prematurely because it's still the easiest way to see an overall diff ;-)
>
> The branch is looking pretty good. I'm glad you were able to verify that the
> OAuth signatures were working, at least for public tweets.

I actually verified that the OAuth signatures worked for *every* API
endpoint *except* sending new direct messages (we can receive them ok,
too). That's what debug_twitter_live.py was all about, every time I'd
write a new protocol operation, I'd modify it to do that new operation,
then run it, and I'd see on my live twitter account that it worked.

> I guess more
> research will have to be done to figure out why private replies are broken.

Yeah, I still have no idea.

> Do you intend to include the debug_twitter_*.py scripts in the branch? If so,
> let's put them in a tools subdirectory (i.e. not in the gwibber Python
> package). That way they'll be easier to omit from the Debian package.

I hadn't originally. At first it just started off as 'from
gwibber.utils.account import AccountManager; a = AccountManager(None);
a._accounts['6/twitter'].protocol.receive()' because I was sick of
typing that out 10,000 times in an interactive python shell every time I
wanted to test a change I made to the code. But once the script was
born, it grew, and then ken expanded it for his own testing purposes,
which is included here as well.

In fact, *all* of the style nits you mention were written by ken ;-)

I had assumed that you would just not merge them. If you think there's
value in keeping them in a tools directory, I guess that's ok. I don't
have a strong opinion either way.

>> + account.protocol.user()
>> + account.protocol.delete('252823527978311680')
>
> What does this number correspond to? Is it private information?

That is a public tweet I made with a previous incarnation of this
script, that I've now deleted by running this script as-is. No worries.

>> + def _get_url(self, url, data=None):
>> + """Access the Twitter API with correct OAuth signed headers."""
>> + # TODO start enforcing some rate limiting.
>
> What are your thoughts on how to do this rate limiting? Is it something that
> Twitter provides through their API? (in a similar manner to Facebook's
> pagination support?). I wonder if there's some commonality refactoring that
> should go on here, at least for determining the constants for the rate
> limiting?

I haven't looked closely at Twitter's rate limiting standards yet, but I
wrote that comment because I ran into Twitter's limit at one point and
they stopped me from downloading tweets that I wanted.

All I know is that each API endpoint has a different maximum number of
requests that can be made "per window" and the length of time of the
window is different, too. So basically I'm envisioning a dict that maps
URLs to ints, the int for a given URL will increase once per each
_get_url invocation, and then we just need to figure out how long to
pause before making the request in order not to hit th...

[lp:~robru/gwibber/twitter]

1478. By Robert Bruce Park on 2012-10-03

Allow debug_twitter_live.py to call any action from the commandline.

Commandline args are passed directly to protocol.__call__(), so any
feature supported by Twitter can be invoked directly by this script.

1479. By Robert Bruce Park on 2012-10-03

Generalize debug_twitter_*.py scripts into debug_*.py scripts.

debug_live has a good docstring that explains its usage.

1480. By Robert Bruce Park on 2012-10-03

Add tests for Twitter operations.

This also tweaks Twitter.tag so that it silently ignores the # symbols
in hash tags, allowing you to include them or not, at your option.
Previously, if you tried to Twitter.tag('#hashtag'), it wouldn't work,
you had to strip the # symbol yourself. Now we accept both.

1481. By Robert Bruce Park on 2012-10-03

Docstring cleanup.

1482. By Robert Bruce Park on 2012-10-03

Fix bug in Base._unpublish

1483. By Robert Bruce Park on 2012-10-03

Improve _seen_ids / _seen_messages consistency.

Also fix a comment.

[Barry Warsaw (barry)]

On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:

>I actually verified that the OAuth signatures worked for *every* API endpoint
>*except* sending new direct messages (we can receive them ok, too). That's
>what debug_twitter_live.py was all about, every time I'd write a new protocol
>operation, I'd modify it to do that new operation, then run it, and I'd see
>on my live twitter account that it worked.

You may have to engage with the Twitter developer community to debug this. I
didn't find anything relevant in a very little bit of googling though. You'd
think that if it were a bug in their API, others would have noticed it.

>I hadn't originally. At first it just started off as 'from
>gwibber.utils.account import AccountManager; a = AccountManager(None);

Just FWIW, I have a patch in my facebook.working branch that makes None the
default argument. You have to be careful not to call the callback if it's
None. I wonder if it's useful to land that in the py3 branch separately?

>In fact, *all* of the style nits you mention were written by ken ;-)

/me hears Kirk's voice scream to the heavens "Keeeeeeennnnnn!"

>I haven't looked closely at Twitter's rate limiting standards yet, but I
>wrote that comment because I ran into Twitter's limit at one point and they
>stopped me from downloading tweets that I wanted.

I think Twitter warns you (via response headers?) if you're getting rate
limited. So I guess you could check for those and automatically back-off if
you see them. It's probably better to at least have some combination of
proactive and reactive response to rate limiting. The gory details:

https://dev.twitter.com/docs/rate-limiting

>>> + # "Client" == "Consumer" in oauthlib parlance.
>>> + client_key = self._account.auth.parameters['ConsumerKey']
>>> + client_secret = self._account.auth.parameters['ConsumerSecret']
>>
>> Can these fail? Are the other parts of your code prepared to handle the
>> resulting KeyErrors if these parameters are missing? Do you have any tests
>> for those situations?
>
>No, these are constants defined by Twitter at app-registration time, stored
>in libaccounts-sso package. Can never fail. They may change in the event that
>somebody nefarious compromises them (which ought to be easy to do for an
>open-source package), but they won't change during runtime.

Cool. In that case, the code above is fine because if they *do* fail, you
want to know via the logged KeyErrors.

>>> + def _publish_tweet(self, tweet):
>>> + """Publish a single tweet into the Dee.SharedModel."""
>>> + tweet_id = tweet.get('id_str', '')
>>
>> Does it still make sense to publish this tweet if there is no id_str?
>> Won't that possibly result in row collisions, since the only unique item in
>> the key will be the empty string for multiple id_str-less entries?
>>
>> I think in my Facebook branch, I just skip the JSON response entry if
>> there's no message_id.
>
>Yeah, that's probably reasonable. I just write 'adict.get' out of habit for
>the sake of being as flexible as possible.

As discussed over IRC, something like:

    tweet_id = tweet.get('id_str')
    if tweet_id is None:
        log.info...

[lp:~robru/gwibber/twitter]

1484. By Robert Bruce Park on 2012-10-03

Fix up Base._unpublish: Better tests, bugs fixed.

Particularly I've fixed the removal of ids when there were duplicates
before, and also I've eliminated the possibility of collisions between
message id's from different services (there used to be a small chance
that twitter and facebook could assign the same message id to
different messages, and then _unpublish would get confused.

And of course, improved test coverage to prove that this is all good
now ;-)

1485. By Robert Bruce Park on 2012-10-03

Typo.

1486. By Robert Bruce Park on 2012-10-03

Cleanup some API formatting.

1487. By Robert Bruce Park on 2012-10-03

Added comment as per barry.

1488. By Robert Bruce Park on 2012-10-03

Undo a change that wasn't necessary.

[Robert Bruce Park (robru)]

On 12-10-03 05:22 PM, Barry Warsaw wrote:
> On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:
>> I actually verified that the OAuth signatures worked for *every* API endpoint
>> *except* sending new direct messages (we can receive them ok, too). That's
>> what debug_twitter_live.py was all about, every time I'd write a new protocol
>> operation, I'd modify it to do that new operation, then run it, and I'd see
>> on my live twitter account that it worked.
>
> You may have to engage with the Twitter developer community to debug this. I
> didn't find anything relevant in a very little bit of googling though. You'd
> think that if it were a bug in their API, others would have noticed it.

Yeah, no idea. Maybe we're the first to port to 1.1 ;-)

I'll look into it a little bit later tonight, got some other
refactoring/cleanup going on.

>> I hadn't originally. At first it just started off as 'from
>> gwibber.utils.account import AccountManager; a = AccountManager(None);
>
> Just FWIW, I have a patch in my facebook.working branch that makes None the
> default argument. You have to be careful not to call the callback if it's
> None. I wonder if it's useful to land that in the py3 branch separately?

I don't think it matters. Maybe it should just default to lambda:None
and then we don't have to worry about whether the callback gets called
or not. For the debug_live.py script, it's not meant to be a
long-running script, just short stints for testing. So it's unlikely for
the callback to be triggered during it's operation.

>> I haven't looked closely at Twitter's rate limiting standards yet, but I
>> wrote that comment because I ran into Twitter's limit at one point and they
>> stopped me from downloading tweets that I wanted.
>
> I think Twitter warns you (via response headers?) if you're getting rate
> limited. So I guess you could check for those and automatically back-off if
> you see them. It's probably better to at least have some combination of
> proactive and reactive response to rate limiting. The gory details:
>
> https://dev.twitter.com/docs/rate-limiting

Yeah, I'm gonna dive into this soon, but so far I've just been focusing
on cleaning up the code that's already there and improving the tests.

>>>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
>>>> + @feature
>>>> + def mentions(self):
>>>> + """Gather the tweets that mention us."""
>>>> + url = self._timeline.format('mentions')
>>>> + for tweet in self._get_url(url):
>>>> + self._publish_tweet(tweet)
>>>
>>> Do you have tests for each of these? I know they *look* ridiculously
>>> similar, but each should have a test.
>>
>> I wasn't really sure how to test them, to be honest.
>
> It's probably good enough to use the same, or very similar sample data, but
> just call .mentions(), etc. in the tests.
>
> Really, the only thing you care about is that you've tested the basic code in
> this (and similar) methods, not in anything it calls. Let's say for example
> that you - or a future developer - typo'd "self._publish_twit()". That's the
> kind of thing you want these tests to catch.

Fair enough. I wrote some t...

[lp:~robru/gwibber/twitter]

1489. By Robert Bruce Park on 2012-10-04

Implement a basic rate limiter.

This had to be done in the Downloader class unfortunately because it
needed direct access to the HTTP headers and I didn't really see any
way to access those outside of the Downloader class.

Luckily it will harmlessly ignore any protocol that doesn't specify
rate limits in the HTTP headers, so it should work well for Twitter
and do nothing for anybody else.

This basically makes no attempt to limit you if you have more than 5
requests remaining in the rate limit window (making it unobtrusive for
cases where the dispatcher is just calling every 5 minutes and it
really should never come near the rate limiter *anyway*), but if
something has gone on such as a user requesting too many manual
refreshes, then it will forcibly pause each request long enough to
avoid hitting Twitter's rate limiter.

1490. By Robert Bruce Park on 2012-10-04

Prevent duplicate message ids from filling up the message_ids column.

It turns out that it's quite easy to see the same message multiple
times, and previously the dupe-checking logic was successful in
stopping the message from being published more than once, it blindly
appended the message_id to the list regardless of whether or not it
was already on the list, so that tended to fill up with junk quick.

With test case.

[Robert Bruce Park (robru)]

Ok, barry, I think we have a pretty solid thing here, rate limiting and everything! Have another look over the diff and let me know what you think ;-)

The only real issue remaining is that send_private still 403s, but I've sent a message to the Twitter support forum, so hopefully they respond to that soon.

[lp:~robru/gwibber/twitter]

1491. By Robert Bruce Park on 2012-10-04

Fill out some comments and docstrings.

1492. By Robert Bruce Park on 2012-10-04

Catch 403s in send_private, with test.

1493. By Robert Bruce Park on 2012-10-04

Add tests for the rate limiter.

=== modified file 'gwibber/gwibber/protocols/twitter.py'
--- gwibber/gwibber/protocols/twitter.py	2012-09-19 22:21:39 +0000
+++ gwibber/gwibber/protocols/twitter.py	2012-10-04 20:29:21 +0000
@@ -14,20 +14,289 @@
 
 """The Twitter protocol plugin."""
 
+
 __all__ = [
     'Twitter',
     ]
 
 
-import gettext
 import logging
 
-from gwibber.utils.base import Base
+from oauthlib.oauth1 import Client
+from urllib.error import HTTPError
+from urllib.parse import quote
+
+from gwibber.utils.authentication import Authentication
+from gwibber.utils.base import Base, feature
+from gwibber.utils.download import get_json
+from gwibber.utils.time import parsetime, iso8601utc
 
 
 log = logging.getLogger('gwibber.service')
-_ = gettext.lgettext
-
-
+
+
+# https://dev.twitter.com/docs/api/1.1
 class Twitter(Base):
-    pass
+    # StatusNet claims to mimick the Twitter API very closely (so
+    # closely to the point that they refer you to the twitter API
+    # reference docs as a starting point for learning about their
+    # API), So these prefixes are defined here as class attributes
+    # instead of the usual module globals, in the hopes that the
+    # StatusNet class will be able to subclass Twitter and change only
+    # the URLs, with minimal other changes, and magically work.
+    _api_base = 'https://api.twitter.com/1.1/{endpoint}.json'
+
+    _timeline = _api_base.format(endpoint='statuses/{}_timeline')
+    _user_timeline = _timeline.format('user') + '?screen_name={}'
+
+    _lists = _api_base.format(endpoint='lists/statuses') + '?list_id={}'
+
+    _destroy = _api_base.format(endpoint='statuses/destroy/{}')
+    _retweet = _api_base.format(endpoint='statuses/retweet/{}')
+
+    _tweet_permalink = 'https://twitter.com/{user_id}/status/{tweet_id}'
+
+    def _locked_login(self, old_token):
+        """Sign in without worrying about concurrent login attempts."""
+        result = Authentication(self._account, log).login()
+        if result is None:
+            log.error('No Twitter authentication results received.')
+            return
+
+        token = result.get('AccessToken')
+        if token is None:
+            log.error('No AccessToken in Twitter session: {!r}', result)
+        else:
+            self._account.access_token = token
+            self._account.secret_token = result.get('TokenSecret')
+            self._account.user_id = result.get('UserId')
+            self._account.user_name = result.get('ScreenName')
+            log.debug('{} UID: {}'.format(self.__class__.__name__,
+                                          self._account.user_id))
+
+    def _get_url(self, url, data=None):
+        """Access the Twitter API with correct OAuth signed headers."""
+        # TODO start enforcing some rate limiting.
+        do_post = data is not None
+
+        # "Client" == "Consumer" in oauthlib parlance.
+        client_key = self._account.auth.parameters['ConsumerKey']
+        client_secret = self._account.auth.parameters['ConsumerSecret']
+
+        # "resource_owner" == secret and token.
+        resource_owner_key = self._get_access_token()
+        resource_owner_secret = self._account.secret_token
+        oauth_client = Client(client_key, client_secret,
+                              resource_owner_key, resource_owner_secret)
+
+        headers = {}
+        if do_post:
+            headers['Content-Type'] = 'application/x-www-form-urlencoded'
+
+        # All we care about is the headers, which will contain the
+        # Authorization header necessary to satisfy OAuth.
+        uri, headers, body = oauth_client.sign(
+            url, body=data, headers=headers,
+            http_method='POST' if do_post else 'GET')
+
+        return get_json(url,
+                        params=data,
+                        headers=headers,
+                        post=do_post)
+
+    def _publish_tweet(self, tweet):
+        """Publish a single tweet into the Dee.SharedModel."""
+        tweet_id = tweet.get('id_str')
+        if tweet_id is None:
+            log.info('We got a tweet with no id! Abort!')
+            return
+
+        user = tweet.get('user', {})
+        screen_name = user.get('screen_name', '')
+        self._publish(
+            message_id=tweet_id,
+            message=tweet.get('text', ''),
+            timestamp=iso8601utc(parsetime(tweet.get('created_at', ''))),
+            stream='messages',
+            sender=user.get('name', ''),
+            sender_nick=screen_name,
+            from_me=(screen_name == self._account.user_id),
+            icon_uri=user.get('profile_image_url_https', ''),
+            liked=tweet.get('favorited', False),
+            url=self._tweet_permalink.format(user_id=screen_name,
+                                             tweet_id=tweet_id),
+            )
+
+# https://dev.twitter.com/docs/api/1.1/get/statuses/home_timeline
+    @feature
+    def home(self):
+        """Gather the user's home timeline."""
+        url = self._timeline.format('home')
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
+    @feature
+    def mentions(self):
+        """Gather the tweets that mention us."""
+        url = self._timeline.format('mentions')
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline
+    @feature
+    def user(self, screen_name=''):
+        """Gather the tweets from a specific user.
+
+        If screen_name is not specified, then gather the tweets
+        written by the currently authenticated user.
+        """
+        url = self._user_timeline.format(screen_name)
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/get/lists/statuses
+    @feature
+    def list(self, list_id):
+        """Gather the tweets from the specified list_id."""
+        url = self._lists.format(list_id)
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/get/lists/list
+    @feature
+    def lists(self):
+        """Gather the tweets from the lists that the we are subscribed to."""
+        url = self._api_base.format(endpoint='lists/list')
+        for twitlist in self._get_url(url):
+            self.list(twitlist.get('id_str', ''))
+
+# https://dev.twitter.com/docs/api/1.1/get/direct_messages
+# https://dev.twitter.com/docs/api/1.1/get/direct_messages/sent
+    @feature
+    def private(self):
+        """Gather the direct messages sent to/from us."""
+        url = self._api_base.format(endpoint='direct_messages')
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+        url = self._api_base.format(endpoint='direct_messages/sent')
+        for tweet in self._get_url(url):
+            self._publish_tweet(tweet)
+
+    @feature
+    def receive(self):
+        """Gather and publish all incoming messages."""
+        # TODO I know mentions and lists are actually incorporated
+        # within the home timeline, but calling them explicitly will
+        # ensure that more of those types of messages appear in the
+        # timeline (eg, so they don't get drown out by everything
+        # else). I'm not sure how necessary it is, though.
+        self.home()
+        self.mentions()
+        self.lists()
+        self.private()
+
+    @feature
+    def send_private(self, screen_name, message):
+        """Send a direct message to the given screen name.
+
+        This will error 403 if the person you are sending to does not
+        follow you.
+        """
+        url = self._api_base.format(endpoint='direct_messages/new')
+        try:
+            tweet = self._get_url(url, dict(text=message, screen_name=screen_name))
+            self._publish_tweet(tweet)
+        except HTTPError as e:
+            log.error('{}: Does that user follow you?'.format(e))
+
+# https://dev.twitter.com/docs/api/1.1/post/statuses/update
+    @feature
+    def send(self, message):
+        """Publish a public tweet."""
+        url = self._api_base.format(endpoint='statuses/update')
+        tweet = self._get_url(url, dict(status=message))
+        self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/post/statuses/update
+    @feature
+    def send_thread(self, message_id, message):
+        """Send a reply message to message_id.
+
+        Note that you have to @mention the message_id owner's screen name in
+        order for Twitter to actually accept this as a reply. Otherwise it will
+        just be an ordinary tweet.
+        """
+        url = self._api_base.format(endpoint='statuses/update')
+        tweet = self._get_url(url, dict(in_reply_to_status_id=message_id,
+                                        status=message))
+        self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
+    @feature
+    def delete(self, message_id):
+        """Delete a tweet that you wrote."""
+        url = self._destroy.format(message_id)
+        tweet = self._get_url(url, dict(trim_user='true'))
+        self._unpublish(message_id)
+
+# https://dev.twitter.com/docs/api/1.1/post/statuses/retweet/%3Aid
+    @feature
+    def retweet(self, message_id):
+        """Republish somebody else's tweet with your name on it."""
+        url = self._retweet.format(message_id)
+        tweet = self._get_url(url, dict(trim_user='true'))
+        self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/post/friendships/destroy
+    @feature
+    def unfollow(self, screen_name):
+        """Stop following the given screen name."""
+        url = self._api_base.format(endpoint='friendships/destroy')
+        self._get_url(url, dict(screen_name=screen_name))
+
+# https://dev.twitter.com/docs/api/1.1/post/friendships/create
+    @feature
+    def follow(self, screen_name):
+        """Start following the given screen name."""
+        url = self._api_base.format(endpoint='friendships/create')
+        self._get_url(url, dict(screen_name=screen_name, follow='true'))
+
+# https://dev.twitter.com/docs/api/1.1/post/favorites/create
+    @feature
+    def like(self, message_id):
+        """Announce to the world your undying love for a tweet."""
+        url = self._api_base.format(endpoint='favorites/create')
+        tweet = self._get_url(url, dict(id=message_id))
+        # I don't think we need to publish this tweet because presumably
+        # the user has clicked the 'favorite' button on the message that's
+        # already in the stream...
+
+# https://dev.twitter.com/docs/api/1.1/post/favorites/destroy
+    @feature
+    def unlike(self, message_id):
+        """Renounce your undying love for a tweet."""
+        url = self._api_base.format(endpoint='favorites/destroy')
+        tweet = self._get_url(url, dict(id=message_id))
+
+# https://dev.twitter.com/docs/api/1.1/get/search/tweets
+    @feature
+    def tag(self, hashtag):
+        """Return a list of some recent tweets mentioning hashtag."""
+        url = self._api_base.format(endpoint='search/tweets')
+
+        response = self._get_url(
+            '{}?q=%23{}'.format(url, hashtag.lstrip('#')))
+        for tweet in response.get('statuses', []):
+            self._publish_tweet(tweet)
+
+# https://dev.twitter.com/docs/api/1.1/get/search/tweets
+    @feature
+    def search(self, query):
+        """Search for any arbitrary string."""
+        url = self._api_base.format(endpoint='search/tweets')
+
+        response = self._get_url('{}?q={}'.format(url, quote(query, safe='')))
+        for tweet in response.get('statuses', []):
+            self._publish_tweet(tweet)
 
=== modified file 'gwibber/gwibber/testing/mocks.py'
--- gwibber/gwibber/testing/mocks.py	2012-09-21 21:45:06 +0000
+++ gwibber/gwibber/testing/mocks.py	2012-10-04 20:29:21 +0000
@@ -71,21 +71,30 @@
 class FakeData:
     """Mimic a urlopen() that returns canned data."""
 
-    def __init__(self, path, resource, charset='utf-8'):
+    def __init__(self, path, resource, charset='utf-8', headers=None):
         # resource_string() always returns bytes.
         self._data = resource_string(path, resource)
         self.call_count = 0
         self._charset = charset
+        self._headers = headers or {}
 
     def __call__(self, url, post_data=None):
         # Ignore url and post_data since the canned data will already
         # represents these results.  We just have to make the API fit.
         # post_data will be missing for GETs.
         charset = self._charset
+        class FakeInfo:
+            def __init__(self, headers=None):
+                self.headers = headers or {}
+            def get(self, key, default=None):
+                return self.headers.get(key, default)
+            def get_content_charset(self):
+                return charset
         class FakeOpen:
-            def __init__(self, data, charset):
+            def __init__(self, data, charset, headers=None):
                 self._data = data
                 self._charset = charset
+                self.headers = headers or {}
             def read(self):
                 return self._data
             def __enter__(self):
@@ -93,12 +102,9 @@
             def __exit__(self, *args, **kws):
                 pass
             def info(self):
-                class FakeInfo:
-                    def get_content_charset(self):
-                        return charset
-                return FakeInfo()
+                return FakeInfo(self.headers)
         self.call_count += 1
-        return FakeOpen(self._data, self._charset)
+        return FakeOpen(self._data, self._charset, self._headers)
 
 
 class SettingsIterMock:
 
=== added file 'gwibber/gwibber/tests/data/twitter-home.dat'
--- gwibber/gwibber/tests/data/twitter-home.dat	1970-01-01 00:00:00 +0000
+++ gwibber/gwibber/tests/data/twitter-home.dat	2012-10-04 20:29:21 +0000
@@ -0,0 +1,344 @@
+    [
+      {
+        "coordinates": null,
+        "truncated": false,
+        "created_at": "Tue Aug 28 21:16:23 +0000 2012",
+        "favorited": false,
+        "id_str": "240558470661799936",
+        "in_reply_to_user_id_str": null,
+        "entities": {
+          "urls": [
+
+          ],
+          "hashtags": [
+
+          ],
+          "user_mentions": [
+
+          ]
+        },
+        "text": "just another test",
+        "contributors": null,
+        "id": 240558470661799936,
+        "retweet_count": 0,
+        "in_reply_to_status_id_str": null,
+        "geo": null,
+        "retweeted": false,
+        "in_reply_to_user_id": null,
+        "place": null,
+        "source": "<a href=\"http://realitytechnicians.com\" rel=\"nofollow\">OAuth Dancer Reborn</a>",
+        "user": {
+          "name": "OAuth Dancer",
+          "profile_sidebar_fill_color": "DDEEF6",
+          "profile_background_tile": true,
+          "profile_sidebar_border_color": "C0DEED",
+          "profile_image_url": "http://a0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg",
+          "created_at": "Wed Mar 03 19:37:35 +0000 2010",
+          "location": "San Francisco, CA",
+          "follow_request_sent": false,
+          "id_str": "119476949",
+          "is_translator": false,
+          "profile_link_color": "0084B4",
+          "entities": {
+            "url": {
+              "urls": [
+                {
+                  "expanded_url": null,
+                  "url": "http://bit.ly/oauth-dancer",
+                  "indices": [
+                    0,
+                    26
+                  ],
+                  "display_url": null
+                }
+              ]
+            },
+            "description": null
+          },
+          "default_profile": false,
+          "url": "http://bit.ly/oauth-dancer",
+          "contributors_enabled": false,
+          "favourites_count": 7,
+          "utc_offset": null,
+          "profile_image_url_https": "https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg",
+          "id": 119476949,
+          "listed_count": 1,
+          "profile_use_background_image": true,
+          "profile_text_color": "333333",
+          "followers_count": 28,
+          "lang": "en",
+          "protected": false,
+          "geo_enabled": true,
+          "notifications": false,
+          "description": "",
+          "profile_background_color": "C0DEED",
+          "verified": false,
+          "time_zone": null,
+          "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/80151733/oauth-dance.png",
+          "statuses_count": 166,
+          "profile_background_image_url": "http://a0.twimg.com/profile_background_images/80151733/oauth-dance.png",
+          "default_profile_image": false,
+          "friends_count": 14,
+          "following": false,
+          "show_all_inline_media": false,
+          "screen_name": "oauth_dancer"
+        },
+        "in_reply_to_screen_name": null,
+        "in_reply_to_status_id": null
+      },
+      {
+        "coordinates": {
+          "coordinates": [
+            -122.25831,
+            37.871609
+          ],
+          "type": "Point"
+        },
+        "truncated": false,
+        "created_at": "Tue Aug 28 21:08:15 +0000 2012",
+        "favorited": false,
+        "id_str": "240556426106372096",
+        "in_reply_to_user_id_str": null,
+        "entities": {
+          "urls": [
+            {
+              "expanded_url": "http://blogs.ischool.berkeley.edu/i290-abdt-s12/",
+              "url": "http://t.co/bfj7zkDJ",
+              "indices": [
+                79,
+                99
+              ],
+              "display_url": "blogs.ischool.berkeley.edu/i290-abdt-s12/"
+            }
+          ],
+          "hashtags": [
+
+          ],
+          "user_mentions": [
+            {
+              "name": "Cal",
+              "id_str": "17445752",
+              "id": 17445752,
+              "indices": [
+                60,
+                64
+              ],
+              "screen_name": "Cal"
+            },
+            {
+              "name": "Othman Laraki",
+              "id_str": "20495814",
+              "id": 20495814,
+              "indices": [
+                70,
+                77
+              ],
+              "screen_name": "othman"
+            }
+          ]
+        },
+        "text": "lecturing at the \"analyzing big data with twitter\" class at @cal with @othman  http://t.co/bfj7zkDJ",
+        "contributors": null,
+        "id": 240556426106372096,
+        "retweet_count": 3,
+        "in_reply_to_status_id_str": null,
+        "geo": {
+          "coordinates": [
+            37.871609,
+            -122.25831
+          ],
+          "type": "Point"
+        },
+        "retweeted": false,
+        "possibly_sensitive": false,
+        "in_reply_to_user_id": null,
+        "place": {
+          "name": "Berkeley",
+          "country_code": "US",
+          "country": "United States",
+          "attributes": {
+          },
+          "url": "http://api.twitter.com/1/geo/id/5ef5b7f391e30aff.json",
+          "id": "5ef5b7f391e30aff",
+          "bounding_box": {
+            "coordinates": [
+              [
+                [
+                  -122.367781,
+                  37.835727
+                ],
+                [
+                  -122.234185,
+                  37.835727
+                ],
+                [
+                  -122.234185,
+                  37.905824
+                ],
+                [
+                  -122.367781,
+                  37.905824
+                ]
+              ]
+            ],
+            "type": "Polygon"
+          },
+          "full_name": "Berkeley, CA",
+          "place_type": "city"
+        },
+        "source": "<a href=\"http://www.apple.com\" rel=\"nofollow\">Safari on iOS</a>",
+        "user": {
+          "name": "Raffi Krikorian",
+          "profile_sidebar_fill_color": "DDEEF6",
+          "profile_background_tile": false,
+          "profile_sidebar_border_color": "C0DEED",
+          "profile_image_url": "http://a0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png",
+          "created_at": "Sun Aug 19 14:24:06 +0000 2007",
+          "location": "San Francisco, California",
+          "follow_request_sent": false,
+          "id_str": "8285392",
+          "is_translator": false,
+          "profile_link_color": "0084B4",
+          "entities": {
+            "url": {
+              "urls": [
+                {
+                  "expanded_url": "http://about.me/raffi.krikorian",
+                  "url": "http://t.co/eNmnM6q",
+                  "indices": [
+                    0,
+                    19
+                  ],
+                  "display_url": "about.me/raffi.krikorian"
+                }
+              ]
+            },
+            "description": {
+              "urls": [
+
+              ]
+            }
+          },
+          "default_profile": true,
+          "url": "http://t.co/eNmnM6q",
+          "contributors_enabled": false,
+          "favourites_count": 724,
+          "utc_offset": -28800,
+          "profile_image_url_https": "https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png",
+          "id": 8285392,
+          "listed_count": 619,
+          "profile_use_background_image": true,
+          "profile_text_color": "333333",
+          "followers_count": 18752,
+          "lang": "en",
+          "protected": false,
+          "geo_enabled": true,
+          "notifications": false,
+          "description": "Director of @twittereng's Platform Services. I break things.",
+          "profile_background_color": "C0DEED",
+          "verified": false,
+          "time_zone": "Pacific Time (US & Canada)",
+          "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png",
+          "statuses_count": 5007,
+          "profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png",
+          "default_profile_image": false,
+          "friends_count": 701,
+          "following": true,
+          "show_all_inline_media": true,
+          "screen_name": "raffi"
+        },
+        "in_reply_to_screen_name": null,
+        "in_reply_to_status_id": null
+      },
+      {
+        "coordinates": null,
+        "truncated": false,
+        "created_at": "Tue Aug 28 19:59:34 +0000 2012",
+        "favorited": false,
+        "id_str": "240539141056638977",
+        "in_reply_to_user_id_str": null,
+        "entities": {
+          "urls": [
+
+          ],
+          "hashtags": [
+
+          ],
+          "user_mentions": [
+
+          ]
+        },
+        "text": "You'd be right more often if you thought you were wrong.",
+        "contributors": null,
+        "id": 240539141056638977,
+        "retweet_count": 1,
+        "in_reply_to_status_id_str": null,
+        "geo": null,
+        "retweeted": false,
+        "in_reply_to_user_id": null,
+        "place": null,
+        "source": "web",
+        "user": {
+          "name": "Taylor Singletary",
+          "profile_sidebar_fill_color": "FBFBFB",
+          "profile_background_tile": true,
+          "profile_sidebar_border_color": "000000",
+          "profile_image_url": "http://a0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg",
+          "created_at": "Wed Mar 07 22:23:19 +0000 2007",
+          "location": "San Francisco, CA",
+          "follow_request_sent": false,
+          "id_str": "819797",
+          "is_translator": false,
+          "profile_link_color": "c71818",
+          "entities": {
+            "url": {
+              "urls": [
+                {
+                  "expanded_url": "http://www.rebelmouse.com/episod/",
+                  "url": "http://t.co/Lxw7upbN",
+                  "indices": [
+                    0,
+                    20
+                  ],
+                  "display_url": "rebelmouse.com/episod/"
+                }
+              ]
+            },
+            "description": {
+              "urls": [
+
+              ]
+            }
+          },
+          "default_profile": false,
+          "url": "http://t.co/Lxw7upbN",
+          "contributors_enabled": false,
+          "favourites_count": 15990,
+          "utc_offset": -28800,
+          "profile_image_url_https": "https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg",
+          "id": 819797,
+          "listed_count": 340,
+          "profile_use_background_image": true,
+          "profile_text_color": "D20909",
+          "followers_count": 7126,
+          "lang": "en",
+          "protected": false,
+          "geo_enabled": true,
+          "notifications": false,
+          "description": "Reality Technician, Twitter API team, synthesizer enthusiast; a most excellent adventure in timelines. I know it's hard to believe in something you can't see.",
+          "profile_background_color": "000000",
+          "verified": false,
+          "time_zone": "Pacific Time (US & Canada)",
+          "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/643655842/hzfv12wini4q60zzrthg.png",
+          "statuses_count": 18076,
+          "profile_background_image_url": "http://a0.twimg.com/profile_background_images/643655842/hzfv12wini4q60zzrthg.png",
+          "default_profile_image": false,
+          "friends_count": 5444,
+          "following": true,
+          "show_all_inline_media": true,
+          "screen_name": "episod"
+        },
+        "in_reply_to_screen_name": null,
+        "in_reply_to_status_id": null
+      }
+    ]
 
=== modified file 'gwibber/gwibber/tests/test_dbus.py'
--- gwibber/gwibber/tests/test_dbus.py	2012-09-25 20:25:42 +0000
+++ gwibber/gwibber/tests/test_dbus.py	2012-10-04 20:29:21 +0000
@@ -111,6 +111,11 @@
             '/com/gwibber/Service')
         iface = dbus.Interface(obj, 'com.Gwibber.Service')
         # TODO Add more cases as more protocols are added.
+        self.assertEqual(json.loads(iface.GetFeatures('twitter')),
+                         ['delete', 'follow', 'home', 'like', 'list', 'lists',
+                          'mentions', 'private', 'receive', 'retweet', 'search',
+                          'send', 'send_private', 'send_thread', 'tag',
+                          'unfollow', 'unlike', 'user'])
         self.assertEqual(json.loads(iface.GetFeatures('flickr')), ['receive'])
         self.assertEqual(json.loads(iface.GetFeatures('foursquare')),
                          ['receive'])
 
=== modified file 'gwibber/gwibber/tests/test_download.py'
--- gwibber/gwibber/tests/test_download.py	2012-09-14 20:29:11 +0000
+++ gwibber/gwibber/tests/test_download.py	2012-10-04 20:29:21 +0000
@@ -249,3 +249,44 @@
                                   headers={'X-Foo': 'baz',
                                            'X-Bar': 'foo'}),
                          dict(foo='baz', bar='foo'))
+
+    @mock.patch('gwibber.utils.download.urlopen',
+                FakeData('gwibber.tests.data', 'twitter-home.dat',
+                         headers={'X-Rate-Limit-Reset':1349382153 + 300,
+                                  'X-Rate-Limit-Remaining': 1}))
+    @mock.patch('gwibber.utils.download.time.sleep')
+    @mock.patch('gwibber.utils.download.time.time', return_value=1349382153)
+    def test_rate_limiter_maximum(self, time, sleep):
+        # First call does not get limited, but establishes the limit
+        get_json('http://example.com/alpha')
+        sleep.assert_called_with(0)
+        # Second call gets called with the established limit
+        get_json('http://example.com/alpha')
+        sleep.assert_called_with(300)
+
+    @mock.patch('gwibber.utils.download.urlopen',
+                FakeData('gwibber.tests.data', 'twitter-home.dat',
+                         headers={'X-Rate-Limit-Reset':1349382153 + 300,
+                                  'X-Rate-Limit-Remaining': 3}))
+    @mock.patch('gwibber.utils.download.time.sleep')
+    @mock.patch('gwibber.utils.download.time.time', return_value=1349382153)
+    def test_rate_limiter_medium(self, time, sleep):
+        # First call does not get limited, but establishes the limit
+        get_json('http://example.com/beta')
+        sleep.assert_called_with(0)
+        # Second call gets called with the established limit
+        get_json('http://example.com/beta')
+        sleep.assert_called_with(100.0)
+
+    @mock.patch('gwibber.utils.download.urlopen',
+                FakeData('gwibber.tests.data', 'twitter-home.dat',
+                         headers={'X-Rate-Limit-Reset':int(time.time()) + 300,
+                                  'X-Rate-Limit-Remaining': 10}))
+    @mock.patch('gwibber.utils.download.time.sleep')
+    def test_rate_limiter_unlimited(self, sleep):
+        # First call does not get limited, but establishes the limit
+        get_json('http://example.com/omega')
+        sleep.assert_called_with(0)
+        # Second call gets called with the established limit
+        get_json('http://example.com/omega')
+        sleep.assert_called_with(0)
 
=== modified file 'gwibber/gwibber/tests/test_protocols.py'
--- gwibber/gwibber/tests/test_protocols.py	2012-09-25 20:25:42 +0000
+++ gwibber/gwibber/tests/test_protocols.py	2012-10-04 20:29:21 +0000
@@ -176,6 +176,7 @@
 
     @mock.patch('gwibber.utils.base.Model', TestModel)
     @mock.patch('gwibber.utils.base._seen_messages', {})
+    @mock.patch('gwibber.utils.base._seen_ids', {})
     def test_one_message(self):
         # Test that publishing a message inserts a row into the model.
         base = Base(FakeAccount())
@@ -215,6 +216,32 @@
 
     @mock.patch('gwibber.utils.base.Model', TestModel)
     @mock.patch('gwibber.utils.base._seen_messages', {})
+    @mock.patch('gwibber.utils.base._seen_ids', {})
+    def test_unpublish(self):
+        base = Base(FakeAccount())
+        self.assertEqual(0, TestModel.get_n_rows())
+        self.assertTrue(base._publish(
+            message_id='1234',
+            sender='fred',
+            message='hello, @jimmy'))
+        self.assertTrue(base._publish(
+            message_id='5678',
+            sender='fred',
+            message='hello, +jimmy'))
+        self.assertEqual(1, TestModel.get_n_rows())
+        self.assertEqual(TestModel[0][0],
+                         [['base', 'faker/than fake', '1234'],
+                          ['base', 'faker/than fake', '5678']])
+        base._unpublish('1234')
+        self.assertEqual(1, TestModel.get_n_rows())
+        self.assertEqual(TestModel[0][0],
+                         [['base', 'faker/than fake', '5678']])
+        base._unpublish('5678')
+        self.assertEqual(0, TestModel.get_n_rows())
+
+    @mock.patch('gwibber.utils.base.Model', TestModel)
+    @mock.patch('gwibber.utils.base._seen_messages', {})
+    @mock.patch('gwibber.utils.base._seen_ids', {})
     def test_duplicate_messages_identified(self):
         # When two messages which are deemed identical, by way of the
         # _make_key() test in base.py, are published, only one ends up in the
@@ -259,6 +286,31 @@
 
     @mock.patch('gwibber.utils.base.Model', TestModel)
     @mock.patch('gwibber.utils.base._seen_messages', {})
+    @mock.patch('gwibber.utils.base._seen_ids', {})
+    def test_duplicate_ids_not_duplicated(self):
+        # When two messages are actually identical (same ids and all),
+        # we need to avoid duplicating the id in the sharedmodel.
+        base = Base(FakeAccount())
+        self.assertEqual(0, TestModel.get_n_rows())
+        self.assertTrue(base._publish(
+            message_id='1234',
+            stream='messages',
+            sender='fred',
+            message='hello, @jimmy'))
+        self.assertTrue(base._publish(
+            message_id='1234',
+            stream='messages',
+            sender='fred',
+            message='hello, @jimmy'))
+        self.assertEqual(1, TestModel.get_n_rows())
+        row = TestModel.get_row(0)
+        # The same message_id should not appear twice.
+        self.assertEqual(row[COLUMN_INDICES['message_ids']],
+                         [['base', 'faker/than fake', '1234']])
+
+    @mock.patch('gwibber.utils.base.Model', TestModel)
+    @mock.patch('gwibber.utils.base._seen_messages', {})
+    @mock.patch('gwibber.utils.base._seen_ids', {})
     def test_similar_messages_allowed(self):
         # Because both the sender and message contribute to the unique key we
         # use to identify messages, if two messages are published with
 
=== modified file 'gwibber/gwibber/tests/test_twitter.py'
--- gwibber/gwibber/tests/test_twitter.py	2012-09-21 14:11:34 +0000
+++ gwibber/gwibber/tests/test_twitter.py	2012-10-04 20:29:21 +0000
@@ -14,6 +14,7 @@
 
 """Test the Twitter plugin."""
 
+
 __all__ = [
     'TestTwitter',
     ]
@@ -21,9 +22,13 @@
 
 import unittest
 
+from gi.repository import Dee
+
 from gwibber.protocols.twitter import Twitter
 from gwibber.testing.helpers import FakeAccount
-from gwibber.testing.mocks import LogMock
+from gwibber.testing.mocks import FakeData, LogMock
+from gwibber.utils.model import COLUMN_TYPES
+
 
 try:
     # Python 3.3
@@ -32,10 +37,12 @@
     import mock
 
 
-# Ensure synchronicity between the main thread and the sub-thread.  Also, set
-# up the loggers for the modules-under-test so that we can assert their error
-# messages.
[e-mail]('gwibber.utils.base.__dict__', {'_SYNCHRONIZE': True})
+# Create a test model that will not interfere with the user's environment.
+# We'll use this object as a mock of the real model.
+TestModel = Dee.SharedModel.new('com.Gwibber.TestSharedModel')
+TestModel.set_schema_full(COLUMN_TYPES)
+
+
 class TestTwitter(unittest.TestCase):
     """Test the Twitter API."""
 
@@ -46,8 +53,289 @@
                                 'gwibber.protocols.twitter')
 
     def tearDown(self):
+        # Ensure that any log entries we haven't tested just get consumed so
+        # as to isolate out test logger from other tests.
         self.log_mock.stop()
 
-    def test_protocol_info(self):
-        # Each protocol carries with it a number of protocol variables.
-        self.assertEqual(self.protocol.__class__.__name__, 'Twitter')
+    @mock.patch('gwibber.utils.authentication.Authentication.login',
+                return_value=None)
+    @mock.patch('gwibber.utils.download.get_json',
+                return_value=None)
+    def test_unsuccessful_authentication(self, *mocks):
+        self.assertFalse(self.protocol._login())
+        self.assertIsNone(self.account.user_name)
+        self.assertIsNone(self.account.user_id)
+
+    @mock.patch('gwibber.utils.authentication.Authentication.login',
+                return_value=dict(AccessToken='some clever fake data',
+                                  TokenSecret='sssssshhh!',
+                                  UserId='rickygervais',
+                                  ScreenName='Ricky Gervais'))
+    def test_successful_authentication(self, *mocks):
+        self.assertTrue(self.protocol._login())
+        self.assertEqual(self.account.user_name, 'Ricky Gervais')
+        self.assertEqual(self.account.user_id, 'rickygervais')
+        self.assertEqual(self.account.access_token, 'some clever fake data')
+        self.assertEqual(self.account.secret_token, 'sssssshhh!')
+
+
+    @mock.patch('gwibber.protocols.twitter.get_json', lambda *x, **y: y)
+    @mock.patch('oauthlib.oauth1.rfc5849.generate_nonce',
+                lambda: 'once upon a nonce')
+    @mock.patch('oauthlib.oauth1.rfc5849.generate_timestamp',
+                lambda: '1348690628')
+    def test_signatures(self, *mocks):
+        self.account.secret_token = 'alpha'
+        self.account.access_token = 'omega'
+        self.account.auth.id = 6
+        self.account.auth.method = 'oauth2'
+        self.account.auth.mechanism = 'HMAC-SHA1'
+        self.account.auth.parameters = dict(ConsumerKey='consume',
+                                            ConsumerSecret='obey')
+        result = '''\
+OAuth oauth_nonce="once%20upon%20a%20nonce", \
+oauth_timestamp="1348690628", \
+oauth_version="1.0", \
+oauth_signature_method="HMAC-SHA1", \
+oauth_consumer_key="consume", \
+oauth_token="omega", \
+oauth_signature="2MlC4DOqcAdCUmU647izPmxiL%2F0%3D"'''
+
+        self.assertEqual(
+            self.protocol._get_url('http://example.com')['headers'],
+            dict(Authorization=result))
+
+    @mock.patch('gwibber.utils.base.Model', TestModel)
+    @mock.patch('gwibber.utils.download.urlopen',
+                FakeData('gwibber.tests.data', 'twitter-home.dat'))
+    @mock.patch('gwibber.protocols.twitter.Twitter._login',
+                return_value=True)
+    def test_home(self, *mocks):
+        self.account.access_token = 'access'
+        self.account.secret_token = 'secret'
+        self.account.auth.parameters = dict(
+            ConsumerKey='key',
+            ConsumerSecret='secret')
+        self.assertEqual(0, TestModel.get_n_rows())
+        self.protocol.home()
+        self.assertEqual(3, TestModel.get_n_rows())
+
+        # This test data was ripped directly from Twitter's API docs.
+        expected = [
+            [[['twitter', 'faker/than fake', '240558470661799936']],
+             'messages', 'OAuth Dancer', 'oauth_dancer', False,
+             '2012-08-28T21:16:23', 'just another test', '',
+             'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
+             'https://twitter.com/oauth_dancer/status/240558470661799936', '',
+             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
+             '', '', '', '', '', '', '', '', '', '',
+             ],
+            [[['twitter', 'faker/than fake', '240556426106372096']],
+             'messages', 'Raffi Krikorian', 'raffi', False,
+             '2012-08-28T21:08:15', 'lecturing at the "analyzing big data ' +
+             'with twitter" class at @cal with @othman  http://t.co/bfj7zkDJ', '',
+             'https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png',
+             'https://twitter.com/raffi/status/240556426106372096', '',
+             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
+             '', '', '', '', '', '', '', '', '', '',
+             ],
+            [[['twitter', 'faker/than fake', '240539141056638977']],
+             'messages', 'Taylor Singletary', 'episod', False,
+             '2012-08-28T19:59:34', 'You\'d be right more often if you thought you were wrong.', '',
+             'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
+             'https://twitter.com/episod/status/240539141056638977', '',
+             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
+             '', '', '', '', '', '', '', '', '', '',
+             ],
+            ]
+        for i, expected_row in enumerate(expected):
+            for got, want in zip(TestModel.get_row(i), expected_row):
+                self.assertEqual(got, want)
+
+    def test_mentions(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.mentions()
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/mentions_timeline.json')
+
+    def test_user(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.user()
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=')
+
+    def test_list(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.list('some_list_id')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/lists/statuses.json?list_id=some_list_id')
+
+    def test_lists(self):
+        get_url = self.protocol._get_url = mock.Mock(
+            return_value=[dict(id_str='twitlist')])
+        publish = self.protocol.list = mock.Mock()
+
+        self.protocol.lists()
+
+        publish.assert_called_with('twitlist')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/lists/list.json')
+
+    def test_private(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.private()
+
+        publish.assert_called_with('tweet')
+        self.assertEqual(
+            get_url.mock_calls,
+            [mock.call('https://api.twitter.com/1.1/direct_messages.json'),
+             mock.call('https://api.twitter.com/1.1/direct_messages/sent.json')])
+
+    def test_send_private(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.send_private('pumpichank', 'Are you mocking me?')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/direct_messages/new.json',
+            dict(text='Are you mocking me?', screen_name='pumpichank'))
+
+    def test_failing_send_private(self):
+        from urllib.error import HTTPError
+        def failing_request(*ignore):
+            raise HTTPError('url', 403, 'Forbidden', 'Forbidden', mock.Mock())
+        get_url = self.protocol._get_url = failing_request
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.send_private('pumpichank', 'Are you mocking me?')
+
+        self.assertEqual(
+            self.log_mock.empty(),
+            'HTTP Error 403: Forbidden: Does that user follow you?\n')
+
+    def test_send(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.send('Hello, twitterverse!')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/update.json',
+            dict(status='Hello, twitterverse!'))
+
+    def test_send_thread(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.send_thread(
+            '1234',
+            'Why yes, I would love to respond to your tweet @pumpichank!')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/update.json',
+            dict(status='Why yes, I would love to respond to your tweet @pumpichank!',
+                 in_reply_to_status_id='1234'))
+
+    def test_delete(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
+        publish = self.protocol._unpublish = mock.Mock()
+
+        self.protocol.delete('1234')
+
+        publish.assert_called_with('1234')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/destroy/1234.json',
+            dict(trim_user='true'))
+
+    def test_retweet(self):
+        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.retweet('1234')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/statuses/retweet/1234.json',
+            dict(trim_user='true'))
+
+    def test_unfollow(self):
+        get_url = self.protocol._get_url = mock.Mock()
+
+        self.protocol.unfollow('pumpichank')
+
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/friendships/destroy.json',
+            dict(screen_name='pumpichank'))
+
+    def test_follow(self):
+        get_url = self.protocol._get_url = mock.Mock()
+
+        self.protocol.follow('pumpichank')
+
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/friendships/create.json',
+            dict(screen_name='pumpichank', follow='true'))
+
+    def test_like(self):
+        get_url = self.protocol._get_url = mock.Mock()
+
+        self.protocol.like('1234')
+
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/favorites/create.json',
+            dict(id='1234'))
+
+    def test_unlike(self):
+        get_url = self.protocol._get_url = mock.Mock()
+
+        self.protocol.unlike('1234')
+
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/favorites/destroy.json',
+            dict(id='1234'))
+
+    def test_tag(self):
+        get_url = self.protocol._get_url = mock.Mock(
+            return_value=dict(statuses=['tweet']))
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.tag('yegbike')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/search/tweets.json?q=%23yegbike')
+
+        self.protocol.tag('#yegbike')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/search/tweets.json?q=%23yegbike')
+
+    def test_search(self):
+        get_url = self.protocol._get_url = mock.Mock(
+            return_value=dict(statuses=['tweet']))
+        publish = self.protocol._publish_tweet = mock.Mock()
+
+        self.protocol.search('hello')
+
+        publish.assert_called_with('tweet')
+        get_url.assert_called_with(
+            'https://api.twitter.com/1.1/search/tweets.json?q=hello')
 
=== modified file 'gwibber/gwibber/utils/base.py'
--- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
+++ gwibber/gwibber/utils/base.py	2012-10-04 20:29:21 +0000
@@ -42,6 +42,7 @@
 # representing the rows matching those keys.  It is used for quickly finding
 # duplicates when we want to insert new rows into the model.
 _seen_messages = {}
+_seen_ids = {}
 
 
 # Protocol __call__() methods run in threads, so we need to serialize
@@ -179,11 +180,10 @@
         # The column value is a list of lists (see gwibber/utils/model.py for
         # details), and because the arguments are themselves a list, this gets
         # initialized as a triply-nested list.
-        args = [[[
-                self.__class__.__name__.lower(),
-                self._account.id,
-                message_id
-                ]]]
+        triple = [self.__class__.__name__.lower(),
+                  self._account.id,
+                  message_id]
+        args = [[triple]]
         # Now iterate through all the column names listed in the SCHEMA,
         # except for the first, since we just composed its value in the
         # preceding line.  Pop matching column values from the kwargs, in the
@@ -218,9 +218,34 @@
                 # the model gets updated, we need to insert into the row, thus
                 # it's best to concatenate the two lists together and store it
                 # back into the column.
-                row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
+                if triple not in row[IDS_IDX]:
+                    row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
+
+            _seen_ids[tuple(triple)] = _seen_messages.get(key)
             return key in _seen_messages
 
+    def _unpublish(self, message_id):
+        """Remove message_id from the Dee.SharedModel."""
+        triple = [self.__class__.__name__.lower(),
+                  self._account.id,
+                  message_id]
+
+        row_iter = _seen_ids.pop(tuple(triple), None)
+        if row_iter is None:
+            log.error('Tried to delete an invalid message id.')
+            return
+
+        row = Model.get_row(row_iter)
+        if len(row[IDS_IDX]) == 1:
+            # Message only exists on one protocol, delete it
+            del _seen_messages[_make_key(row)]
+            Model.remove(row_iter)
+        else:
+            # Message exists on other protocols too, only drop id
+            row[IDS_IDX] = [ids for ids
+                            in row[IDS_IDX]
+                            if message_id not in ids]
+
     def _get_access_token(self):
         """Return an access token, logging in if necessary."""
         if self._account.access_token is None:
 
=== modified file 'gwibber/gwibber/utils/download.py'
--- gwibber/gwibber/utils/download.py	2012-09-14 20:29:11 +0000
+++ gwibber/gwibber/utils/download.py	2012-10-04 20:29:21 +0000
@@ -20,6 +20,7 @@
     ]
 
 
+import time
 import json
 import logging
 
@@ -31,12 +32,18 @@
 log = logging.getLogger('gwibber.service')
 
 
+# This stores some information about the current rate limits imposed
+# upon us, and persists that data between instances of the Downloader.
+_rate_limits = {}
+
+
 class Downloader:
     """Convenient downloading wrapper."""
 
     def __init__(self, url, params=None, post=False,
                  username=None, password=None,
                  headers=None):
+        self.base_url = url.split('?')[0]
         self.url = url
         self.params = params
         self.post = post
@@ -47,11 +54,19 @@
 
     def _download(self):
         """Return the results as a decoded unicode string."""
+        # Downloads will be happening in threads, so this won't
+        # block the whole app ;-)
+        time.sleep(_rate_limits.get(self.base_url, 0))
+
         data = None
         url = self.url
         headers = self.headers.copy()
         if self.params is not None:
-            params = urlencode(self.params)
+            # urlencode() does not have an option to use quote()
+            # instead of quote_plus(), but Twitter requires
+            # percent-encoded spaces, and this is harmless to any
+            # other protocol.
+            params = urlencode(self.params).replace('+', '%20')
             if self.post:
                 data = params.encode('utf-8')
                 headers['Content-Type'] = (
@@ -89,6 +104,24 @@
         with urlopen(request) as result:
             payload = result.read()
             info = result.info()
+
+        rate_reset = info.get('X-Rate-Limit-Reset') # in UTC epoch seconds
+        rate_count = info.get('X-Rate-Limit-Remaining')
+        if None not in (rate_reset, rate_count):
+            rate_reset = int(rate_reset)
+            rate_count = int(rate_count)
+            rate_delta = rate_reset - time.time()
+            if rate_count > 5:
+                # Ehhh, let's not impede the user if they have more than 5
+                # requests remaining in this window.
+                _rate_limits[self.base_url] = 0
+            elif rate_count <= 1:
+                # Avoid division by zero... wait the full length of time!
+                _rate_limits[self.base_url] = rate_delta
+            else:
+                wait_secs = rate_delta / rate_count
+                _rate_limits[self.base_url] = wait_secs
+
         # RFC 4627 $3.  JSON text SHALL be encoded in Unicode.  The default
         # encoding is UTF-8.  Since the first two characters of a JSON text
         # will always be ASCII characters [RFC0020], it is possible to
 
=== removed directory 'gwibber/microblog/plugins/twitter'
=== removed file 'gwibber/microblog/plugins/twitter/__init__.py'
--- gwibber/microblog/plugins/twitter/__init__.py	2012-06-15 13:53:08 +0000
+++ gwibber/microblog/plugins/twitter/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,819 +0,0 @@
-import cgi
-from gettext import lgettext as _
-from oauth import oauth
-
-from gwibber.microblog import network, util
-from gwibber.microblog.util import resources
-from gwibber.microblog.util.time import parsetime
-from gwibber.microblog.util.auth import Authentication
-
-import logging
-logger = logging.getLogger("Twitter")
-logger.debug("Initializing.")
-
-PROTOCOL_INFO = {
-  "name": "Twitter",
-  "version": "1.0",
-  
-  "config": [
-    "private:secret_token",
-    "access_token",
-    "username",
-    "color",
-    "receive_enabled",
-    "send_enabled",
-  ],
- 
-  "authtype": "oauth1a",
-  "color": "#729FCF",
-
-  "features": [
-    "send",
-    "receive",
-    "search",
-    "tag",
-    "reply",
-    "responses",
-    "private",
-    "public",
-    "delete",
-    "follow",
-    "unfollow",
-    "profile",
-    "retweet",
-    "like",
-    "send_thread",
-    "send_private",
-    "user_messages",
-    "sinceid",
-    "lists",
-    "list",
-  ],
-
-  "default_streams": [
-    "receive",
-    "images",
-    "responses",
-    "private",
-    "lists",
-  ],
-}
-
-URL_PREFIX = "https://twitter.com"
-API_PREFIX = "https://api.twitter.com/1"
-
-class Client ():
-  """Querys Twitter and converts the data.
-  
-  The Client class is responsible for querying Twitter and turning the data obtained
-  into data that Gwibber can understand. Twitter uses a version of OAuth for security.
-  
-  Tokens have already been obtained when the account was set up in Gwibber and are used to 
-  authenticate when getting data.
-  
-  """
-  def __init__(self, acct):
-    self.service = util.getbus("Service")
-    self.account = acct
-    self._loop = None
-
-    self.sigmethod = oauth.OAuthSignatureMethod_HMAC_SHA1()
-    self.token = None
-    parameters = self.account["auth"]["parameters"]
-    self.consumer = oauth.OAuthConsumer(parameters["ConsumerKey"],
-                                        parameters["ConsumerSecret"])
-
-  def _login(self):
-    old_token = self.account.get("secret_token", None)
-    with self.account.login_lock:
-      # Perform the login only if it wasn't already performed by another thread
-      # while we were waiting to the lock
-      if self.account.get("secret_token", None) == old_token:
-        self._locked_login(old_token)
-
-    self.token = oauth.OAuthToken(self.account["access_token"],
-                                  self.account["secret_token"])
-    return "access_token" in self.account and \
-        self.account["access_token"] != old_token
-
-  def _locked_login(self, old_token):
-    logger.debug("Re-authenticating" if old_token else "Logging in")
-
-    auth = Authentication(self.account, logger)
-    reply = auth.login()
-    if reply and reply.has_key("AccessToken"):
-      self.account["access_token"] = reply["AccessToken"]
-      self.account["secret_token"] = reply["TokenSecret"]
-      self.account["uid"] = reply["UserId"]
-      self.account["username"] = reply["ScreenName"]
-      logger.debug("User id is: %s, name is %s" % (self.account["uid"],
-                                                   self.account["username"]))
-    else:
-      logger.error("Didn't find token in session: %s", (reply,))
-
-  def _common(self, data):
-    """Parses messages into Gwibber compatible forms.  
-    
-    This function is common to all tweet types
-    and includes parsing of user mentions, hashtags, 
-    urls, images and videos.
-    
-    Arguments: 
-    data -- A data object obtained from Twitter containing a complete tweet
-    
-    Returns: 
-    m -- A data object compatible with inserting into the Gwibber database for that tweet
-    
-    """
-    m = {}
-    try:
-      m["mid"] = str(data["id"])
-      m["service"] = "twitter"
-      m["account"] = self.account["id"]
-      if data.has_key("created_at"):
-        m["time"] = parsetime(data["created_at"])
-      m["text"] = util.unescape(data["text"])
-      m["text"] = cgi.escape(m["text"])
-      m["content"] = m["text"]
-      
-      # Go through the entities in the tweet and use them to linkify/filter tweets as appropriate
-      if data.has_key("entities"):
-
-        #Get mention entries
-        if data["entities"].has_key("user_mentions"):
-          names = []
-          for mention in data["entities"]["user_mentions"]:
-            if not mention["screen_name"] in names:
-              try:
-                m["content"] = m["content"].replace("@" + mention["screen_name"], "@<a href='gwibber:/user?acct=" + m["account"] + "&name=@" + mention["screen_name"] + "'>" + mention["screen_name"] + "</a>")
-              except:
-                pass
-            names.append(mention["screen_name"])  
-
-        #Get hashtag entities
-        if data["entities"].has_key("hashtags"):
-          hashtags = []
-          for tag in data["entities"]["hashtags"]:
-            if not tag["text"] in hashtags:
-              try:
-                m["content"] = m["content"].replace("#" + tag["text"], "#<a href='gwibber:/tag?acct=" + m["account"] + "&query=#" + tag["text"] + "'>" + tag["text"] + "</a>")
-              except:
-                pass
-            hashtags.append(tag["text"])
-        
-        # Get url entities - These usually go in the link stream, but if they're pictures or videos, they should go in the proper stream
-        if data["entities"].has_key("urls"):
-          for urls in data["entities"]["urls"]:
-              url = cgi.escape (urls["url"])
-              expanded_url = url
-              if urls.has_key("expanded_url"):
-                if not urls["expanded_url"] is None:
-                  expanded_url = cgi.escape(urls["expanded_url"])
-
-              display_url  = url
-              if urls.has_key("display_url"):
-                display_url = cgi.escape (urls["display_url"])
- 
-              if url == m["content"]:
-                m["content"] = "<a href='" + url + "' title='" + expanded_url + "'>" + display_url + "</a>"
-              else:
-                try:
-                  startindex = m["content"].index(url)
-                  endindex   = startindex + len(url)
-                  start      = m["content"][0:startindex]
-                  end        = m["content"][endindex:]
-                  m["content"] = start + "<a href='" + url + "' title='" + expanded_url + "'>" + display_url + "</a>" + end
-                except:
-                  logger.debug ("Failed to set url for ID: %s",  m["mid"])
-
-              m["type"] = "link"
-
-              images = util.imgpreview(expanded_url)
-              videos = util.videopreview(expanded_url)
-              if images:
-                m["images"] = images
-                m["type"] = "photo"
-              elif videos:
-                m["images"] = videos
-                m["type"] = "video"
-              else:
-                # Well, it's not anything else, so it must be a link
-                m["link"] = {}
-                m["link"]["picture"] = ""
-                m["link"]["name"] = ""
-                m["link"]["description"] = m["content"]
-                m["link"]["url"] = url
-                m["link"]["icon"] = ""
-                m["link"]["caption"] = ""
-                m["link"]["properties"] = {}
-              
-        if data["entities"].has_key("media"):
-          for media in data["entities"]["media"]:
-            try:
-              url = cgi.escape (media["url"])
-              media_url_https = media["media_url_https"]
-              expanded_url = url
-              if media.has_key("expanded_url"):
-                expanded_url = cgi.escape(media["expanded_url"])
-                
-              display_url  = url
-              if media.has_key("display_url"):
-                display_url = cgi.escape (media["display_url"])
- 
-              startindex = m["content"].index(url)
-              endindex   = startindex + len(url)
-              start      = m["content"][0:startindex]
-              end        = m["content"][endindex:]
-              m["content"] = start + "<a href='" + url + "' title='" + expanded_url  + "'>" + display_url + "</a>" + end
-              
-              if media["type"] == "photo":
-                m["type"] = "photo"
-                m["photo"] = {}
-                m["photo"]["picture"] = media_url_https
-                m["photo"]["url"] = None
-                m["photo"]["name"] = None
-
-            except:
-              pass
-
-      else:
-        m["content"] = util.linkify(util.unescape(m["text"]),
-          ((util.PARSE_HASH, '#<a href="gwibber:/tag?acct=%s&query=\\1">\\1</a>' % m["account"]),
-          (util.PARSE_NICK, '@<a href="gwibber:/user?acct=%s&name=\\1">\\1</a>' % m["account"])), escape=True)
-
-      m["html"] = m["content"]
-      
-      m["to_me"] = ("@%s" % self.account["username"]) in data["text"]   # Check if it's a reply directed at the user
-      m["favorited"] = data.get("favorited", False)                     # Check if the tweet has been favourited
-
-    except: 
-      logger.error("%s failure - %s", PROTOCOL_INFO["name"], data)
-      return {}
- 
-    return m
-
-  def _user(self, user):
-    """Parses the user portion of a tweet.
-    
-    Arguments:
-    user -- A user object from a tweet
-    
-    Returns:
-    A user object in a format compatible with Gwibber's database.
-    
-    """
-    return {
-        "name": user.get("name", None),
-        "nick": user.get("screen_name", None),
-        "id": user.get("id", None),
-        "location": user.get("location", None),
-        "followers": user.get("followers_count", None),
-        "friends": user.get("friends_count", None),
-        "description": user.get("description", None),
-        "following": user.get("following", None),
-        "protected": user.get("protected", None),
-        "statuses": user.get("statuses_count", None),
-        "image": user.get("profile_image_url", None),
-        "website": user.get("url", None),
-        "url": "/".join((URL_PREFIX, user.get("screen_name", ""))) or None,
-        "is_me": user.get("screen_name", None) == self.account["username"],
-    }
-    
-  def _message(self, data):
-    """Parses messages into gwibber compatible forms.
-    
-    This is the initial function for tweet parsing and parses 
-    retweeted status (the shared-by portion), source (the program 
-    the tweet was tweeted from) and reply details (the in-reply-to 
-    portion). It sends the rest to _common() for further parsing.
-    
-    Arguments: 
-    data -- A data object obtained from Twitter containing a complete tweet
-    
-    Returns: 
-    m -- A data object compatible with inserting into the Gwibber database for that tweet
-    
-    """
-    if type(data) != dict:
-      logger.error("Cannot parse message data: %s", str(data))
-      return {}
-
-    n = {}
-    if data.has_key("retweeted_status"):
-      n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
-      if data.has_key("created_at"):
-        n["time"] = parsetime(data["created_at"])
-      data = data["retweeted_status"]
-    else:
-      n["retweeted_by"] = None
-      if data.has_key("created_at"):
-        n["time"] = parsetime(data["created_at"])
-
-    m = self._common(data)
-    for k in n:
-      m[k] = n[k]
-
-    m["source"] = data.get("source", False)
-    
-    if data.has_key("in_reply_to_status_id"):
-      if data["in_reply_to_status_id"]:
-        m["reply"] = {}
-        m["reply"]["id"] = data["in_reply_to_status_id"]
-        m["reply"]["nick"] = data["in_reply_to_screen_name"]
-        if m["reply"]["id"] and m["reply"]["nick"]:
-          m["reply"]["url"] = "/".join((URL_PREFIX, m["reply"]["nick"], "statuses", str(m["reply"]["id"])))
-        else:
-          m["reply"]["url"] = None
-
-    m["sender"] = self._user(data["user"] if "user" in data else data["sender"])
-    m["url"] = "/".join((m["sender"]["url"], "statuses", str(m.get("mid", None))))
-
-    return m
-
-  def _responses(self, data):
-    """Sets the message type if the message should be in the replies stream. 
-    
-    It sends the rest to _message() for further parsing.
-    
-    Arguments: 
-    data -- A data object obtained from Twitter containing a complete tweet
-    
-    Returns: 
-    m -- A data object compatible with inserting into the Gwibber database for that tweet
-    
-    """
-    m = self._message(data)
-    m["type"] = None
-
-    return m
-
-  def _private(self, data):
-    """Sets the message type and privacy.
-    
-    Sets the message type and privacy if the message should be in the private stream.
-    Also parses the recipient as both sent & recieved messages can be in the private stream.
-    It sends the rest to _message() for further parsing
-    
-    Arguments: 
-    data -- A data object obtained from Twitter containing a complete tweet
-    
-    Returns: 
-    m -- A data object compatible with inserting into the Gwibber database for that tweet
-    
-    """
-    m = self._message(data)
-    m["private"] = True
-    m["type"] = None
-
-    m["recipient"] = {}
-    m["recipient"]["name"] = data["recipient"]["name"]
-    m["recipient"]["nick"] = data["recipient"]["screen_name"]
-    m["recipient"]["id"] = data["recipient"]["id"]
-    m["recipient"]["image"] = data["recipient"]["profile_image_url"]
-    m["recipient"]["location"] = data["recipient"]["location"]
-    m["recipient"]["url"] = "/".join((URL_PREFIX, m["recipient"]["nick"]))
-    m["recipient"]["is_me"] = m["recipient"]["nick"] == self.account["username"]
-    m["to_me"] = m["recipient"]["is_me"]
-
-    return m
-
-  def _result(self, data):
-    """Called when a search is done in Gwibber.  
-    
-    Parses the sender and sends the rest to _common()
-    for further parsing.
-    
-    Arguments: 
-    data -- A data object obtained from Twitter containing a complete tweet
-    
-    Returns: 
-    m -- A data object compatible with inserting into the Gwibber database for that tweet
-    
-    """
-    m = self._common(data)
-    
-    if data["to_user_id"]:
-      m["reply"] = {}
-      m["reply"]["id"] = data["to_user_id"]
-      m["reply"]["nick"] = data["to_user"]
-
-    m["sender"] = {}
-    m["sender"]["nick"] = data["from_user"]
-    m["sender"]["id"] = data["from_user_id"]
-    m["sender"]["image"] = data["profile_image_url"]
-    m["sender"]["url"] = "/".join((URL_PREFIX, m["sender"]["nick"]))
-    m["sender"]["is_me"] = m["sender"]["nick"] == self.account["username"]
-    m["url"] = "/".join((m["sender"]["url"], "statuses", str(m["mid"])))
-    return m
-
-  def _profile(self, data):
-    """Called when a user is clicked on.
-    
-    Args: 
-    data -- A data object obtained from Twitter containing a complete user
-    
-    Returns: 
-    A data object compatible with inserting into the Gwibber database for that user.
-    
-    """
-    if "error" in data:
-      return {
-          "error": data["error"]
-        }
-    return {
-        "name": data.get("name", data["screen_name"]),
-        "service": "twitter",
-        "stream": "profile",
-        "account": self.account["id"],
-        "mid": data["id"],
-        "text": data.get("description", ""),
-        "nick": data["screen_name"],
-        "url": data.get("url", ""),
-        "protected": data.get("protected", False),
-        "statuses": data.get("statuses_count", 0),
-        "followers": data.get("followers_count", 0),
-        "friends": data.get("friends_count", 0),
-        "following": data.get("following", 0),
-        "favourites": data.get("favourites_count", 0),
-        "image": data["profile_image_url"],
-        "utc_offset": data.get("utc_offset", 0),
-        "id": data["id"],
-        "lang": data.get("lang", "en"),
-        "verified": data.get("verified", False),
-        "geo_enabled": data.get("geo_enabled", False),
-        "time_zone": data.get("time_zone", "")
-    }
-
-  def _list(self, data):    
-    """Called when a list is clicked on.
-    
-    Args: 
-    data -- A data object obtained from Twitter containing a complete list
-    
-    Returns: 
-    A data object compatible with inserting into the Gwibber database for that list.
-    
-    """
-    return {
-        "mid": data["id"],
-        "service": "twitter",
-        "account": self.account["id"],
-        "time": 0,
-        "text": data["description"],
-        "html": data["description"],
-        "content": data["description"],
-        "url": "/".join((URL_PREFIX, data["uri"])),
-        "sender": self._user(data["user"]),
-        "name": data["name"],
-        "nick": data["slug"],
-        "key": data["slug"],
-        "full": data["full_name"],
-        "uri": data["uri"],
-        "mode": data["mode"],
-        "members": data["member_count"],
-        "followers": data["subscriber_count"],
-        "kind": "list",
-    }
-
-  def _get(self, path, parse="message", post=False, single=False, **args):
-    """Establishes a connection with Twitter and gets the data requested.  
-    
-    Requires authentication.
-    
-    Arguments: 
-    path -- The end of the url to look up on Twitter
-    parse -- The function to use to parse the data returned (message by default)
-    post -- True if using POST, for example the send operation. False if using GET, most operations other than send. (False by default)
-    single -- True if a single checkin is requested, False if multiple (False by default)
-    **args -- Arguments to be added to the URL when accessed. 
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by the parse function.
-    
-    """
-    if not self.token and not self._login():
-      logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), "Auth needs updating")
-      logger.error("%s", logstr)
-      return [{"error": {"type": "auth", "account": self.account, "message": _("Authentication failed, please re-authorize")}}]
-
-    url = "/".join((API_PREFIX, path))
-
-    request = oauth.OAuthRequest.from_consumer_and_token(self.consumer, self.token,
-        http_method=post and "POST" or "GET", http_url=url, parameters=util.compact(args))
-    request.sign_request(self.sigmethod, self.consumer, self.token)
-
-    if post:
-      headers = request.to_header()
-      data = network.Download(url, util.compact(args), post, header=headers).get_json()
-    else:
-      data = network.Download(request.to_url(), None, post).get_json()
-    resources.dump(self.account["service"], self.account["id"], data)
-
-    if isinstance(data, dict) and data.get("errors", 0):
-      if "authenticate" in data["errors"][0]["message"]:
-        # Try again, if we get a new token
-        if self._login():
-          logger.debug("Authentication error, logging in again")
-          return self._get(path, parse, post, single, args)
-        else:
-          logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), data["errors"][0]["message"])
-          logger.error("%s", logstr)
-          return [{"error": {"type": "auth", "account": self.account, "message": data["errors"][0]["message"]}}]
-      else:
-        for error in data["errors"]:
-          logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Unknown failure"), error["message"])
-          return [{"error": {"type": "unknown", "account": self.account, "message": error["message"]}}]
-    elif isinstance(data, dict) and data.get("error", 0):
-      if "Incorrect signature" in data["error"]:
-        logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Request failed"), data["error"])
-        logger.error("%s", logstr)
-        return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
-    elif isinstance(data, str):
-      logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Request failed"), data)
-      logger.error("%s", logstr)
-      return [{"error": {"type": "request", "account": self.account, "message": data}}]
-
-    if parse == "follow" or parse == "unfollow":
-      if isinstance(data, dict) and data.get("error", 0):
-        logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("%s failed" % parse), data["error"])
-        logger.error("%s", logstr)
-        return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
-      else:
-        return [["friendships", {"type": parse, "account": self.account["id"], "service": self.account["service"],"user_id": data["id"], "nick": data["screen_name"]}]]
-    
-    if parse == "profile" and isinstance(data, dict):
-      return self._profile(data)
-
-    if parse == "list":
-      return [self._list(l) for l in data["lists"]]
-
-    if single: return [getattr(self, "_%s" % parse)(data)]
-    if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
-    else: return []
-
-  def _search(self, **args):
-    """Establishes a connection with Twitter and gets the results of a search.
-    
-    Does not require authentication
-    
-    Arguments: 
-    **args -- The search terms
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _result().
-    
-    """
-    data = network.Download("http://search.twitter.com/search.json", util.compact(args))
-    data = data.get_json()["results"]
-
-    if type(data) != list:
-      logger.error("Cannot parse search data: %s", str(data))
-      return []
-
-    return [self._result(m) for m in data]
-
-  def __call__(self, opname, **args):
-    return getattr(self, opname)(**args)
-  
-  def receive(self, count=util.COUNT, since=None):
-    """Gets the latest tweets and adds them to the database.  
-    
-    Arguments: 
-    count -- Number of updates to get
-    since -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _message().
-    
-    """
-    return self._get("statuses/home_timeline.json", include_entities=1, count=count, since_id=since)
-
-  def responses(self, count=util.COUNT, since=None):
-    """Gets the latest replies and adds them to the database.
-    
-    Arguments: 
-    count  -- Number of updates to get
-    since  -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _responses().
-    
-    """
-    return self._get("statuses/mentions.json", "responses", include_entities=1, count=count, since_id=since)
-
-  def private(self, count=util.COUNT, since=None):
-    """Gets the latest direct messages sent and recieved and adds them to the database.
-    
-    Args: 
-    count -- Number of updates to get
-    since -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _private().
-    
-    """
-    private = self._get("direct_messages.json", "private", include_entities=1, count=count, since_id=since) or []
-    private_sent = self._get("direct_messages/sent.json", "private", count=count, since_id=since) or []
-    return private + private_sent
-
-  def public(self):
-    """Gets the latest tweets from the public timeline and adds them to the database.
-    
-    Arguments: 
-    None
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _message().
-    
-    """
-    return self._get("statuses/public_timeline.json", include_entities=1)
-
-  def lists(self, **args):
-    """Gets subscribed lists and adds them to the database.
-    
-    Arguments: 
-    None
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _list().
-    
-    """
-    if not "username" in self.account:
-      self._login()
-    following = self._get("%s/lists/subscriptions.json" % self.account["username"], "list") or []
-    lists = self._get("%s/lists.json" % self.account["username"], "list") or []
-    return following + lists
-
-  def list(self, user, id, count=util.COUNT, since=None):    
-    """Gets the latest tweets from subscribed lists and adds them to the database.
-    
-    Arguments: 
-    user -- The user's name whose lists are to be got
-    id -- The user's id whose lists are to be got
-    count -- Number of updates to get
-    since -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _message().
-    
-    """
-    return self._get("%s/lists/%s/statuses.json" % (user, id), include_entities=1, per_page=count, since_id=since)
-
-  def search(self, query, count=util.COUNT, since=None):
-    """Gets the latest results from a search and adds them to the database.
-    
-    Arguments: 
-    query -- The search query
-    count -- Number of updates to get
-    since -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _search().
-    
-    """
-    return self._search(include_entities=1, q=query, rpp=count, since_id=since)
-
-  def tag(self, query, count=util.COUNT, since=None):
-    """Gets the latest results from a hashtag search and adds them to the database.
-    
-    Arguments: 
-    query -- The search query (hashtag without the #)
-    count -- Number of updates to get
-    since -- Time to get updates since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _search().
-    
-    """
-    return self._search(q="#%s" % query, count=count, since_id=since)
-
-  def delete(self, message):
-    """Deletes a specified tweet from Twitter.
-    
-    Arguments: 
-    message -- A Gwibber compatible message object (from gwibber's database)
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("statuses/destroy/%s.json" % message["mid"], None, post=True, do=1)
-
-  def like(self, message):
-    """Favourites a specified tweet on Twitter.
-    
-    Arguments: 
-    message -- A Gwibber compatible message object (from Gwibber's database)
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("favorites/create/%s.json" % message["mid"], None, post=True, do=1)
-
-  def send(self, message):
-    """Sends a tweet to Twitter.
-    
-    Arguments: 
-    message -- The tweet's text
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("statuses/update.json", post=True, single=True,
-        status=message)
-  
-  def send_private(self, message, private):
-    """Sends a direct message to Twitter.
-    
-    Arguments: 
-    message -- The tweet's text
-    private -- A gwibber compatible user object (from gwibber's database)
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("direct_messages/new.json", "private", post=True, single=True,
-        text=message, screen_name=private["sender"]["nick"])
-
-  def send_thread(self, message, target):
-    """Sends a reply to a user on Twitter.
-    
-    Arguments: 
-    message -- The tweet's text
-    target -- A Gwibber compatible user object (from Gwibber's database)
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("statuses/update.json", post=True, single=True,
-        status=message, in_reply_to_status_id=target["mid"])
-
-  def retweet(self, message):
-    """Retweets a tweet.
-    
-    Arguments: 
-    message -- A Gwibber compatible message object (from gwibber's database)
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("statuses/retweet/%s.json" % message["mid"], None, post=True, do=1)
-
-  def follow(self, screen_name):
-    """Follows a user.
-    
-    Arguments: 
-    screen_name -- The screen name (@someone without the @) of the user to be followed
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("friendships/create.json", screen_name=screen_name, post=True, parse="follow")
-
-  def unfollow(self, screen_name):
-    """Unfollows a user.
-    
-    Arguments: 
-    screen_name -- The screen name (@someone without the @) of the user to be unfollowed
-    
-    Returns:
-    Nothing
-    
-    """
-    return self._get("friendships/destroy.json", screen_name=screen_name, post=True, parse="unfollow")
-
-  def profile(self, id=None, count=None, since=None):
-    """Gets a user's profile.
-    
-    Arguments: 
-    id -- The user's screen name
-    count -- Number of tweets to get
-    since -- Time to get tweets since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _profile().
-    
-    """
-    return self._get("users/show.json", screen_name=id, count=count, since_id=since, parse="profile")
-
-  def user_messages(self, id=None, count=util.COUNT, since=None):
-    """Gets a user's profile & timeline.
-    
-    Arguments: 
-    id -- The user's screen name
-    count -- Number of tweets to get
-    since -- Time to get tweets since
-    
-    Returns:
-    A list of Gwibber compatible objects which have been parsed by _profile().
-    
-    """
-    profiles = [self.profile(id)] or []
-    messages = self._get("statuses/user_timeline.json", id=id, include_entities=1, count=count, since_id=since) or []
-    return messages + profiles
 
=== added directory 'gwibber/tools'
=== added file 'gwibber/tools/debug_live.py'
--- gwibber/tools/debug_live.py	1970-01-01 00:00:00 +0000
+++ gwibber/tools/debug_live.py	2012-10-04 20:29:21 +0000
@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+
+"""Usage: ./tools/debug_live.py PROTOCOL OPERATION [OPTIONS]
+
+Where PROTOCOL is a protocol supported by Gwibber, such as 'twitter',
+OPERATION is an instance method defined in that protocol's class, and
+OPTIONS are whatever arguments you'd like to pass to that method (if
+any), such as message id's or a status message.
+
+Examples:
+
+./tools/debug_live.py twitter home
+./tools/debug_live.py twitter send 'Hello, world!'
+
+This tool is provided to aid with rapid feedback of changes made to
+the gwibber source tree, and as such is designed to be run from the
+same directory that contains 'setup.py'. It is not intended for use
+with an installed gwibber package.
+"""
+
+import sys
+
+sys.path.insert(0, '.')
+
+if len(sys.argv) < 3:
+    sys.exit(__doc__)
+
+protocol = sys.argv[1]
+args = sys.argv[2:]
+
+from gwibber.utils.account import AccountManager
+from gwibber.utils.model import Model
+from gwibber.utils.base import Base
+from gi.repository import GObject
+
+# Disable threading for easier testing.
+Base._SYNCHRONIZE = True
+
+def refresh(account):
+    print()
+    print('#' * 80)
+    print('Performing "{}" operation!'.format(args[0]))
+    print('#' * 80)
+
+    account.protocol(*args)
+    for row in Model:
+        print([col for col in row])
+        print()
+    print('ROWS: ', len(Model))
+    return True
+
+if __name__ == '__main__':
+
+    found = False
+    a = AccountManager(None)
+
+    for account_id, account in a._accounts.items():
+        if account_id.endswith(protocol):
+            found = True
+            refresh(account)
+            GObject.timeout_add_seconds(300, refresh, account)
+
+    if not found:
+        print('No {} account found in your Ubuntu Online Accounts!'.format(
+                protocol))
+    else:
+        GObject.MainLoop().run()
 
=== added file 'gwibber/tools/debug_slave.py'
--- gwibber/tools/debug_slave.py	1970-01-01 00:00:00 +0000
+++ gwibber/tools/debug_slave.py	2012-10-04 20:29:21 +0000
@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+
+from gi.repository import Dee
+from gi.repository import GObject
+
+
+class Slave:
+  def __init__(self):
+    model_name = 'com.Gwibber.Streams'
+    print('Joining model ' + model_name)
+    self.model = Dee.SharedModel.new(model_name)
+    self.model.connect('row-added', self.on_row_added)
+
+  def on_row_added(self, model, itr):
+    row = self.model.get_row(itr)
+    print(row)
+    print('ROWS: ', len(self.model))
+
+if __name__ == '__main__':
+
+  s = Slave()
+  GObject.MainLoop().run()