Gwibber

Merge lp:~robru/gwibber/twitter into lp:~barry/gwibber/py3

twitter
Merge into py3

Proposed by Robert Bruce Park on 2012-10-01

Status:	Merged
Merged at revision:	1445
Proposed branch:	lp:~robru/gwibber/twitter
Merge into:	lp:~barry/gwibber/py3
Diff against target:	2245 lines (+1180/-847) 12 files modified gwibber/gwibber/protocols/twitter.py (+275/-6) gwibber/gwibber/testing/mocks.py (+13/-7) gwibber/gwibber/tests/data/twitter-home.dat (+344/-0) gwibber/gwibber/tests/test_dbus.py (+5/-0) gwibber/gwibber/tests/test_download.py (+41/-0) gwibber/gwibber/tests/test_protocols.py (+52/-0) gwibber/gwibber/tests/test_twitter.py (+296/-8) gwibber/gwibber/utils/base.py (+31/-6) gwibber/gwibber/utils/download.py (+34/-1) gwibber/microblog/plugins/twitter/__init__.py (+0/-819) gwibber/tools/debug_live.py (+67/-0) gwibber/tools/debug_slave.py (+22/-0)
To merge this branch:	bzr merge lp:~robru/gwibber/twitter
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Barry Warsaw		2012-10-01	Pending
Review via email: mp+127373@code.launchpad.net

Description of the change

Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit prematurely because it's still the easiest way to see an overall diff ;-)

lp:~robru/gwibber/twitter updated on 2012-10-01

1474. By Robert Bruce Park on 2012-10-01

Implement Base._unpublish for the sake of deleting messages.

This required adding a new module global to base.py, _seen_ids, which
is a dict mapping message ids to SharedModel row iters. Without this
it would have been necessary to iterate over the entire model every
time you wanted to delete a single row.

1475. By Robert Bruce Park on 2012-10-01

Add a test case for Base._unpublish.

1476. By Robert Bruce Park on 2012-10-01

Fill out supported features in test_dbus.

All tests pass!

1477. By Robert Bruce Park on 2012-10-01

Uncomment line ;-)

Revision history for this message

Barry Warsaw (barry) wrote on 2012-10-02:

Download full text (29.8 KiB)

On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:

>Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>prematurely because it's still the easiest way to see an overall diff ;-)

The branch is looking pretty good. I'm glad you were able to verify that the
OAuth signatures were working, at least for public tweets. I guess more
research will have to be done to figure out why private replies are broken.

Anyway, here's a review of the branch so far. In generally, it looks pretty
good. There are a few error cases that need to be thought about, a few
additional tests that I think need to be written, and just a few minor style
nits here and there. But overall, it's looking fantastic and the small
problems should be easy to fix. I think this can be landed pretty quickly.

Detailed comments follow.

Do you intend to include the debug_twitter_*.py scripts in the branch? If so,
let's put them in a tools subdirectory (i.e. not in the gwibber Python
package). That way they'll be easier to omit from the Debian package.
Also...

=== added file 'gwibber/debug_twitter_live.py'
--- gwibber/debug_twitter_live.py 1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_live.py 2012-10-02 14:33:58 +0000
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +
> +from gwibber.utils.account import AccountManager
> +from gwibber.utils.model import Model
> +from gi.repository import GObject
> +
> +def refresh (account):
> + print ("############################# Receive ####################################")

This can probably be fit into 79 characters. :)

Also, no space between print and open paren. (PEP 8)

> + account.protocol.user()
> + account.protocol.delete('252823527978311680')

What does this number correspond to? Is it private information?

> + for row in Model:
> + print([col for col in row])
> + print()
> + print ("ROWS: ", len(Model))

extra space

> + return True
> +
> +if __name__ == "__main__":
> +
> + found = False
> + a = AccountManager(None)
> +
> + for account_id, account in a._accounts.items():
> + if account_id.endswith('twitter'):
> + found = True
> + refresh(account)
> + GObject.timeout_add_seconds(300, refresh, account)
> +
> + if not found:
> + print('No Twitter account found in your Ubuntu Online Accounts!')
> + else:
> + GObject.MainLoop().run()

=== added file 'gwibber/debug_twitter_slave.py'
--- gwibber/debug_twitter_slave.py 1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_slave.py 2012-10-02 14:33:58 +0000
> @@ -0,0 +1,23 @@
> +#!/usr/bin/env python3
> +
> +from gi.repository import Dee
> +from gi.repository import GObject
> +
> +
> +class Slave:
> + def __init__(self):

4 space indents (PEP 8).

> + model_name = "com.Gwibber.Streams"
> + print ("Joining model %s" % model_name)

extra space.

> + self.model = Dee.SharedModel.new(model_name)
> + self.model.connect("row-added", self.on_row_added)
> +
> + def on_row_added (self, model, itr):
> + row = self.model.get_row (itr)
> + print (row)
> + print ("ROWS: ", len(self.model))

extra spaces.

> +
> +if __name__ == "__main__":
> +
> + s =...

On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:

>Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>prematurely because it's still the easiest way to see an overall diff ;-)

The branch is looking pretty good.  I'm glad you were able to verify that the
OAuth signatures were working, at least for public tweets.  I guess more
research will have to be done to figure out why private replies are broken.

Anyway, here's a review of the branch so far.  In generally, it looks pretty
good.  There are a few error cases that need to be thought about, a few
additional tests that I think need to be written, and just a few minor style
nits here and there.  But overall, it's looking fantastic and the small
problems should be easy to fix.  I think this can be landed pretty quickly.

Detailed comments follow.

Do you intend to include the debug_twitter_*.py scripts in the branch?  If so,
let's put them in a tools subdirectory (i.e. not in the gwibber Python
package).  That way they'll be easier to omit from the Debian package.
Also...

=== added file 'gwibber/debug_twitter_live.py'
--- gwibber/debug_twitter_live.py	1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_live.py	2012-10-02 14:33:58 +0000
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +
> +from gwibber.utils.account import AccountManager
> +from gwibber.utils.model import Model
> +from gi.repository import GObject
> +
> +def refresh (account):
> +    print ("############################# Receive ####################################")

This can probably be fit into 79 characters. :)

Also, no space between print and open paren.  (PEP 8)

> +    account.protocol.user()
> +    account.protocol.delete('252823527978311680')

What does this number correspond to?  Is it private information?

> +    for row in Model:
> +        print([col for col in row])
> +        print()
> +    print ("ROWS: ", len(Model))

extra space

> +    return True
> +
> +if __name__ == "__main__":
> +
> +    found = False
> +    a = AccountManager(None)
> +
> +    for account_id, account in a._accounts.items():
> +        if account_id.endswith('twitter'):
> +            found = True
> +            refresh(account)
> +            GObject.timeout_add_seconds(300, refresh, account)
> +
> +    if not found:
> +        print('No Twitter account found in your Ubuntu Online Accounts!')
> +    else:
> +        GObject.MainLoop().run()

=== added file 'gwibber/debug_twitter_slave.py'
--- gwibber/debug_twitter_slave.py	1970-01-01 00:00:00 +0000
+++ gwibber/debug_twitter_slave.py	2012-10-02 14:33:58 +0000
> @@ -0,0 +1,23 @@
> +#!/usr/bin/env python3
> +
> +from gi.repository import Dee
> +from gi.repository import GObject
> +
> +
> +class Slave:
> +  def __init__(self):

4 space indents (PEP 8).

> +    model_name = "com.Gwibber.Streams"
> +    print ("Joining model %s" % model_name)

extra space.

> +    self.model = Dee.SharedModel.new(model_name)
> +    self.model.connect("row-added", self.on_row_added)
> +
> +  def on_row_added (self, model, itr):
> +    row = self.model.get_row (itr)
> +    print (row)
> +    print ("ROWS: ", len(self.model))

extra spaces.

> +
> +if __name__ == "__main__":
> +
> +  s = Slave ()
> +  GObject.MainLoop().run()
> +

=== modified file 'gwibber/gwibber/protocols/twitter.py'
--- gwibber/gwibber/protocols/twitter.py	2012-09-19 22:21:39 +0000
+++ gwibber/gwibber/protocols/twitter.py	2012-10-02 14:33:58 +0000
> @@ -14,20 +14,266 @@
>  
>  """The Twitter protocol plugin."""
>  
> +
>  __all__ = [
>      'Twitter',
>      ]
>  
>  
> -import gettext
>  import logging
>  
> -from gwibber.utils.base import Base
> +from oauthlib.oauth1 import Client
> +from urllib.parse import quote
> +
> +from gwibber.utils.authentication import Authentication
> +from gwibber.utils.base import Base, feature
> +from gwibber.utils.download import get_json
> +from gwibber.utils.time import parsetime, iso8601utc
>  
>  
>  log = logging.getLogger('gwibber.service')
> -_ = gettext.lgettext
>  
>  
>  class Twitter(Base):
> -    pass
> +    # StatusNet claims to mimick the Twitter API very closely (so
> +    # closely to the point that they refer you to the twitter API
> +    # reference docs as a starting point for learning about their
> +    # API), So these prefixes are defined here as class attributes
> +    # instead of the usual module globals, in the hopes that the
> +    # StatusNet class will be able to subclass Twitter and change only
> +    # the URLs, with minimal other changes, and magically work.
> +    _api_base = 'https://api.twitter.com/1.1/{endpoint}.json'
> +
> +    _timeline = _api_base.format(endpoint='statuses/{}_timeline')
> +    _user_timeline = _timeline.format('user') + '?screen_name={}'
> +
> +    _lists = _api_base.format(endpoint='lists/statuses') + '?list_id={}'
> +
> +    _url_prefix = 'https://twitter.com/'
> +    _tweet_permalink = 'https://twitter.com/{user_id}/status/{tweet_id}'
> +
> +    def _locked_login(self, old_token):
> +        result = Authentication(self._account, log).login()
> +        if result is None:
> +            log.error('No Twitter authentication results received.')
> +            return
> +
> +        token = result.get('AccessToken')
> +        if token is None:
> +            log.error('No AccessToken in Twitter session: {!r}', result)
> +        else:
> +            self._account.access_token = token
> +            self._account.secret_token = result.get('TokenSecret')
> +            self._account.user_id = result.get('UserId')
> +            self._account.user_name = result.get('ScreenName')
> +            log.debug('{} UID: {}'.format(self.__class__.__name__,
> +                                          self._account.user_id))
> +
> +    def _get_url(self, url, data=None):
> +        """Access the Twitter API with correct OAuth signed headers."""
> +        # TODO start enforcing some rate limiting.

What are your thoughts on how to do this rate limiting?  Is it something that
Twitter provides through their API?  (in a similar manner to Facebook's
pagination support?).  I wonder if there's some commonality refactoring that
should go on here, at least for determining the constants for the rate
limiting?

> +        do_post = data is not None
> +
> +        # "Client" == "Consumer" in oauthlib parlance.
> +        client_key = self._account.auth.parameters['ConsumerKey']
> +        client_secret = self._account.auth.parameters['ConsumerSecret']

Can these fail?  Are the other parts of your code prepared to handle the
resulting KeyErrors if these parameters are missing?  Do you have any tests
for those situations?

> +
> +        # "resource_owner" == secret and token.
> +        resource_owner_key = self._get_access_token()
> +        resource_owner_secret = self._account.secret_token
> +        oauth_client = Client(client_key, client_secret,
> +                              resource_owner_key, resource_owner_secret)
> +
> +        headers = {}
> +        if do_post:
> +            headers['Content-Type'] = 'application/x-www-form-urlencoded'
> +
> +        # All we care about is the headers, which will contain the
> +        # Authorization header necessary to satisfy OAuth.
> +        uri, headers, body = oauth_client.sign(
> +            url, body=data, headers=headers,
> +            http_method='POST' if do_post else 'GET')
> +
> +        return get_json(url,
> +                        params=data,
> +                        headers=headers,
> +                        post=do_post)
> +
> +    def _publish_tweet(self, tweet):
> +        """Publish a single tweet into the Dee.SharedModel."""
> +        tweet_id = tweet.get('id_str', '')

Does it still make sense to publish this tweet if there is no id_str?  Won't
that possibly result in row collisions, since the only unique item in the key
will be the empty string for multiple id_str-less entries?

I think in my Facebook branch, I just skip the JSON response entry if there's
no message_id.

> +        user = tweet.get('user', {})
> +        screen_name = user.get('screen_name', '')
> +        self._publish(
> +            message_id=tweet_id,
> +            message=tweet.get('text', ''),
> +            timestamp=iso8601utc(parsetime(tweet.get('created_at', ''))),
> +            stream='messages',
> +            sender=user.get('name', ''),
> +            sender_nick=screen_name,
> +            from_me=(screen_name == self._account.user_id),
> +            icon_uri=user.get('profile_image_url_https', ''),
> +            liked=tweet.get('favorited', False),
> +            url=self._tweet_permalink.format(user_id=screen_name,
> +                                             tweet_id=tweet_id),
> +            )
> +
> +# https://dev.twitter.com/docs/api/1.1/get/statuses/home_timeline
> +    @feature
> +    def home(self):
> +        """Gather the user's home timeline."""
> +        url = self._timeline.format('home')
> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
> +    @feature
> +    def mentions(self):
> +        """Gather the tweets that mention us."""
> +        url = self._timeline.format('mentions')
> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)

Do you have tests for each of these?  I know they *look* ridiculously similar,
but each should have a test.

> +
> +# https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline
> +    @feature
> +    def user(self, screen_name=''):
> +        """Gather the tweets from a specific user.
> +
> +        If screen_name is not specified, then gather the tweets
> +        written by the currently authenticated user.
> +        """
> +        url = self._user_timeline.format(screen_name)

So, if screen_name is not given, the resulting url will be something like:

https://api.twitter.com/1.1/statuses/user_timeline?screen_name=

is that a valid url for getting the tweets of the current user?  Or do you
need to check for screen_name missing and insert self.user_id or some such?
If the latter is the case, then you should default the screen_name argument to
None and check for that.

> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/get/lists/statuses
> +    @feature
> +    def list(self, list_id):
> +        """Gather the tweets from the specified list_id."""
> +        url = self._lists.format(list_id)
> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/get/lists/list
> +    @feature
> +    def lists(self):
> +        """Gather the tweets from the lists that the we are subscribed to."""
> +        url = self._api_base.format(endpoint='lists/list')
> +        for twitlist in self._get_url(url):
> +            self.list(twitlist.get('id_str', ''))

I think these methods are untested.

Also, what happens if get_json() returns an error?  Are these methods prepared
to handle problems in communicating with the service?  Of course, they're
going to be run in a sub-thread, but is there something better you can do than
let the sub-thread exceptions just fill up the log file?  Can you maybe catch
exceptions and print something more useful to the log file?

> +
> +# https://dev.twitter.com/docs/api/1.1/get/direct_messages
> +# https://dev.twitter.com/docs/api/1.1/get/direct_messages/sent
> +    @feature
> +    def private(self):
> +        """Gather the direct messages sent to/from us."""
> +        url = self._api_base.format(endpoint='direct_messages')
> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)
> +
> +        url = self._api_base.format(endpoint='direct_messages/sent')
> +        for tweet in self._get_url(url):
> +            self._publish_tweet(tweet)

Tests?

Also, this seems like quite a common idiom:

for tweet in self._get_url(url):
            self._publish_tweet(tweet)

Is there any opportunity for refactoring some commonality?

> +
> +    @feature
> +    def receive(self):
> +        """Gather and publish all incoming messages."""
> +        # TODO I know mentions and lists are actually incorporated
> +        # within the home timeline, but calling them explicitely will
> +        # ensure that more of those types of messages appear in the
> +        # timeline (eg, so they don't get drown out by everything
> +        # else). I'm not sure how necessary it is, though.
> +        self.home()
> +        self.mentions()
> +        self.lists()
> +        self.private()

What happens if you get an exception in one of these streams?  Does it make
sense to still try to publish the other streams?

> +
> +    @feature
> +    def send_private(self, screen_name, message):
> +        # TODO: This is utterly broken. 403s no matter what.

:(  Does it work in Quantal gwibber?

> +        url = self._api_base.format(endpoint='direct_messages/new')
> +        tweet = self._get_url(url, dict(text=message, screen_name=screen_name))
> +        self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/post/statuses/update
> +    @feature
> +    def send(self, message):
> +        url = self._api_base.format(endpoint='statuses/update')
> +        tweet = self._get_url(url, dict(status=message))
> +        self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/post/statuses/update
> +    @feature
> +    def send_thread(self, message_id, message):
> +        """Send a reply message to message_id.
> +
> +        Note that you have to @mention the message_id owner's screen name in
> +        order for Twitter to actually accept this as a reply. Otherwise it will
> +        just be an ordinary tweet."""

Closing triple-quote should be on the next line, by itself, under the 'j' in
'just'.

> +        url = self._api_base.format(endpoint='statuses/update')
> +        tweet = self._get_url(url, dict(in_reply_to_status_id=message_id,
> +                                        status=message))
> +        self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
> +    @feature
> +    def delete(self, message_id):
> +        url = self._api_base.format(endpoint='statuses/destroy/%s' % message_id)

Any reason you're using % interpolation here?  Perhaps either use {} and
.format(), or just concatenation (e.g.: 'statuses/destroy/' + message_id)?

> +        tweet = self._get_url(url, dict(trim_user='true'))
> +        self._unpublish(message_id)
> +
> +# https://dev.twitter.com/docs/api/1.1/post/statuses/retweet/%3Aid
> +    @feature
> +    def retweet(self, message_id):
> +        url = self._api_base.format(endpoint='statuses/retweet/%s' % message_id)

Same.

> +        tweet = self._get_url(url, dict(trim_user='true'))
> +        self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/post/friendships/destroy
> +    @feature
> +    def unfollow(self, screen_name):
> +        url = self._api_base.format(endpoint='friendships/destroy')
> +        self._get_url(url, dict(screen_name=screen_name))
> +
> +# https://dev.twitter.com/docs/api/1.1/post/friendships/create
> +    @feature
> +    def follow(self, screen_name):
> +        url = self._api_base.format(endpoint='friendships/create')
> +        self._get_url(url, dict(screen_name=screen_name, follow='true'))
> +
> +# https://dev.twitter.com/docs/api/1.1/post/favorites/create
> +    @feature
> +    def like(self, message_id):
> +        url = self._api_base.format(endpoint='favorites/create')
> +        tweet = self._get_url(url, dict(id=message_id))
> +        # I don't think we need to publish this tweet because presumably
> +        # the user has clicked the 'favorite' button on the message that's
> +        # already in the stream...
> +
> +# https://dev.twitter.com/docs/api/1.1/post/favorites/destroy
> +    @feature
> +    def unlike(self, message_id):
> +        url = self._api_base.format(endpoint='favorites/destroy')
> +        tweet = self._get_url(url, dict(id=message_id))
> +
> +# https://dev.twitter.com/docs/api/1.1/get/search/tweets
> +    @feature
> +    def tag(self, hashtag):
> +        """Return a list of some recent tweets mentioning hashtag.
> +
> +        Note, don't include the # symbol in the call to this method."""

Similar comment to above re: closing triple quotes.

> +        url = self._api_base.format(endpoint='search/tweets')
> +
> +        response = self._get_url('{}?q=%23{}'.format(url, hashtag))
> +        for tweet in response.get('statuses', []):
> +            self._publish_tweet(tweet)
> +
> +# https://dev.twitter.com/docs/api/1.1/get/search/tweets
> +    @feature
> +    def search(self, query):
> +        """Search for any arbitrary string."""
> +        url = self._api_base.format(endpoint='search/tweets')
> +
> +        response = self._get_url('{}?q={}'.format(url, quote(query, safe='')))
> +        for tweet in response.get('statuses', []):
> +            self._publish_tweet(tweet)

=== modified file 'gwibber/gwibber/tests/test_protocols.py'
--- gwibber/gwibber/tests/test_protocols.py	2012-09-25 20:25:42 +0000
+++ gwibber/gwibber/tests/test_protocols.py	2012-10-02 14:33:58 +0000
> @@ -176,6 +176,7 @@
>  
>      @mock.patch('gwibber.utils.base.Model', TestModel)
>      @mock.patch('gwibber.utils.base._seen_messages', {})
> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>      def test_one_message(self):
>          # Test that publishing a message inserts a row into the model.
>          base = Base(FakeAccount())
> @@ -215,6 +216,27 @@
>  
>      @mock.patch('gwibber.utils.base.Model', TestModel)
>      @mock.patch('gwibber.utils.base._seen_messages', {})
> +    @mock.patch('gwibber.utils.base._seen_ids', {})

Do you really need to mock both _seen_messages and _seen_ids?  What's the
difference between these two?  Can they ever get out of sync?  Do we need
additional tests to ensure that the constraints of these two data structures
are always correct?  Do we need additional tests for when they contain cache
hits?

> +    def test_unpublish(self):
> +        base = Base(FakeAccount())
> +        self.assertEqual(0, TestModel.get_n_rows())
> +        self.assertTrue(base._publish(
> +            message_id='1234',
> +            stream='messages',
> +            sender='fred',
> +            sender_nick='freddy',
> +            from_me=True,
> +            timestamp='today',
> +            message='hello, @jimmy',
> +            likes=10,
> +            liked=True))

Since this is specifically a test of deleting rows, you probably don't
necessarily need to test the return value of _publish().  I guess it can't
hurt either.

> +        self.assertEqual(1, TestModel.get_n_rows())
> +        base._unpublish('1234')
> +        self.assertEqual(0, TestModel.get_n_rows())
> +
> +    @mock.patch('gwibber.utils.base.Model', TestModel)
> +    @mock.patch('gwibber.utils.base._seen_messages', {})
> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>      def test_duplicate_messages_identified(self):
>          # When two messages which are deemed identical, by way of the
>          # _make_key() test in base.py, are published, only one ends up in the
> @@ -259,6 +281,7 @@
>  
>      @mock.patch('gwibber.utils.base.Model', TestModel)
>      @mock.patch('gwibber.utils.base._seen_messages', {})
> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>      def test_similar_messages_allowed(self):
>          # Because both the sender and message contribute to the unique key we
>          # use to identify messages, if two messages are published with

=== modified file 'gwibber/gwibber/tests/test_twitter.py'
--- gwibber/gwibber/tests/test_twitter.py	2012-09-21 14:11:34 +0000
+++ gwibber/gwibber/tests/test_twitter.py	2012-10-02 14:33:58 +0000
> @@ -14,6 +14,7 @@
>  
>  """Test the Twitter plugin."""
>  
> +
>  __all__ = [
>      'TestTwitter',
>      ]
> @@ -21,9 +22,13 @@
>  
>  import unittest
>  
> +from gi.repository import Dee
> +
>  from gwibber.protocols.twitter import Twitter
>  from gwibber.testing.helpers import FakeAccount
> -from gwibber.testing.mocks import LogMock
> +from gwibber.testing.mocks import FakeData, LogMock
> +from gwibber.utils.model import COLUMN_TYPES
> +
>  
>  try:
>      # Python 3.3
> @@ -32,10 +37,12 @@
>      import mock
>  
>  
> -# Ensure synchronicity between the main thread and the sub-thread.  Also, set
> -# up the loggers for the modules-under-test so that we can assert their error
> -# messages.
> -@mock.patch.dict('gwibber.utils.base.__dict__', {'_SYNCHRONIZE': True})
> +# Create a test model that will not interfere with the user's environment.
> +# We'll use this object as a mock of the real model.
> +TestModel = Dee.SharedModel.new('com.Gwibber.TestSharedModel')
> +TestModel.set_schema_full(COLUMN_TYPES)
> +
> +
>  class TestTwitter(unittest.TestCase):
>      """Test the Twitter API."""
>  
> @@ -46,8 +53,101 @@
>                                  'gwibber.protocols.twitter')
>  
>      def tearDown(self):
> +        # Ensure that any log entries we haven't tested just get consumed so
> +        # as to isolate out test logger from other tests.
>          self.log_mock.stop()

The diff doesn't show it, but does setUp() set Base._SYNCHRONIZE = True?

>  
> -    def test_protocol_info(self):
> -        # Each protocol carries with it a number of protocol variables.
> -        self.assertEqual(self.protocol.__class__.__name__, 'Twitter')
> +    @mock.patch('gwibber.utils.authentication.Authentication.login',
> +                return_value=None)
> +    @mock.patch('gwibber.utils.download.get_json',
> +                return_value=None)
> +    def test_unsuccessful_authentication(self, *mocks):
> +        self.assertFalse(self.protocol._login())
> +        self.assertIsNone(self.account.user_name)
> +        self.assertIsNone(self.account.user_id)
> +
> +    @mock.patch('gwibber.utils.authentication.Authentication.login',
> +                return_value=dict(AccessToken='some clever fake data',
> +                                  TokenSecret='sssssshhh!',
> +                                  UserId='rickygervais',
> +                                  ScreenName='Ricky Gervais'))
> +    def test_successful_authentication(self, *mocks):
> +        self.assertTrue(self.protocol._login())
> +        self.assertEqual(self.account.user_name, 'Ricky Gervais')
> +        self.assertEqual(self.account.user_id, 'rickygervais')
> +        self.assertEqual(self.account.access_token, 'some clever fake data')
> +        self.assertEqual(self.account.secret_token, 'sssssshhh!')
> +
> +

Extra blank line.

> +    @mock.patch('gwibber.protocols.twitter.get_json', lambda *x, **y: y)
> +    @mock.patch('oauthlib.oauth1.rfc5849.generate_nonce',
> +                lambda: 'once upon a nonce')
> +    @mock.patch('oauthlib.oauth1.rfc5849.generate_timestamp',
> +                lambda: '1348690628')
> +    def test_signatures(self, *mocks):
> +        self.account.secret_token = 'alpha'
> +        self.account.access_token = 'omega'
> +        self.account.auth.id = 6
> +        self.account.auth.method = 'oauth2'
> +        self.account.auth.mechanism = 'HMAC-SHA1'
> +        self.account.auth.parameters = dict(ConsumerKey='consume',
> +                                            ConsumerSecret='obey')
> +        result = '''\
> +OAuth oauth_nonce="once%20upon%20a%20nonce", \
> +oauth_timestamp="1348690628", \
> +oauth_version="1.0", \
> +oauth_signature_method="HMAC-SHA1", \
> +oauth_consumer_key="consume", \
> +oauth_token="omega", \
> +oauth_signature="2MlC4DOqcAdCUmU647izPmxiL%2F0%3D"'''
> +
> +        self.assertEqual(
> +            self.protocol._get_url('http://example.com')['headers'],
> +            dict(Authorization=result))
> +
> +    @mock.patch('gwibber.utils.base.Model', TestModel)
> +    @mock.patch('gwibber.utils.download.urlopen',
> +                FakeData('gwibber.tests.data', 'twitter-home.dat'))
> +    @mock.patch('gwibber.protocols.twitter.Twitter._login',
> +                return_value=True)
> +    def test_home(self, *mocks):
> +        self.account.access_token = 'access'
> +        self.account.secret_token = 'secret'
> +        self.account.auth.parameters = dict(
> +            ConsumerKey='key',
> +            ConsumerSecret='secret')
> +        self.assertEqual(0, TestModel.get_n_rows())
> +        self.protocol.home()
> +        self.assertEqual(3, TestModel.get_n_rows())
> +
> +        # This test data was ripped directly from Twitter's API docs.
> +        expected = [
> +            [[['twitter', 'faker/than fake', '240558470661799936']],
> +             'messages', 'OAuth Dancer', 'oauth_dancer', False,
> +             '2012-08-28T21:16:23', 'just another test', '',
> +             'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
> +             'https://twitter.com/oauth_dancer/status/240558470661799936', '',
> +             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
> +             '', '', '', '', '', '', '', '', '', '',
> +             ],
> +            [[['twitter', 'faker/than fake', '240556426106372096']],
> +             'messages', 'Raffi Krikorian', 'raffi', False,
> +             '2012-08-28T21:08:15', 'lecturing at the "analyzing big data ' +
> +             'with twitter" class at @cal with @othman  http://t.co/bfj7zkDJ', '',
> +             'https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png',
> +             'https://twitter.com/raffi/status/240556426106372096', '',
> +             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
> +             '', '', '', '', '', '', '', '', '', '',
> +             ],
> +            [[['twitter', 'faker/than fake', '240539141056638977']],
> +             'messages', 'Taylor Singletary', 'episod', False,
> +             '2012-08-28T19:59:34', 'You\'d be right more often if you thought you were wrong.', '',
> +             'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
> +             'https://twitter.com/episod/status/240539141056638977', '',
> +             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
> +             '', '', '', '', '', '', '', '', '', '',
> +             ],
> +            ]
> +        for i, expected_row in enumerate(expected):
> +            for got, want in zip(TestModel.get_row(i), expected_row):
> +                self.assertEqual(got, want)

=== modified file 'gwibber/gwibber/utils/base.py'
--- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
+++ gwibber/gwibber/utils/base.py	2012-10-02 14:33:58 +0000
> @@ -42,6 +42,7 @@
>  # representing the rows matching those keys.  It is used for quickly finding
>  # duplicates when we want to insert new rows into the model.
>  _seen_messages = {}
> +_seen_ids = {}

What is _seen_ids for and how is it used differently than _seen_messages?  Do
we still need both of these?  If so, can you adjust the comment to explain
what these are for?

>  
>  
>  # Protocol __call__() methods run in threads, so we need to serialize
> @@ -204,7 +205,7 @@
>              row_iter = _seen_messages.get(key)
>              if row_iter is None:
>                  # We haven't seen this message before.
> -                _seen_messages[key] = Model.append(*args)
> +                _seen_ids[message_id] = _seen_messages[key] = Model.append(*args)
>              else:
>                  # We have seen this before, so append to the matching column's
>                  # message_ids list, this message's id.
> @@ -221,6 +222,28 @@
>                  row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
>              return key in _seen_messages
>  
> +    def _unpublish(self, message_id):
> +        """Remove message_id from the Dee.SharedModel."""
> +        triple = [self.__class__.__name__.lower(),
> +                  self._account.id,
> +                  message_id]
> +
> +        row_iter = _seen_ids.pop(message_id)
> +        if row_iter is None:
> +            log.error('Tried to delete an invalid message id.')
> +            return

Have you tested the 'miss' logic?  What happens when you .pop() with a missing
key?  (this is a trick question :)

What happens if the .pop() does work?  Does anything need to happen to
_seen_messages to keep it in sync?  Isn't the row associated with the row_iter
going to be removed?  If so, what happens to the same row_iter sitting as a
value in the _seen_messages mapping?

> +
> +        row = Model.get_row(row_iter)
> +        if triple in row[IDS_IDX]:
> +            if len(row[IDS_IDX]) == 1:
> +                # Message only exists on Twitter, delete it

Actually, the method isn't Twitter specific is it?  If you think _unpublish()
is only useful for Twitter, should it really go in the base class?  Or maybe
the comment just needs updating?

> +                Model.remove(row_iter)
> +            else:
> +                # Message exists on other protocols too, only drop id
> +                row[IDS_IDX] = [ids for ids
> +                                in row[IDS_IDX]
> +                                if message_id not in ids]
> +
>      def _get_access_token(self):
>          """Return an access token, logging in if necessary."""
>          if self._account.access_token is None:

=== modified file 'gwibber/gwibber/utils/download.py'
--- gwibber/gwibber/utils/download.py	2012-09-14 20:29:11 +0000
+++ gwibber/gwibber/utils/download.py	2012-10-02 14:33:58 +0000
> @@ -51,11 +51,11 @@
>          url = self.url
>          headers = self.headers.copy()
>          if self.params is not None:
> -            params = urlencode(self.params)
> +            params = urlencode(self.params).replace('+', '%20')

Why do this?  (Yes, I know you explained it to me, but pretend I'm a
first-time reader of the code.  Explain it to them! :).

>              if self.post:
>                  data = params.encode('utf-8')
>                  headers['Content-Type'] = (
> -                    'application/x-www-form-urlencoded; charset=utf-8')
> +                    'application/x-www-form-urlencoded')

Why do this too?  This one you haven't explained even to me! :)

>              else:
>                  # Do a GET with an encoded query string.
>                  url = '{}?{}'.format(self.url, params)

Revision history for this message

Robert Bruce Park (robru) wrote on 2012-10-03:

Download full text (19.3 KiB)

On 12-10-02 02:13 PM, Barry Warsaw wrote:
> On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:
>
>> Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>> prematurely because it's still the easiest way to see an overall diff ;-)
>
> The branch is looking pretty good. I'm glad you were able to verify that the
> OAuth signatures were working, at least for public tweets.

I actually verified that the OAuth signatures worked for *every* API
endpoint *except* sending new direct messages (we can receive them ok,
too). That's what debug_twitter_live.py was all about, every time I'd
write a new protocol operation, I'd modify it to do that new operation,
then run it, and I'd see on my live twitter account that it worked.

> I guess more
> research will have to be done to figure out why private replies are broken.

Yeah, I still have no idea.

> Do you intend to include the debug_twitter_*.py scripts in the branch? If so,
> let's put them in a tools subdirectory (i.e. not in the gwibber Python
> package). That way they'll be easier to omit from the Debian package.

I hadn't originally. At first it just started off as 'from
gwibber.utils.account import AccountManager; a = AccountManager(None);
a._accounts['6/twitter'].protocol.receive()' because I was sick of
typing that out 10,000 times in an interactive python shell every time I
wanted to test a change I made to the code. But once the script was
born, it grew, and then ken expanded it for his own testing purposes,
which is included here as well.

In fact, *all* of the style nits you mention were written by ken ;-)

I had assumed that you would just not merge them. If you think there's
value in keeping them in a tools directory, I guess that's ok. I don't
have a strong opinion either way.

>> + account.protocol.user()
>> + account.protocol.delete('252823527978311680')
>
> What does this number correspond to? Is it private information?

That is a public tweet I made with a previous incarnation of this
script, that I've now deleted by running this script as-is. No worries.

>> + def _get_url(self, url, data=None):
>> + """Access the Twitter API with correct OAuth signed headers."""
>> + # TODO start enforcing some rate limiting.
>
> What are your thoughts on how to do this rate limiting? Is it something that
> Twitter provides through their API? (in a similar manner to Facebook's
> pagination support?). I wonder if there's some commonality refactoring that
> should go on here, at least for determining the constants for the rate
> limiting?

I haven't looked closely at Twitter's rate limiting standards yet, but I
wrote that comment because I ran into Twitter's limit at one point and
they stopped me from downloading tweets that I wanted.

On 12-10-02 02:13 PM, Barry Warsaw wrote:
> On Oct 01, 2012, at 09:30 PM, Robert Bruce Park wrote:
>
>> Twitter branch mostly done. Some TODOs remain. I am mp'ing a little bit
>> prematurely because it's still the easiest way to see an overall diff ;-)
>
> The branch is looking pretty good.  I'm glad you were able to verify that the
> OAuth signatures were working, at least for public tweets.

I actually verified that the OAuth signatures worked for *every* API 
endpoint *except* sending new direct messages (we can receive them ok, 
too). That's what debug_twitter_live.py was all about, every time I'd 
write a new protocol operation, I'd modify it to do that new operation, 
then run it, and I'd see on my live twitter account that it worked.

>  I guess more
> research will have to be done to figure out why private replies are broken.

Yeah, I still have no idea.

> Do you intend to include the debug_twitter_*.py scripts in the branch?  If so,
> let's put them in a tools subdirectory (i.e. not in the gwibber Python
> package).  That way they'll be easier to omit from the Debian package.

I hadn't originally. At first it just started off as 'from 
gwibber.utils.account import AccountManager; a = AccountManager(None); 
a._accounts['6/twitter'].protocol.receive()' because I was sick of 
typing that out 10,000 times in an interactive python shell every time I 
wanted to test a change I made to the code. But once the script was 
born, it grew, and then ken expanded it for his own testing purposes, 
which is included here as well.

In fact, *all* of the style nits you mention were written by ken ;-)

I had assumed that you would just not merge them. If you think there's 
value in keeping them in a tools directory, I guess that's ok. I don't 
have a strong opinion either way.

>> +    account.protocol.user()
>> +    account.protocol.delete('252823527978311680')
>
> What does this number correspond to?  Is it private information?

That is a public tweet I made with a previous incarnation of this 
script, that I've now deleted by running this script as-is. No worries.

>> +    def _get_url(self, url, data=None):
>> +        """Access the Twitter API with correct OAuth signed headers."""
>> +        # TODO start enforcing some rate limiting.
>
> What are your thoughts on how to do this rate limiting?  Is it something that
> Twitter provides through their API?  (in a similar manner to Facebook's
> pagination support?).  I wonder if there's some commonality refactoring that
> should go on here, at least for determining the constants for the rate
> limiting?

I haven't looked closely at Twitter's rate limiting standards yet, but I 
wrote that comment because I ran into Twitter's limit at one point and 
they stopped me from downloading tweets that I wanted.

All I know is that each API endpoint has a different maximum number of 
requests that can be made "per window" and the length of time of the 
window is different, too. So basically I'm envisioning a dict that maps 
URLs to ints, the int for a given URL will increase once per each 
_get_url invocation, and then we just need to figure out how long to 
pause before making the request in order not to hit the limit. Pausing 
shouldn't be a big deal because everything is threaded anyway, so 
unrelated requests can continue in other threads while a rate-limited 
request can sleep peacefully in it's own thread.

>> +        # "Client" == "Consumer" in oauthlib parlance.
>> +        client_key = self._account.auth.parameters['ConsumerKey']
>> +        client_secret = self._account.auth.parameters['ConsumerSecret']
>
> Can these fail?  Are the other parts of your code prepared to handle the
> resulting KeyErrors if these parameters are missing?  Do you have any tests
> for those situations?

No, these are constants defined by Twitter at app-registration time, 
stored in libaccounts-sso package. Can never fail. They may change in 
the event that somebody nefarious compromises them (which ought to be 
easy to do for an open-source package), but they won't change during 
runtime.

>> +    def _publish_tweet(self, tweet):
>> +        """Publish a single tweet into the Dee.SharedModel."""
>> +        tweet_id = tweet.get('id_str', '')
>
> Does it still make sense to publish this tweet if there is no id_str?  Won't
> that possibly result in row collisions, since the only unique item in the key
> will be the empty string for multiple id_str-less entries?
>
> I think in my Facebook branch, I just skip the JSON response entry if there's
> no message_id.

Yeah, that's probably reasonable. I just write 'adict.get' out of habit 
for the sake of being as flexible as possible.

>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
>> +    @feature
>> +    def mentions(self):
>> +        """Gather the tweets that mention us."""
>> +        url = self._timeline.format('mentions')
>> +        for tweet in self._get_url(url):
>> +            self._publish_tweet(tweet)
>
> Do you have tests for each of these?  I know they *look* ridiculously similar,
> but each should have a test.

I wasn't really sure how to test them, to be honest. 'test_home' is 
really an indirect test of _get_url and _publish_tweet, and then these 
individual operations, I wasn't sure how to test them without touching 
the live service. Or the opposite way, a nearly no-op test that mocks 
out *everything* so aggressively and then just tests that it calls the 
method that it calls. I guess that's what you're looking for, then? Mock 
out self._get_url and self._publish_tweet and just test that _get_url 
was called with the right URL, and then _publish_tweet was called one or 
more times?

>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline
>> +    @feature
>> +    def user(self, screen_name=''):
>> +        """Gather the tweets from a specific user.
>> +
>> +        If screen_name is not specified, then gather the tweets
>> +        written by the currently authenticated user.
>> +        """
>> +        url = self._user_timeline.format(screen_name)
>
> So, if screen_name is not given, the resulting url will be something like:
>
> https://api.twitter.com/1.1/statuses/user_timeline?screen_name=
>
> is that a valid url for getting the tweets of the current user?  Or do you
> need to check for screen_name missing and insert self.user_id or some such?
> If the latter is the case, then you should default the screen_name argument to
> None and check for that.

Nope, '...?screen_name=' is a valid way of requesting the current user's 
tweets. I tested this all with debug_twitter_live.py as I was writing it ;-)

>> +# https://dev.twitter.com/docs/api/1.1/get/lists/list
>> +    @feature
>> +    def lists(self):
>> +        """Gather the tweets from the lists that the we are subscribed to."""
>> +        url = self._api_base.format(endpoint='lists/list')
>> +        for twitlist in self._get_url(url):
>> +            self.list(twitlist.get('id_str', ''))
>
> I think these methods are untested.
>
> Also, what happens if get_json() returns an error?  Are these methods prepared
> to handle problems in communicating with the service?  Of course, they're
> going to be run in a sub-thread, but is there something better you can do than
> let the sub-thread exceptions just fill up the log file?  Can you maybe catch
> exceptions and print something more useful to the log file?

Well, I forget the exact error code, but in the event that we hit the 
rate limit, get_json will raise an exception with an appropriate HTTP 
error code and a meaningful string along with it, such as 'rate limit 
exceeded' or something similar. I didn't really see any issue with that 
just bubbling straight up into the log file. No need to duplicate the 
same error-checking code directly in every single method.

> Also, this seems like quite a common idiom:
>
>          for tweet in self._get_url(url):
>              self._publish_tweet(tweet)
>
> Is there any opportunity for refactoring some commonality?

Originally I had written the method as _publish_tweets (plural) and it 
was built-in with the ability to iterate over multiple tweets. That 
worked for a good long while, but then there were just a couple API 
endpoints that only returned a single tweet object rather than a list of 
tweets, and so then I needed to refactor the method to be able to handle 
both cases, and this is what I came up with.

If you really want, we can have something like this:

def _publish_tweets(self, url):
     for tweet in self._get_url(url):
         self._publish_tweet(tweet)

But I just thought it was too simple to even bother.

>> +    @feature
>> +    def receive(self):
>> +        """Gather and publish all incoming messages."""
>> +        # TODO I know mentions and lists are actually incorporated
>> +        # within the home timeline, but calling them explicitely will
>> +        # ensure that more of those types of messages appear in the
>> +        # timeline (eg, so they don't get drown out by everything
>> +        # else). I'm not sure how necessary it is, though.
>> +        self.home()
>> +        self.mentions()
>> +        self.lists()
>> +        self.private()
>
> What happens if you get an exception in one of these streams?  Does it make
> sense to still try to publish the other streams?

What kind of exceptions are you suspecting there to be? The only thing I 
can think of would be in the event of a failed login, but if that was 
the case then they would all fail equally, and really the user would 
have to go back to UOA and re-register his account properly. Not a lot 
we can do about that here.

>
>> +
>> +    @feature
>> +    def send_private(self, screen_name, message):
>> +        # TODO: This is utterly broken. 403s no matter what.
>
> :(  Does it work in Quantal gwibber?

It seems to. As you said, more research is necessary.

One problem is that quantal gwibber is using exclusively Twitter API v1, 
which is deprecated and will be going away actually quite soon (good 
timing for this rewrite!). My code is all using the newer API v1.1, so 
it's not always helpful to look at what quantal gwibber does, because 
the API isn't always the same.

>> +# https://dev.twitter.com/docs/api/1.1/post/statuses/update
>> +    @feature
>> +    def send_thread(self, message_id, message):
>> +        """Send a reply message to message_id.
>> +
>> +        Note that you have to @mention the message_id owner's screen name in
>> +        order for Twitter to actually accept this as a reply. Otherwise it will
>> +        just be an ordinary tweet."""
>
> Closing triple-quote should be on the next line, by itself, under the 'j' in
> 'just'.

Oh yeah, oops.

>> +# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
>> +    @feature
>> +    def delete(self, message_id):
>> +        url = self._api_base.format(endpoint='statuses/destroy/%s' % message_id)
>
> Any reason you're using % interpolation here?  Perhaps either use {} and
> .format(), or just concatenation (e.g.: 'statuses/destroy/' + message_id)?

Yeah, it looks really ugly to me when I see 
'string{}'.format('foo{}'.format('bar')), but I also wanted to allow 
message_id to be an int rather than requiring it be a string, which is 
what + concatenation would have done.

% interpolation just made the int/str conversion easy without being too 
verbose.

>> +# https://dev.twitter.com/docs/api/1.1/post/statuses/retweet/%3Aid
>> +    @feature
>> +    def retweet(self, message_id):
>> +        url = self._api_base.format(endpoint='statuses/retweet/%s' % message_id)
>
> Same.

Same. :-P

>>       @mock.patch('gwibber.utils.base.Model', TestModel)
>>       @mock.patch('gwibber.utils.base._seen_messages', {})
>> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>
> Do you really need to mock both _seen_messages and _seen_ids?  What's the
> difference between these two?  Can they ever get out of sync?  Do we need
> additional tests to ensure that the constraints of these two data structures
> are always correct?  Do we need additional tests for when they contain cache
> hits?

Sorry, I was in a rush when I wrote this. Base._publish now adds to both 
of these so it seemed safer to mock both for consistency, but it's not 
clear to me right off the top of my head whether it's strictly necessary 
for the test itself.

>> +    def test_unpublish(self):
>> +        base = Base(FakeAccount())
>> +        self.assertEqual(0, TestModel.get_n_rows())
>> +        self.assertTrue(base._publish(
>> +            message_id='1234',
>> +            stream='messages',
>> +            sender='fred',
>> +            sender_nick='freddy',
>> +            from_me=True,
>> +            timestamp='today',
>> +            message='hello, @jimmy',
>> +            likes=10,
>> +            liked=True))
>
> Since this is specifically a test of deleting rows, you probably don't
> necessarily need to test the return value of _publish().  I guess it can't
> hurt either.

Yeah, this was just a quick copy&paste from a different test, in order 
to set up an environment in which to test message deletion.

>>       def tearDown(self):
>> +        # Ensure that any log entries we haven't tested just get consumed so
>> +        # as to isolate out test logger from other tests.
>>           self.log_mock.stop()
>
> The diff doesn't show it, but does setUp() set Base._SYNCHRONIZE = True?

No, I don't need the synchronize stuff because I write the tests to just 
test the synchronous methods directly. I never invoke the 
thread-starting __call__ in the tests.

> === modified file 'gwibber/gwibber/utils/base.py'
> --- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
> +++ gwibber/gwibber/utils/base.py	2012-10-02 14:33:58 +0000
>> @@ -42,6 +42,7 @@
>>   # representing the rows matching those keys.  It is used for quickly finding
>>   # duplicates when we want to insert new rows into the model.
>>   _seen_messages = {}
>> +_seen_ids = {}
>
> What is _seen_ids for and how is it used differently than _seen_messages?  Do
> we still need both of these?  If so, can you adjust the comment to explain
> what these are for?

_seen_ids maps message ids to rowiters. It was the only way I could 
think to allow deletion of messages without iterating over the whole 
SharedModel terribly. Because at delete-time, we only have access to the 
message_id, not the sender & message data that would be required to look 
up the rowiter in _seen_messages.

>> +    def _unpublish(self, message_id):
>> +        """Remove message_id from the Dee.SharedModel."""
>> +        triple = [self.__class__.__name__.lower(),
>> +                  self._account.id,
>> +                  message_id]
>> +
>> +        row_iter = _seen_ids.pop(message_id)
>> +        if row_iter is None:
>> +            log.error('Tried to delete an invalid message id.')
>> +            return
>
> Have you tested the 'miss' logic?  What happens when you .pop() with a missing
> key?  (this is a trick question :)

Well, I haven't specified a default, which means it will raise KeyError 
if the id is not present. This doesn't particularly bother me, because 
a) the threading logic will safely log and ignore the exception, and b) 
this can only be invoked by clicking on the delete button on a message 
that's being displayed in gwibber, so the message_id being valid is 
almost *tautological*: if message_id wasn't valid, then how the hell did 
this method even get invoked?

> What happens if the .pop() does work?  Does anything need to happen to
> _seen_messages to keep it in sync?  Isn't the row associated with the row_iter
> going to be removed?  If so, what happens to the same row_iter sitting as a
> value in the _seen_messages mapping?

Right, they would become out of sync. But only insofar as there will be 
a sender/message key pointing at a rowiter that points at a row that 
doesn't exist. I'm not certain that this is particularly harmful, beyond 
being a minor memory leak.

I suppose in some extreme case if a person deletes a message and then 
tries to submit an identical message, it would cause us to attempt to 
append that message onto a row that doesn't exist. I don't really see 
that being a useful real-world scenario, because in my mind the 
use-cases for deleting a message would be to either a) leave it deleted, 
or b) to submit a *different* message in it's place.

Anyway, do you know of an easy way to do a dict.pop() that pops based on 
values instead of keys? Or are we going to have to scan the whole dict 
in an inefficient way? Or is there maybe some kind of alternate data 
structure we should be looking at? Some kind two-way dict where both 
keys and values are hashed and indexed?

>
>> +
>> +        row = Model.get_row(row_iter)
>> +        if triple in row[IDS_IDX]:
>> +            if len(row[IDS_IDX]) == 1:
>> +                # Message only exists on Twitter, delete it
>
> Actually, the method isn't Twitter specific is it?  If you think _unpublish()
> is only useful for Twitter, should it really go in the base class?  Or maybe
> the comment just needs updating?

Sorry, yeah, comment needs updating. It is a general-purpose method. It 
would need to be defined in the base class anyway because of it's use of 
IDS_IDX, Model, etc. Those are things that aren't imported into 
twitter.py, and would be sloppy to do so when the imports are all 
already here.

> === modified file 'gwibber/gwibber/utils/download.py'
> --- gwibber/gwibber/utils/download.py	2012-09-14 20:29:11 +0000
> +++ gwibber/gwibber/utils/download.py	2012-10-02 14:33:58 +0000
>> @@ -51,11 +51,11 @@
>>           url = self.url
>>           headers = self.headers.copy()
>>           if self.params is not None:
>> -            params = urlencode(self.params)
>> +            params = urlencode(self.params).replace('+', '%20')
>
> Why do this?  (Yes, I know you explained it to me, but pretend I'm a
> first-time reader of the code.  Explain it to them! :).

Because urlencode does not provide any option to use quote() instead of 
quote_plus() which it enforces, but Twitter requires percent-encoded 
spaces, and this is harmless to any other protocol. Also it's easier 
than re-implementing urlencode from scratch. hooray for code reuse!

>
>>               if self.post:
>>                   data = params.encode('utf-8')
>>                   headers['Content-Type'] = (
>> -                    'application/x-www-form-urlencoded; charset=utf-8')
>> +                    'application/x-www-form-urlencoded')
>
> Why do this too?  This one you haven't explained even to me! :)

It's because oauthlib is incredibly brittle. It scans for precisely the 
string 'application/x-www-form-urlencoded' and raises an exception if 
it's not exactly that. So it seems this was necessary in order to get 
OAuth signatures to even function.

On second thought however, it's not clear to me what oauthlib actually 
*does* with that header when it's signing (I'm pretty sure the value of 
that header is not included in the base string that gets signed). So it 
might be worthwhile to undo this change and then see if anything breaks. 
This was just one of those things that I did back when I was really 
struggling to get any signature working at all, and then after I got it 
working, I kept it because a) I forgot that I'd done it, and b) it 
didn't occur to me until just now that it might not have been necessary 
to make the signatures work.

So in other words, I'll test this ;-)

Sorry I'm not on IRC, I just got back into town from my big train trip 
and I'm having some troubles with emacs24/system updaty type problems. 
I'll get it working shortly and then fix up this mp for you by tonight.

lp:~robru/gwibber/twitter updated on 2012-10-03

1478. By Robert Bruce Park on 2012-10-03

Allow debug_twitter_live.py to call any action from the commandline.

Commandline args are passed directly to protocol.__call__(), so any
feature supported by Twitter can be invoked directly by this script.

1479. By Robert Bruce Park on 2012-10-03

Generalize debug_twitter_*.py scripts into debug_*.py scripts.

debug_live has a good docstring that explains its usage.

1480. By Robert Bruce Park on 2012-10-03

Add tests for Twitter operations.

This also tweaks Twitter.tag so that it silently ignores the # symbols
in hash tags, allowing you to include them or not, at your option.
Previously, if you tried to Twitter.tag('#hashtag'), it wouldn't work,
you had to strip the # symbol yourself. Now we accept both.

1481. By Robert Bruce Park on 2012-10-03

Docstring cleanup.

1482. By Robert Bruce Park on 2012-10-03

Fix bug in Base._unpublish

1483. By Robert Bruce Park on 2012-10-03

Improve _seen_ids / _seen_messages consistency.

Also fix a comment.

Revision history for this message

Barry Warsaw (barry) wrote on 2012-10-03:

Download full text (17.3 KiB)

On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:

>I actually verified that the OAuth signatures worked for *every* API endpoint
>*except* sending new direct messages (we can receive them ok, too). That's
>what debug_twitter_live.py was all about, every time I'd write a new protocol
>operation, I'd modify it to do that new operation, then run it, and I'd see
>on my live twitter account that it worked.

You may have to engage with the Twitter developer community to debug this. I
didn't find anything relevant in a very little bit of googling though. You'd
think that if it were a bug in their API, others would have noticed it.

>I hadn't originally. At first it just started off as 'from
>gwibber.utils.account import AccountManager; a = AccountManager(None);

Just FWIW, I have a patch in my facebook.working branch that makes None the
default argument. You have to be careful not to call the callback if it's
None. I wonder if it's useful to land that in the py3 branch separately?

>In fact, *all* of the style nits you mention were written by ken ;-)

/me hears Kirk's voice scream to the heavens "Keeeeeeennnnnn!"

>I haven't looked closely at Twitter's rate limiting standards yet, but I
>wrote that comment because I ran into Twitter's limit at one point and they
>stopped me from downloading tweets that I wanted.

I think Twitter warns you (via response headers?) if you're getting rate
limited. So I guess you could check for those and automatically back-off if
you see them. It's probably better to at least have some combination of
proactive and reactive response to rate limiting. The gory details:

https://dev.twitter.com/docs/rate-limiting

>>> + # "Client" == "Consumer" in oauthlib parlance.
>>> + client_key = self._account.auth.parameters['ConsumerKey']
>>> + client_secret = self._account.auth.parameters['ConsumerSecret']
>>
>> Can these fail? Are the other parts of your code prepared to handle the
>> resulting KeyErrors if these parameters are missing? Do you have any tests
>> for those situations?
>
>No, these are constants defined by Twitter at app-registration time, stored
>in libaccounts-sso package. Can never fail. They may change in the event that
>somebody nefarious compromises them (which ought to be easy to do for an
>open-source package), but they won't change during runtime.

Cool. In that case, the code above is fine because if they *do* fail, you
want to know via the logged KeyErrors.

>>> + def _publish_tweet(self, tweet):
>>> + """Publish a single tweet into the Dee.SharedModel."""
>>> + tweet_id = tweet.get('id_str', '')
>>
>> Does it still make sense to publish this tweet if there is no id_str?
>> Won't that possibly result in row collisions, since the only unique item in
>> the key will be the empty string for multiple id_str-less entries?
>>
>> I think in my Facebook branch, I just skip the JSON response entry if
>> there's no message_id.
>
>Yeah, that's probably reasonable. I just write 'adict.get' out of habit for
>the sake of being as flexible as possible.

As discussed over IRC, something like:

    tweet_id = tweet.get('id_str')
    if tweet_id is None:
        log.info...

On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:

You may have to engage with the Twitter developer community to debug this.  I
didn't find anything relevant in a very little bit of googling though.  You'd
think that if it were a bug in their API, others would have noticed it.

>I hadn't originally. At first it just started off as 'from
>gwibber.utils.account import AccountManager; a = AccountManager(None);

Just FWIW, I have a patch in my facebook.working branch that makes None the
default argument.  You have to be careful not to call the callback if it's
None.  I wonder if it's useful to land that in the py3 branch separately?

>In fact, *all* of the style nits you mention were written by ken ;-)

/me hears Kirk's voice scream to the heavens "Keeeeeeennnnnn!"

I think Twitter warns you (via response headers?) if you're getting rate
limited.  So I guess you could check for those and automatically back-off if
you see them.  It's probably better to at least have some combination of
proactive and reactive response to rate limiting.  The gory details:

https://dev.twitter.com/docs/rate-limiting

>>> +        # "Client" == "Consumer" in oauthlib parlance.
>>> +        client_key = self._account.auth.parameters['ConsumerKey']
>>> +        client_secret = self._account.auth.parameters['ConsumerSecret']
>>
>> Can these fail?  Are the other parts of your code prepared to handle the
>> resulting KeyErrors if these parameters are missing?  Do you have any tests
>> for those situations?
>
>No, these are constants defined by Twitter at app-registration time, stored
>in libaccounts-sso package. Can never fail. They may change in the event that
>somebody nefarious compromises them (which ought to be easy to do for an
>open-source package), but they won't change during runtime.

Cool.  In that case, the code above is fine because if they *do* fail, you
want to know via the logged KeyErrors.

>>> +    def _publish_tweet(self, tweet):
>>> +        """Publish a single tweet into the Dee.SharedModel."""
>>> +        tweet_id = tweet.get('id_str', '')
>>
>> Does it still make sense to publish this tweet if there is no id_str?
>> Won't that possibly result in row collisions, since the only unique item in
>> the key will be the empty string for multiple id_str-less entries?
>>
>> I think in my Facebook branch, I just skip the JSON response entry if
>> there's no message_id.
>
>Yeah, that's probably reasonable. I just write 'adict.get' out of habit for
>the sake of being as flexible as possible.

As discussed over IRC, something like:

tweet_id = tweet.get('id_str')
    if tweet_id is None:
        log.info('WTF? no id_str')
        # skip this entry
        continue # or whatever

>>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
>>> +    @feature
>>> +    def mentions(self):
>>> +        """Gather the tweets that mention us."""
>>> +        url = self._timeline.format('mentions')
>>> +        for tweet in self._get_url(url):
>>> +            self._publish_tweet(tweet)
>>
>> Do you have tests for each of these?  I know they *look* ridiculously
>> similar, but each should have a test.
>
>I wasn't really sure how to test them, to be honest.

It's probably good enough to use the same, or very similar sample data, but
just call .mentions(), etc. in the tests.

Really, the only thing you care about is that you've tested the basic code in
this (and similar) methods, not in anything it calls.  Let's say for example
that you - or a future developer - typo'd "self._publish_twit()".  That's the
kind of thing you want these tests to catch.

>'test_home' is really an indirect test of _get_url and _publish_tweet, and
>then these individual operations, I wasn't sure how to test them without
>touching the live service. Or the opposite way, a nearly no-op test that
>mocks out *everything* so aggressively and then just tests that it calls the
>method that it calls. I guess that's what you're looking for, then? Mock out
>self._get_url and self._publish_tweet and just test that _get_url was called
>with the right URL, and then _publish_tweet was called one or more times?

You probably don't need to mock _publish_tweet() since you can easily test the
actual results of that by looking at the model.  We already have mocks that
should have the effect of mocking _get_url(), but yeah, that's the level at
which I'd test it.  Maybe also check that the url gotten was the one you
expect.

>> Also, what happens if get_json() returns an error?  Are these methods
>> prepared to handle problems in communicating with the service?  Of course,
>> they're going to be run in a sub-thread, but is there something better you
>> can do than let the sub-thread exceptions just fill up the log file?  Can
>> you maybe catch exceptions and print something more useful to the log file?
>
>Well, I forget the exact error code, but in the event that we hit the rate
>limit, get_json will raise an exception with an appropriate HTTP error code
>and a meaningful string along with it, such as 'rate limit exceeded' or
>something similar. I didn't really see any issue with that just bubbling
>straight up into the log file. No need to duplicate the same error-checking
>code directly in every single method.

Okay.

>> Also, this seems like quite a common idiom:
>>
>>          for tweet in self._get_url(url):
>>              self._publish_tweet(tweet)
>>
>> Is there any opportunity for refactoring some commonality?
>
>Originally I had written the method as _publish_tweets (plural) and it was
>built-in with the ability to iterate over multiple tweets. That worked for a
>good long while, but then there were just a couple API endpoints that only
>returned a single tweet object rather than a list of tweets, and so then I
>needed to refactor the method to be able to handle both cases, and this is
>what I came up with.

Cool.

>If you really want, we can have something like this:
>
>def _publish_tweets(self, url):
>     for tweet in self._get_url(url):
>         self._publish_tweet(tweet)
>
>But I just thought it was too simple to even bother.

Yeah, I think what you have is fine.  With the additional tests described
above, you should be good to go.

>>> +    @feature
>>> +    def receive(self):
>>> +        """Gather and publish all incoming messages."""
>>> +        # TODO I know mentions and lists are actually incorporated
>>> +        # within the home timeline, but calling them explicitely will

Oh, btw.  s/explicitely/explicitly/

>>> +        # ensure that more of those types of messages appear in the
>>> +        # timeline (eg, so they don't get drown out by everything
>>> +        # else). I'm not sure how necessary it is, though.
>>> +        self.home()
>>> +        self.mentions()
>>> +        self.lists()
>>> +        self.private()
>>
>> What happens if you get an exception in one of these streams?  Does it make
>> sense to still try to publish the other streams?
>
>What kind of exceptions are you suspecting there to be?

Well, let's say for example that Twitter has an outage, inconveniently timed
for when you call .lists() but by the time you call .private() it would
succeed.  Does it make sense that the db will have the results for .home() and
.mentions() but that's it?

Maybe that's okay, and the error will just get logged, and next time around it
will all sync up.  OTOH, maybe you want all or nothing, or maybe you'd want as
much as will succeed.  I think once you decide, a comment would explain the
rationale for the future reader.

>One problem is that quantal gwibber is using exclusively Twitter API v1,
>which is deprecated and will be going away actually quite soon (good timing
>for this rewrite!). My code is all using the newer API v1.1, so it's not
>always helpful to look at what quantal gwibber does, because the API isn't
>always the same.

So it could be a bug in v1.1 of the API, not present in v1.0, and it's
possible that it either hasn't been reported yet, or we haven't found the bug
report.

>>> +# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
>>> +    @feature
>>> +    def delete(self, message_id):
>>> +        url = self._api_base.format(endpoint='statuses/destroy/%s' % message_id)
>>
>> Any reason you're using % interpolation here?  Perhaps either use {} and
>> .format(), or just concatenation (e.g.: 'statuses/destroy/' + message_id)?
>
>Yeah, it looks really ugly to me when I see
>'string{}'.format('foo{}'.format('bar')), but I also wanted to allow
>message_id to be an int rather than requiring it be a string, which is what +
>concatenation would have done.
>
>% interpolation just made the int/str conversion easy without being too
>verbose.

Does it make sense to be even more strict about int-i-ness of message_id?

The other option of course is to add more class attributes with endpoints like
the above, so you'd have something like:

url = self._status_destroy.format(message_id)

>>>       @mock.patch('gwibber.utils.base.Model', TestModel)
>>>       @mock.patch('gwibber.utils.base._seen_messages', {})
>>> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>>
>> Do you really need to mock both _seen_messages and _seen_ids?  What's the
>> difference between these two?  Can they ever get out of sync?  Do we need
>> additional tests to ensure that the constraints of these two data structures
>> are always correct?  Do we need additional tests for when they contain cache
>> hits?
>
>Sorry, I was in a rush when I wrote this. Base._publish now adds to both of
>these so it seemed safer to mock both for consistency, but it's not clear to
>me right off the top of my head whether it's strictly necessary for the test
>itself.

Easy to find out. :)

>>>       def tearDown(self):
>>> +        # Ensure that any log entries we haven't tested just get consumed so
>>> +        # as to isolate out test logger from other tests.
>>>           self.log_mock.stop()
>>
>> The diff doesn't show it, but does setUp() set Base._SYNCHRONIZE = True?
>
>No, I don't need the synchronize stuff because I write the tests to just test
>the synchronous methods directly. I never invoke the thread-starting __call__
>in the tests.

Good point.

>> === modified file 'gwibber/gwibber/utils/base.py'
>> --- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
>> +++ gwibber/gwibber/utils/base.py	2012-10-02 14:33:58 +0000
>>> @@ -42,6 +42,7 @@
>>>   # representing the rows matching those keys.  It is used for quickly finding
>>>   # duplicates when we want to insert new rows into the model.
>>>   _seen_messages = {}
>>> +_seen_ids = {}
>>
>> What is _seen_ids for and how is it used differently than _seen_messages?  Do
>> we still need both of these?  If so, can you adjust the comment to explain
>> what these are for?
>
>_seen_ids maps message ids to rowiters. It was the only way I could think to
>allow deletion of messages without iterating over the whole SharedModel
>terribly. Because at delete-time, we only have access to the message_id, not
>the sender & message data that would be required to look up the rowiter in
>_seen_messages.

I think, but you'd have to verify it, that _seen_messages doesn't need to be a
mapping then.  It can be just a set().  IOW, you don't need to map the
extended ids to rowiters, just the message_ids to rowiters.  So the checks
would be something like:

key = _make_key(args)
    if key in _seen_messages:
        row = Model.get_row(_seen_ids[message_id])
        row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
    else:
        _seen_ids[message_id] = Model.append(*args)
        _seen_messages.add(key)

>>> +    def _unpublish(self, message_id):
>>> +        """Remove message_id from the Dee.SharedModel."""
>>> +        triple = [self.__class__.__name__.lower(),
>>> +                  self._account.id,
>>> +                  message_id]
>>> +
>>> +        row_iter = _seen_ids.pop(message_id)
>>> +        if row_iter is None:
>>> +            log.error('Tried to delete an invalid message id.')
>>> +            return
>>
>> Have you tested the 'miss' logic?  What happens when you .pop() with a
>> missing key?  (this is a trick question :)
>
>Well, I haven't specified a default, which means it will raise KeyError if
>the id is not present. This doesn't particularly bother me, because a) the
>threading logic will safely log and ignore the exception, and b) this can
>only be invoked by clicking on the delete button on a message that's being
>displayed in gwibber, so the message_id being valid is almost *tautological*:
>if message_id wasn't valid, then how the hell did this method even get
>invoked?

Right, but the point is, can row_iter ever actually be None?  As you say, if
you pop a missing message_id, you'll get a KeyError instead of a None return.
Is it even possible to insert a message_id into _seen_ids that has None
instead of an actual rowiter?

Or put another way, no test probably reaches this code, and a coverage tool
would show that.  So, is the test and logging necessary?  If not, and you end
up removing it, please do add a comment above the .pop() that explains the
rational for not checking for KeyErrors, otherwise Future Reader will be
confused.

>> What happens if the .pop() does work?  Does anything need to happen to
>> _seen_messages to keep it in sync?  Isn't the row associated with the
>> row_iter going to be removed?  If so, what happens to the same row_iter
>> sitting as a value in the _seen_messages mapping?
>
>Right, they would become out of sync. But only insofar as there will be a
>sender/message key pointing at a rowiter that points at a row that doesn't
>exist. I'm not certain that this is particularly harmful, beyond being a
>minor memory leak.
>
>I suppose in some extreme case if a person deletes a message and then tries
>to submit an identical message, it would cause us to attempt to append that
>message onto a row that doesn't exist. I don't really see that being a useful
>real-world scenario, because in my mind the use-cases for deleting a message
>would be to either a) leave it deleted, or b) to submit a *different* message
>in it's place.
>
>Anyway, do you know of an easy way to do a dict.pop() that pops based on
>values instead of keys? Or are we going to have to scan the whole dict in an
>inefficient way? Or is there maybe some kind of alternate data structure we
>should be looking at? Some kind two-way dict where both keys and values are
>hashed and indexed?

My concern is that if the two data structures get out of sync, this will lead
to hard to debug errors.  It's a wart asking for trouble.

What I'm thinking is something like this instead, building on my above idea
that _seen_messages would be a set, and the observation that rowiters can
never be None.

_seen_messages.discard(triple)
        row_iter = _seen_ids.pop(message_id, None)
        if row_iter is None:
            # same as it ever was

>> === modified file 'gwibber/gwibber/utils/download.py'
>> --- gwibber/gwibber/utils/download.py	2012-09-14 20:29:11 +0000
>> +++ gwibber/gwibber/utils/download.py	2012-10-02 14:33:58 +0000
>>> @@ -51,11 +51,11 @@
>>>           url = self.url
>>>           headers = self.headers.copy()
>>>           if self.params is not None:
>>> -            params = urlencode(self.params)
>>> +            params = urlencode(self.params).replace('+', '%20')
>>
>> Why do this?  (Yes, I know you explained it to me, but pretend I'm a
>> first-time reader of the code.  Explain it to them! :).
>
>Because urlencode does not provide any option to use quote() instead of
>quote_plus() which it enforces, but Twitter requires percent-encoded spaces,
>and this is harmless to any other protocol. Also it's easier than
>re-implementing urlencode from scratch. hooray for code reuse!

Right!  It was a trick question, since I really just want that paragraph, or
something very much like it, in a comment above the code, to help our dear
Future Reader.

>>
>>>               if self.post:
>>>                   data = params.encode('utf-8')
>>>                   headers['Content-Type'] = (
>>> -                    'application/x-www-form-urlencoded; charset=utf-8')
>>> +                    'application/x-www-form-urlencoded')
>>
>> Why do this too?  This one you haven't explained even to me! :)
>
>It's because oauthlib is incredibly brittle. It scans for precisely the
>string 'application/x-www-form-urlencoded' and raises an exception if it's
>not exactly that. So it seems this was necessary in order to get OAuth
>signatures to even function.
>
>On second thought however, it's not clear to me what oauthlib actually *does*
>with that header when it's signing (I'm pretty sure the value of that header
>is not included in the base string that gets signed). So it might be
>worthwhile to undo this change and then see if anything breaks. This was just
>one of those things that I did back when I was really struggling to get any
>signature working at all, and then after I got it working, I kept it because
>a) I forgot that I'd done it, and b) it didn't occur to me until just now
>that it might not have been necessary to make the signatures work.
>
>So in other words, I'll test this ;-)

Excellent!

lp:~robru/gwibber/twitter updated on 2012-10-03

1484. By Robert Bruce Park on 2012-10-03

Fix up Base._unpublish: Better tests, bugs fixed.

Particularly I've fixed the removal of ids when there were duplicates
before, and also I've eliminated the possibility of collisions between
message id's from different services (there used to be a small chance
that twitter and facebook could assign the same message id to
different messages, and then _unpublish would get confused.

And of course, improved test coverage to prove that this is all good
now ;-)

1485. By Robert Bruce Park on 2012-10-03

Typo.

1486. By Robert Bruce Park on 2012-10-03

Cleanup some API formatting.

1487. By Robert Bruce Park on 2012-10-03

Added comment as per barry.

1488. By Robert Bruce Park on 2012-10-03

Undo a change that wasn't necessary.

Revision history for this message

Robert Bruce Park (robru) wrote on 2012-10-03:

Download full text (17.0 KiB)

On 12-10-03 05:22 PM, Barry Warsaw wrote:
> On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:
>> I actually verified that the OAuth signatures worked for *every* API endpoint
>> *except* sending new direct messages (we can receive them ok, too). That's
>> what debug_twitter_live.py was all about, every time I'd write a new protocol
>> operation, I'd modify it to do that new operation, then run it, and I'd see
>> on my live twitter account that it worked.
>
> You may have to engage with the Twitter developer community to debug this. I
> didn't find anything relevant in a very little bit of googling though. You'd
> think that if it were a bug in their API, others would have noticed it.

Yeah, no idea. Maybe we're the first to port to 1.1 ;-)

I'll look into it a little bit later tonight, got some other
refactoring/cleanup going on.

>> I hadn't originally. At first it just started off as 'from
>> gwibber.utils.account import AccountManager; a = AccountManager(None);
>
> Just FWIW, I have a patch in my facebook.working branch that makes None the
> default argument. You have to be careful not to call the callback if it's
> None. I wonder if it's useful to land that in the py3 branch separately?

I don't think it matters. Maybe it should just default to lambda:None
and then we don't have to worry about whether the callback gets called
or not. For the debug_live.py script, it's not meant to be a
long-running script, just short stints for testing. So it's unlikely for
the callback to be triggered during it's operation.

>> I haven't looked closely at Twitter's rate limiting standards yet, but I
>> wrote that comment because I ran into Twitter's limit at one point and they
>> stopped me from downloading tweets that I wanted.
>
> I think Twitter warns you (via response headers?) if you're getting rate
> limited. So I guess you could check for those and automatically back-off if
> you see them. It's probably better to at least have some combination of
> proactive and reactive response to rate limiting. The gory details:
>
> https://dev.twitter.com/docs/rate-limiting

Yeah, I'm gonna dive into this soon, but so far I've just been focusing
on cleaning up the code that's already there and improving the tests.

>>>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
>>>> + @feature
>>>> + def mentions(self):
>>>> + """Gather the tweets that mention us."""
>>>> + url = self._timeline.format('mentions')
>>>> + for tweet in self._get_url(url):
>>>> + self._publish_tweet(tweet)
>>>
>>> Do you have tests for each of these? I know they *look* ridiculously
>>> similar, but each should have a test.
>>
>> I wasn't really sure how to test them, to be honest.
>
> It's probably good enough to use the same, or very similar sample data, but
> just call .mentions(), etc. in the tests.
>
> Really, the only thing you care about is that you've tested the basic code in
> this (and similar) methods, not in anything it calls. Let's say for example
> that you - or a future developer - typo'd "self._publish_twit()". That's the
> kind of thing you want these tests to catch.

Fair enough. I wrote some t...

On 12-10-03 05:22 PM, Barry Warsaw wrote:
> On Oct 03, 2012, at 12:34 PM, Robert Bruce Park wrote:
>> I actually verified that the OAuth signatures worked for *every* API endpoint
>> *except* sending new direct messages (we can receive them ok, too). That's
>> what debug_twitter_live.py was all about, every time I'd write a new protocol
>> operation, I'd modify it to do that new operation, then run it, and I'd see
>> on my live twitter account that it worked.
>
> You may have to engage with the Twitter developer community to debug this.  I
> didn't find anything relevant in a very little bit of googling though.  You'd
> think that if it were a bug in their API, others would have noticed it.

Yeah, no idea. Maybe we're the first to port to 1.1 ;-)

I'll look into it a little bit later tonight, got some other 
refactoring/cleanup going on.

>> I hadn't originally. At first it just started off as 'from
>> gwibber.utils.account import AccountManager; a = AccountManager(None);
>
> Just FWIW, I have a patch in my facebook.working branch that makes None the
> default argument.  You have to be careful not to call the callback if it's
> None.  I wonder if it's useful to land that in the py3 branch separately?

I don't think it matters. Maybe it should just default to lambda:None 
and then we don't have to worry about whether the callback gets called 
or not. For the debug_live.py script, it's not meant to be a 
long-running script, just short stints for testing. So it's unlikely for 
the callback to be triggered during it's operation.

>> I haven't looked closely at Twitter's rate limiting standards yet, but I
>> wrote that comment because I ran into Twitter's limit at one point and they
>> stopped me from downloading tweets that I wanted.
>
> I think Twitter warns you (via response headers?) if you're getting rate
> limited.  So I guess you could check for those and automatically back-off if
> you see them.  It's probably better to at least have some combination of
> proactive and reactive response to rate limiting.  The gory details:
>
> https://dev.twitter.com/docs/rate-limiting

Yeah, I'm gonna dive into this soon, but so far I've just been focusing 
on cleaning up the code that's already there and improving the tests.

>>>> +# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
>>>> +    @feature
>>>> +    def mentions(self):
>>>> +        """Gather the tweets that mention us."""
>>>> +        url = self._timeline.format('mentions')
>>>> +        for tweet in self._get_url(url):
>>>> +            self._publish_tweet(tweet)
>>>
>>> Do you have tests for each of these?  I know they *look* ridiculously
>>> similar, but each should have a test.
>>
>> I wasn't really sure how to test them, to be honest.
>
> It's probably good enough to use the same, or very similar sample data, but
> just call .mentions(), etc. in the tests.
>
> Really, the only thing you care about is that you've tested the basic code in
> this (and similar) methods, not in anything it calls.  Let's say for example
> that you - or a future developer - typo'd "self._publish_twit()".  That's the
> kind of thing you want these tests to catch.

Fair enough. I wrote some tests that do that already ;-)

>> 'test_home' is really an indirect test of _get_url and _publish_tweet, and
>> then these individual operations, I wasn't sure how to test them without
>> touching the live service. Or the opposite way, a nearly no-op test that
>> mocks out *everything* so aggressively and then just tests that it calls the
>> method that it calls. I guess that's what you're looking for, then? Mock out
>> self._get_url and self._publish_tweet and just test that _get_url was called
>> with the right URL, and then _publish_tweet was called one or more times?
>
> You probably don't need to mock _publish_tweet() since you can easily test the
> actual results of that by looking at the model.  We already have mocks that
> should have the effect of mocking _get_url(), but yeah, that's the level at
> which I'd test it.  Maybe also check that the url gotten was the one you
> expect.

Yeah, I mocked out _get_url and _publish_tweet and then just test that 
_get_url was called with the right URL, and that _publish_tweet got 
called with the tweet. Quite a bit simpler than the test_home method, so 
that I don't have to test the model for having the right data in it 
every time.

>>>> +    @feature
>>>> +    def receive(self):
>>>> +        """Gather and publish all incoming messages."""
>>>> +        # TODO I know mentions and lists are actually incorporated
>>>> +        # within the home timeline, but calling them explicitely will
>
> Oh, btw.  s/explicitely/explicitly/

Fixed.

>>>> +        # ensure that more of those types of messages appear in the
>>>> +        # timeline (eg, so they don't get drown out by everything
>>>> +        # else). I'm not sure how necessary it is, though.
>>>> +        self.home()
>>>> +        self.mentions()
>>>> +        self.lists()
>>>> +        self.private()
>>>
>>> What happens if you get an exception in one of these streams?  Does it make
>>> sense to still try to publish the other streams?
>>
>> What kind of exceptions are you suspecting there to be?
>
> Well, let's say for example that Twitter has an outage, inconveniently timed
> for when you call .lists() but by the time you call .private() it would
> succeed.  Does it make sense that the db will have the results for .home() and
> .mentions() but that's it?
>
> Maybe that's okay, and the error will just get logged, and next time around it
> will all sync up.  OTOH, maybe you want all or nothing, or maybe you'd want as
> much as will succeed.  I think once you decide, a comment would explain the
> rationale for the future reader.

To be honest, I can't really answer this question because I have no idea 
which streams even matter at all. The thing is that the tweets that 
you'd get back from .mentions(), .lists(), and .private() all actually 
also show up in .home(), which means that it's very possible that 
receive() itself can go away, and just be an alias for .home().

The only reason that I wrote it the way that it is, is that it was meant 
to approximate the old "default_streams" config that twitter used to 
have. Each method returns approximately 18 tweets (15-20 in my 
experience). So if we just have .home(), the user gets approximately 18 
tweets that are a mixture of all types, but with .receive() the way it 
is, the end result is that our stream gets filled with 18 public tweets, 
18 tweets mentioning us, 18 tweets from each list that we follow, and 18 
private tweets, and 18 private tweets that we've sent.

I think it really only makes a difference the first time the client is 
run (first EVER, I mean), because then the stream will be empty and 
we'll fill it up with quite a lot of tweets this way. After that, the 
tweets will be cached and this method will only result in lots of 
attempted duplicates not being published.

I guess also if the user has been offline for a while (say, long enough 
for 18 public tweets to have been tweeted since the last private message 
to them), then this implementation of .receive() allows those private 
messages to still make it into the stream rather than just being lost to 
the abyss, so I guess that's an argument in favor.

I think we need to think about this some more and decide how we want it 
to behave and what the right course of action is, in terms of how the 
stream gets populated and how to handle errors then.

>>>> +# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
>>>> +    @feature
>>>> +    def delete(self, message_id):
>>>> +        url = self._api_base.format(endpoint='statuses/destroy/%s' % message_id)
>>>
>>> Any reason you're using % interpolation here?  Perhaps either use {} and
>>> .format(), or just concatenation (e.g.: 'statuses/destroy/' + message_id)?
>>
>> Yeah, it looks really ugly to me when I see
>> 'string{}'.format('foo{}'.format('bar')), but I also wanted to allow
>> message_id to be an int rather than requiring it be a string, which is what +
>> concatenation would have done.
>>
>> % interpolation just made the int/str conversion easy without being too
>> verbose.
>
> Does it make sense to be even more strict about int-i-ness of message_id?

No, twitter plays it pretty loosey-goosey about when ids are strs vs. 
ints. Not to mention that basically everything is a string in json. If 
we did decide to be strict, i think it would actually be more correct to 
enforce strings and disallow ints, because that would mean less type 
conversions here and there, but personally I think it's easier and 
better to just support both, so that we can allow whatever's easier at 
the time to happen. who knows, maybe it'll be easier for ken's vala to 
work in ints.

> The other option of course is to add more class attributes with endpoints like
> the above, so you'd have something like:
>
>      url = self._status_destroy.format(message_id)

That I like. Done.

>>>>        @mock.patch('gwibber.utils.base.Model', TestModel)
>>>>        @mock.patch('gwibber.utils.base._seen_messages', {})
>>>> +    @mock.patch('gwibber.utils.base._seen_ids', {})
>>>
>>> Do you really need to mock both _seen_messages and _seen_ids?  What's the
>>> difference between these two?  Can they ever get out of sync?  Do we need
>>> additional tests to ensure that the constraints of these two data structures
>>> are always correct?  Do we need additional tests for when they contain cache
>>> hits?
>>
>> Sorry, I was in a rush when I wrote this. Base._publish now adds to both of
>> these so it seemed safer to mock both for consistency, but it's not clear to
>> me right off the top of my head whether it's strictly necessary for the test
>> itself.
>
> Easy to find out. :)

I think it is necessary, now that I think about it, because they 
generally need to be in sync while the program is operating, so mocking 
them both out makes the test environment behave more like the real 
world. It's better this way ;-)

>>> === modified file 'gwibber/gwibber/utils/base.py'
>>> --- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
>>> +++ gwibber/gwibber/utils/base.py	2012-10-02 14:33:58 +0000
>>>> @@ -42,6 +42,7 @@
>>>>    # representing the rows matching those keys.  It is used for quickly finding
>>>>    # duplicates when we want to insert new rows into the model.
>>>>    _seen_messages = {}
>>>> +_seen_ids = {}
>>>
>>> What is _seen_ids for and how is it used differently than _seen_messages?  Do
>>> we still need both of these?  If so, can you adjust the comment to explain
>>> what these are for?
>>
>> _seen_ids maps message ids to rowiters. It was the only way I could think to
>> allow deletion of messages without iterating over the whole SharedModel
>> terribly. Because at delete-time, we only have access to the message_id, not
>> the sender & message data that would be required to look up the rowiter in
>> _seen_messages.
>
> I think, but you'd have to verify it, that _seen_messages doesn't need to be a
> mapping then.  It can be just a set().  IOW, you don't need to map the
> extended ids to rowiters, just the message_ids to rowiters.  So the checks
> would be something like:
>
>      key = _make_key(args)
>      if key in _seen_messages:
>          row = Model.get_row(_seen_ids[message_id])
>          row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
>      else:
>          _seen_ids[message_id] = Model.append(*args)
>          _seen_messages.add(key)

This pseudocode looks nice in theory but I played with it a bit and 
couldn't quite get it to work. You're welcome to try but I don't think 
it's a big deal.

>>>> +    def _unpublish(self, message_id):
>>>> +        """Remove message_id from the Dee.SharedModel."""
>>>> +        triple = [self.__class__.__name__.lower(),
>>>> +                  self._account.id,
>>>> +                  message_id]
>>>> +
>>>> +        row_iter = _seen_ids.pop(message_id)
>>>> +        if row_iter is None:
>>>> +            log.error('Tried to delete an invalid message id.')
>>>> +            return
>>>
>>> Have you tested the 'miss' logic?  What happens when you .pop() with a
>>> missing key?  (this is a trick question :)
>>
>> Well, I haven't specified a default, which means it will raise KeyError if
>> the id is not present. This doesn't particularly bother me, because a) the
>> threading logic will safely log and ignore the exception, and b) this can
>> only be invoked by clicking on the delete button on a message that's being
>> displayed in gwibber, so the message_id being valid is almost *tautological*:
>> if message_id wasn't valid, then how the hell did this method even get
>> invoked?
>
> Right, but the point is, can row_iter ever actually be None?  As you say, if
> you pop a missing message_id, you'll get a KeyError instead of a None return.
> Is it even possible to insert a message_id into _seen_ids that has None
> instead of an actual rowiter?

Right, it's not. As I mentioned, I had written this code in a bit of a 
rush. Had somebody standing over my shoulder waiting for me to come to 
dinner ;-)

I've cleaned it up now, I specify 'None' in the call to .pop() so that I 
can catch that error and log it.

> Or put another way, no test probably reaches this code, and a coverage tool
> would show that.  So, is the test and logging necessary?  If not, and you end
> up removing it, please do add a comment above the .pop() that explains the
> rational for not checking for KeyErrors, otherwise Future Reader will be
> confused.

Sorry, I was a bit confused before. The way you'd quoted it made me 
forget that I had that test there in the first place. I fixed .pop() to 
return None instead of raise KeyError.

>>> What happens if the .pop() does work?  Does anything need to happen to
>>> _seen_messages to keep it in sync?  Isn't the row associated with the
>>> row_iter going to be removed?  If so, what happens to the same row_iter
>>> sitting as a value in the _seen_messages mapping?
>>
>> Right, they would become out of sync. But only insofar as there will be a
>> sender/message key pointing at a rowiter that points at a row that doesn't
>> exist. I'm not certain that this is particularly harmful, beyond being a
>> minor memory leak.
>>
>> I suppose in some extreme case if a person deletes a message and then tries
>> to submit an identical message, it would cause us to attempt to append that
>> message onto a row that doesn't exist. I don't really see that being a useful
>> real-world scenario, because in my mind the use-cases for deleting a message
>> would be to either a) leave it deleted, or b) to submit a *different* message
>> in it's place.
>>
>> Anyway, do you know of an easy way to do a dict.pop() that pops based on
>> values instead of keys? Or are we going to have to scan the whole dict in an
>> inefficient way? Or is there maybe some kind of alternate data structure we
>> should be looking at? Some kind two-way dict where both keys and values are
>> hashed and indexed?
>
> My concern is that if the two data structures get out of sync, this will lead
> to hard to debug errors.  It's a wart asking for trouble.
>
> What I'm thinking is something like this instead, building on my above idea
> that _seen_messages would be a set, and the observation that rowiters can
> never be None.
>
>          _seen_messages.discard(triple)
>          row_iter = _seen_ids.pop(message_id, None)
>          if row_iter is None:
>              # same as it ever was

Ok, right. I haven't been able to make _seen_messages be a set, but I 
did think of a nice way to 'del' the deleted row out of it to keep them 
in sync. Please see the updated diff.

>>>
>>>>                if self.post:
>>>>                    data = params.encode('utf-8')
>>>>                    headers['Content-Type'] = (
>>>> -                    'application/x-www-form-urlencoded; charset=utf-8')
>>>> +                    'application/x-www-form-urlencoded')
>>>
>>> Why do this too?  This one you haven't explained even to me! :)
>>
>> It's because oauthlib is incredibly brittle. It scans for precisely the
>> string 'application/x-www-form-urlencoded' and raises an exception if it's
>> not exactly that. So it seems this was necessary in order to get OAuth
>> signatures to even function.
>>
>> On second thought however, it's not clear to me what oauthlib actually *does*
>> with that header when it's signing (I'm pretty sure the value of that header
>> is not included in the base string that gets signed). So it might be
>> worthwhile to undo this change and then see if anything breaks. This was just
>> one of those things that I did back when I was really struggling to get any
>> signature working at all, and then after I got it working, I kept it because
>> a) I forgot that I'd done it, and b) it didn't occur to me until just now
>> that it might not have been necessary to make the signatures work.
>>
>> So in other words, I'll test this ;-)
>
> Excellent!

Yeah, that turned out to be a red herring. I *do* have to specify 
exactly the 'x-www-form-urlencoded' thing when I'm generating the 
oauthlib signature, but it wasn't necessary to change this here.

Alright, I'm gonna break for dinner, then come back and work on the rate 
limiting, and also look into the broken direct messages. Then we're done!

lp:~robru/gwibber/twitter updated on 2012-10-04

1489. By Robert Bruce Park on 2012-10-04

Implement a basic rate limiter.

This had to be done in the Downloader class unfortunately because it
needed direct access to the HTTP headers and I didn't really see any
way to access those outside of the Downloader class.

Luckily it will harmlessly ignore any protocol that doesn't specify
rate limits in the HTTP headers, so it should work well for Twitter
and do nothing for anybody else.

This basically makes no attempt to limit you if you have more than 5
requests remaining in the rate limit window (making it unobtrusive for
cases where the dispatcher is just calling every 5 minutes and it
really should never come near the rate limiter *anyway*), but if
something has gone on such as a user requesting too many manual
refreshes, then it will forcibly pause each request long enough to
avoid hitting Twitter's rate limiter.

1490. By Robert Bruce Park on 2012-10-04

Prevent duplicate message ids from filling up the message_ids column.

It turns out that it's quite easy to see the same message multiple
times, and previously the dupe-checking logic was successful in
stopping the message from being published more than once, it blindly
appended the message_id to the list regardless of whether or not it
was already on the list, so that tended to fill up with junk quick.

With test case.

Revision history for this message

Robert Bruce Park (robru) wrote on 2012-10-04:

Ok, barry, I think we have a pretty solid thing here, rate limiting and everything! Have another look over the diff and let me know what you think ;-)

The only real issue remaining is that send_private still 403s, but I've sent a message to the Twitter support forum, so hopefully they respond to that soon.

lp:~robru/gwibber/twitter updated on 2012-10-04

1491. By Robert Bruce Park on 2012-10-04: Fill out some comments and docstrings.
1492. By Robert Bruce Park on 2012-10-04: Catch 403s in send_private, with test.
1493. By Robert Bruce Park on 2012-10-04: Add tests for the rate limiter.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Barry Warsaw

Robert Bruce Park

 === modified file 'gwibber/gwibber/protocols/twitter.py'
 --- gwibber/gwibber/protocols/twitter.py	2012-09-19 22:21:39 +0000
 +++ gwibber/gwibber/protocols/twitter.py	2012-10-04 20:29:21 +0000
@@ -14,20 +14,289 @@
  """The Twitter protocol plugin."""
++
  __all__ = [
      'Twitter',
+     ]
--import gettext
  import logging
--from gwibber.utils.base import Base
++from oauthlib.oauth1 import Client
++from urllib.error import HTTPError
++from urllib.parse import quote
++
++from gwibber.utils.authentication import Authentication
++from gwibber.utils.base import Base, feature
++from gwibber.utils.download import get_json
++from gwibber.utils.time import parsetime, iso8601utc
  log = logging.getLogger('gwibber.service')
--_ = gettext.lgettext
--
--
++
++
++# https://dev.twitter.com/docs/api/1.1
  class Twitter(Base):
--    pass
++    # StatusNet claims to mimick the Twitter API very closely (so
++    # closely to the point that they refer you to the twitter API
++    # reference docs as a starting point for learning about their
++    # API), So these prefixes are defined here as class attributes
++    # instead of the usual module globals, in the hopes that the
++    # StatusNet class will be able to subclass Twitter and change only
++    # the URLs, with minimal other changes, and magically work.
++    _api_base = 'https://api.twitter.com/1.1/{endpoint}.json'
++
++    _timeline = _api_base.format(endpoint='statuses/{}_timeline')
++    _user_timeline = _timeline.format('user') + '?screen_name={}'
++
++    _lists = _api_base.format(endpoint='lists/statuses') + '?list_id={}'
++
++    _destroy = _api_base.format(endpoint='statuses/destroy/{}')
++    _retweet = _api_base.format(endpoint='statuses/retweet/{}')
++
++    _tweet_permalink = 'https://twitter.com/{user_id}/status/{tweet_id}'
++
++    def _locked_login(self, old_token):
++        """Sign in without worrying about concurrent login attempts."""
++        result = Authentication(self._account, log).login()
++        if result is None:
++            log.error('No Twitter authentication results received.')
++            return
++
++        token = result.get('AccessToken')
++        if token is None:
++            log.error('No AccessToken in Twitter session: {!r}', result)
++        else:
++            self._account.access_token = token
++            self._account.secret_token = result.get('TokenSecret')
++            self._account.user_id = result.get('UserId')
++            self._account.user_name = result.get('ScreenName')
++            log.debug('{} UID: {}'.format(self.__class__.__name__,
++                                          self._account.user_id))
++
++    def _get_url(self, url, data=None):
++        """Access the Twitter API with correct OAuth signed headers."""
++        # TODO start enforcing some rate limiting.
++        do_post = data is not None
++
++        # "Client" == "Consumer" in oauthlib parlance.
++        client_key = self._account.auth.parameters['ConsumerKey']
++        client_secret = self._account.auth.parameters['ConsumerSecret']
++
++        # "resource_owner" == secret and token.
++        resource_owner_key = self._get_access_token()
++        resource_owner_secret = self._account.secret_token
++        oauth_client = Client(client_key, client_secret,
++                              resource_owner_key, resource_owner_secret)
++
++        headers = {}
++        if do_post:
++            headers['Content-Type'] = 'application/x-www-form-urlencoded'
++
++        # All we care about is the headers, which will contain the
++        # Authorization header necessary to satisfy OAuth.
++        uri, headers, body = oauth_client.sign(
++            url, body=data, headers=headers,
++            http_method='POST' if do_post else 'GET')
++
++        return get_json(url,
++                        params=data,
++                        headers=headers,
++                        post=do_post)
++
++    def _publish_tweet(self, tweet):
++        """Publish a single tweet into the Dee.SharedModel."""
++        tweet_id = tweet.get('id_str')
++        if tweet_id is None:
++            log.info('We got a tweet with no id! Abort!')
++            return
++
++        user = tweet.get('user', {})
++        screen_name = user.get('screen_name', '')
++        self._publish(
++            message_id=tweet_id,
++            message=tweet.get('text', ''),
++            timestamp=iso8601utc(parsetime(tweet.get('created_at', ''))),
++            stream='messages',
++            sender=user.get('name', ''),
++            sender_nick=screen_name,
++            from_me=(screen_name == self._account.user_id),
++            icon_uri=user.get('profile_image_url_https', ''),
++            liked=tweet.get('favorited', False),
++            url=self._tweet_permalink.format(user_id=screen_name,
++                                             tweet_id=tweet_id),
++            )
++
++# https://dev.twitter.com/docs/api/1.1/get/statuses/home_timeline
++    @feature
++    def home(self):
++        """Gather the user's home timeline."""
++        url = self._timeline.format('home')
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/get/statuses/mentions_timeline
++    @feature
++    def mentions(self):
++        """Gather the tweets that mention us."""
++        url = self._timeline.format('mentions')
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline
++    @feature
++    def user(self, screen_name=''):
++        """Gather the tweets from a specific user.
++
++        If screen_name is not specified, then gather the tweets
++        written by the currently authenticated user.
++        """
++        url = self._user_timeline.format(screen_name)
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/get/lists/statuses
++    @feature
++    def list(self, list_id):
++        """Gather the tweets from the specified list_id."""
++        url = self._lists.format(list_id)
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/get/lists/list
++    @feature
++    def lists(self):
++        """Gather the tweets from the lists that the we are subscribed to."""
++        url = self._api_base.format(endpoint='lists/list')
++        for twitlist in self._get_url(url):
++            self.list(twitlist.get('id_str', ''))
++
++# https://dev.twitter.com/docs/api/1.1/get/direct_messages
++# https://dev.twitter.com/docs/api/1.1/get/direct_messages/sent
++    @feature
++    def private(self):
++        """Gather the direct messages sent to/from us."""
++        url = self._api_base.format(endpoint='direct_messages')
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++        url = self._api_base.format(endpoint='direct_messages/sent')
++        for tweet in self._get_url(url):
++            self._publish_tweet(tweet)
++
++    @feature
++    def receive(self):
++        """Gather and publish all incoming messages."""
++        # TODO I know mentions and lists are actually incorporated
++        # within the home timeline, but calling them explicitly will
++        # ensure that more of those types of messages appear in the
++        # timeline (eg, so they don't get drown out by everything
++        # else). I'm not sure how necessary it is, though.
++        self.home()
++        self.mentions()
++        self.lists()
++        self.private()
++
++    @feature
++    def send_private(self, screen_name, message):
++        """Send a direct message to the given screen name.
++
++        This will error 403 if the person you are sending to does not
++        follow you.
++        """
++        url = self._api_base.format(endpoint='direct_messages/new')
++        try:
++            tweet = self._get_url(url, dict(text=message, screen_name=screen_name))
++            self._publish_tweet(tweet)
++        except HTTPError as e:
++            log.error('{}: Does that user follow you?'.format(e))
++
++# https://dev.twitter.com/docs/api/1.1/post/statuses/update
++    @feature
++    def send(self, message):
++        """Publish a public tweet."""
++        url = self._api_base.format(endpoint='statuses/update')
++        tweet = self._get_url(url, dict(status=message))
++        self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/post/statuses/update
++    @feature
++    def send_thread(self, message_id, message):
++        """Send a reply message to message_id.
++
++        Note that you have to @mention the message_id owner's screen name in
++        order for Twitter to actually accept this as a reply. Otherwise it will
++        just be an ordinary tweet.
++        """
++        url = self._api_base.format(endpoint='statuses/update')
++        tweet = self._get_url(url, dict(in_reply_to_status_id=message_id,
++                                        status=message))
++        self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/post/statuses/destroy/%3Aid
++    @feature
++    def delete(self, message_id):
++        """Delete a tweet that you wrote."""
++        url = self._destroy.format(message_id)
++        tweet = self._get_url(url, dict(trim_user='true'))
++        self._unpublish(message_id)
++
++# https://dev.twitter.com/docs/api/1.1/post/statuses/retweet/%3Aid
++    @feature
++    def retweet(self, message_id):
++        """Republish somebody else's tweet with your name on it."""
++        url = self._retweet.format(message_id)
++        tweet = self._get_url(url, dict(trim_user='true'))
++        self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/post/friendships/destroy
++    @feature
++    def unfollow(self, screen_name):
++        """Stop following the given screen name."""
++        url = self._api_base.format(endpoint='friendships/destroy')
++        self._get_url(url, dict(screen_name=screen_name))
++
++# https://dev.twitter.com/docs/api/1.1/post/friendships/create
++    @feature
++    def follow(self, screen_name):
++        """Start following the given screen name."""
++        url = self._api_base.format(endpoint='friendships/create')
++        self._get_url(url, dict(screen_name=screen_name, follow='true'))
++
++# https://dev.twitter.com/docs/api/1.1/post/favorites/create
++    @feature
++    def like(self, message_id):
++        """Announce to the world your undying love for a tweet."""
++        url = self._api_base.format(endpoint='favorites/create')
++        tweet = self._get_url(url, dict(id=message_id))
++        # I don't think we need to publish this tweet because presumably
++        # the user has clicked the 'favorite' button on the message that's
++        # already in the stream...
++
++# https://dev.twitter.com/docs/api/1.1/post/favorites/destroy
++    @feature
++    def unlike(self, message_id):
++        """Renounce your undying love for a tweet."""
++        url = self._api_base.format(endpoint='favorites/destroy')
++        tweet = self._get_url(url, dict(id=message_id))
++
++# https://dev.twitter.com/docs/api/1.1/get/search/tweets
++    @feature
++    def tag(self, hashtag):
++        """Return a list of some recent tweets mentioning hashtag."""
++        url = self._api_base.format(endpoint='search/tweets')
++
++        response = self._get_url(
++            '{}?q=%23{}'.format(url, hashtag.lstrip('#')))
++        for tweet in response.get('statuses', []):
++            self._publish_tweet(tweet)
++
++# https://dev.twitter.com/docs/api/1.1/get/search/tweets
++    @feature
++    def search(self, query):
++        """Search for any arbitrary string."""
++        url = self._api_base.format(endpoint='search/tweets')
++
++        response = self._get_url('{}?q={}'.format(url, quote(query, safe='')))
++        for tweet in response.get('statuses', []):
++            self._publish_tweet(tweet)
 === modified file 'gwibber/gwibber/testing/mocks.py'
 --- gwibber/gwibber/testing/mocks.py	2012-09-21 21:45:06 +0000
 +++ gwibber/gwibber/testing/mocks.py	2012-10-04 20:29:21 +0000
@@ -71,21 +71,30 @@
  class FakeData:
      """Mimic a urlopen() that returns canned data."""
--    def __init__(self, path, resource, charset='utf-8'):
++    def __init__(self, path, resource, charset='utf-8', headers=None):
          # resource_string() always returns bytes.
          self._data = resource_string(path, resource)
          self.call_count = 0
          self._charset = charset
++        self._headers = headers or {}
      def __call__(self, url, post_data=None):
          # Ignore url and post_data since the canned data will already
          # represents these results.  We just have to make the API fit.
          # post_data will be missing for GETs.
          charset = self._charset
++        class FakeInfo:
++            def __init__(self, headers=None):
++                self.headers = headers or {}
++            def get(self, key, default=None):
++                return self.headers.get(key, default)
++            def get_content_charset(self):
++                return charset
          class FakeOpen:
--            def __init__(self, data, charset):
++            def __init__(self, data, charset, headers=None):
                  self._data = data
                  self._charset = charset
++                self.headers = headers or {}
              def read(self):
                  return self._data
              def __enter__(self):
@@ -93,12 +102,9 @@
              def __exit__(self, *args, **kws):
                  pass
              def info(self):
--                class FakeInfo:
--                    def get_content_charset(self):
--                        return charset
--                return FakeInfo()
++                return FakeInfo(self.headers)
          self.call_count += 1
--        return FakeOpen(self._data, self._charset)
++        return FakeOpen(self._data, self._charset, self._headers)
  class SettingsIterMock:
 === added file 'gwibber/gwibber/tests/data/twitter-home.dat'
 --- gwibber/gwibber/tests/data/twitter-home.dat	1970-01-01 00:00:00 +0000
 +++ gwibber/gwibber/tests/data/twitter-home.dat	2012-10-04 20:29:21 +0000
@@ -0,0 +1,344 @@
++    [
++      {
++        "coordinates": null,
++        "truncated": false,
++        "created_at": "Tue Aug 28 21:16:23 +0000 2012",
++        "favorited": false,
++        "id_str": "240558470661799936",
++        "in_reply_to_user_id_str": null,
++        "entities": {
++          "urls": [
++
++          ],
++          "hashtags": [
++
++          ],
++          "user_mentions": [
++
++          ]
++        },
++        "text": "just another test",
++        "contributors": null,
++        "id": 240558470661799936,
++        "retweet_count": 0,
++        "in_reply_to_status_id_str": null,
++        "geo": null,
++        "retweeted": false,
++        "in_reply_to_user_id": null,
++        "place": null,
++        "source": "<a href=\"http://realitytechnicians.com\" rel=\"nofollow\">OAuth Dancer Reborn</a>",
++        "user": {
++          "name": "OAuth Dancer",
++          "profile_sidebar_fill_color": "DDEEF6",
++          "profile_background_tile": true,
++          "profile_sidebar_border_color": "C0DEED",
++          "profile_image_url": "http://a0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg",
++          "created_at": "Wed Mar 03 19:37:35 +0000 2010",
++          "location": "San Francisco, CA",
++          "follow_request_sent": false,
++          "id_str": "119476949",
++          "is_translator": false,
++          "profile_link_color": "0084B4",
++          "entities": {
++            "url": {
++              "urls": [
++                {
++                  "expanded_url": null,
++                  "url": "http://bit.ly/oauth-dancer",
++                  "indices": [
++                    0,
++                    26
++                  ],
++                  "display_url": null
++                }
++              ]
++            },
++            "description": null
++          },
++          "default_profile": false,
++          "url": "http://bit.ly/oauth-dancer",
++          "contributors_enabled": false,
++          "favourites_count": 7,
++          "utc_offset": null,
++          "profile_image_url_https": "https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg",
++          "id": 119476949,
++          "listed_count": 1,
++          "profile_use_background_image": true,
++          "profile_text_color": "333333",
++          "followers_count": 28,
++          "lang": "en",
++          "protected": false,
++          "geo_enabled": true,
++          "notifications": false,
++          "description": "",
++          "profile_background_color": "C0DEED",
++          "verified": false,
++          "time_zone": null,
++          "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/80151733/oauth-dance.png",
++          "statuses_count": 166,
++          "profile_background_image_url": "http://a0.twimg.com/profile_background_images/80151733/oauth-dance.png",
++          "default_profile_image": false,
++          "friends_count": 14,
++          "following": false,
++          "show_all_inline_media": false,
++          "screen_name": "oauth_dancer"
++        },
++        "in_reply_to_screen_name": null,
++        "in_reply_to_status_id": null
++      },
++      {
++        "coordinates": {
++          "coordinates": [
++            -122.25831,
++            37.871609
++          ],
++          "type": "Point"
++        },
++        "truncated": false,
++        "created_at": "Tue Aug 28 21:08:15 +0000 2012",
++        "favorited": false,
++        "id_str": "240556426106372096",
++        "in_reply_to_user_id_str": null,
++        "entities": {
++          "urls": [
++            {
++              "expanded_url": "http://blogs.ischool.berkeley.edu/i290-abdt-s12/",
++              "url": "http://t.co/bfj7zkDJ",
++              "indices": [
++                79,
++                99
++              ],
++              "display_url": "blogs.ischool.berkeley.edu/i290-abdt-s12/"
++            }
++          ],
++          "hashtags": [
++
++          ],
++          "user_mentions": [
++            {
++              "name": "Cal",
++              "id_str": "17445752",
++              "id": 17445752,
++              "indices": [
++                60,
++                64
++              ],
++              "screen_name": "Cal"
++            },
++            {
++              "name": "Othman Laraki",
++              "id_str": "20495814",
++              "id": 20495814,
++              "indices": [
++                70,
++                77
++              ],
++              "screen_name": "othman"
++            }
++          ]
++        },
++        "text": "lecturing at the \"analyzing big data with twitter\" class at @cal with @othman  http://t.co/bfj7zkDJ",
++        "contributors": null,
++        "id": 240556426106372096,
++        "retweet_count": 3,
++        "in_reply_to_status_id_str": null,
++        "geo": {
++          "coordinates": [
++            37.871609,
++            -122.25831
++          ],
++          "type": "Point"
++        },
++        "retweeted": false,
++        "possibly_sensitive": false,
++        "in_reply_to_user_id": null,
++        "place": {
++          "name": "Berkeley",
++          "country_code": "US",
++          "country": "United States",
++          "attributes": {
++          },
++          "url": "http://api.twitter.com/1/geo/id/5ef5b7f391e30aff.json",
++          "id": "5ef5b7f391e30aff",
++          "bounding_box": {
++            "coordinates": [
++              [
++                [
++                  -122.367781,
++                  37.835727
++                ],
++                [
++                  -122.234185,
++                  37.835727
++                ],
++                [
++                  -122.234185,
++                  37.905824
++                ],
++                [
++                  -122.367781,
++                  37.905824
++                ]
++              ]
++            ],
++            "type": "Polygon"
++          },
++          "full_name": "Berkeley, CA",
++          "place_type": "city"
++        },
++        "source": "<a href=\"http://www.apple.com\" rel=\"nofollow\">Safari on iOS</a>",
++        "user": {
++          "name": "Raffi Krikorian",
++          "profile_sidebar_fill_color": "DDEEF6",
++          "profile_background_tile": false,
++          "profile_sidebar_border_color": "C0DEED",
++          "profile_image_url": "http://a0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png",
++          "created_at": "Sun Aug 19 14:24:06 +0000 2007",
++          "location": "San Francisco, California",
++          "follow_request_sent": false,
++          "id_str": "8285392",
++          "is_translator": false,
++          "profile_link_color": "0084B4",
++          "entities": {
++            "url": {
++              "urls": [
++                {
++                  "expanded_url": "http://about.me/raffi.krikorian",
++                  "url": "http://t.co/eNmnM6q",
++                  "indices": [
++                    0,
++                    19
++                  ],
++                  "display_url": "about.me/raffi.krikorian"
++                }
++              ]
++            },
++            "description": {
++              "urls": [
++
++              ]
++            }
++          },
++          "default_profile": true,
++          "url": "http://t.co/eNmnM6q",
++          "contributors_enabled": false,
++          "favourites_count": 724,
++          "utc_offset": -28800,
++          "profile_image_url_https": "https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png",
++          "id": 8285392,
++          "listed_count": 619,
++          "profile_use_background_image": true,
++          "profile_text_color": "333333",
++          "followers_count": 18752,
++          "lang": "en",
++          "protected": false,
++          "geo_enabled": true,
++          "notifications": false,
++          "description": "Director of @twittereng's Platform Services. I break things.",
++          "profile_background_color": "C0DEED",
++          "verified": false,
++          "time_zone": "Pacific Time (US & Canada)",
++          "profile_background_image_url_https": "https://si0.twimg.com/images/themes/theme1/bg.png",
++          "statuses_count": 5007,
++          "profile_background_image_url": "http://a0.twimg.com/images/themes/theme1/bg.png",
++          "default_profile_image": false,
++          "friends_count": 701,
++          "following": true,
++          "show_all_inline_media": true,
++          "screen_name": "raffi"
++        },
++        "in_reply_to_screen_name": null,
++        "in_reply_to_status_id": null
++      },
++      {
++        "coordinates": null,
++        "truncated": false,
++        "created_at": "Tue Aug 28 19:59:34 +0000 2012",
++        "favorited": false,
++        "id_str": "240539141056638977",
++        "in_reply_to_user_id_str": null,
++        "entities": {
++          "urls": [
++
++          ],
++          "hashtags": [
++
++          ],
++          "user_mentions": [
++
++          ]
++        },
++        "text": "You'd be right more often if you thought you were wrong.",
++        "contributors": null,
++        "id": 240539141056638977,
++        "retweet_count": 1,
++        "in_reply_to_status_id_str": null,
++        "geo": null,
++        "retweeted": false,
++        "in_reply_to_user_id": null,
++        "place": null,
++        "source": "web",
++        "user": {
++          "name": "Taylor Singletary",
++          "profile_sidebar_fill_color": "FBFBFB",
++          "profile_background_tile": true,
++          "profile_sidebar_border_color": "000000",
++          "profile_image_url": "http://a0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg",
++          "created_at": "Wed Mar 07 22:23:19 +0000 2007",
++          "location": "San Francisco, CA",
++          "follow_request_sent": false,
++          "id_str": "819797",
++          "is_translator": false,
++          "profile_link_color": "c71818",
++          "entities": {
++            "url": {
++              "urls": [
++                {
++                  "expanded_url": "http://www.rebelmouse.com/episod/",
++                  "url": "http://t.co/Lxw7upbN",
++                  "indices": [
++                    0,
++                    20
++                  ],
++                  "display_url": "rebelmouse.com/episod/"
++                }
++              ]
++            },
++            "description": {
++              "urls": [
++
++              ]
++            }
++          },
++          "default_profile": false,
++          "url": "http://t.co/Lxw7upbN",
++          "contributors_enabled": false,
++          "favourites_count": 15990,
++          "utc_offset": -28800,
++          "profile_image_url_https": "https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg",
++          "id": 819797,
++          "listed_count": 340,
++          "profile_use_background_image": true,
++          "profile_text_color": "D20909",
++          "followers_count": 7126,
++          "lang": "en",
++          "protected": false,
++          "geo_enabled": true,
++          "notifications": false,
++          "description": "Reality Technician, Twitter API team, synthesizer enthusiast; a most excellent adventure in timelines. I know it's hard to believe in something you can't see.",
++          "profile_background_color": "000000",
++          "verified": false,
++          "time_zone": "Pacific Time (US & Canada)",
++          "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/643655842/hzfv12wini4q60zzrthg.png",
++          "statuses_count": 18076,
++          "profile_background_image_url": "http://a0.twimg.com/profile_background_images/643655842/hzfv12wini4q60zzrthg.png",
++          "default_profile_image": false,
++          "friends_count": 5444,
++          "following": true,
++          "show_all_inline_media": true,
++          "screen_name": "episod"
++        },
++        "in_reply_to_screen_name": null,
++        "in_reply_to_status_id": null
++      }
++    ]
 === modified file 'gwibber/gwibber/tests/test_dbus.py'
 --- gwibber/gwibber/tests/test_dbus.py	2012-09-25 20:25:42 +0000
 +++ gwibber/gwibber/tests/test_dbus.py	2012-10-04 20:29:21 +0000
@@ -111,6 +111,11 @@
              '/com/gwibber/Service')
          iface = dbus.Interface(obj, 'com.Gwibber.Service')
          # TODO Add more cases as more protocols are added.
++        self.assertEqual(json.loads(iface.GetFeatures('twitter')),
++                         ['delete', 'follow', 'home', 'like', 'list', 'lists',
++                          'mentions', 'private', 'receive', 'retweet', 'search',
++                          'send', 'send_private', 'send_thread', 'tag',
++                          'unfollow', 'unlike', 'user'])
          self.assertEqual(json.loads(iface.GetFeatures('flickr')), ['receive'])
          self.assertEqual(json.loads(iface.GetFeatures('foursquare')),
                           ['receive'])
 === modified file 'gwibber/gwibber/tests/test_download.py'
 --- gwibber/gwibber/tests/test_download.py	2012-09-14 20:29:11 +0000
 +++ gwibber/gwibber/tests/test_download.py	2012-10-04 20:29:21 +0000
@@ -249,3 +249,44 @@
                                    headers={'X-Foo': 'baz',
                                             'X-Bar': 'foo'}),
                           dict(foo='baz', bar='foo'))
++
++    @mock.patch('gwibber.utils.download.urlopen',
++                FakeData('gwibber.tests.data', 'twitter-home.dat',
++                         headers={'X-Rate-Limit-Reset':1349382153 + 300,
++                                  'X-Rate-Limit-Remaining': 1}))
++    @mock.patch('gwibber.utils.download.time.sleep')
++    @mock.patch('gwibber.utils.download.time.time', return_value=1349382153)
++    def test_rate_limiter_maximum(self, time, sleep):
++        # First call does not get limited, but establishes the limit
++        get_json('http://example.com/alpha')
++        sleep.assert_called_with(0)
++        # Second call gets called with the established limit
++        get_json('http://example.com/alpha')
++        sleep.assert_called_with(300)
++
++    @mock.patch('gwibber.utils.download.urlopen',
++                FakeData('gwibber.tests.data', 'twitter-home.dat',
++                         headers={'X-Rate-Limit-Reset':1349382153 + 300,
++                                  'X-Rate-Limit-Remaining': 3}))
++    @mock.patch('gwibber.utils.download.time.sleep')
++    @mock.patch('gwibber.utils.download.time.time', return_value=1349382153)
++    def test_rate_limiter_medium(self, time, sleep):
++        # First call does not get limited, but establishes the limit
++        get_json('http://example.com/beta')
++        sleep.assert_called_with(0)
++        # Second call gets called with the established limit
++        get_json('http://example.com/beta')
++        sleep.assert_called_with(100.0)
++
++    @mock.patch('gwibber.utils.download.urlopen',
++                FakeData('gwibber.tests.data', 'twitter-home.dat',
++                         headers={'X-Rate-Limit-Reset':int(time.time()) + 300,
++                                  'X-Rate-Limit-Remaining': 10}))
++    @mock.patch('gwibber.utils.download.time.sleep')
++    def test_rate_limiter_unlimited(self, sleep):
++        # First call does not get limited, but establishes the limit
++        get_json('http://example.com/omega')
++        sleep.assert_called_with(0)
++        # Second call gets called with the established limit
++        get_json('http://example.com/omega')
++        sleep.assert_called_with(0)
 === modified file 'gwibber/gwibber/tests/test_protocols.py'
 --- gwibber/gwibber/tests/test_protocols.py	2012-09-25 20:25:42 +0000
 +++ gwibber/gwibber/tests/test_protocols.py	2012-10-04 20:29:21 +0000
@@ -176,6 +176,7 @@
      @mock.patch('gwibber.utils.base.Model', TestModel)
      @mock.patch('gwibber.utils.base._seen_messages', {})
++    @mock.patch('gwibber.utils.base._seen_ids', {})
      def test_one_message(self):
          # Test that publishing a message inserts a row into the model.
          base = Base(FakeAccount())
@@ -215,6 +216,32 @@
      @mock.patch('gwibber.utils.base.Model', TestModel)
      @mock.patch('gwibber.utils.base._seen_messages', {})
++    @mock.patch('gwibber.utils.base._seen_ids', {})
++    def test_unpublish(self):
++        base = Base(FakeAccount())
++        self.assertEqual(0, TestModel.get_n_rows())
++        self.assertTrue(base._publish(
++            message_id='1234',
++            sender='fred',
++            message='hello, @jimmy'))
++        self.assertTrue(base._publish(
++            message_id='5678',
++            sender='fred',
++            message='hello, +jimmy'))
++        self.assertEqual(1, TestModel.get_n_rows())
++        self.assertEqual(TestModel[0][0],
++                         [['base', 'faker/than fake', '1234'],
++                          ['base', 'faker/than fake', '5678']])
++        base._unpublish('1234')
++        self.assertEqual(1, TestModel.get_n_rows())
++        self.assertEqual(TestModel[0][0],
++                         [['base', 'faker/than fake', '5678']])
++        base._unpublish('5678')
++        self.assertEqual(0, TestModel.get_n_rows())
++
++    @mock.patch('gwibber.utils.base.Model', TestModel)
++    @mock.patch('gwibber.utils.base._seen_messages', {})
++    @mock.patch('gwibber.utils.base._seen_ids', {})
      def test_duplicate_messages_identified(self):
          # When two messages which are deemed identical, by way of the
          # _make_key() test in base.py, are published, only one ends up in the
@@ -259,6 +286,31 @@
      @mock.patch('gwibber.utils.base.Model', TestModel)
      @mock.patch('gwibber.utils.base._seen_messages', {})
++    @mock.patch('gwibber.utils.base._seen_ids', {})
++    def test_duplicate_ids_not_duplicated(self):
++        # When two messages are actually identical (same ids and all),
++        # we need to avoid duplicating the id in the sharedmodel.
++        base = Base(FakeAccount())
++        self.assertEqual(0, TestModel.get_n_rows())
++        self.assertTrue(base._publish(
++            message_id='1234',
++            stream='messages',
++            sender='fred',
++            message='hello, @jimmy'))
++        self.assertTrue(base._publish(
++            message_id='1234',
++            stream='messages',
++            sender='fred',
++            message='hello, @jimmy'))
++        self.assertEqual(1, TestModel.get_n_rows())
++        row = TestModel.get_row(0)
++        # The same message_id should not appear twice.
++        self.assertEqual(row[COLUMN_INDICES['message_ids']],
++                         [['base', 'faker/than fake', '1234']])
++
++    @mock.patch('gwibber.utils.base.Model', TestModel)
++    @mock.patch('gwibber.utils.base._seen_messages', {})
++    @mock.patch('gwibber.utils.base._seen_ids', {})
      def test_similar_messages_allowed(self):
          # Because both the sender and message contribute to the unique key we
          # use to identify messages, if two messages are published with
 === modified file 'gwibber/gwibber/tests/test_twitter.py'
 --- gwibber/gwibber/tests/test_twitter.py	2012-09-21 14:11:34 +0000
 +++ gwibber/gwibber/tests/test_twitter.py	2012-10-04 20:29:21 +0000
@@ -14,6 +14,7 @@
  """Test the Twitter plugin."""
++
  __all__ = [
      'TestTwitter',
+     ]
@@ -21,9 +22,13 @@
  import unittest
++from gi.repository import Dee
++
  from gwibber.protocols.twitter import Twitter
  from gwibber.testing.helpers import FakeAccount
--from gwibber.testing.mocks import LogMock
++from gwibber.testing.mocks import FakeData, LogMock
++from gwibber.utils.model import COLUMN_TYPES
++
  try:
      # Python 3.3
@@ -32,10 +37,12 @@
      import mock
--# Ensure synchronicity between the main thread and the sub-thread.  Also, set
--# up the loggers for the modules-under-test so that we can assert their error
--# messages.
--@mock.patch.dict('gwibber.utils.base.__dict__', {'_SYNCHRONIZE': True})
++# Create a test model that will not interfere with the user's environment.
++# We'll use this object as a mock of the real model.
++TestModel = Dee.SharedModel.new('com.Gwibber.TestSharedModel')
++TestModel.set_schema_full(COLUMN_TYPES)
++
++
  class TestTwitter(unittest.TestCase):
      """Test the Twitter API."""
@@ -46,8 +53,289 @@
                                  'gwibber.protocols.twitter')
      def tearDown(self):
++        # Ensure that any log entries we haven't tested just get consumed so
++        # as to isolate out test logger from other tests.
          self.log_mock.stop()
--    def test_protocol_info(self):
--        # Each protocol carries with it a number of protocol variables.
--        self.assertEqual(self.protocol.__class__.__name__, 'Twitter')
++    @mock.patch('gwibber.utils.authentication.Authentication.login',
++                return_value=None)
++    @mock.patch('gwibber.utils.download.get_json',
++                return_value=None)
++    def test_unsuccessful_authentication(self, *mocks):
++        self.assertFalse(self.protocol._login())
++        self.assertIsNone(self.account.user_name)
++        self.assertIsNone(self.account.user_id)
++
++    @mock.patch('gwibber.utils.authentication.Authentication.login',
++                return_value=dict(AccessToken='some clever fake data',
++                                  TokenSecret='sssssshhh!',
++                                  UserId='rickygervais',
++                                  ScreenName='Ricky Gervais'))
++    def test_successful_authentication(self, *mocks):
++        self.assertTrue(self.protocol._login())
++        self.assertEqual(self.account.user_name, 'Ricky Gervais')
++        self.assertEqual(self.account.user_id, 'rickygervais')
++        self.assertEqual(self.account.access_token, 'some clever fake data')
++        self.assertEqual(self.account.secret_token, 'sssssshhh!')
++
++
++    @mock.patch('gwibber.protocols.twitter.get_json', lambda *x, **y: y)
++    @mock.patch('oauthlib.oauth1.rfc5849.generate_nonce',
++                lambda: 'once upon a nonce')
++    @mock.patch('oauthlib.oauth1.rfc5849.generate_timestamp',
++                lambda: '1348690628')
++    def test_signatures(self, *mocks):
++        self.account.secret_token = 'alpha'
++        self.account.access_token = 'omega'
++        self.account.auth.id = 6
++        self.account.auth.method = 'oauth2'
++        self.account.auth.mechanism = 'HMAC-SHA1'
++        self.account.auth.parameters = dict(ConsumerKey='consume',
++                                            ConsumerSecret='obey')
++        result = '''\
++OAuth oauth_nonce="once%20upon%20a%20nonce", \
++oauth_timestamp="1348690628", \
++oauth_version="1.0", \
++oauth_signature_method="HMAC-SHA1", \
++oauth_consumer_key="consume", \
++oauth_token="omega", \
++oauth_signature="2MlC4DOqcAdCUmU647izPmxiL%2F0%3D"'''
++
++        self.assertEqual(
++            self.protocol._get_url('http://example.com')['headers'],
++            dict(Authorization=result))
++
++    @mock.patch('gwibber.utils.base.Model', TestModel)
++    @mock.patch('gwibber.utils.download.urlopen',
++                FakeData('gwibber.tests.data', 'twitter-home.dat'))
++    @mock.patch('gwibber.protocols.twitter.Twitter._login',
++                return_value=True)
++    def test_home(self, *mocks):
++        self.account.access_token = 'access'
++        self.account.secret_token = 'secret'
++        self.account.auth.parameters = dict(
++            ConsumerKey='key',
++            ConsumerSecret='secret')
++        self.assertEqual(0, TestModel.get_n_rows())
++        self.protocol.home()
++        self.assertEqual(3, TestModel.get_n_rows())
++
++        # This test data was ripped directly from Twitter's API docs.
++        expected = [
++            [[['twitter', 'faker/than fake', '240558470661799936']],
++             'messages', 'OAuth Dancer', 'oauth_dancer', False,
++             '2012-08-28T21:16:23', 'just another test', '',
++             'https://si0.twimg.com/profile_images/730275945/oauth-dancer_normal.jpg',
++             'https://twitter.com/oauth_dancer/status/240558470661799936', '',
++             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
++             '', '', '', '', '', '', '', '', '', '',
++             ],
++            [[['twitter', 'faker/than fake', '240556426106372096']],
++             'messages', 'Raffi Krikorian', 'raffi', False,
++             '2012-08-28T21:08:15', 'lecturing at the "analyzing big data ' +
++             'with twitter" class at @cal with @othman  http://t.co/bfj7zkDJ', '',
++             'https://si0.twimg.com/profile_images/1270234259/raffi-headshot-casual_normal.png',
++             'https://twitter.com/raffi/status/240556426106372096', '',
++             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
++             '', '', '', '', '', '', '', '', '', '',
++             ],
++            [[['twitter', 'faker/than fake', '240539141056638977']],
++             'messages', 'Taylor Singletary', 'episod', False,
++             '2012-08-28T19:59:34', 'You\'d be right more often if you thought you were wrong.', '',
++             'https://si0.twimg.com/profile_images/2546730059/f6a8zq58mg1hn0ha8vie_normal.jpeg',
++             'https://twitter.com/episod/status/240539141056638977', '',
++             '', '', '', 0.0, False, '', '', '', '', '', '', '', '', '', '', '',
++             '', '', '', '', '', '', '', '', '', '',
++             ],
++            ]
++        for i, expected_row in enumerate(expected):
++            for got, want in zip(TestModel.get_row(i), expected_row):
++                self.assertEqual(got, want)
++
++    def test_mentions(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.mentions()
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/mentions_timeline.json')
++
++    def test_user(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.user()
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=')
++
++    def test_list(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.list('some_list_id')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/lists/statuses.json?list_id=some_list_id')
++
++    def test_lists(self):
++        get_url = self.protocol._get_url = mock.Mock(
++            return_value=[dict(id_str='twitlist')])
++        publish = self.protocol.list = mock.Mock()
++
++        self.protocol.lists()
++
++        publish.assert_called_with('twitlist')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/lists/list.json')
++
++    def test_private(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value=['tweet'])
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.private()
++
++        publish.assert_called_with('tweet')
++        self.assertEqual(
++            get_url.mock_calls,
++            [mock.call('https://api.twitter.com/1.1/direct_messages.json'),
++             mock.call('https://api.twitter.com/1.1/direct_messages/sent.json')])
++
++    def test_send_private(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.send_private('pumpichank', 'Are you mocking me?')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/direct_messages/new.json',
++            dict(text='Are you mocking me?', screen_name='pumpichank'))
++
++    def test_failing_send_private(self):
++        from urllib.error import HTTPError
++        def failing_request(*ignore):
++            raise HTTPError('url', 403, 'Forbidden', 'Forbidden', mock.Mock())
++        get_url = self.protocol._get_url = failing_request
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.send_private('pumpichank', 'Are you mocking me?')
++
++        self.assertEqual(
++            self.log_mock.empty(),
++            'HTTP Error 403: Forbidden: Does that user follow you?\n')
++
++    def test_send(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.send('Hello, twitterverse!')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/update.json',
++            dict(status='Hello, twitterverse!'))
++
++    def test_send_thread(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.send_thread(
++            '1234',
++            'Why yes, I would love to respond to your tweet @pumpichank!')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/update.json',
++            dict(status='Why yes, I would love to respond to your tweet @pumpichank!',
++                 in_reply_to_status_id='1234'))
++
++    def test_delete(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
++        publish = self.protocol._unpublish = mock.Mock()
++
++        self.protocol.delete('1234')
++
++        publish.assert_called_with('1234')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/destroy/1234.json',
++            dict(trim_user='true'))
++
++    def test_retweet(self):
++        get_url = self.protocol._get_url = mock.Mock(return_value='tweet')
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.retweet('1234')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/statuses/retweet/1234.json',
++            dict(trim_user='true'))
++
++    def test_unfollow(self):
++        get_url = self.protocol._get_url = mock.Mock()
++
++        self.protocol.unfollow('pumpichank')
++
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/friendships/destroy.json',
++            dict(screen_name='pumpichank'))
++
++    def test_follow(self):
++        get_url = self.protocol._get_url = mock.Mock()
++
++        self.protocol.follow('pumpichank')
++
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/friendships/create.json',
++            dict(screen_name='pumpichank', follow='true'))
++
++    def test_like(self):
++        get_url = self.protocol._get_url = mock.Mock()
++
++        self.protocol.like('1234')
++
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/favorites/create.json',
++            dict(id='1234'))
++
++    def test_unlike(self):
++        get_url = self.protocol._get_url = mock.Mock()
++
++        self.protocol.unlike('1234')
++
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/favorites/destroy.json',
++            dict(id='1234'))
++
++    def test_tag(self):
++        get_url = self.protocol._get_url = mock.Mock(
++            return_value=dict(statuses=['tweet']))
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.tag('yegbike')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/search/tweets.json?q=%23yegbike')
++
++        self.protocol.tag('#yegbike')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/search/tweets.json?q=%23yegbike')
++
++    def test_search(self):
++        get_url = self.protocol._get_url = mock.Mock(
++            return_value=dict(statuses=['tweet']))
++        publish = self.protocol._publish_tweet = mock.Mock()
++
++        self.protocol.search('hello')
++
++        publish.assert_called_with('tweet')
++        get_url.assert_called_with(
++            'https://api.twitter.com/1.1/search/tweets.json?q=hello')
 === modified file 'gwibber/gwibber/utils/base.py'
 --- gwibber/gwibber/utils/base.py	2012-09-25 20:25:42 +0000
 +++ gwibber/gwibber/utils/base.py	2012-10-04 20:29:21 +0000
@@ -42,6 +42,7 @@
  # representing the rows matching those keys.  It is used for quickly finding
  # duplicates when we want to insert new rows into the model.
  _seen_messages = {}
++_seen_ids = {}
  # Protocol __call__() methods run in threads, so we need to serialize
@@ -179,11 +180,10 @@
          # The column value is a list of lists (see gwibber/utils/model.py for
          # details), and because the arguments are themselves a list, this gets
          # initialized as a triply-nested list.
--        args = [[[
--                self.__class__.__name__.lower(),
--                self._account.id,
--                message_id
--                ]]]
++        triple = [self.__class__.__name__.lower(),
++                  self._account.id,
++                  message_id]
++        args = [[triple]]
          # Now iterate through all the column names listed in the SCHEMA,
          # except for the first, since we just composed its value in the
          # preceding line.  Pop matching column values from the kwargs, in the
@@ -218,9 +218,34 @@
                  # the model gets updated, we need to insert into the row, thus
                  # it's best to concatenate the two lists together and store it
                  # back into the column.
--                row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
++                if triple not in row[IDS_IDX]:
++                    row[IDS_IDX] = row[IDS_IDX] + args[IDS_IDX]
++
++            _seen_ids[tuple(triple)] = _seen_messages.get(key)
              return key in _seen_messages
++    def _unpublish(self, message_id):
++        """Remove message_id from the Dee.SharedModel."""
++        triple = [self.__class__.__name__.lower(),
++                  self._account.id,
++                  message_id]
++
++        row_iter = _seen_ids.pop(tuple(triple), None)
++        if row_iter is None:
++            log.error('Tried to delete an invalid message id.')
++            return
++
++        row = Model.get_row(row_iter)
++        if len(row[IDS_IDX]) == 1:
++            # Message only exists on one protocol, delete it
++            del _seen_messages[_make_key(row)]
++            Model.remove(row_iter)
++        else:
++            # Message exists on other protocols too, only drop id
++            row[IDS_IDX] = [ids for ids
++                            in row[IDS_IDX]
++                            if message_id not in ids]
++
      def _get_access_token(self):
          """Return an access token, logging in if necessary."""
          if self._account.access_token is None:
 === modified file 'gwibber/gwibber/utils/download.py'
 --- gwibber/gwibber/utils/download.py	2012-09-14 20:29:11 +0000
 +++ gwibber/gwibber/utils/download.py	2012-10-04 20:29:21 +0000
@@ -20,6 +20,7 @@
+     ]
++import time
  import json
  import logging
@@ -31,12 +32,18 @@
  log = logging.getLogger('gwibber.service')
++# This stores some information about the current rate limits imposed
++# upon us, and persists that data between instances of the Downloader.
++_rate_limits = {}
++
++
  class Downloader:
      """Convenient downloading wrapper."""
      def __init__(self, url, params=None, post=False,
                   username=None, password=None,
                   headers=None):
++        self.base_url = url.split('?')[0]
          self.url = url
          self.params = params
          self.post = post
@@ -47,11 +54,19 @@
      def _download(self):
          """Return the results as a decoded unicode string."""
++        # Downloads will be happening in threads, so this won't
++        # block the whole app ;-)
++        time.sleep(_rate_limits.get(self.base_url, 0))
++
          data = None
          url = self.url
          headers = self.headers.copy()
          if self.params is not None:
--            params = urlencode(self.params)
++            # urlencode() does not have an option to use quote()
++            # instead of quote_plus(), but Twitter requires
++            # percent-encoded spaces, and this is harmless to any
++            # other protocol.
++            params = urlencode(self.params).replace('+', '%20')
              if self.post:
                  data = params.encode('utf-8')
                  headers['Content-Type'] = (
@@ -89,6 +104,24 @@
          with urlopen(request) as result:
              payload = result.read()
              info = result.info()
++
++        rate_reset = info.get('X-Rate-Limit-Reset') # in UTC epoch seconds
++        rate_count = info.get('X-Rate-Limit-Remaining')
++        if None not in (rate_reset, rate_count):
++            rate_reset = int(rate_reset)
++            rate_count = int(rate_count)
++            rate_delta = rate_reset - time.time()
++            if rate_count > 5:
++                # Ehhh, let's not impede the user if they have more than 5
++                # requests remaining in this window.
++                _rate_limits[self.base_url] = 0
++            elif rate_count <= 1:
++                # Avoid division by zero... wait the full length of time!
++                _rate_limits[self.base_url] = rate_delta
++            else:
++                wait_secs = rate_delta / rate_count
++                _rate_limits[self.base_url] = wait_secs
++
          # RFC 4627 $3.  JSON text SHALL be encoded in Unicode.  The default
          # encoding is UTF-8.  Since the first two characters of a JSON text
          # will always be ASCII characters [RFC0020], it is possible to
 === removed directory 'gwibber/microblog/plugins/twitter'
 === removed file 'gwibber/microblog/plugins/twitter/__init__.py'
 --- gwibber/microblog/plugins/twitter/__init__.py	2012-06-15 13:53:08 +0000
 +++ gwibber/microblog/plugins/twitter/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,819 +0,0 @@
--import cgi
--from gettext import lgettext as _
--from oauth import oauth
--
--from gwibber.microblog import network, util
--from gwibber.microblog.util import resources
--from gwibber.microblog.util.time import parsetime
--from gwibber.microblog.util.auth import Authentication
--
--import logging
--logger = logging.getLogger("Twitter")
--logger.debug("Initializing.")
--
--PROTOCOL_INFO = {
--  "name": "Twitter",
--  "version": "1.0",
--
--  "config": [
--    "private:secret_token",
--    "access_token",
--    "username",
--    "color",
--    "receive_enabled",
--    "send_enabled",
--  ],
--
--  "authtype": "oauth1a",
--  "color": "#729FCF",
--
--  "features": [
--    "send",
--    "receive",
--    "search",
--    "tag",
--    "reply",
--    "responses",
--    "private",
--    "public",
--    "delete",
--    "follow",
--    "unfollow",
--    "profile",
--    "retweet",
--    "like",
--    "send_thread",
--    "send_private",
--    "user_messages",
--    "sinceid",
--    "lists",
--    "list",
--  ],
--
--  "default_streams": [
--    "receive",
--    "images",
--    "responses",
--    "private",
--    "lists",
--  ],
--}
--
--URL_PREFIX = "https://twitter.com"
--API_PREFIX = "https://api.twitter.com/1"
--
--class Client ():
--  """Querys Twitter and converts the data.
--
--  The Client class is responsible for querying Twitter and turning the data obtained
--  into data that Gwibber can understand. Twitter uses a version of OAuth for security.
--
--  Tokens have already been obtained when the account was set up in Gwibber and are used to
--  authenticate when getting data.
--
--  """
--  def __init__(self, acct):
--    self.service = util.getbus("Service")
--    self.account = acct
--    self._loop = None
--
--    self.sigmethod = oauth.OAuthSignatureMethod_HMAC_SHA1()
--    self.token = None
--    parameters = self.account["auth"]["parameters"]
--    self.consumer = oauth.OAuthConsumer(parameters["ConsumerKey"],
--                                        parameters["ConsumerSecret"])
--
--  def _login(self):
--    old_token = self.account.get("secret_token", None)
--    with self.account.login_lock:
--      # Perform the login only if it wasn't already performed by another thread
--      # while we were waiting to the lock
--      if self.account.get("secret_token", None) == old_token:
--        self._locked_login(old_token)
--
--    self.token = oauth.OAuthToken(self.account["access_token"],
--                                  self.account["secret_token"])
--    return "access_token" in self.account and \
--        self.account["access_token"] != old_token
--
--  def _locked_login(self, old_token):
--    logger.debug("Re-authenticating" if old_token else "Logging in")
--
--    auth = Authentication(self.account, logger)
--    reply = auth.login()
--    if reply and reply.has_key("AccessToken"):
--      self.account["access_token"] = reply["AccessToken"]
--      self.account["secret_token"] = reply["TokenSecret"]
--      self.account["uid"] = reply["UserId"]
--      self.account["username"] = reply["ScreenName"]
--      logger.debug("User id is: %s, name is %s" % (self.account["uid"],
--                                                   self.account["username"]))
--    else:
--      logger.error("Didn't find token in session: %s", (reply,))
--
--  def _common(self, data):
--    """Parses messages into Gwibber compatible forms.
--
--    This function is common to all tweet types
--    and includes parsing of user mentions, hashtags,
--    urls, images and videos.
--
--    Arguments:
--    data -- A data object obtained from Twitter containing a complete tweet
--
--    Returns:
--    m -- A data object compatible with inserting into the Gwibber database for that tweet
--
--    """
--    m = {}
--    try:
--      m["mid"] = str(data["id"])
--      m["service"] = "twitter"
--      m["account"] = self.account["id"]
--      if data.has_key("created_at"):
--        m["time"] = parsetime(data["created_at"])
--      m["text"] = util.unescape(data["text"])
--      m["text"] = cgi.escape(m["text"])
--      m["content"] = m["text"]
--
--      # Go through the entities in the tweet and use them to linkify/filter tweets as appropriate
--      if data.has_key("entities"):
--
--        #Get mention entries
--        if data["entities"].has_key("user_mentions"):
--          names = []
--          for mention in data["entities"]["user_mentions"]:
--            if not mention["screen_name"] in names:
--              try:
--                m["content"] = m["content"].replace("@" + mention["screen_name"], "@<a href='gwibber:/user?acct=" + m["account"] + "&name=@" + mention["screen_name"] + "'>" + mention["screen_name"] + "</a>")
--              except:
--                pass
--            names.append(mention["screen_name"])
--
--        #Get hashtag entities
--        if data["entities"].has_key("hashtags"):
--          hashtags = []
--          for tag in data["entities"]["hashtags"]:
--            if not tag["text"] in hashtags:
--              try:
--                m["content"] = m["content"].replace("#" + tag["text"], "#<a href='gwibber:/tag?acct=" + m["account"] + "&query=#" + tag["text"] + "'>" + tag["text"] + "</a>")
--              except:
--                pass
--            hashtags.append(tag["text"])
--
--        # Get url entities - These usually go in the link stream, but if they're pictures or videos, they should go in the proper stream
--        if data["entities"].has_key("urls"):
--          for urls in data["entities"]["urls"]:
--              url = cgi.escape (urls["url"])
--              expanded_url = url
--              if urls.has_key("expanded_url"):
--                if not urls["expanded_url"] is None:
--                  expanded_url = cgi.escape(urls["expanded_url"])
--
--              display_url  = url
--              if urls.has_key("display_url"):
--                display_url = cgi.escape (urls["display_url"])
--
--              if url == m["content"]:
--                m["content"] = "<a href='" + url + "' title='" + expanded_url + "'>" + display_url + "</a>"
--              else:
--                try:
--                  startindex = m["content"].index(url)
--                  endindex   = startindex + len(url)
--                  start      = m["content"][0:startindex]
--                  end        = m["content"][endindex:]
--                  m["content"] = start + "<a href='" + url + "' title='" + expanded_url + "'>" + display_url + "</a>" + end
--                except:
--                  logger.debug ("Failed to set url for ID: %s",  m["mid"])
--
--              m["type"] = "link"
--
--              images = util.imgpreview(expanded_url)
--              videos = util.videopreview(expanded_url)
--              if images:
--                m["images"] = images
--                m["type"] = "photo"
--              elif videos:
--                m["images"] = videos
--                m["type"] = "video"
--              else:
--                # Well, it's not anything else, so it must be a link
--                m["link"] = {}
--                m["link"]["picture"] = ""
--                m["link"]["name"] = ""
--                m["link"]["description"] = m["content"]
--                m["link"]["url"] = url
--                m["link"]["icon"] = ""
--                m["link"]["caption"] = ""
--                m["link"]["properties"] = {}
--
--        if data["entities"].has_key("media"):
--          for media in data["entities"]["media"]:
--            try:
--              url = cgi.escape (media["url"])
--              media_url_https = media["media_url_https"]
--              expanded_url = url
--              if media.has_key("expanded_url"):
--                expanded_url = cgi.escape(media["expanded_url"])
--
--              display_url  = url
--              if media.has_key("display_url"):
--                display_url = cgi.escape (media["display_url"])
--
--              startindex = m["content"].index(url)
--              endindex   = startindex + len(url)
--              start      = m["content"][0:startindex]
--              end        = m["content"][endindex:]
--              m["content"] = start + "<a href='" + url + "' title='" + expanded_url  + "'>" + display_url + "</a>" + end
--
--              if media["type"] == "photo":
--                m["type"] = "photo"
--                m["photo"] = {}
--                m["photo"]["picture"] = media_url_https
--                m["photo"]["url"] = None
--                m["photo"]["name"] = None
--
--            except:
--              pass
--
--      else:
--        m["content"] = util.linkify(util.unescape(m["text"]),
--          ((util.PARSE_HASH, '#<a href="gwibber:/tag?acct=%s&query=\\1">\\1</a>' % m["account"]),
--          (util.PARSE_NICK, '@<a href="gwibber:/user?acct=%s&name=\\1">\\1</a>' % m["account"])), escape=True)
--
--      m["html"] = m["content"]
--
--      m["to_me"] = ("@%s" % self.account["username"]) in data["text"]   # Check if it's a reply directed at the user
--      m["favorited"] = data.get("favorited", False)                     # Check if the tweet has been favourited
--
--    except:
--      logger.error("%s failure - %s", PROTOCOL_INFO["name"], data)
--      return {}
--
--    return m
--
--  def _user(self, user):
--    """Parses the user portion of a tweet.
--
--    Arguments:
--    user -- A user object from a tweet
--
--    Returns:
--    A user object in a format compatible with Gwibber's database.
--
--    """
--    return {
--        "name": user.get("name", None),
--        "nick": user.get("screen_name", None),
--        "id": user.get("id", None),
--        "location": user.get("location", None),
--        "followers": user.get("followers_count", None),
--        "friends": user.get("friends_count", None),
--        "description": user.get("description", None),
--        "following": user.get("following", None),
--        "protected": user.get("protected", None),
--        "statuses": user.get("statuses_count", None),
--        "image": user.get("profile_image_url", None),
--        "website": user.get("url", None),
--        "url": "/".join((URL_PREFIX, user.get("screen_name", ""))) or None,
--        "is_me": user.get("screen_name", None) == self.account["username"],
--    }
--
--  def _message(self, data):
--    """Parses messages into gwibber compatible forms.
--
--    This is the initial function for tweet parsing and parses
--    retweeted status (the shared-by portion), source (the program
--    the tweet was tweeted from) and reply details (the in-reply-to
--    portion). It sends the rest to _common() for further parsing.
--
--    Arguments:
--    data -- A data object obtained from Twitter containing a complete tweet
--
--    Returns:
--    m -- A data object compatible with inserting into the Gwibber database for that tweet
--
--    """
--    if type(data) != dict:
--      logger.error("Cannot parse message data: %s", str(data))
--      return {}
--
--    n = {}
--    if data.has_key("retweeted_status"):
--      n["retweeted_by"] = self._user(data["user"] if "user" in data else data["sender"])
--      if data.has_key("created_at"):
--        n["time"] = parsetime(data["created_at"])
--      data = data["retweeted_status"]
--    else:
--      n["retweeted_by"] = None
--      if data.has_key("created_at"):
--        n["time"] = parsetime(data["created_at"])
--
--    m = self._common(data)
--    for k in n:
--      m[k] = n[k]
--
--    m["source"] = data.get("source", False)
--
--    if data.has_key("in_reply_to_status_id"):
--      if data["in_reply_to_status_id"]:
--        m["reply"] = {}
--        m["reply"]["id"] = data["in_reply_to_status_id"]
--        m["reply"]["nick"] = data["in_reply_to_screen_name"]
--        if m["reply"]["id"] and m["reply"]["nick"]:
--          m["reply"]["url"] = "/".join((URL_PREFIX, m["reply"]["nick"], "statuses", str(m["reply"]["id"])))
--        else:
--          m["reply"]["url"] = None
--
--    m["sender"] = self._user(data["user"] if "user" in data else data["sender"])
--    m["url"] = "/".join((m["sender"]["url"], "statuses", str(m.get("mid", None))))
--
--    return m
--
--  def _responses(self, data):
--    """Sets the message type if the message should be in the replies stream.
--
--    It sends the rest to _message() for further parsing.
--
--    Arguments:
--    data -- A data object obtained from Twitter containing a complete tweet
--
--    Returns:
--    m -- A data object compatible with inserting into the Gwibber database for that tweet
--
--    """
--    m = self._message(data)
--    m["type"] = None
--
--    return m
--
--  def _private(self, data):
--    """Sets the message type and privacy.
--
--    Sets the message type and privacy if the message should be in the private stream.
--    Also parses the recipient as both sent & recieved messages can be in the private stream.
--    It sends the rest to _message() for further parsing
--
--    Arguments:
--    data -- A data object obtained from Twitter containing a complete tweet
--
--    Returns:
--    m -- A data object compatible with inserting into the Gwibber database for that tweet
--
--    """
--    m = self._message(data)
--    m["private"] = True
--    m["type"] = None
--
--    m["recipient"] = {}
--    m["recipient"]["name"] = data["recipient"]["name"]
--    m["recipient"]["nick"] = data["recipient"]["screen_name"]
--    m["recipient"]["id"] = data["recipient"]["id"]
--    m["recipient"]["image"] = data["recipient"]["profile_image_url"]
--    m["recipient"]["location"] = data["recipient"]["location"]
--    m["recipient"]["url"] = "/".join((URL_PREFIX, m["recipient"]["nick"]))
--    m["recipient"]["is_me"] = m["recipient"]["nick"] == self.account["username"]
--    m["to_me"] = m["recipient"]["is_me"]
--
--    return m
--
--  def _result(self, data):
--    """Called when a search is done in Gwibber.
--
--    Parses the sender and sends the rest to _common()
--    for further parsing.
--
--    Arguments:
--    data -- A data object obtained from Twitter containing a complete tweet
--
--    Returns:
--    m -- A data object compatible with inserting into the Gwibber database for that tweet
--
--    """
--    m = self._common(data)
--
--    if data["to_user_id"]:
--      m["reply"] = {}
--      m["reply"]["id"] = data["to_user_id"]
--      m["reply"]["nick"] = data["to_user"]
--
--    m["sender"] = {}
--    m["sender"]["nick"] = data["from_user"]
--    m["sender"]["id"] = data["from_user_id"]
--    m["sender"]["image"] = data["profile_image_url"]
--    m["sender"]["url"] = "/".join((URL_PREFIX, m["sender"]["nick"]))
--    m["sender"]["is_me"] = m["sender"]["nick"] == self.account["username"]
--    m["url"] = "/".join((m["sender"]["url"], "statuses", str(m["mid"])))
--    return m
--
--  def _profile(self, data):
--    """Called when a user is clicked on.
--
--    Args:
--    data -- A data object obtained from Twitter containing a complete user
--
--    Returns:
--    A data object compatible with inserting into the Gwibber database for that user.
--
--    """
--    if "error" in data:
--      return {
--          "error": data["error"]
--        }
--    return {
--        "name": data.get("name", data["screen_name"]),
--        "service": "twitter",
--        "stream": "profile",
--        "account": self.account["id"],
--        "mid": data["id"],
--        "text": data.get("description", ""),
--        "nick": data["screen_name"],
--        "url": data.get("url", ""),
--        "protected": data.get("protected", False),
--        "statuses": data.get("statuses_count", 0),
--        "followers": data.get("followers_count", 0),
--        "friends": data.get("friends_count", 0),
--        "following": data.get("following", 0),
--        "favourites": data.get("favourites_count", 0),
--        "image": data["profile_image_url"],
--        "utc_offset": data.get("utc_offset", 0),
--        "id": data["id"],
--        "lang": data.get("lang", "en"),
--        "verified": data.get("verified", False),
--        "geo_enabled": data.get("geo_enabled", False),
--        "time_zone": data.get("time_zone", "")
--    }
--
--  def _list(self, data):
--    """Called when a list is clicked on.
--
--    Args:
--    data -- A data object obtained from Twitter containing a complete list
--
--    Returns:
--    A data object compatible with inserting into the Gwibber database for that list.
--
--    """
--    return {
--        "mid": data["id"],
--        "service": "twitter",
--        "account": self.account["id"],
--        "time": 0,
--        "text": data["description"],
--        "html": data["description"],
--        "content": data["description"],
--        "url": "/".join((URL_PREFIX, data["uri"])),
--        "sender": self._user(data["user"]),
--        "name": data["name"],
--        "nick": data["slug"],
--        "key": data["slug"],
--        "full": data["full_name"],
--        "uri": data["uri"],
--        "mode": data["mode"],
--        "members": data["member_count"],
--        "followers": data["subscriber_count"],
--        "kind": "list",
--    }
--
--  def _get(self, path, parse="message", post=False, single=False, **args):
--    """Establishes a connection with Twitter and gets the data requested.
--
--    Requires authentication.
--
--    Arguments:
--    path -- The end of the url to look up on Twitter
--    parse -- The function to use to parse the data returned (message by default)
--    post -- True if using POST, for example the send operation. False if using GET, most operations other than send. (False by default)
--    single -- True if a single checkin is requested, False if multiple (False by default)
--    **args -- Arguments to be added to the URL when accessed.
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by the parse function.
--
--    """
--    if not self.token and not self._login():
--      logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), "Auth needs updating")
--      logger.error("%s", logstr)
--      return [{"error": {"type": "auth", "account": self.account, "message": _("Authentication failed, please re-authorize")}}]
--
--    url = "/".join((API_PREFIX, path))
--
--    request = oauth.OAuthRequest.from_consumer_and_token(self.consumer, self.token,
--        http_method=post and "POST" or "GET", http_url=url, parameters=util.compact(args))
--    request.sign_request(self.sigmethod, self.consumer, self.token)
--
--    if post:
--      headers = request.to_header()
--      data = network.Download(url, util.compact(args), post, header=headers).get_json()
--    else:
--      data = network.Download(request.to_url(), None, post).get_json()
--    resources.dump(self.account["service"], self.account["id"], data)
--
--    if isinstance(data, dict) and data.get("errors", 0):
--      if "authenticate" in data["errors"][0]["message"]:
--        # Try again, if we get a new token
--        if self._login():
--          logger.debug("Authentication error, logging in again")
--          return self._get(path, parse, post, single, args)
--        else:
--          logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Authentication failed"), data["errors"][0]["message"])
--          logger.error("%s", logstr)
--          return [{"error": {"type": "auth", "account": self.account, "message": data["errors"][0]["message"]}}]
--      else:
--        for error in data["errors"]:
--          logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Unknown failure"), error["message"])
--          return [{"error": {"type": "unknown", "account": self.account, "message": error["message"]}}]
--    elif isinstance(data, dict) and data.get("error", 0):
--      if "Incorrect signature" in data["error"]:
--        logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Request failed"), data["error"])
--        logger.error("%s", logstr)
--        return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
--    elif isinstance(data, str):
--      logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("Request failed"), data)
--      logger.error("%s", logstr)
--      return [{"error": {"type": "request", "account": self.account, "message": data}}]
--
--    if parse == "follow" or parse == "unfollow":
--      if isinstance(data, dict) and data.get("error", 0):
--        logstr = """%s: %s - %s""" % (PROTOCOL_INFO["name"], _("%s failed" % parse), data["error"])
--        logger.error("%s", logstr)
--        return [{"error": {"type": "auth", "account": self.account, "message": data["error"]}}]
--      else:
--        return [["friendships", {"type": parse, "account": self.account["id"], "service": self.account["service"],"user_id": data["id"], "nick": data["screen_name"]}]]
--
--    if parse == "profile" and isinstance(data, dict):
--      return self._profile(data)
--
--    if parse == "list":
--      return [self._list(l) for l in data["lists"]]
--
--    if single: return [getattr(self, "_%s" % parse)(data)]
--    if parse: return [getattr(self, "_%s" % parse)(m) for m in data]
--    else: return []
--
--  def _search(self, **args):
--    """Establishes a connection with Twitter and gets the results of a search.
--
--    Does not require authentication
--
--    Arguments:
--    **args -- The search terms
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _result().
--
--    """
--    data = network.Download("http://search.twitter.com/search.json", util.compact(args))
--    data = data.get_json()["results"]
--
--    if type(data) != list:
--      logger.error("Cannot parse search data: %s", str(data))
--      return []
--
--    return [self._result(m) for m in data]
--
--  def __call__(self, opname, **args):
--    return getattr(self, opname)(**args)
--
--  def receive(self, count=util.COUNT, since=None):
--    """Gets the latest tweets and adds them to the database.
--
--    Arguments:
--    count -- Number of updates to get
--    since -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _message().
--
--    """
--    return self._get("statuses/home_timeline.json", include_entities=1, count=count, since_id=since)
--
--  def responses(self, count=util.COUNT, since=None):
--    """Gets the latest replies and adds them to the database.
--
--    Arguments:
--    count  -- Number of updates to get
--    since  -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _responses().
--
--    """
--    return self._get("statuses/mentions.json", "responses", include_entities=1, count=count, since_id=since)
--
--  def private(self, count=util.COUNT, since=None):
--    """Gets the latest direct messages sent and recieved and adds them to the database.
--
--    Args:
--    count -- Number of updates to get
--    since -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _private().
--
--    """
--    private = self._get("direct_messages.json", "private", include_entities=1, count=count, since_id=since) or []
--    private_sent = self._get("direct_messages/sent.json", "private", count=count, since_id=since) or []
--    return private + private_sent
--
--  def public(self):
--    """Gets the latest tweets from the public timeline and adds them to the database.
--
--    Arguments:
--    None
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _message().
--
--    """
--    return self._get("statuses/public_timeline.json", include_entities=1)
--
--  def lists(self, **args):
--    """Gets subscribed lists and adds them to the database.
--
--    Arguments:
--    None
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _list().
--
--    """
--    if not "username" in self.account:
--      self._login()
--    following = self._get("%s/lists/subscriptions.json" % self.account["username"], "list") or []
--    lists = self._get("%s/lists.json" % self.account["username"], "list") or []
--    return following + lists
--
--  def list(self, user, id, count=util.COUNT, since=None):
--    """Gets the latest tweets from subscribed lists and adds them to the database.
--
--    Arguments:
--    user -- The user's name whose lists are to be got
--    id -- The user's id whose lists are to be got
--    count -- Number of updates to get
--    since -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _message().
--
--    """
--    return self._get("%s/lists/%s/statuses.json" % (user, id), include_entities=1, per_page=count, since_id=since)
--
--  def search(self, query, count=util.COUNT, since=None):
--    """Gets the latest results from a search and adds them to the database.
--
--    Arguments:
--    query -- The search query
--    count -- Number of updates to get
--    since -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _search().
--
--    """
--    return self._search(include_entities=1, q=query, rpp=count, since_id=since)
--
--  def tag(self, query, count=util.COUNT, since=None):
--    """Gets the latest results from a hashtag search and adds them to the database.
--
--    Arguments:
--    query -- The search query (hashtag without the #)
--    count -- Number of updates to get
--    since -- Time to get updates since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _search().
--
--    """
--    return self._search(q="#%s" % query, count=count, since_id=since)
--
--  def delete(self, message):
--    """Deletes a specified tweet from Twitter.
--
--    Arguments:
--    message -- A Gwibber compatible message object (from gwibber's database)
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("statuses/destroy/%s.json" % message["mid"], None, post=True, do=1)
--
--  def like(self, message):
--    """Favourites a specified tweet on Twitter.
--
--    Arguments:
--    message -- A Gwibber compatible message object (from Gwibber's database)
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("favorites/create/%s.json" % message["mid"], None, post=True, do=1)
--
--  def send(self, message):
--    """Sends a tweet to Twitter.
--
--    Arguments:
--    message -- The tweet's text
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("statuses/update.json", post=True, single=True,
--        status=message)
--
--  def send_private(self, message, private):
--    """Sends a direct message to Twitter.
--
--    Arguments:
--    message -- The tweet's text
--    private -- A gwibber compatible user object (from gwibber's database)
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("direct_messages/new.json", "private", post=True, single=True,
--        text=message, screen_name=private["sender"]["nick"])
--
--  def send_thread(self, message, target):
--    """Sends a reply to a user on Twitter.
--
--    Arguments:
--    message -- The tweet's text
--    target -- A Gwibber compatible user object (from Gwibber's database)
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("statuses/update.json", post=True, single=True,
--        status=message, in_reply_to_status_id=target["mid"])
--
--  def retweet(self, message):
--    """Retweets a tweet.
--
--    Arguments:
--    message -- A Gwibber compatible message object (from gwibber's database)
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("statuses/retweet/%s.json" % message["mid"], None, post=True, do=1)
--
--  def follow(self, screen_name):
--    """Follows a user.
--
--    Arguments:
--    screen_name -- The screen name (@someone without the @) of the user to be followed
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("friendships/create.json", screen_name=screen_name, post=True, parse="follow")
--
--  def unfollow(self, screen_name):
--    """Unfollows a user.
--
--    Arguments:
--    screen_name -- The screen name (@someone without the @) of the user to be unfollowed
--
--    Returns:
--    Nothing
--
--    """
--    return self._get("friendships/destroy.json", screen_name=screen_name, post=True, parse="unfollow")
--
--  def profile(self, id=None, count=None, since=None):
--    """Gets a user's profile.
--
--    Arguments:
--    id -- The user's screen name
--    count -- Number of tweets to get
--    since -- Time to get tweets since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _profile().
--
--    """
--    return self._get("users/show.json", screen_name=id, count=count, since_id=since, parse="profile")
--
--  def user_messages(self, id=None, count=util.COUNT, since=None):
--    """Gets a user's profile & timeline.
--
--    Arguments:
--    id -- The user's screen name
--    count -- Number of tweets to get
--    since -- Time to get tweets since
--
--    Returns:
--    A list of Gwibber compatible objects which have been parsed by _profile().
--
--    """
--    profiles = [self.profile(id)] or []
--    messages = self._get("statuses/user_timeline.json", id=id, include_entities=1, count=count, since_id=since) or []
--    return messages + profiles
 === added directory 'gwibber/tools'
 === added file 'gwibber/tools/debug_live.py'
 --- gwibber/tools/debug_live.py	1970-01-01 00:00:00 +0000
 +++ gwibber/tools/debug_live.py	2012-10-04 20:29:21 +0000
@@ -0,0 +1,67 @@
++#!/usr/bin/env python3
++
++"""Usage: ./tools/debug_live.py PROTOCOL OPERATION [OPTIONS]
++
++Where PROTOCOL is a protocol supported by Gwibber, such as 'twitter',
++OPERATION is an instance method defined in that protocol's class, and
++OPTIONS are whatever arguments you'd like to pass to that method (if
++any), such as message id's or a status message.
++
++Examples:
++
++./tools/debug_live.py twitter home
++./tools/debug_live.py twitter send 'Hello, world!'
++
++This tool is provided to aid with rapid feedback of changes made to
++the gwibber source tree, and as such is designed to be run from the
++same directory that contains 'setup.py'. It is not intended for use
++with an installed gwibber package.
++"""
++
++import sys
++
++sys.path.insert(0, '.')
++
++if len(sys.argv) < 3:
++    sys.exit(__doc__)
++
++protocol = sys.argv[1]
++args = sys.argv[2:]
++
++from gwibber.utils.account import AccountManager
++from gwibber.utils.model import Model
++from gwibber.utils.base import Base
++from gi.repository import GObject
++
++# Disable threading for easier testing.
++Base._SYNCHRONIZE = True
++
++def refresh(account):
++    print()
++    print('#' * 80)
++    print('Performing "{}" operation!'.format(args[0]))
++    print('#' * 80)
++
++    account.protocol(*args)
++    for row in Model:
++        print([col for col in row])
++        print()
++    print('ROWS: ', len(Model))
++    return True
++
++if __name__ == '__main__':
++
++    found = False
++    a = AccountManager(None)
++
++    for account_id, account in a._accounts.items():
++        if account_id.endswith(protocol):
++            found = True
++            refresh(account)
++            GObject.timeout_add_seconds(300, refresh, account)
++
++    if not found:
++        print('No {} account found in your Ubuntu Online Accounts!'.format(
++                protocol))
++    else:
++        GObject.MainLoop().run()
 === added file 'gwibber/tools/debug_slave.py'
 --- gwibber/tools/debug_slave.py	1970-01-01 00:00:00 +0000
 +++ gwibber/tools/debug_slave.py	2012-10-04 20:29:21 +0000
@@ -0,0 +1,22 @@
++#!/usr/bin/env python3
++
++from gi.repository import Dee
++from gi.repository import GObject
++
++
++class Slave:
++  def __init__(self):
++    model_name = 'com.Gwibber.Streams'
++    print('Joining model ' + model_name)
++    self.model = Dee.SharedModel.new(model_name)
++    self.model.connect('row-added', self.on_row_added)
++
++  def on_row_added(self, model, itr):
++    row = self.model.get_row(itr)
++    print(row)
++    print('ROWS: ', len(self.model))
++
++if __name__ == '__main__':
++
++  s = Slave()
++  GObject.MainLoop().run()

Gwibber

Merge lp:~robru/gwibber/twitter into lp:~barry/gwibber/py3

Commit message

Description of the change

Preview Diff

Subscribers