Merge lp:~kai-mast/friends/links into lp:friends

[Kai Mast]

This branch likifies mentions and hashtags. Hashtags open a link with the search in a browser. Friends has support for searches but afaik there is no way to add links to them. So this is the best we can do right now. Imo that makes tweets (especially retweets) much more interakctive.

I also fixed a problem when there are multiple links in a tweet and also added a test so this stays fixed.

[Robert Bruce Park (robru)]

Wow, congrats on the test coverage, that's really impressive ;-)

One thing that I'm a little bit confused about. You named a variable "urls_sorted" but I don't actually see any sorting going on there. Then you go on to use a concept of 'offset' in order to keep track of where each link should go, and I find it a bit sloppy.

What I'd like to see instead, is just iterate over urls_sorted in a sorted way, with descending values of 'begin' (eg, linkify the last URL first and progress backwards through the string). That way there's never an offset to have to worry about as the change in string length does not affect links that appear prior to the change being made.

Another (small) problem is that you used iter() in a useless way. Friends is python3-only, so dict.items() always returns an iter and your extra call to iter() doesn't do anything.

So instead it should probably look like this:

    for key, url in sorted(urls_sorted.items(), reverse=True):

And then...

    if content:
        message = ''.join([message[:begin], content, message[end:]])

Also, I'm slightly annoyed by the inconsistency between the _linkify_mention function which is nearly identical to other linkification techniques that are inlined in the larger function. I'm sure there can be a general purpose _linkify method that can be used to linkify all kinds of links, in a much more consistent and space-efficient manner (ie, fewer lines of code).

review: Needs Fixing

[Kai Mast (kai-mast)]

Oh okay I thought dictionaries where sorted by default (I am so used to C++ ;) )

I haven't even thought of going through the message in reverse to keep track off the offset. Good point.

[Kai Mast (kai-mast)]

Better?

I am not sure how to use just one linkify function because some have absolute addresses and some don't, so I now have linkify and linkify_mention, where the latter calls the first.

I also made the links use https which should be used whenever possible in my opinion.

[Robert Bruce Park (robru)]

This is a ton better, thank you!

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
No commit message was specified in the merge proposal. Hit 'Add commit message' on the merge proposal web page or follow the link below. You can approve the merge proposal yourself to rerun.
https://code.launchpad.net/~kai-mast/friends/links/+merge/202954/+edit-commit-message

review: Needs Fixing (continuous-integration)

[Robert Bruce Park (robru)]

One thing to watch out for is that you have a ton of trailing whitespace all over. I don't like to see that in my code ;-)

Don't worry, I'm cleaning it up for this merge, but if you could start stripping trailing whitespace before committing I'd appreciate it.

=== modified file 'friends/protocols/twitter.py'
--- friends/protocols/twitter.py	2013-12-10 07:04:05 +0000
+++ friends/protocols/twitter.py	2014-01-24 10:56:16 +0000
@@ -127,29 +127,51 @@
 
         #Resolve t.co
         #TODO support more than one url and/or media file
-        for url in (entities.get('urls', []) + entities.get('media', [])):
-            begin, end = url.get('indices', (None, None))
-
-            expanded_url = url.get('expanded_url', '')
-            display_url = url.get('display_url', '')
-            other_url = url.get('url', '')
+        urls = {}
+
+        for url in (entities.get('urls', []) + entities.get('media', []) + entities.get('user_mentions', []) + entities.get('hashtags', [])):
+            begin, end = url.get('indices', (None, None))
+			
+            #Drop invalid entities (just to be safe)
+            if None not in (begin, end):	
+                urls[begin] = url
+        
+        for key, url in sorted(urls.items(), reverse = True):
+            begin, end = url.get('indices', (None, None))
+
+            expanded_url = url.get('expanded_url')
+            display_url = url.get('display_url')
+            other_url = url.get('url')
+            
+            mention_name = url.get('screen_name')
 
             picture_url = url.get('media_url', picture_url)
+            
+            hashtag = url.get('text')
+            
+            content = None
 
             # Friends has no notion of display URLs, so this is handled at the protocol level
-            if None not in (begin, end):
+            if (other_url or expanded_url):
+                content = self._linkify(expanded_url or other_url, display_url or other_url)
+             
+            # Linkify hashtags until supported by friends
+            if hashtag:
+                content = self._linkify('https://twitter.com/search?q=%23' + hashtag + '&src=hash', '#' + hashtag)
+            
+            # Linkify a mention until they are supported natively by friends
+            if mention_name:
+                content = self._linkify_mention(mention_name)
+
+            if content:
                 message = ''.join([
                     message[:begin],
-                    '<a href="',
-                    (expanded_url or other_url),
-                    '">',
-                    (display_url or other_url),
-                    '</a>',
+                    content,
                     message[end:]])
-
+   
         if retweet:
-            message = 'RT @{}: {}'.format(
-                retweet.get('user', {}).get('screen_name', ''),
+            message = 'RT {}: {}'.format(
+                self._linkify_mention(retweet.get('user', {}).get('screen_name', '')),
                 message
             )
 
@@ -168,6 +190,17 @@
             link_picture=picture_url,
             )
         return permalink
+        
+    def _linkify_mention(self, name):
+        return self._linkify('https://twitter.com/' + name, '@' + name)
+        
+    def _linkify(self, address, name):
+        return ''.join([
+                    '<a href="',
+                    address,
+                    '">',
+                    name,
+                    '</a>'])
 
     def _append_since(self, url, stream='messages'):
         since = self._tweet_ids.get(stream)
 
=== added file 'friends/tests/data/twitter-hashtags.dat'
--- friends/tests/data/twitter-hashtags.dat	1970-01-01 00:00:00 +0000
+++ friends/tests/data/twitter-hashtags.dat	2014-01-24 10:56:16 +0000
@@ -0,0 +1,94 @@
+{
+  "created_at": "Fri Jan 17 14:23:41 +0000 2014",
+  "id": 424185261375766500,
+  "id_str": "424185261375766530",
+  "text": "A service that filters food pictures from Instagram. #MillionDollarIdea",
+  "source": "web",
+  "truncated": false,
+  "in_reply_to_status_id": null,
+  "in_reply_to_status_id_str": null,
+  "in_reply_to_user_id": null,
+  "in_reply_to_user_id_str": null,
+  "in_reply_to_screen_name": null,
+  "user": {
+    "id": 17339829,
+    "id_str": "17339829",
+    "name": "Kai Mast",
+    "screen_name": "Kai_Mast",
+    "location": "Bamberg",
+    "description": "Computer Science Student, Grad School Applicant, C++ Fanboy",
+    "url": "http://t.co/1myW31Mlhl",
+    "entities": {
+      "url": {
+        "urls": [
+          {
+            "url": "http://t.co/1myW31Mlhl",
+            "expanded_url": "http://kai-mast.de",
+            "display_url": "kai-mast.de",
+            "indices": [
+              0,
+              22
+            ]
+          }
+        ]
+      },
+      "description": {
+        "urls": []
+      }
+    },
+    "protected": false,
+    "followers_count": 415,
+    "friends_count": 904,
+    "listed_count": 36,
+    "created_at": "Wed Nov 12 14:23:09 +0000 2008",
+    "favourites_count": 939,
+    "utc_offset": 3600,
+    "time_zone": "Berlin",
+    "geo_enabled": true,
+    "verified": false,
+    "statuses_count": 7886,
+    "lang": "en",
+    "contributors_enabled": false,
+    "is_translator": false,
+    "profile_background_color": "8B542B",
+    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme8/bg.gif",
+    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme8/bg.gif",
+    "profile_background_tile": false,
+    "profile_image_url": "http://pbs.twimg.com/profile_images/424181800701673473/Q6Ggqg7P_normal.png",
+    "profile_image_url_https": "https://pbs.twimg.com/profile_images/424181800701673473/Q6Ggqg7P_normal.png",
+    "profile_banner_url": "https://pbs.twimg.com/profile_banners/17339829/1360006443",
+    "profile_link_color": "9D582E",
+    "profile_sidebar_border_color": "D9B17E",
+    "profile_sidebar_fill_color": "EADEAA",
+    "profile_text_color": "333333",
+    "profile_use_background_image": true,
+    "default_profile": false,
+    "default_profile_image": false,
+    "following": false,
+    "follow_request_sent": false,
+    "notifications": false
+  },
+  "geo": null,
+  "coordinates": null,
+  "place": null,
+  "contributors": null,
+  "retweet_count": 0,
+  "favorite_count": 0,
+  "entities": {
+    "hashtags": [
+      {
+        "text": "MillionDollarIdea",
+        "indices": [
+          53,
+          71
+        ]
+      }
+    ],
+    "symbols": [],
+    "urls": [],
+    "user_mentions": []
+  },
+  "favorited": false,
+  "retweeted": false,
+  "lang": "en"
+}
 
=== added file 'friends/tests/data/twitter-multiple-links.dat'
--- friends/tests/data/twitter-multiple-links.dat	1970-01-01 00:00:00 +0000
+++ friends/tests/data/twitter-multiple-links.dat	2014-01-24 10:56:16 +0000
@@ -0,0 +1,144 @@
+{
+  "created_at": "Thu Jan 23 11:40:35 +0000 2014",
+  "id": 426318539796930560,
+  "id_str": "426318539796930560",
+  "text": "An old people's home has recreated famous movie scenes for a wonderful calendar http://t.co/jjqteYzur0 http://t.co/JxQTPG7WLL",
+  "source": "<a href=\"https://about.twitter.com/products/tweetdeck\" rel=\"nofollow\">TweetDeck</a>",
+  "truncated": false,
+  "in_reply_to_status_id": null,
+  "in_reply_to_status_id_str": null,
+  "in_reply_to_user_id": null,
+  "in_reply_to_user_id_str": null,
+  "in_reply_to_screen_name": null,
+  "user": {
+    "id": 16973333,
+    "id_str": "16973333",
+    "name": "The Independent",
+    "screen_name": "Independent",
+    "location": "London, United Kingdom",
+    "description": "News, comment and features from The Independent. Also follow: @IndyVoices, \r\n@IndyPolitics, @IndyWorld and our journalists at http://t.co/YjS7NcXK4A",
+    "url": "http://t.co/wDS5ly0QoO",
+    "entities": {
+      "url": {
+        "urls": [
+          {
+            "url": "http://t.co/wDS5ly0QoO",
+            "expanded_url": "http://www.independent.co.uk",
+            "display_url": "independent.co.uk",
+            "indices": [
+              0,
+              22
+            ]
+          }
+        ]
+      },
+      "description": {
+        "urls": [
+          {
+            "url": "http://t.co/YjS7NcXK4A",
+            "expanded_url": "http://ind.pn/Wdlm9a",
+            "display_url": "ind.pn/Wdlm9a",
+            "indices": [
+              126,
+              148
+            ]
+          }
+        ]
+      }
+    },
+    "protected": false,
+    "followers_count": 505759,
+    "friends_count": 1747,
+    "listed_count": 7890,
+    "created_at": "Sun Oct 26 00:00:29 +0000 2008",
+    "favourites_count": 29,
+    "utc_offset": 0,
+    "time_zone": "London",
+    "geo_enabled": false,
+    "verified": true,
+    "statuses_count": 58003,
+    "lang": "en",
+    "contributors_enabled": false,
+    "is_translator": false,
+    "profile_background_color": "EBEBEB",
+    "profile_background_image_url": "http://a0.twimg.com/profile_background_images/378800000119288704/4ac964c83462c88837dc1e735aa1a45e.png",
+    "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/378800000119288704/4ac964c83462c88837dc1e735aa1a45e.png",
+    "profile_background_tile": true,
+    "profile_image_url": "http://pbs.twimg.com/profile_images/378800000706113664/d1a957578723e496c025be1e2577d06d_normal.jpeg",
+    "profile_image_url_https": "https://pbs.twimg.com/profile_images/378800000706113664/d1a957578723e496c025be1e2577d06d_normal.jpeg",
+    "profile_link_color": "FC051A",
+    "profile_sidebar_border_color": "FFFFFF",
+    "profile_sidebar_fill_color": "FFFFFF",
+    "profile_text_color": "333333",
+    "profile_use_background_image": true,
+    "default_profile": false,
+    "default_profile_image": false,
+    "following": false,
+    "follow_request_sent": false,
+    "notifications": false
+  },
+  "geo": null,
+  "coordinates": null,
+  "place": null,
+  "contributors": null,
+  "retweet_count": 131,
+  "favorite_count": 83,
+  "entities": {
+    "hashtags": [],
+    "symbols": [],
+    "urls": [
+      {
+        "url": "http://t.co/jjqteYzur0",
+        "expanded_url": "http://ind.pn/1g3wX9q",
+        "display_url": "ind.pn/1g3wX9q",
+        "indices": [
+          80,
+          102
+        ]
+      }
+    ],
+    "user_mentions": [],
+    "media": [
+      {
+        "id": 426318539692056600,
+        "id_str": "426318539692056576",
+        "indices": [
+          103,
+          125
+        ],
+        "media_url": "http://pbs.twimg.com/media/BeqWc_-CIAAhmdc.jpg",
+        "media_url_https": "https://pbs.twimg.com/media/BeqWc_-CIAAhmdc.jpg",
+        "url": "http://t.co/JxQTPG7WLL",
+        "display_url": "pic.twitter.com/JxQTPG7WLL",
+        "expanded_url": "http://twitter.com/Independent/status/426318539796930560/photo/1",
+        "type": "photo",
+        "sizes": {
+          "thumb": {
+            "w": 150,
+            "h": 150,
+            "resize": "crop"
+          },
+          "large": {
+            "w": 1024,
+            "h": 682,
+            "resize": "fit"
+          },
+          "small": {
+            "w": 340,
+            "h": 226,
+            "resize": "fit"
+          },
+          "medium": {
+            "w": 600,
+            "h": 400,
+            "resize": "fit"
+          }
+        }
+      }
+    ]
+  },
+  "favorited": false,
+  "retweeted": false,
+  "possibly_sensitive": false,
+  "lang": "en"
+}
 
=== modified file 'friends/tests/test_twitter.py'
--- friends/tests/test_twitter.py	2013-12-05 20:45:19 +0000
+++ friends/tests/test_twitter.py	2014-01-24 10:56:16 +0000
@@ -145,7 +145,8 @@
             ['twitter', 88, '240556426106372096',
              'messages', 'Raffi Krikorian', '8285392', 'raffi', False,
              '2012-08-28T21:08:15Z', 'lecturing at the "analyzing big data '
-             'with twitter" class at @cal with @othman '
+             'with twitter" class at <a href="https://twitter.com/Cal">@Cal</a>'
+             ' with <a href="https://twitter.com/othman">@othman</a> '
              '<a href="http://twitter.com/yunorno/status/114080493036773378/photo/1">'
              'pic.twitter.com/rJC5Pxsu</a>', 
              'https://si0.twimg.com/profile_images/1270234259/'
@@ -468,7 +469,7 @@
         expected_row = [
             'twitter', 88, '324220250889543682',
             'messages', 'Robert Bruce', '836242932', 'therealrobru', True,
-            '2013-04-16T17:58:26Z', 'RT @tarek_ziade: Just found a "Notification '
+            '2013-04-16T17:58:26Z', 'RT <a href="https://twitter.com/tarek_ziade">@tarek_ziade</a>: Just found a "Notification '
             'of Inspection" card in the bottom of my bag. looks like they were '
             'curious about those raspberry-pi :O',
             'https://si0.twimg.com/profile_images/2631306428/'
@@ -478,6 +479,78 @@
             ]
         self.assertEqual(list(TestModel.get_row(0)), expected_row)
 
+        
+    @mock.patch('friends.utils.base.Model', TestModel)
+    @mock.patch('friends.utils.http.Soup.Message',
+                FakeSoupMessage('friends.tests.data', 'twitter-multiple-links.dat'))
+    @mock.patch('friends.protocols.twitter.Twitter._login',
+                return_value=True)
+    @mock.patch('friends.utils.base._seen_ids', {})
+    def test_multiple_links(self, *mocks):
+        self.account.access_token = 'access'
+        self.account.secret_token = 'secret'
+        self.account.user_name = 'Independent'
+        self.account.auth.parameters = dict(
+            ConsumerKey='key',
+            ConsumerSecret='secret')
+            
+        self.assertEqual(0, TestModel.get_n_rows())
+        self.assertEqual(
+            self.protocol.send('some message'),
+            'https://twitter.com/Independent/status/426318539796930560')
+        self.assertEqual(1, TestModel.get_n_rows())
+
+        self.maxDiff = None
+        expected_row = [
+            'twitter', 88, '426318539796930560',
+            'messages',  'The Independent', '16973333', 'Independent', True,
+            '2014-01-23T11:40:35Z',
+            'An old people\'s home has recreated famous movie scenes for a wonderful calendar '
+            '<a href="http://ind.pn/1g3wX9q">ind.pn/1g3wX9q</a> '
+            '<a href="http://twitter.com/Independent/status/426318539796930560/photo/1">pic.twitter.com/JxQTPG7WLL</a>',
+            'https://pbs.twimg.com/profile_images/378800000706113664/d1a957578723e496c025be1e2577d06d.jpeg',
+            'https://twitter.com/Independent/status/426318539796930560',
+            0, False, 
+            'http://pbs.twimg.com/media/BeqWc_-CIAAhmdc.jpg',
+			'', '', '', '', '', '', 0.0, 0.0,
+            ]
+        self.assertEqual(list(TestModel.get_row(0)), expected_row)
+
+    @mock.patch('friends.utils.base.Model', TestModel)
+    @mock.patch('friends.utils.http.Soup.Message',
+                FakeSoupMessage('friends.tests.data', 'twitter-hashtags.dat'))
+    @mock.patch('friends.protocols.twitter.Twitter._login',
+                return_value=True)
+    @mock.patch('friends.utils.base._seen_ids', {})
+    def test_multiple_links(self, *mocks):
+        self.account.access_token = 'access'
+        self.account.secret_token = 'secret'
+        self.account.user_name = 'Independent'
+        self.account.auth.parameters = dict(
+            ConsumerKey='key',
+            ConsumerSecret='secret')
+            
+        self.assertEqual(0, TestModel.get_n_rows())
+        self.assertEqual(
+            self.protocol.send('some message'),
+            'https://twitter.com/Kai_Mast/status/424185261375766530')
+        self.assertEqual(1, TestModel.get_n_rows())
+
+        self.maxDiff = None
+        expected_row = [
+            'twitter', 88, '424185261375766530',
+            'messages',  'Kai Mast', '17339829', 'Kai_Mast', False,
+            '2014-01-17T14:23:41Z',
+            'A service that filters food pictures from Instagram. '
+            '<a href="https://twitter.com/search?q=%23MillionDollarIdea&src=hash">#MillionDollarIdea</a>',
+            'https://pbs.twimg.com/profile_images/424181800701673473/Q6Ggqg7P.png',
+            'https://twitter.com/Kai_Mast/status/424185261375766530',
+            0, False, 
+            '',
+			'', '', '', '', '', '', 0.0, 0.0,
+            ]
+        self.assertEqual(list(TestModel.get_row(0)), expected_row)
+
     def test_unfollow(self):
         get_url = self.protocol._get_url = mock.Mock()