Merge lp:~brian.curtin/ubuntu-sso-client/py3-urllib into lp:ubuntu-sso-client

[Brian Curtin]

Try Python 3 urllib names and fall back to Python 2 as needed.

Python 3 reorganizes the urllib and urllib2 packages into a new structure, so we need to change our imports to work for both. Using the try/except dance, prefer the Python 3 way and fall back to the Python 2 way on ImportError.

In order to ease compatibility, some of the imports are changed to import names rather than the entire module, as there is no good way to write multi-version compatible code that imports the higher level names urllib or urllib2.

[lp:~brian.curtin/ubuntu-sso-client/py3-urllib]

977. By Brian Curtin on 2012-06-27

Tag 1018587

978. By Brian Curtin on 2012-06-27

Add test_webclient changes

[Alejandro J. Cura (alecu)]

== Python Lint Notices ==

ubuntu_sso/gtk/tests/test_gui.py:
    450: [W0511] FIXME - New GTK+ 3.5 breaks this check - see bug #1014772

ubuntu_sso/main/darwin.py:
    60: [W0511] TODO

ubuntu_sso/qt/__init__.py:
    50: [W0511] TODO

ubuntu_sso/tests/test_account.py:
    35: [W0611] Unused import urllib2

ubuntu_sso/utils/__init__.py:
    39: [F0401] Unable to import 'urllib.parse'
    39: [E0611] No name 'parse' in module 'urllib'
    40: [F0401] Unable to import 'urllib.request'
    40: [E0611] No name 'request' in module 'urllib'

ubuntu_sso/utils/webclient/common.py:
    284: [W0511] TODO
    40: [F0401] Unable to import 'urllib.parse'
    40: [E0611] No name 'parse' in module 'urllib'

ubuntu_sso/utils/webclient/restful.py:
    34: [F0401] Unable to import 'urllib.parse'
    34: [E0611] No name 'parse' in module 'urllib'

ubuntu_sso/utils/webclient/tests/test_restful.py:
    34: [F0401] Unable to import 'urllib.parse'
    34: [E0611] No name 'parse' in module 'urllib'
    128: [E1103, RestfulClientTestCase.test_restful_namespace_added_to_url] Instance of 'ParseResult' has no 'path' member (but some types could not be inferred)

ubuntu_sso/utils/webclient/tests/test_webclient.py:
    37: [F0401] Unable to import 'urllib.parse'
    37: [E0611] No name 'parse' in module 'urllib'

ubuntu_sso/utils/webclient/txweb.py:
    34: [F0401] Unable to import 'urllib.parse'
    34: [E0611] No name 'parse' in module 'urllib'
    102: [E1103, WebClient.raw_request] Instance of 'ParseResult' has no 'scheme' member (but some types could not be inferred)
    103: [E1103, WebClient.raw_request] Instance of 'ParseResult' has no 'netloc' member (but some types could not be inferred)

review: Needs Fixing

[lp:~brian.curtin/ubuntu-sso-client/py3-urllib]

979. By Brian Curtin on 2012-06-29

Add more pylint disable/enable pairs to please Linux

[Brian Curtin (brian.curtin)]

This should be fixed.

[Alejandro J. Cura (alecu)]

== Python Lint Notices ==

ubuntu_sso/gtk/tests/test_gui.py:
    450: [W0511] FIXME - New GTK+ 3.5 breaks this check - see bug #1014772

ubuntu_sso/main/darwin.py:
    60: [W0511] TODO

ubuntu_sso/qt/__init__.py:
    50: [W0511] TODO

ubuntu_sso/tests/test_account.py:
    35: [W0611] Unused import urllib2

ubuntu_sso/utils/webclient/common.py:
    286: [W0511] TODO

ubuntu_sso/utils/webclient/tests/test_restful.py:
    130: [E1103, RestfulClientTestCase.test_restful_namespace_added_to_url] Instance of 'ParseResult' has no 'path' member (but some types could not be inferred)

ubuntu_sso/utils/webclient/txweb.py:
    104: [E1103, WebClient.raw_request] Instance of 'ParseResult' has no 'scheme' member (but some types could not be inferred)
    105: [E1103, WebClient.raw_request] Instance of 'ParseResult' has no 'netloc' member (but some types could not be inferred)

review: Needs Fixing

[lp:~brian.curtin/ubuntu-sso-client/py3-urllib]

980. By Brian Curtin on 2012-07-03

Remove unused urllib2 import

981. By Brian Curtin on 2012-07-03

disable E1103 for an invalid complaint about a missing member

982. By Brian Curtin on 2012-07-03

Add disabling of E1103 as both Python 2 and Python 3 way of doing urlparse do have scheme and netloc attributes

[Alejandro J. Cura (alecu)]

+1

review: Approve

[dobey (dobey)]

review: Approve

=== modified file 'ubuntu_sso/keyring/__init__.py'
--- ubuntu_sso/keyring/__init__.py	2012-04-19 19:29:15 +0000
+++ ubuntu_sso/keyring/__init__.py	2012-07-03 21:26:20 +0000
@@ -30,7 +30,13 @@
 
 import socket
 import sys
-import urllib
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import quote
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urllib import quote
 
 from twisted.internet.defer import inlineCallbacks, returnValue
 
@@ -58,9 +64,9 @@
 
 def get_old_token_name(app_name):
     """Build the token name (old style). Return an unicode."""
-    quoted_app_name = urllib.quote(app_name)
+    quoted_app_name = quote(app_name)
     computer_name = gethostname()
-    quoted_computer_name = urllib.quote(computer_name)
+    quoted_computer_name = quote(computer_name)
 
     assert isinstance(computer_name, unicode)
     assert isinstance(quoted_computer_name, unicode)

=== modified file 'ubuntu_sso/keyring/linux.py'
--- ubuntu_sso/keyring/linux.py	2012-04-19 19:29:15 +0000
+++ ubuntu_sso/keyring/linux.py	2012-07-03 21:26:20 +0000
@@ -29,8 +29,13 @@
 # files in the program, then also delete it here.
 """Handle keys in the local kerying."""
 
-import urllib
-import urlparse
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import parse_qsl, urlencode
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urllib import urlencode
+    from urlparse import parse_qsl
 
 from twisted.internet.defer import inlineCallbacks, returnValue
 
@@ -78,7 +83,7 @@
     def set_credentials(self, app_name, cred):
         """Set the credentials of the Ubuntu SSO item."""
         # Creates the secret from the credentials
-        secret = urllib.urlencode(cred)
+        secret = urlencode(cred)
 
         attr = self._get_keyring_attr(app_name)
         # Add our SSO credentials to the keyring
@@ -95,7 +100,7 @@
         item = yield self._find_keyring_item(app_name, attr=attr)
         if item is not None:
             yield self.set_credentials(app_name,
-                                       dict(urlparse.parse_qsl(item.secret)))
+                                       dict(parse_qsl(item.secret)))
             yield item.delete()
 
         result = yield self._find_keyring_item(app_name)
@@ -114,7 +119,7 @@
         if item is not None:
             logger.debug("Parsing secret.")
             secret = yield item.get_value()
-            returnValue(dict(urlparse.parse_qsl(secret)))
+            returnValue(dict(parse_qsl(secret)))
         else:
             # if no item found, try getting the old credentials
             if app_name == U1_APP_NAME:

=== modified file 'ubuntu_sso/tests/test_account.py'
--- ubuntu_sso/tests/test_account.py	2012-06-22 16:55:35 +0000
+++ ubuntu_sso/tests/test_account.py	2012-07-03 21:26:20 +0000
@@ -32,12 +32,12 @@
 """Tests for the SSO account code."""
 
 import os
-import urllib2
 
 from twisted.trial.unittest import TestCase
 from twisted.internet import defer
 
 from ubuntu_sso import account
+from ubuntu_sso import utils
 from ubuntu_sso.account import (
     Account,
     AuthenticationError,
@@ -215,7 +215,7 @@
             self.addCleanup(f.close)
             return f
 
-        self.patch(urllib2, 'urlopen', fake_urlopen)  # fd to the path
+        self.patch(utils, 'urlopen', fake_urlopen)  # fd to the path
         self.processor = Account()
         self.register_kwargs = dict(email=EMAIL, password=PASSWORD,
                                     displayname=NAME,

=== modified file 'ubuntu_sso/utils/__init__.py'
--- ubuntu_sso/utils/__init__.py	2012-06-22 16:12:03 +0000
+++ ubuntu_sso/utils/__init__.py	2012-07-03 21:26:20 +0000
@@ -32,10 +32,17 @@
 import os
 import sys
 import time
-import urllib2
 
 from oauth import oauth
-from urlparse import urlparse
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import urlparse
+    from urllib.request import Request, urlopen
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urlparse import urlparse
+    from urllib2 import Request, urlopen
 
 from twisted.internet import defer
 
@@ -121,7 +128,7 @@
     return result
 
 
-class RequestHead(urllib2.Request):
+class RequestHead(Request):
     """A request with the method set to HEAD."""
 
     _request_method = "HEAD"
@@ -147,7 +154,7 @@
         """Get the time at the server."""
         headers = {"Cache-Control": "no-cache"}
         request = RequestHead(self.SERVER_URL, headers=headers)
-        response = urllib2.urlopen(request)
+        response = urlopen(request)
         date_string = response.info()["Date"]
         # delay import, otherwise a default reactor gets installed
         from twisted.web import http

=== modified file 'ubuntu_sso/utils/webclient/common.py'
--- ubuntu_sso/utils/webclient/common.py	2012-06-22 16:12:03 +0000
+++ ubuntu_sso/utils/webclient/common.py	2012-07-03 21:26:20 +0000
@@ -35,7 +35,13 @@
 from httplib2 import iri2uri
 from oauth import oauth
 from twisted.internet import defer
-from urlparse import urlparse
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import urlparse
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urlparse import urlparse
 
 from ubuntu_sso import USER_SUCCESS, UI_PROXY_CREDS_DIALOG
 from ubuntu_sso.logger import setup_logging

=== modified file 'ubuntu_sso/utils/webclient/restful.py'
--- ubuntu_sso/utils/webclient/restful.py	2012-06-14 20:52:29 +0000
+++ ubuntu_sso/utils/webclient/restful.py	2012-07-03 21:26:20 +0000
@@ -29,7 +29,13 @@
 """A proxy-enabled restful client."""
 
 import json
-import urllib
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import urlencode
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urllib import urlencode
 
 from twisted.internet import defer
 
@@ -67,7 +73,7 @@
             params[key] = json.dumps(value)
         namespace, operation = method.split(".")
         params["ws.op"] = operation
-        encoded_args = urllib.urlencode(params)
+        encoded_args = urlencode(params)
         iri = self.service_iri + namespace
         creds = self.oauth_credentials
         logger.debug('Performing REST call to %r.', iri)

=== modified file 'ubuntu_sso/utils/webclient/tests/test_restful.py'
--- ubuntu_sso/utils/webclient/tests/test_restful.py	2012-06-14 20:46:24 +0000
+++ ubuntu_sso/utils/webclient/tests/test_restful.py	2012-07-03 21:26:20 +0000
@@ -29,7 +29,13 @@
 """Tests for the proxy-enabled restful client."""
 
 import logging
-import urlparse
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import urlparse, parse_qs, parse_qsl
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urlparse import urlparse, parse_qs, parse_qsl
 
 from twisted.internet import defer
 from ubuntuone.devtools.handlers import MementoHandler
@@ -119,8 +125,8 @@
         yield self.rc.restcall(SAMPLE_OPERATION, **SAMPLE_ARGS)
         iri, _, _ = self.wc.called[0]
         uri = iri.encode("ascii")
-        url = urlparse.urlparse(uri)
-        # pylint: disable=E1101
+        url = urlparse(uri)
+        # pylint: disable=E1101,E1103
         self.assertTrue(url.path.endswith(SAMPLE_NAMESPACE),
                         "The namespace is included in url")
 
@@ -129,7 +135,7 @@
         """The restful method is added to the params."""
         yield self.rc.restcall(SAMPLE_OPERATION, **SAMPLE_ARGS)
         _, _, webcall_kwargs = self.wc.called[0]
-        wc_params = urlparse.parse_qs(webcall_kwargs["post_content"])
+        wc_params = parse_qs(webcall_kwargs["post_content"])
         self.assertEqual(wc_params["ws.op"][0], SAMPLE_METHOD)
 
     @defer.inlineCallbacks
@@ -137,7 +143,7 @@
         """The keyword arguments are used as json in the webcall."""
         yield self.rc.restcall(SAMPLE_OPERATION, **SAMPLE_ARGS)
         _, _, webcall_kwargs = self.wc.called[0]
-        params = urlparse.parse_qsl(webcall_kwargs["post_content"])
+        params = parse_qsl(webcall_kwargs["post_content"])
         result = {}
         for key, value in params:
             if key == "ws.op":

=== modified file 'ubuntu_sso/utils/webclient/tests/test_webclient.py'
--- ubuntu_sso/utils/webclient/tests/test_webclient.py	2012-06-22 16:51:06 +0000
+++ ubuntu_sso/utils/webclient/tests/test_webclient.py	2012-07-03 21:26:20 +0000
@@ -32,15 +32,21 @@
 import os
 import shutil
 import sys
-import urllib
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import (urlencode, unquote, urlparse,
+                              urljoin, parse_qsl)
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urllib import urlencode, unquote
+    from urlparse import urlparse, urljoin, parse_qsl
 
 from OpenSSL import crypto
 from socket import gethostname
 from twisted.cred import checkers, portal
 from twisted.internet import defer
 from twisted.web import guard, http, resource
-from urllib2 import urlparse
-
 from ubuntuone.devtools.handlers import MementoHandler
 from ubuntuone.devtools.testcases import TestCase
 from ubuntuone.devtools.testcases.squid import SquidTestCase
@@ -326,7 +332,7 @@
     @defer.inlineCallbacks
     def test_post_with_args(self):
         """Test a post request with arguments."""
-        args = urllib.urlencode(SAMPLE_POST_PARAMS)
+        args = urlencode(SAMPLE_POST_PARAMS)
         iri = self.base_iri + VERIFYPOSTPARAMS + "?" + args
         headers = {
             "content-type": "application/x-www-form-urlencoded",
@@ -743,7 +749,7 @@
         method, params_string = header.split(" ", 1)
         for p in params_string.split(","):
             k, v = p.strip().split("=")
-            params[k] = urllib.unquote(v[1:-1])
+            params[k] = unquote(v[1:-1])
 
         return method, params
 
@@ -804,12 +810,12 @@
         result = yield self.wc.build_signed_iri(iri, SAMPLE_CREDENTIALS,
                                           parameters=params)
 
-        scheme, netloc, path, params, query, _ = urlparse.urlparse(result)
+        scheme, netloc, path, params, query, _ = urlparse(result)
 
-        self.assertEqual(urlparse.urljoin(scheme + '://' + netloc, path), uri)
+        self.assertEqual(urljoin(scheme + '://' + netloc, path), uri)
 
         expected = request.parameters
-        actual = dict(urlparse.parse_qsl(query))
+        actual = dict(parse_qsl(query))
         for i in ('oauth_nonce', 'oauth_signature'):
             expected.pop(i)
             actual.pop(i)

=== modified file 'ubuntu_sso/utils/webclient/txweb.py'
--- ubuntu_sso/utils/webclient/txweb.py	2012-04-23 17:49:31 +0000
+++ ubuntu_sso/utils/webclient/txweb.py	2012-07-03 21:26:20 +0000
@@ -29,7 +29,13 @@
 """A webclient backend that uses twisted.web.client."""
 
 import base64
-import urlparse
+
+try:
+    # pylint: disable=E0611,F0401
+    from urllib.parse import urlparse
+    # pylint: enable=E0611,F0401
+except ImportError:
+    from urlparse import urlparse
 
 from twisted.internet import defer
 
@@ -92,12 +98,12 @@
         # delay import, otherwise a default reactor gets installed
         from twisted.web import error
 
-        parsed_url = urlparse.urlparse(uri)
+        parsed_url = urlparse(uri)
 
-        # pylint: disable=E1101
+        # pylint: disable=E1101,E1103
         https = parsed_url.scheme == "https"
         host = parsed_url.netloc.split(":")[0]
-        # pylint: enable=E1101
+        # pylint: enable=E1101,E1103
         if parsed_url.port is None:
             port = 443 if https else 80
         else: