Merge lp:~harlowja/cloud-init/url-ssl-fixings into lp:~cloud-init-dev/cloud-init/trunk

[Joshua Harlow]

[lp:~harlowja/cloud-init/url-ssl-fixings]

695. By harlowja <email address hidden> on 2013-02-22

More work on requests integration.

696. By harlowja <email address hidden> on 2013-02-22

Update to code on trunk.

697. By harlowja <email address hidden> on 2013-02-22

Why did this file showup.

698. By harlowja <email address hidden> on 2013-02-24

Get tests working and further adjustments.

699. By harlowja <email address hidden> on 2013-02-24

Fix the maas callback mechanism now that requests is used.

[Joshua Harlow (harlowja)]

So I looked into the requests code, they are using in a underlying layer the HTTP[d]Connection objects and the regular ssl object, this should allow it to use the system certificates as a fallback. So thats good news!

[Scott Moser (smoser)]

I'm sorry this is such a pain.
$ python -c 'from cloudinit import util; util.read_file_or_url("http://brickies.net")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cloudinit/util.py", line 689, in read_file_or_url
    ssl_details=ssl_details)
  File "cloudinit/url_helper.py", line 163, in readurl
    r = requests.request(**req_args)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
TypeError: request() got an unexpected keyword argument 'config'

$ dpkg-query --show python-requests
python-requests 1.1.0-1

See https://launchpad.net/ubuntu/+source/requests for all the ubuntu versions:
 precise: 0.8.2-1
 quantal: 0.12.1-1ubuntu6
 raring: 1.1.0-1
 latest-release: 1.1.0

Your committed code does work on precise and quantal.

review: Needs Fixing

[Scott Moser (smoser)]

Also:
$ python -c 'from cloudinit import util; import sys; print util.read_file_or_url(sys.argv[1])' https://stackoverflow.com/
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cloudinit/util.py", line 701, in read_file_or_url
    ssl_details=ssl_details)
  File "cloudinit/url_helper.py", line 173, in readurl
    if isinstance(e, (exceptions.HTTPError)) and e.response:
AttributeError: 'HTTPError' object has no attribute 'response'

$ dpkg-query --show python-requests
python-requests 0.8.2-1

[lp:~harlowja/cloud-init/url-ssl-fixings]

700. By Joshua Harlow on 2013-03-14

Update to handle requests >= 1.0 which doesn't use the config dict.

701. By Joshua Harlow on 2013-03-14

Fix how the http error doesn't always have the response attached
in earlier versions of requests (pre 0.10.8).

[Joshua Harlow (harlowja)]

Ok dokie, think those are all fixed by more checks on the requests version and the http error exception object.

[lp:~harlowja/cloud-init/url-ssl-fixings]

702. By Scott Moser on 2013-03-19

merge from trunk at revno 799

703. By Scott Moser on 2013-03-19

merge from trunk rev 800

704. By Scott Moser on 2013-03-19

appease pylint and pep8

* cloudinit/distros/parsers/resolv_conf.py
  added some pylint overrides with 'plXXXXX' syntax.
  example: # pl51222 pylint: disable=E0102

  The pl51222 there means: http://www.logilab.org/ticket/51222

  This specific issue is present in 12.04 pylint, but not 13.04.

* pylint doesn't like the requests special handling we have.
  which makes sense as it is only checking versus one specific version.
* general pep8 and pylint cleanups.

705. By Scott Moser on 2013-03-19

do not bother retrying on ssl errors

if the error is an ssl error, its extremely unlikely that it would be fixed by
waiting a few seconds and trying again.

706. By Scott Moser on 2013-03-19

set 'allow_redirects' to True by default

the previous implementation of url_helper.readurl() would default
to allow_redirects being true.

So, for backwards compat, we should keep that behavior.

707. By Scott Moser on 2013-03-19

make get_instance_userdata and get_instance_metadata more like botos

this shouldn't change anything, only the signatures of the methods.

708. By Scott Moser on 2013-03-19

fix typo

709. By Joshua Harlow on 2013-03-19

Move back to using boto for now.

710. By Joshua Harlow on 2013-03-19

Add doc about issue 1401 in boto.

=== modified file 'Requires'
--- Requires	2012-07-09 20:41:45 +0000
+++ Requires	2013-03-19 22:58:20 +0000
@@ -10,11 +10,6 @@
 # datasource is removed, this is no longer needed
 oauth
 
-# This is used to fetch the ec2 metadata into a easily
-# parseable format, instead of having to have cloud-init perform
-# those same fetchs and decodes and signing (...) that ec2 requires.
-boto
-
 # This is only needed for places where we need to support configs in a manner
 # that the built-in config parser is not sufficent (ie
 # when we need to preserve comments, or do not have a top-level
@@ -26,3 +21,9 @@
 
 # The new main entrypoint uses argparse instead of optparse
 argparse
+
+# Requests handles ssl correctly!
+requests
+
+# Boto for ec2
+boto

=== modified file 'cloudinit/config/cc_phone_home.py'
--- cloudinit/config/cc_phone_home.py	2012-10-28 02:25:48 +0000
+++ cloudinit/config/cc_phone_home.py	2013-03-19 22:58:20 +0000
@@ -19,7 +19,6 @@
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 from cloudinit import templater
-from cloudinit import url_helper as uhelp
 from cloudinit import util
 
 from cloudinit.settings import PER_INSTANCE
@@ -112,7 +111,9 @@
     }
     url = templater.render_string(url, url_params)
     try:
-        uhelp.readurl(url, data=real_submit_keys, retries=tries, sec_between=3)
+        util.read_file_or_url(url, data=real_submit_keys,
+                              retries=tries, sec_between=3,
+                              ssl_details=util.fetch_ssl_details(cloud.paths))
     except:
         util.logexc(log, ("Failed to post phone home data to"
                           " %s in %s tries"), url, tries)

=== modified file 'cloudinit/config/cc_power_state_change.py'
--- cloudinit/config/cc_power_state_change.py	2013-03-07 19:54:25 +0000
+++ cloudinit/config/cc_power_state_change.py	2013-03-19 22:58:20 +0000
@@ -100,7 +100,7 @@
         proc = subprocess.Popen(exe_args, stdin=subprocess.PIPE,
                                 stdout=output, stderr=subprocess.STDOUT)
         proc.communicate(data_in)
-        ret = proc.returncode
+        ret = proc.returncode  # pylint: disable=E1101
     except Exception:
         doexit(EXIT_FAIL)
     doexit(ret)

=== modified file 'cloudinit/distros/parsers/resolv_conf.py'
--- cloudinit/distros/parsers/resolv_conf.py	2012-11-12 22:30:08 +0000
+++ cloudinit/distros/parsers/resolv_conf.py	2013-03-19 22:58:20 +0000
@@ -137,8 +137,8 @@
         self._contents.append(('option', ['search', s_list, '']))
         return flat_sds
 
-    @local_domain.setter
-    def local_domain(self, domain):
+    @local_domain.setter  # pl51222 pylint: disable=E1101
+    def local_domain(self, domain):  # pl51222 pylint: disable=E0102
         self.parse()
         self._remove_option('domain')
         self._contents.append(('option', ['domain', str(domain), '']))

=== modified file 'cloudinit/ec2_utils.py'
--- cloudinit/ec2_utils.py	2012-11-13 19:02:03 +0000
+++ cloudinit/ec2_utils.py	2013-03-19 22:58:20 +0000
@@ -28,6 +28,10 @@
 # would have existed) do not exist due to the blocking
 # that occurred.
 
+# TODO: https://github.com/boto/boto/issues/1401
+# When boto finally moves to using requests, we should be able
+# to provide it ssl details, it does not yet, so we can't provide them...
+
 
 def _unlazy_dict(mp):
     if not isinstance(mp, (dict)):
@@ -57,3 +61,4 @@
     if not isinstance(metadata, (dict)):
         metadata = {}
     return _unlazy_dict(metadata)
+

=== modified file 'cloudinit/sources/DataSourceCloudStack.py'
--- cloudinit/sources/DataSourceCloudStack.py	2013-03-07 21:27:47 +0000
+++ cloudinit/sources/DataSourceCloudStack.py	2013-03-19 22:58:20 +0000
@@ -25,7 +25,7 @@
 import os
 import time
 
-from cloudinit import ec2_utils as ec2
+from cloudinit import ec2_utils
 from cloudinit import log as logging
 from cloudinit import sources
 from cloudinit import url_helper as uhelp
@@ -101,10 +101,11 @@
             if not self.wait_for_metadata_service():
                 return False
             start_time = time.time()
-            self.userdata_raw = ec2.get_instance_userdata(self.api_ver,
-                self.metadata_address)
-            self.metadata = ec2.get_instance_metadata(self.api_ver,
-                                                      self.metadata_address)
+            md_addr = self.metadata_address
+            self.userdata_raw = ec2_utils.get_instance_userdata(self.api_ver,
+                                                                md_addr)
+            self.metadata = ec2_utils.get_instance_metadata(self.api_ver,
+                                                            md_addr)
             LOG.debug("Crawl of metadata service took %s seconds",
                       int(time.time() - start_time))
             return True

=== modified file 'cloudinit/sources/DataSourceEc2.py'
--- cloudinit/sources/DataSourceEc2.py	2013-03-07 03:24:05 +0000
+++ cloudinit/sources/DataSourceEc2.py	2013-03-19 22:58:20 +0000
@@ -23,7 +23,7 @@
 import os
 import time
 
-from cloudinit import ec2_utils as ec2
+from cloudinit import ec2_utils
 from cloudinit import log as logging
 from cloudinit import sources
 from cloudinit import url_helper as uhelp
@@ -61,10 +61,11 @@
             if not self.wait_for_metadata_service():
                 return False
             start_time = time.time()
-            self.userdata_raw = ec2.get_instance_userdata(self.api_ver,
-                self.metadata_address)
-            self.metadata = ec2.get_instance_metadata(self.api_ver,
-                                                      self.metadata_address)
+            md_addr = self.metadata_address
+            self.userdata_raw = ec2_utils.get_instance_userdata(self.api_ver,
+                                                                md_addr)
+            self.metadata = ec2_utils.get_instance_metadata(self.api_ver,
+                                                            md_addr)
             LOG.debug("Crawl of metadata service took %s seconds",
                        int(time.time() - start_time))
             return True
@@ -133,7 +134,7 @@
 
         start_time = time.time()
         url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
-                                timeout=timeout, status_cb=LOG.warn)
+                                 timeout=timeout, status_cb=LOG.warn)
 
         if url:
             LOG.debug("Using metadata source: '%s'", url2base[url])

=== modified file 'cloudinit/sources/DataSourceMAAS.py'
--- cloudinit/sources/DataSourceMAAS.py	2013-03-07 03:24:05 +0000
+++ cloudinit/sources/DataSourceMAAS.py	2013-03-19 22:58:20 +0000
@@ -27,7 +27,7 @@
 
 from cloudinit import log as logging
 from cloudinit import sources
-from cloudinit import url_helper as uhelp
+from cloudinit import url_helper
 from cloudinit import util
 
 LOG = logging.getLogger(__name__)
@@ -80,7 +80,8 @@
             self.base_url = url
 
             (userdata, metadata) = read_maas_seed_url(self.base_url,
-                                                      self.md_headers)
+                                                      self._md_headers,
+                                                      paths=self.paths)
             self.userdata_raw = userdata
             self.metadata = metadata
             return True
@@ -88,7 +89,7 @@
             util.logexc(LOG, "Failed fetching metadata from url %s", url)
             return False
 
-    def md_headers(self, url):
+    def _md_headers(self, url):
         mcfg = self.ds_cfg
 
         # If we are missing token_key, token_secret or consumer_key
@@ -132,36 +133,37 @@
         starttime = time.time()
         check_url = "%s/%s/meta-data/instance-id" % (url, MD_VERSION)
         urls = [check_url]
-        url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
-                                 timeout=timeout, exception_cb=self._except_cb,
-                                 headers_cb=self.md_headers)
+        url = url_helper.wait_for_url(urls=urls, max_wait=max_wait,
+                                      timeout=timeout,
+                                      exception_cb=self._except_cb,
+                                      headers_cb=self._md_headers)
 
         if url:
             LOG.debug("Using metadata source: '%s'", url)
         else:
             LOG.critical("Giving up on md from %s after %i seconds",
-                            urls, int(time.time() - starttime))
+                         urls, int(time.time() - starttime))
 
         return bool(url)
 
     def _except_cb(self, msg, exception):
-        if not (isinstance(exception, urllib2.HTTPError) and
+        if not (isinstance(exception, url_helper.UrlError) and
                 (exception.code == 403 or exception.code == 401)):
             return
+
         if 'date' not in exception.headers:
-            LOG.warn("date field not in %d headers" % exception.code)
+            LOG.warn("Missing header 'date' in %s response", exception.code)
             return
 
         date = exception.headers['date']
-
         try:
             ret_time = time.mktime(parsedate(date))
-        except:
-            LOG.warn("failed to convert datetime '%s'")
+        except Exception as e:
+            LOG.warn("Failed to convert datetime '%s': %s", date, e)
             return
 
         self.oauth_clockskew = int(ret_time - time.time())
-        LOG.warn("set oauth clockskew to %d" % self.oauth_clockskew)
+        LOG.warn("Setting oauth clockskew to %d", self.oauth_clockskew)
         return
 
 
@@ -189,11 +191,11 @@
 
 
 def read_maas_seed_url(seed_url, header_cb=None, timeout=None,
-    version=MD_VERSION):
+                       version=MD_VERSION, paths=None):
     """
     Read the maas datasource at seed_url.
-    header_cb is a method that should return a headers dictionary that will
-    be given to urllib2.Request()
+      - header_cb is a method that should return a headers dictionary for
+        a given url
 
     Expected format of seed_url is are the following files:
       * <seed_url>/<version>/meta-data/instance-id
@@ -221,13 +223,17 @@
         else:
             headers = {}
         try:
-            resp = uhelp.readurl(url, headers=headers, timeout=timeout)
+            ssl_details = util.fetch_ssl_details(paths)
+            resp = util.read_file_or_url(url,
+                                         headers=headers,
+                                         timeout=timeout,
+                                         ssl_details=ssl_details)
             if resp.ok():
                 md[name] = str(resp)
             else:
                 LOG.warn(("Fetching from %s resulted in"
                           " an invalid http code %s"), url, resp.code)
-        except urllib2.HTTPError as e:
+        except url_helper.UrlError as e:
             if e.code != 404:
                 raise
     return check_seed_contents(md, seed_url)
@@ -370,7 +376,8 @@
         if args.subcmd == "check-seed":
             if args.url.startswith("http"):
                 (userdata, metadata) = read_maas_seed_url(args.url,
-                    header_cb=my_headers, version=args.apiver)
+                                                          header_cb=my_headers,
+                                                          version=args.apiver)
             else:
                 (userdata, metadata) = read_maas_seed_url(args.url)
             print "=== userdata ==="

=== modified file 'cloudinit/url_helper.py'
--- cloudinit/url_helper.py	2012-09-24 21:13:38 +0000
+++ cloudinit/url_helper.py	2013-03-19 22:58:20 +0000
@@ -20,43 +20,55 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from contextlib import closing
-
-import errno
-import socket
 import time
-import urllib
-import urllib2
+
+import requests
+from requests import exceptions
+
+from urlparse import (urlparse, urlunparse)
 
 from cloudinit import log as logging
 from cloudinit import version
 
 LOG = logging.getLogger(__name__)
 
+# Check if requests has ssl support (added in requests >= 0.8.8)
+SSL_ENABLED = False
+CONFIG_ENABLED = False  # This was added in 0.7 (but taken out in >=1.0)
+try:
+    from distutils.version import LooseVersion
+    import pkg_resources
+    _REQ = pkg_resources.get_distribution('requests')
+    _REQ_VER = LooseVersion(_REQ.version)  # pylint: disable=E1103
+    if _REQ_VER >= LooseVersion('0.8.8'):
+        SSL_ENABLED = True
+    if _REQ_VER >= LooseVersion('0.7.0') and _REQ_VER < LooseVersion('1.0.0'):
+        CONFIG_ENABLED = True
+except:
+    pass
+
+
+def _cleanurl(url):
+    parsed_url = list(urlparse(url, scheme='http'))  # pylint: disable=E1123
+    if not parsed_url[1] and parsed_url[2]:
+        # Swap these since this seems to be a common
+        # occurrence when given urls like 'www.google.com'
+        parsed_url[1] = parsed_url[2]
+        parsed_url[2] = ''
+    return urlunparse(parsed_url)
+
 
 class UrlResponse(object):
-    def __init__(self, status_code, contents=None, headers=None):
-        self._status_code = status_code
-        self._contents = contents
-        self._headers = headers
-
-    @property
-    def code(self):
-        return self._status_code
+    def __init__(self, response):
+        self._response = response
 
     @property
     def contents(self):
-        return self._contents
+        return self._response.content
 
     @property
-    def headers(self):
-        return self._headers
-
-    def __str__(self):
-        if not self.contents:
-            return ''
-        else:
-            return str(self.contents)
+    def url(self):
+        return self._response.url
 
     def ok(self, redirects_ok=False):
         upper = 300
@@ -67,72 +79,117 @@
         else:
             return False
 
-
-def readurl(url, data=None, timeout=None,
-            retries=0, sec_between=1, headers=None):
-
-    req_args = {}
-    req_args['url'] = url
-    if data is not None:
-        req_args['data'] = urllib.urlencode(data)
-
+    @property
+    def headers(self):
+        return self._response.headers
+
+    @property
+    def code(self):
+        return self._response.status_code
+
+    def __str__(self):
+        return self.contents
+
+
+class UrlError(IOError):
+    def __init__(self, cause, code=None, headers=None):
+        IOError.__init__(self, str(cause))
+        self.cause = cause
+        self.code = code
+        self.headers = headers
+        if self.headers is None:
+            self.headers = {}
+
+
+def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
+            headers=None, ssl_details=None, check_status=True,
+            allow_redirects=True):
+    url = _cleanurl(url)
+    req_args = {
+        'url': url,
+    }
+    scheme = urlparse(url).scheme  # pylint: disable=E1101
+    if scheme == 'https' and ssl_details:
+        if not SSL_ENABLED:
+            LOG.warn("SSL is not enabled, cert. verification can not occur!")
+        else:
+            if 'ca_certs' in ssl_details and ssl_details['ca_certs']:
+                req_args['verify'] = ssl_details['ca_certs']
+            else:
+                req_args['verify'] = True
+            if 'cert_file' in ssl_details and 'key_file' in ssl_details:
+                req_args['cert'] = [ssl_details['cert_file'],
+                                    ssl_details['key_file']]
+            elif 'cert_file' in ssl_details:
+                req_args['cert'] = str(ssl_details['cert_file'])
+
+    req_args['allow_redirects'] = allow_redirects
+    req_args['method'] = 'GET'
+    if timeout is not None:
+        req_args['timeout'] = max(float(timeout), 0)
+    if data:
+        req_args['method'] = 'POST'
+    # It doesn't seem like config
+    # was added in older library versions (or newer ones either), thus we
+    # need to manually do the retries if it wasn't...
+    if CONFIG_ENABLED:
+        req_config = {
+            'store_cookies': False,
+        }
+        # Don't use the retry support built-in
+        # since it doesn't allow for 'sleep_times'
+        # in between tries....
+        # if retries:
+        #     req_config['max_retries'] = max(int(retries), 0)
+        req_args['config'] = req_config
+    manual_tries = 1
+    if retries:
+        manual_tries = max(int(retries) + 1, 1)
     if not headers:
         headers = {
             'User-Agent': 'Cloud-Init/%s' % (version.version_string()),
         }
-
     req_args['headers'] = headers
-    req = urllib2.Request(**req_args)
-
-    retries = max(retries, 0)
-    attempts = retries + 1
-
-    excepts = []
-    LOG.debug(("Attempting to open '%s' with %s attempts"
-               " (%s retries, timeout=%s) to be performed"),
-              url, attempts, retries, timeout)
-    open_args = {}
-    if timeout is not None:
-        open_args['timeout'] = int(timeout)
-    for i in range(0, attempts):
+    LOG.debug("Attempting to open '%s' with %s configuration", url, req_args)
+    if data:
+        # Do this after the log (it might be large)
+        req_args['data'] = data
+    if sec_between is None:
+        sec_between = -1
+    excps = []
+    # Handle retrying ourselves since the built-in support
+    # doesn't handle sleeping between tries...
+    for i in range(0, manual_tries):
         try:
-            with closing(urllib2.urlopen(req, **open_args)) as rh:
-                content = rh.read()
-                status = rh.getcode()
-                if status is None:
-                    # This seems to happen when files are read...
-                    status = 200
-                headers = {}
-                if rh.headers:
-                    headers = dict(rh.headers)
-                LOG.debug("Read from %s (%s, %sb) after %s attempts",
-                          url, status, len(content), (i + 1))
-                return UrlResponse(status, content, headers)
-        except urllib2.HTTPError as e:
-            excepts.append(e)
-        except urllib2.URLError as e:
-            # This can be a message string or
-            # another exception instance
-            # (socket.error for remote URLs, OSError for local URLs).
-            if (isinstance(e.reason, (OSError)) and
-                e.reason.errno == errno.ENOENT):
-                excepts.append(e.reason)
+            r = requests.request(**req_args)
+            if check_status:
+                r.raise_for_status()  # pylint: disable=E1103
+            LOG.debug("Read from %s (%s, %sb) after %s attempts", url,
+                      r.status_code, len(r.content),  # pylint: disable=E1103
+                      (i + 1))
+            # Doesn't seem like we can make it use a different
+            # subclass for responses, so add our own backward-compat
+            # attrs
+            return UrlResponse(r)
+        except exceptions.RequestException as e:
+            if (isinstance(e, (exceptions.HTTPError))
+                and hasattr(e, 'response')  # This appeared in v 0.10.8
+                and e.response):
+                excps.append(UrlError(e, code=e.response.status_code,
+                                      headers=e.response.headers))
             else:
-                excepts.append(e)
-        except Exception as e:
-            excepts.append(e)
-        if i + 1 < attempts:
-            LOG.debug("Please wait %s seconds while we wait to try again",
-                     sec_between)
-            time.sleep(sec_between)
-
-    # Didn't work out
-    LOG.debug("Failed reading from %s after %s attempts", url, attempts)
-
-    # It must of errored at least once for code
-    # to get here so re-raise the last error
-    LOG.debug("%s errors occured, re-raising the last one", len(excepts))
-    raise excepts[-1]
+                excps.append(UrlError(e))
+                if SSL_ENABLED and isinstance(e, exceptions.SSLError):
+                    # ssl exceptions are not going to get fixed by waiting a
+                    # few seconds
+                    break
+            if i + 1 < manual_tries and sec_between > 0:
+                LOG.debug("Please wait %s seconds while we wait to try again",
+                          sec_between)
+                time.sleep(sec_between)
+    if excps:
+        raise excps[-1]
+    return None  # Should throw before this...
 
 
 def wait_for_url(urls, max_wait=None, timeout=None,
@@ -143,7 +200,7 @@
     max_wait:  roughly the maximum time to wait before giving up
                The max time is *actually* len(urls)*timeout as each url will
                be tried once and given the timeout provided.
-    timeout:   the timeout provided to urllib2.urlopen
+    timeout:   the timeout provided to urlopen
     status_cb: call method with string message when a url is not available
     headers_cb: call method with single argument of url to get headers
                 for request.
@@ -190,36 +247,40 @@
                     timeout = int((start_time + max_wait) - now)
 
             reason = ""
+            e = None
             try:
                 if headers_cb is not None:
                     headers = headers_cb(url)
                 else:
                     headers = {}
 
-                resp = readurl(url, headers=headers, timeout=timeout)
-                if not resp.contents:
-                    reason = "empty response [%s]" % (resp.code)
-                    e = ValueError(reason)
-                elif not resp.ok():
-                    reason = "bad status code [%s]" % (resp.code)
-                    e = ValueError(reason)
+                response = readurl(url, headers=headers, timeout=timeout,
+                                   check_status=False)
+                if not response.contents:
+                    reason = "empty response [%s]" % (response.code)
+                    e = UrlError(ValueError(reason),
+                                 code=response.code, headers=response.headers)
+                elif not response.ok():
+                    reason = "bad status code [%s]" % (response.code)
+                    e = UrlError(ValueError(reason),
+                                 code=response.code, headers=response.headers)
                 else:
                     return url
-            except urllib2.HTTPError as e:
-                reason = "http error [%s]" % e.code
-            except urllib2.URLError as e:
-                reason = "url error [%s]" % e.reason
-            except socket.timeout as e:
-                reason = "socket timeout [%s]" % e
+            except UrlError as e:
+                reason = "request error [%s]" % e
             except Exception as e:
                 reason = "unexpected error [%s]" % e
 
             time_taken = int(time.time() - start_time)
             status_msg = "Calling '%s' failed [%s/%ss]: %s" % (url,
-                                                             time_taken,
-                                                             max_wait, reason)
+                                                               time_taken,
+                                                               max_wait,
+                                                               reason)
             status_cb(status_msg)
             if exception_cb:
+                # This can be used to alter the headers that will be sent
+                # in the future, for example this is what the MAAS datasource
+                # does.
                 exception_cb(msg=status_msg, exception=e)
 
         if timeup(max_wait, start_time):

=== modified file 'cloudinit/user_data.py'
--- cloudinit/user_data.py	2012-10-10 16:27:28 +0000
+++ cloudinit/user_data.py	2013-03-19 22:58:20 +0000
@@ -29,7 +29,6 @@
 
 from cloudinit import handlers
 from cloudinit import log as logging
-from cloudinit import url_helper
 from cloudinit import util
 
 LOG = logging.getLogger(__name__)
@@ -60,6 +59,7 @@
 class UserDataProcessor(object):
     def __init__(self, paths):
         self.paths = paths
+        self.ssl_details = util.fetch_ssl_details(paths)
 
     def process(self, blob):
         accumulating_msg = MIMEMultipart()
@@ -173,7 +173,8 @@
             if include_once_on and os.path.isfile(include_once_fn):
                 content = util.load_file(include_once_fn)
             else:
-                resp = url_helper.readurl(include_url)
+                resp = util.read_file_or_url(include_url,
+                                             ssl_details=self.ssl_details)
                 if include_once_on and resp.ok():
                     util.write_file(include_once_fn, str(resp), mode=0600)
                 if resp.ok():

=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2013-03-13 14:43:40 +0000
+++ cloudinit/util.py	2013-03-19 22:58:20 +0000
@@ -52,7 +52,7 @@
 from cloudinit import mergers
 from cloudinit import safeyaml
 from cloudinit import type_utils
-from cloudinit import url_helper as uhelp
+from cloudinit import url_helper
 from cloudinit import version
 
 from cloudinit.settings import (CFG_BUILTIN)
@@ -71,6 +71,31 @@
 CONTAINER_TESTS = ['running-in-container', 'lxc-is-container']
 
 
+# Made to have same accessors as UrlResponse so that the
+# read_file_or_url can return this or that object and the
+# 'user' of those objects will not need to know the difference.
+class StringResponse(object):
+    def __init__(self, contents, code=200):
+        self.code = code
+        self.headers = {}
+        self.contents = contents
+        self.url = None
+
+    def ok(self, *args, **kwargs):  # pylint: disable=W0613
+        if self.code != 200:
+            return False
+        return True
+
+    def __str__(self):
+        return self.contents
+
+
+class FileResponse(StringResponse):
+    def __init__(self, path, contents, code=200):
+        StringResponse.__init__(self, contents, code=code)
+        self.url = path
+
+
 class ProcessExecutionError(IOError):
 
     MESSAGE_TMPL = ('%(description)s\n'
@@ -462,7 +487,7 @@
             new_fp = open(arg, owith)
         elif mode == "|":
             proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
+            new_fp = proc.stdin  # pylint: disable=E1101
         else:
             raise TypeError("Invalid type for output format: %s" % outfmt)
 
@@ -484,7 +509,7 @@
             new_fp = open(arg, owith)
         elif mode == "|":
             proc = subprocess.Popen(arg, shell=True, stdin=subprocess.PIPE)
-            new_fp = proc.stdin
+            new_fp = proc.stdin  # pylint: disable=E1101
         else:
             raise TypeError("Invalid type for error format: %s" % errfmt)
 
@@ -606,18 +631,62 @@
         fill['user-data'] = ud
         fill['meta-data'] = md
         return True
-    except OSError as e:
+    except IOError as e:
         if e.errno == errno.ENOENT:
             return False
         raise
 
 
-def read_file_or_url(url, timeout=5, retries=10, file_retries=0):
+def fetch_ssl_details(paths=None):
+    ssl_details = {}
+    # Lookup in these locations for ssl key/cert files
+    ssl_cert_paths = [
+        '/var/lib/cloud/data/ssl',
+        '/var/lib/cloud/instance/data/ssl',
+    ]
+    if paths:
+        ssl_cert_paths.extend([
+            os.path.join(paths.get_ipath_cur('data'), 'ssl'),
+            os.path.join(paths.get_cpath('data'), 'ssl'),
+        ])
+    ssl_cert_paths = uniq_merge(ssl_cert_paths)
+    ssl_cert_paths = [d for d in ssl_cert_paths if d and os.path.isdir(d)]
+    cert_file = None
+    for d in ssl_cert_paths:
+        if os.path.isfile(os.path.join(d, 'cert.pem')):
+            cert_file = os.path.join(d, 'cert.pem')
+            break
+    key_file = None
+    for d in ssl_cert_paths:
+        if os.path.isfile(os.path.join(d, 'key.pem')):
+            key_file = os.path.join(d, 'key.pem')
+            break
+    if cert_file and key_file:
+        ssl_details['cert_file'] = cert_file
+        ssl_details['key_file'] = key_file
+    elif cert_file:
+        ssl_details['cert_file'] = cert_file
+    return ssl_details
+
+
+def read_file_or_url(url, timeout=5, retries=10,
+                     headers=None, data=None, sec_between=1, ssl_details=None):
+    url = url.lstrip()
     if url.startswith("/"):
         url = "file://%s" % url
-    if url.startswith("file://"):
-        retries = file_retries
-    return uhelp.readurl(url, timeout=timeout, retries=retries)
+    if url.lower().startswith("file://"):
+        if data:
+            LOG.warn("Unable to post data to file resource %s", url)
+        file_path = url[len("file://"):]
+        return FileResponse(file_path, contents=load_file(file_path))
+    else:
+        return url_helper.readurl(url,
+                                  timeout=timeout,
+                                  retries=retries,
+                                  headers=headers,
+                                  data=data,
+                                  sec_between=sec_between,
+                                  ssl_details=ssl_details)
 
 
 def load_yaml(blob, default=None, allowed=(dict,)):
@@ -834,7 +903,7 @@
     if not url:
         return (None, None, None)
 
-    resp = uhelp.readurl(url)
+    resp = read_file_or_url(url)
     if resp.contents.startswith(starts) and resp.ok():
         return (key, url, str(resp))
 
@@ -1409,7 +1478,7 @@
         (out, err) = sp.communicate(data)
     except OSError as e:
         raise ProcessExecutionError(cmd=args, reason=e)
-    rc = sp.returncode
+    rc = sp.returncode  # pylint: disable=E1101
     if rc not in rcs:
         raise ProcessExecutionError(stdout=out, stderr=err,
                                     exit_code=rc,

=== modified file 'setup.py'
--- setup.py	2013-02-21 14:07:54 +0000
+++ setup.py	2013-03-19 22:58:20 +0000
@@ -61,9 +61,10 @@
     sp = subprocess.Popen(cmd, stdout=stdout,
                     stderr=stderr, stdin=None)
     (out, err) = sp.communicate()
-    if sp.returncode not in [0]:
+    ret = sp.returncode  # pylint: disable=E1101
+    if ret not in [0]:
         raise RuntimeError("Failed running %s [rc=%s] (%s, %s)" 
-                            % (cmd, sp.returncode, out, err))
+                            % (cmd, ret, out, err))
     return (out, err)
 
 

=== modified file 'tests/unittests/test__init__.py'
--- tests/unittests/test__init__.py	2013-03-07 22:13:05 +0000
+++ tests/unittests/test__init__.py	2013-03-19 22:58:20 +0000
@@ -195,8 +195,8 @@
 
         mock_readurl = self.mocker.replace(url_helper.readurl,
                                            passthrough=False)
-        mock_readurl(url)
-        self.mocker.result(url_helper.UrlResponse(200, payload))
+        mock_readurl(url, ARGS, KWARGS)
+        self.mocker.result(util.StringResponse(payload))
         self.mocker.replay()
 
         self.assertEqual((key, url, None),
@@ -211,8 +211,8 @@
 
         mock_readurl = self.mocker.replace(url_helper.readurl,
                                            passthrough=False)
-        mock_readurl(url)
-        self.mocker.result(url_helper.UrlResponse(200, payload))
+        mock_readurl(url, ARGS, KWARGS)
+        self.mocker.result(util.StringResponse(payload))
         self.mocker.replay()
 
         self.assertEqual((key, url, payload),
@@ -225,7 +225,7 @@
         cmdline = "ro %s=%s bar=1" % (key, url)
 
         self.mocker.replace(url_helper.readurl, passthrough=False)
-        self.mocker.result(url_helper.UrlResponse(400))
+        self.mocker.result(util.StringResponse(""))
         self.mocker.replay()
 
         self.assertEqual((None, None, None),

=== modified file 'tests/unittests/test_datasource/test_maas.py'
--- tests/unittests/test_datasource/test_maas.py	2013-02-07 17:08:30 +0000
+++ tests/unittests/test_datasource/test_maas.py	2013-03-19 22:58:20 +0000
@@ -3,12 +3,13 @@
 
 from cloudinit.sources import DataSourceMAAS
 from cloudinit import url_helper
+from cloudinit import util
 from tests.unittests.helpers import populate_dir
 
-from mocker import MockerTestCase
-
-
-class TestMAASDataSource(MockerTestCase):
+import mocker
+
+
+class TestMAASDataSource(mocker.MockerTestCase):
 
     def setUp(self):
         super(TestMAASDataSource, self).setUp()
@@ -115,9 +116,11 @@
 
         for key in valid_order:
             url = "%s/%s/%s" % (my_seed, my_ver, key)
-            mock_request(url, headers=my_headers, timeout=None)
+            mock_request(url, headers=my_headers, timeout=mocker.ANY,
+                         data=mocker.ANY, sec_between=mocker.ANY,
+                         ssl_details=mocker.ANY, retries=mocker.ANY)
             resp = valid.get(key)
-            self.mocker.result(url_helper.UrlResponse(200, resp))
+            self.mocker.result(util.StringResponse(resp))
         self.mocker.replay()
 
         (userdata, metadata) = DataSourceMAAS.read_maas_seed_url(my_seed,

=== modified file 'tests/unittests/test_util.py'
--- tests/unittests/test_util.py	2013-03-11 02:47:46 +0000
+++ tests/unittests/test_util.py	2013-03-19 22:58:20 +0000
@@ -1,3 +1,5 @@
+# pylint: disable=C0301
+# the mountinfo data lines are too long
 import os
 import stat
 import yaml