Merge ~larsks/cloud-init:bug/ec2-tests into cloud-init:master

[Lars Kellogg-Stedman]

test_ec2: metadata tests were mocking wrong urls

The ec2 metadata tests were only mocking one version of the metadata
api, but requests were made against both. This fixes _setup_ds to
register mock data at both versions of the API, and adds a fallback
handler to raise an http error on requests to un-mocked requests.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:177c0edb53d53479839f5a723783a72a8ae17b5e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/209/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/209/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9e5a3601ffb3d191b53b520741b5dcebfa425452
https://jenkins.ubuntu.com/server/job/cloud-init-ci/213/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/213/rebuild

review: Needs Fixing (continuous-integration)

Unmerged commits

1884378... by Lars Kellogg-Stedman on 2017-08-25

test_ec2: metadata tests were mocking wrong urls

The ec2 metadata tests were only mocking one version of the metadata
api, but requests were made against both. This fixes _setup_ds to
register mock data at both versions of the API.

29025c3... by Lars Kellogg-Stedman on 2016-12-08

url_helper: fail gracefully if oauthlib is not available

We are unable to ship python-oauthlib in RHEL. This commit allows
imports of url_helper to succeed even when oauthlib is unavailable
and OauthUrlHelper.oauth_headers to raise a NotImplementedException
when called.

diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
index 7cf76aa..c83061a 100644
--- a/cloudinit/url_helper.py
+++ b/cloudinit/url_helper.py
@@ -17,7 +17,11 @@ import time
 from email.utils import parsedate
 from functools import partial
 
-import oauthlib.oauth1 as oauth1
+try:
+    import oauthlib.oauth1 as oauth1
+except ImportError:
+    oauth1 = None
+
 from requests import exceptions
 
 from six.moves.urllib.parse import (
@@ -488,6 +492,10 @@ class OauthUrlHelper(object):
 
 def oauth_headers(url, consumer_key, token_key, token_secret, consumer_secret,
                   timestamp=None):
+
+    if oauth1 is None:
+        raise NotImplementedError('oauth support is not available')
+
     if timestamp:
         timestamp = str(timestamp)
     else:
diff --git a/tests/unittests/test_datasource/test_ec2.py b/tests/unittests/test_datasource/test_ec2.py
index 33d0261..7c9ac70 100644
--- a/tests/unittests/test_datasource/test_ec2.py
+++ b/tests/unittests/test_datasource/test_ec2.py
@@ -167,17 +167,36 @@ class TestEc2(test_helpers.HttprettyTestCase):
             self.datasource.min_metadata_version, 'meta-data', ''])
 
     @property
+    def metadata_urls(self):
+        return ['/'.join([self.metadata_addr, version, 'meta-data'])
+                for version in
+                [self.datasource.min_metadata_version] +
+                self.datasource.extended_metadata_versions]
+
+    @property
     def userdata_url(self):
         return '/'.join([
             self.metadata_addr,
             self.datasource.min_metadata_version, 'user-data'])
 
+    @property
+    def userdata_urls(self):
+        return ['/'.join([self.metadata_addr, version, 'user-data'])
+                for version in
+                [self.datasource.min_metadata_version] +
+                self.datasource.extended_metadata_versions]
+
     def _patch_add_cleanup(self, mpath, *args, **kwargs):
         p = mock.patch(mpath, *args, **kwargs)
         p.start()
         self.addCleanup(p.stop)
 
+    def bad_uri_handler(self, req, uri, headers):
+        print('Request for invalid URL:', uri)
+        return (999, headers, 'Invalid request for %s' % uri)
+
     def _setup_ds(self, sys_cfg, platform_data, md, ud=None):
+        self.uris = []
         distro = {}
         paths = helpers.Paths({})
         if sys_cfg is None:
@@ -189,8 +208,10 @@ class TestEc2(test_helpers.HttprettyTestCase):
                 return_value=platform_data)
 
         if md:
-            register_mock_metaserver(self.metadata_url, md)
-            register_mock_metaserver(self.userdata_url, ud)
+            for url in self.metadata_urls:
+                register_mock_metaserver(url, md)
+            for url in self.userdata_urls:
+                register_mock_metaserver(url, ud)
 
         return ds
 
@@ -268,6 +289,7 @@ class TestEc2(test_helpers.HttprettyTestCase):
         Then the metadata services is crawled for more network config info.
         When the platform data is valid, return True.
         """
+
         m_is_bsd.return_value = False
         m_dhcp.return_value = [{
             'interface': 'eth9', 'fixed-address': '192.168.2.9',
@@ -278,6 +300,7 @@ class TestEc2(test_helpers.HttprettyTestCase):
             platform_data=self.valid_platform_data,
             sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
             md=DEFAULT_METADATA)
+
         ret = ds.get_data()
         self.assertTrue(ret)
         m_dhcp.assert_called_once_with()