Merge lp:~edouardb/cloud-init/scaleway-datasource into lp:~cloud-init-dev/cloud-init/trunk

[edouardb]

Add Datasource for Scaleway's metadata service

[Julien Castets (jcastets)]

Unlike other providers, the Scaleway user-data API is restricted to privileged ports (< 1024) to prevent non-root users accessing to it.

We added a new parameter to readurl to specify the requests session object, to bind on a specific port.

[Scott Moser (smoser)]

Hey,

This looks well done, thanks.
A couple comments

a.) we'll need some unit tests to ensure we dont inadvertently break this.
b.) is there some way (anyway) we can detect if we're on scaleway? As it is right now, it looks like we're just going to block and retry for the availability of the MD. That is much less than ideal, and the only current "on by default" datasource thtat does that is EC2 (which only gets that privilege from being first). Other vendors provide some dmi data or another quick local test.
c.) you'll need to sign the Canonical Contributors License Agreement (http://www.ubuntu.com/legal/contributors)
d.) vendor-data would be nice (and helpful to you as the operator of the cloud.

again, though. Thanks, it looks really good.
Feel free to ping in #cloud-init if you have questions.

[lp:~edouardb/cloud-init/scaleway-datasource]

1152. By Edouard Bonlieu <email address hidden> on 2015-10-21

Add a check to ensure we are on Scaleway

1153. By Edouard Bonlieu <email address hidden> on 2015-10-21

Pass userdata url and retries as parameters

1154. By Edouard Bonlieu <email address hidden> on 2015-10-28

Merge jcastets PR

[Julien Castets (jcastets)]

Hi,

a) Done
b) Done. Unfortunately, there's no way to ensure you're running on Scaleway without hitting a network resource
c) Done
d) Indeed. Can we consider adding them later?

[Scott Moser (smoser)]

Julien,

Thanks.
for 'd', sure you can add vendor-data later.
if its not in the cloud provider anyway, not much use for cluod-init to support it.

The big issue is just 'b'. We can't enable by default without solid way of knowing that we should hit a http source that might hang indefinitely.

I'll revewi shortly.

[Julien Castets (jcastets)]

Great, thanks :) Waiting for your review then.

[Scott Moser (smoser)]

two nitpicks. but this looks good other than failing tests.
b

[Manfred Touron (moul)]

To check if you are on scaleway:

    $ test -f /run/oc-metadata.cache; echo $?
    0

This file is populated when something fetches the api metadata.

Our initrd (https://github.com/scaleway/initrd/tree/master/Linux) will create this file on each boots automatically.

[Scott Moser (smoser)]

Hi Eduardo,
Looking at this again.

Could you please sign the contributors agreement please feel free to contact me if you have any questions (freenode 'smoser') http://www.ubuntu.com/legal/contributors

Second, i think the check for /run/oc-metadata.cache is probably ok, as long as that is created early enough in boot (prior to cloud-init.service or upstart job running).

[Scott Moser (smoser)]

Hello,
Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html .

Unmerged revisions

1154. By Edouard Bonlieu <email address hidden> on 2015-10-28

Merge jcastets PR

1153. By Edouard Bonlieu <email address hidden> on 2015-10-21

Pass userdata url and retries as parameters

1152. By Edouard Bonlieu <email address hidden> on 2015-10-21

Add a check to ensure we are on Scaleway

1151. By Edouard Bonlieu <email address hidden> on 2015-10-15

Add Datasource for Scaleway's metadata service (https://www.scaleway.com)

1150. By Edouard Bonlieu <email address hidden> on 2015-10-15

Add optional session parameter to readurl

=== added file 'cloudinit/sources/DataSourceScaleway.py'
--- cloudinit/sources/DataSourceScaleway.py	1970-01-01 00:00:00 +0000
+++ cloudinit/sources/DataSourceScaleway.py	2015-10-28 09:51:17 +0000
@@ -0,0 +1,216 @@
+# vi: ts=4 expandtab
+#
+#    Author: Edouard Bonlieu <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import functools
+import errno
+import json
+import time
+
+from requests.packages.urllib3.poolmanager import PoolManager
+import requests
+
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper
+from cloudinit import util
+
+
+LOG = logging.getLogger(__name__)
+
+BUILTIN_DS_CONFIG = {
+    'metadata_url': 'http://169.254.42.42/conf?format=json',
+    'userdata_url': 'http://169.254.42.42/user_data/cloud-init'
+}
+
+DEF_MD_RETRIES = 5
+DEF_MD_TIMEOUT = 10
+
+
+def on_scaleway(user_data_url, retries=5):
+    """ Check if we are on Scaleway.
+
+    If Scaleway's user-data API isn't queried from a privileged source port
+    (ie. below 1024), it returns HTTP/403.
+    """
+    for _ in range(retries):
+        try:
+            code = requests.head(user_data_url).status_code
+            if code not in (403, 429) and code < 500:
+                return False
+            if code == 403:
+                return True
+        except (requests.exceptions.ConnectionError,
+                requests.exceptions.Timeout):
+            return False
+
+        time.sleep(1)  # be nice, and wait a bit before retrying
+    return False
+
+
+class SourceAddressAdapter(requests.adapters.HTTPAdapter):
+    """ Adapter for requests to choose the local address to bind to.
+    """
+
+    def __init__(self, source_address, **kwargs):
+        self.source_address = source_address
+        super(SourceAddressAdapter, self).__init__(**kwargs)
+
+    def init_poolmanager(self, connections, maxsize, block=False):
+        self.poolmanager = PoolManager(num_pools=connections,
+                                       maxsize=maxsize,
+                                       block=block,
+                                       source_address=self.source_address)
+
+
+def _get_user_data(userdata_address, timeout, retries, session):
+    """ Retrieve user data.
+
+    Scaleway userdata API returns HTTP/404 if user data is not set.
+
+    This function wraps `url_helper.readurl` but instead of considering
+    HTTP/404 as an error that requires a retry, it considers it as empty user
+    data.
+
+    Also, user data API require the source port to be below 1024. If requests
+    raises ConnectionError (EADDRINUSE), we raise immediately instead of
+    retrying. This way, the caller can retry to call this function on an other
+    port.
+    """
+    try:
+        # exception_cb is used to re-raise the exception if the API responds
+        # HTTP/404.
+        resp = url_helper.readurl(
+            userdata_address,
+            data=None,
+            timeout=timeout,
+            retries=retries,
+            session=session,
+            exception_cb=lambda _, exc: exc.code == 404 or isinstance(
+                exc.cause, requests.exceptions.ConnectionError
+            )
+        )
+        return util.decode_binary(resp.contents)
+    except url_helper.UrlError as exc:
+        # Empty user data
+        if exc.code == 404:
+            return None
+
+        # `retries` is reached, re-raise
+        raise
+
+
+class DataSourceScaleway(sources.DataSource):
+
+    def __init__(self, sys_cfg, distro, paths):
+        LOG.debug('Init scw')
+        sources.DataSource.__init__(self, sys_cfg, distro, paths)
+
+        self.metadata = {}
+        self.ds_cfg = util.mergemanydict([
+            util.get_cfg_by_path(sys_cfg, ["datasource", "Scaleway"], {}),
+            BUILTIN_DS_CONFIG
+        ])
+
+        self.metadata_address = self.ds_cfg['metadata_url']
+        self.userdata_address = self.ds_cfg['userdata_url']
+
+        self.retries = self.ds_cfg.get('retries', DEF_MD_RETRIES)
+        self.timeout = self.ds_cfg.get('timeout', DEF_MD_TIMEOUT)
+
+    def _get_metadata(self):
+        resp = url_helper.readurl(
+            self.metadata_address,
+            timeout=self.timeout,
+            retries=self.retries
+        )
+        metadata = json.loads(util.decode_binary(resp.contents))
+        LOG.debug('metadata downloaded')
+
+        # try to make a request on the first privileged port available
+        for port in range(1, 1024):
+            try:
+                LOG.debug(
+                    'Trying to get user data (bind on port %d)...' % port
+                )
+                session = requests.Session()
+                session.mount(
+                    'http://',
+                    SourceAddressAdapter(source_address=('0.0.0.0', port))
+                )
+                user_data = _get_user_data(
+                    self.userdata_address,
+                    timeout=self.timeout,
+                    retries=self.retries,
+                    session=session
+                )
+                LOG.debug('user-data downloaded')
+                return metadata, user_data
+
+            except url_helper.UrlError as exc:
+                # local port already in use, try next port
+                if isinstance(exc.cause, requests.exceptions.ConnectionError):
+                    continue
+
+                # unexpected exception
+                raise
+
+    def get_data(self):
+        if on_scaleway(self.ds_cfg['userdata_url'], self.retries) is False:
+            return False
+
+        metadata, metadata['user-data'] = self._get_metadata()
+        self.metadata = {
+            'id': metadata['id'],
+            'hostname': metadata['hostname'],
+            'user-data': metadata['user-data'],
+            'ssh_public_keys': [
+                key['key'] for key in metadata['ssh_public_keys']
+            ]
+        }
+        return True
+
+    @property
+    def launch_index(self):
+        return None
+
+    def get_instance_id(self):
+        return self.metadata['id']
+
+    def get_public_ssh_keys(self):
+        return self.metadata['ssh_public_keys']
+
+    def get_hostname(self, fqdn=False, resolve_ip=False):
+        return self.metadata['hostname']
+
+    def get_userdata_raw(self):
+        return self.metadata['user-data']
+
+    @property
+    def availability_zone(self):
+        return None
+
+    @property
+    def region(self):
+        return None
+
+
+datasources = [
+    (DataSourceScaleway, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)

=== modified file 'cloudinit/url_helper.py'
--- cloudinit/url_helper.py	2015-09-29 21:17:49 +0000
+++ cloudinit/url_helper.py	2015-10-28 09:51:17 +0000
@@ -183,7 +183,8 @@
 
 def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
             headers=None, headers_cb=None, ssl_details=None,
-            check_status=True, allow_redirects=True, exception_cb=None):
+            check_status=True, allow_redirects=True, exception_cb=None,
+            session=None):
     url = _cleanurl(url)
     req_args = {
         'url': url,
@@ -242,7 +243,9 @@
             LOG.debug("[%s/%s] open '%s' with %s configuration", i,
                       manual_tries, url, filtered_req_args)
 
-            r = requests.request(**req_args)
+            if session is None:
+                session = requests.Session()
+            r = session.request(**req_args)
             if check_status:
                 r.raise_for_status()
             LOG.debug("Read from %s (%s, %sb) after %s attempts", url,

=== added file 'tests/unittests/test_datasource/test_scaleway.py'
--- tests/unittests/test_datasource/test_scaleway.py	1970-01-01 00:00:00 +0000
+++ tests/unittests/test_datasource/test_scaleway.py	2015-10-28 09:51:17 +0000
@@ -0,0 +1,193 @@
+#
+#    Copyright (C) 2015 Julien Castets
+#
+#    Author: Julien Castets <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+import requests
+
+from cloudinit import settings
+from cloudinit import helpers
+from cloudinit.sources import DataSourceScaleway
+
+from .. import helpers as test_helpers
+
+
+httpretty = test_helpers.import_httpretty()
+
+
+class UserDataResponses(object):
+    """ Possible responses of the API endpoint
+    169.254.42.42/user_data/cloud-init.
+
+    HEAD requests are made to check if the server is on Scaleway.
+    GET requests are made to get user data.
+    """
+
+    FAKE_USER_DATA = '#!/bin/bash\necho "user-data"'
+
+    @staticmethod
+    def head_ok(method, uri, headers):
+        """ To ensure it's running on Scaleway, the datasource makes a HEAD
+        request to 169.254.42.42/user_data/cloud-init and expects a HTTP/403
+        response, because this endpoint needs to be queried with a privileged
+        source port (below 2014).
+        """
+        return 403, headers, ''
+
+    @staticmethod
+    def connection_error(method, uri, headers):
+        """ Unable to connect to the user data API.
+        """
+        raise requests.exceptions.ConnectionError()
+
+    @staticmethod
+    def rate_limited(method, uri, headers):
+        return 429, headers, ''
+
+    @staticmethod
+    def api_error(method, uri, headers):
+        return 500, headers, ''
+
+    @classmethod
+    def get_ok(cls, method, uri, headers):
+        return 200, headers, cls.FAKE_USER_DATA
+
+    @staticmethod
+    def empty(method, uri, headers):
+        """ No user data for this server.
+        """
+        return 404, headers, ''
+
+
+class MetadataResponses(object):
+    """ Possible responses of the metadata API.
+    """
+
+    FAKE_METADATA = {
+        'id': '00000000-0000-0000-0000-000000000000',
+        'hostname': 'scaleway.host',
+        'ssh_public_keys': [{
+            'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA',
+            'fingerprint': '2048 06:ae:...  login (RSA)'
+        }, {
+            'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABCCCCC',
+            'fingerprint': '2048 06:ff:...  login2 (RSA)'
+        }]
+    }
+
+    @classmethod
+    def get_ok(cls, method, uri, headers):
+        return 200, headers, json.dumps(cls.FAKE_METADATA)
+
+
+class TestDataSourceScaleway(test_helpers.HttprettyTestCase):
+
+    def setUp(self):
+        self.datasource = DataSourceScaleway.DataSourceScaleway(
+            settings.CFG_BUILTIN, None, helpers.Paths({})
+        )
+        super(TestDataSourceScaleway, self).setUp()
+
+        self.metadata_url = \
+            DataSourceScaleway.BUILTIN_DS_CONFIG['metadata_url']
+        self.userdata_url = \
+            DataSourceScaleway.BUILTIN_DS_CONFIG['userdata_url']
+
+    @httpretty.activate
+    @test_helpers.mock.patch('time.sleep', return_value=None)
+    def test_on_scaleway(self, sleep):
+        # Test API ok
+        httpretty.register_uri(httpretty.HEAD, self.userdata_url,
+                               body=UserDataResponses.head_ok)
+        self.assertTrue(DataSourceScaleway.on_scaleway(self.userdata_url))
+
+        # API returns something else than 403: we're not on scaleway
+        httpretty.register_uri(httpretty.HEAD, self.userdata_url,
+                               body='ok')
+        self.assertFalse(DataSourceScaleway.on_scaleway(self.userdata_url))
+
+        # Connection error
+        httpretty.register_uri(httpretty.HEAD, self.userdata_url,
+                               body=UserDataResponses.connection_error)
+        self.assertFalse(DataSourceScaleway.on_scaleway(self.userdata_url))
+
+        # Rate limited 2 times, then API error, then ok
+        httpretty.register_uri(
+            httpretty.HEAD, self.userdata_url,
+            responses=[
+                httpretty.Response(body=UserDataResponses.rate_limited),
+                httpretty.Response(body=UserDataResponses.rate_limited),
+                httpretty.Response(body=UserDataResponses.api_error),
+                httpretty.Response(body=UserDataResponses.head_ok),
+            ]
+        )
+        self.assertTrue(DataSourceScaleway.on_scaleway(self.userdata_url))
+        self.assertEqual(sleep.call_count, 3)
+
+    @httpretty.activate
+    @test_helpers.mock.patch('time.sleep', return_value=None)
+    def test_metadata(self, sleep):
+        # Not on scaleway
+        httpretty.register_uri(httpretty.HEAD, self.userdata_url,
+                               body=UserDataResponses.connection_error)
+        self.assertFalse(self.datasource.get_data())
+
+        # Make on_scaleway return true
+        httpretty.register_uri(httpretty.HEAD, self.userdata_url,
+                               body=UserDataResponses.head_ok)
+
+        # Make user data API return a valid response
+        httpretty.register_uri(httpretty.GET, self.metadata_url,
+                               body=MetadataResponses.get_ok)
+        httpretty.register_uri(httpretty.GET, self.userdata_url,
+                               body=UserDataResponses.get_ok)
+        self.datasource.get_data()
+
+        self.assertEqual(self.datasource.get_instance_id(),
+                         MetadataResponses.FAKE_METADATA['id'])
+        self.assertEqual(self.datasource.get_public_ssh_keys(), [
+            elem['key'] for elem in
+            MetadataResponses.FAKE_METADATA['ssh_public_keys']
+        ])
+        self.assertEqual(self.datasource.get_hostname(),
+                         MetadataResponses.FAKE_METADATA['hostname'])
+        self.assertEqual(self.datasource.get_userdata_raw(),
+                         UserDataResponses.FAKE_USER_DATA)
+        self.assertIsNone(self.datasource.availability_zone)
+        self.assertIsNone(self.datasource.region)
+
+        # Make user data API return HTTP/404, which means there is no user data
+        # for the server.
+        httpretty.register_uri(httpretty.GET, self.userdata_url,
+                               body=UserDataResponses.empty)
+        self.datasource.get_data()
+        self.assertIsNone(self.datasource.get_userdata_raw())
+
+        # Make user data API rate limit 2 times, then ConnectionError (ie.
+        # local port is used), then API ok
+        httpretty.register_uri(
+            httpretty.GET, self.userdata_url,
+            responses=[
+                httpretty.Response(body=UserDataResponses.rate_limited),
+                httpretty.Response(body=UserDataResponses.rate_limited),
+                httpretty.Response(body=UserDataResponses.connection_error),
+                httpretty.Response(body=UserDataResponses.get_ok),
+            ]
+        )
+        self.datasource.get_data()
+        self.assertEqual(self.datasource.get_userdata_raw(),
+                         UserDataResponses.FAKE_USER_DATA)