cloud-init

Merge ~jcastets/cloud-init:scaleway-datasource into cloud-init:master

  1. Git
  2. lp:~jcastets/cloud-init
  3. scaleway-datasource
  4. Merge into master
Proposed by Julien Castets
Status: Merged
Approved by: Scott Moser
Approved revision: f61323fb96a58688be73566eaa30c8f7c3b3adc6
Merged at revision: e80517ae6aea49c9ab3bd622a33fee44014f485f
Proposed branch: ~jcastets/cloud-init:scaleway-datasource
Merge into: cloud-init:master
Diff against target: 561 lines (+510/-3)
4 files modified
cloudinit/sources/DataSourceScaleway.py (+223/-0)
cloudinit/url_helper.py (+8/-2)
tests/unittests/test_datasource/test_scaleway.py (+262/-0)
tools/ds-identify (+17/-1)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Scott Moser Approve
Server Team CI bot continuous-integration Approve
Review via email: mp+325740@code.launchpad.net

Description of the change

Implements Scaleway datasource with user and vendor data.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:adfca12743e2dd78ac1ad7f83253f41c06eb3795
https://jenkins.ubuntu.com/server/job/cloud-init-ci/536/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/536
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/536
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/536
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/536
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/536

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/536/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

Some things, and some content inline:
 * We'll need some unit tests for this. Otherwise it is at increased risk of being inadvertently broken.
 * We will need to add knowledge of the datasource to tools/ds-identify (without that your datasource will only ever be considered if it is a single entry in the configured list)
 * we really, *REALLY* want a positive non-network identification. Without such a thing, we can't enable the datasource by default, meaning Ubuntu or other images that would work elsewhere wont work on your platform.

Also, give a nicer commit message:
  Summary
  <blank line>
  More information
  ...
  <blank line>
  LP: #XXXXXXX

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:a0748674f74e97e70a3274d312c9592b28a6dc08
https://jenkins.ubuntu.com/server/job/cloud-init-ci/537/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/537
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/537
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/537
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/537
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/537

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/537/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

over all nice.
some comments.

thank you, Julien.

Revision history for this message
Scott Moser (smoser) wrote :

I'm going to move this to 'work in progress'
Please address the comments / tests and move it back to 'Needs Review'.

review: Needs Fixing
Revision history for this message
Julien Castets (jcastets) wrote :

Everything should be fixed.

* I updated the header of cloudinit/sources/DatasourceScaleway. Is it what you expect?
* on_scaleway doesn't rely on network anymore. It checks if "scaleway" is in /var/run/scaleway, in the commandline, or in DMI data.
* Made some unittests
* requests.requests creates a requests.Session object and calls session.request: https://github.com/requests/requests/blob/master/requests/api.py#L57 ; so my change should be backward compatible

If you need any precision or change, feel free to ask.

Since the datasource no longer makes network requests, is there a chance to enable it by default?

Thanks,

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:2d3032a34fd109b89918bc7ddd1cf1611e1419c6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/2/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/2/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/2/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/2/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/2/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/2/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/2/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:174de3110c0a48a494091c5b29703898b5eefeed
https://jenkins.ubuntu.com/server/job/cloud-init-ci/3/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/3/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/3/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/3/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/3/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/3/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/3/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:edea9c27ba59e631fecb64262fb1c7db97cd9c07
https://jenkins.ubuntu.com/server/job/cloud-init-ci/4/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/4/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/4/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/4/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/4/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/4/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/4/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:44618aa9204fd20ec1cf4765b98c42f9849f8df4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/5/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/5/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/5/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/5/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/5/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:e53e67513c61fbfc79c9a80d7f2619825282676f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/6/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/6
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/6
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/6
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/6
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/6

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/6/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

Some comments inline. mostly questions.
Would it be easier for you to use requests directly rather than going through urlhelper ?

~jcastets/cloud-init:scaleway-datasource updated
c8201d5... by Julien Castets

Scaleway: fix typo

55eb390... by Julien Castets

Scaleway: add logging

c28b9e4... by Julien Castets

Scaleway: fix header copyright header

f52c323... by Julien Castets

Scaleway: remove inline pylint bypass

80be5ac... by Julien Castets

Scaleway: fix docstring format

3fddcf0... by Julien Castets

Scaleway: assert sleep is called in tests

Revision history for this message
Julien Castets (jcastets) wrote :

> Some comments inline. mostly questions.

Everything should be fixed. I can rebase my commits into one if you want me to.

> Would it be easier for you to use requests directly rather than going through
> urlhelper ?

I'd prefer not to. url_helper is doing some logging, sets the user-agent, gracefully handles SSL errors... even if it seems hackish, using url_helper.readurl is IMO the best way to do.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:3fddcf07e447cdc87abbb9a3a13744de17d889c0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/rebuild

review: Approve (continuous-integration)
~jcastets/cloud-init:scaleway-datasource updated
f61323f... by Julien Castets

Scaleway: split unittests

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:f61323fb96a58688be73566eaa30c8f7c3b3adc6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/40/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/40/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

I approve of this at this point.
The use of urllib3 through requests could be a problem in the future, but
should not cause any regression potential here as this is a new
datasource.

I plan on adding the additional changes at
 http://paste.ubuntu.com/25114217/

Just for our future reference / context.

Revision history for this message
Scott Moser (smoser) wrote :

chad does this MP pluls above patch look ok to you ?

review: Approve
Revision history for this message
Chad Smith (chad.smith) wrote :

Unit test decomposition looks good; thank you for that. I'd avoid wrapper functions in the future like install_mocks just so we can see mocked return_values are seen local to the unit test instead of having to look up at another method defintion to find out what it really is set to. I get that in this case it gives and easy eye in the 3 unit tests toward reading the True,False flags for whether the mock is 'active'.

  Per the content of the merge proposal, the altered readurl in cloudinit.url_helper with the explicit session context manager doesn't adversely affect us on existing deployments +1 there. I also agree that any potential maintenance of the Scaleway datasource importing request.packages.urllib3 on other distributions will probably be minimal/infrequent, and if there is adverse impact, it will be limited to Scaleway cloud-init users.

Approved with Scott's comments for context and s/priviledged/privileged/.

review: Approve
Revision history for this message
Julien Castets (jcastets) wrote :

Thanks a lot for your help :)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/sources/DataSourceScaleway.py b/cloudinit/sources/DataSourceScaleway.py
2new file mode 100644
3index 0000000..93b4be9
4--- /dev/null
5+++ b/cloudinit/sources/DataSourceScaleway.py
6@@ -0,0 +1,223 @@
7+# Author: Julien Castets <castets.j@gmail.com>
8+#
9+# This file is part of cloud-init. See LICENSE file for license information.
10+
11+# Scaleway API:
12+# https://developer.scaleway.com/#metadata
13+
14+import json
15+import os
16+import socket
17+import time
18+
19+import requests
20+
21+# pylint fails to import the two modules below.
22+# pylint: disable=E0401
23+from requests.packages.urllib3.connection import HTTPConnection
24+from requests.packages.urllib3.poolmanager import PoolManager
25+
26+from cloudinit import log as logging
27+from cloudinit import sources
28+from cloudinit import url_helper
29+from cloudinit import util
30+
31+
32+LOG = logging.getLogger(__name__)
33+
34+DS_BASE_URL = 'http://169.254.42.42'
35+
36+BUILTIN_DS_CONFIG = {
37+ 'metadata_url': DS_BASE_URL + '/conf?format=json',
38+ 'userdata_url': DS_BASE_URL + '/user_data/cloud-init',
39+ 'vendordata_url': DS_BASE_URL + '/vendor_data/cloud-init'
40+}
41+
42+DEF_MD_RETRIES = 5
43+DEF_MD_TIMEOUT = 10
44+
45+
46+def on_scaleway():
47+ """
48+ There are three ways to detect if you are on Scaleway:
49+
50+ * check DMI data: not yet implemented by Scaleway, but the check is made to
51+ be future-proof.
52+ * the initrd created the file /var/run/scaleway.
53+ * "scaleway" is in the kernel cmdline.
54+ """
55+ vendor_name = util.read_dmi_data('system-manufacturer')
56+ if vendor_name == 'Scaleway':
57+ return True
58+
59+ if os.path.exists('/var/run/scaleway'):
60+ return True
61+
62+ cmdline = util.get_cmdline()
63+ if 'scaleway' in cmdline:
64+ return True
65+
66+ return False
67+
68+
69+class SourceAddressAdapter(requests.adapters.HTTPAdapter):
70+ """
71+ Adapter for requests to choose the local address to bind to.
72+ """
73+ def __init__(self, source_address, **kwargs):
74+ self.source_address = source_address
75+ super(SourceAddressAdapter, self).__init__(**kwargs)
76+
77+ def init_poolmanager(self, connections, maxsize, block=False):
78+ socket_options = HTTPConnection.default_socket_options + [
79+ (socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
80+ ]
81+ self.poolmanager = PoolManager(num_pools=connections,
82+ maxsize=maxsize,
83+ block=block,
84+ source_address=self.source_address,
85+ socket_options=socket_options)
86+
87+
88+def query_data_api_once(api_address, timeout, requests_session):
89+ """
90+ Retrieve user data or vendor data.
91+
92+ Scaleway user/vendor data API returns HTTP/404 if user/vendor data is not
93+ set.
94+
95+ This function calls `url_helper.readurl` but instead of considering
96+ HTTP/404 as an error that requires a retry, it considers it as empty
97+ user/vendor data.
98+
99+ Also, be aware the user data/vendor API requires the source port to be
100+ below 1024 to ensure the client is root (since non-root users can't bind
101+ ports below 1024). If requests raises ConnectionError (EADDRINUSE), the
102+ caller should retry to call this function on an other port.
103+ """
104+ try:
105+ resp = url_helper.readurl(
106+ api_address,
107+ data=None,
108+ timeout=timeout,
109+ # It's the caller's responsability to recall this function in case
110+ # of exception. Don't let url_helper.readurl() retry by itself.
111+ retries=0,
112+ session=requests_session,
113+ # If the error is a HTTP/404 or a ConnectionError, go into raise
114+ # block below.
115+ exception_cb=lambda _, exc: exc.code == 404 or (
116+ isinstance(exc.cause, requests.exceptions.ConnectionError)
117+ )
118+ )
119+ return util.decode_binary(resp.contents)
120+ except url_helper.UrlError as exc:
121+ # Empty user data.
122+ if exc.code == 404:
123+ return None
124+ raise
125+
126+
127+def query_data_api(api_type, api_address, retries, timeout):
128+ """
129+ Get user or vendor data.
130+
131+ Handle the retrying logic in case the source port is used.
132+ """
133+ # Query user/vendor data. Try to make a request on the first privileged
134+ # port available.
135+ for port in range(1, max(retries, 2)):
136+ try:
137+ LOG.debug(
138+ 'Trying to get %s data (bind on port %d)...',
139+ api_type, port
140+ )
141+ requests_session = requests.Session()
142+ requests_session.mount(
143+ 'http://',
144+ SourceAddressAdapter(source_address=('0.0.0.0', port))
145+ )
146+ data = query_data_api_once(
147+ api_address,
148+ timeout=timeout,
149+ requests_session=requests_session
150+ )
151+ LOG.debug('%s-data downloaded', api_type)
152+ return data
153+
154+ except url_helper.UrlError as exc:
155+ # Local port already in use or HTTP/429.
156+ LOG.warning('Error while trying to get %s data: %s', api_type, exc)
157+ time.sleep(5)
158+ last_exc = exc
159+ continue
160+
161+ # Max number of retries reached.
162+ raise last_exc
163+
164+
165+class DataSourceScaleway(sources.DataSource):
166+
167+ def __init__(self, sys_cfg, distro, paths):
168+ super(DataSourceScaleway, self).__init__(sys_cfg, distro, paths)
169+
170+ self.ds_cfg = util.mergemanydict([
171+ util.get_cfg_by_path(sys_cfg, ["datasource", "Scaleway"], {}),
172+ BUILTIN_DS_CONFIG
173+ ])
174+
175+ self.metadata_address = self.ds_cfg['metadata_url']
176+ self.userdata_address = self.ds_cfg['userdata_url']
177+ self.vendordata_address = self.ds_cfg['vendordata_url']
178+
179+ self.retries = int(self.ds_cfg.get('retries', DEF_MD_RETRIES))
180+ self.timeout = int(self.ds_cfg.get('timeout', DEF_MD_TIMEOUT))
181+
182+ def get_data(self):
183+ if not on_scaleway():
184+ return False
185+
186+ resp = url_helper.readurl(self.metadata_address,
187+ timeout=self.timeout,
188+ retries=self.retries)
189+ self.metadata = json.loads(util.decode_binary(resp.contents))
190+
191+ self.userdata_raw = query_data_api(
192+ 'user-data', self.userdata_address,
193+ self.retries, self.timeout
194+ )
195+ self.vendordata_raw = query_data_api(
196+ 'vendor-data', self.vendordata_address,
197+ self.retries, self.timeout
198+ )
199+ return True
200+
201+ @property
202+ def launch_index(self):
203+ return None
204+
205+ def get_instance_id(self):
206+ return self.metadata['id']
207+
208+ def get_public_ssh_keys(self):
209+ return [key['key'] for key in self.metadata['ssh_public_keys']]
210+
211+ def get_hostname(self, fqdn=False, resolve_ip=False):
212+ return self.metadata['hostname']
213+
214+ @property
215+ def availability_zone(self):
216+ return None
217+
218+ @property
219+ def region(self):
220+ return None
221+
222+
223+datasources = [
224+ (DataSourceScaleway, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
225+]
226+
227+
228+def get_datasource_list(depends):
229+ return sources.list_from_depends(depends, datasources)
230diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
231index d2b92e6..7cf76aa 100644
232--- a/cloudinit/url_helper.py
233+++ b/cloudinit/url_helper.py
234@@ -172,7 +172,8 @@ def _get_ssl_args(url, ssl_details):
235
236 def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
237 headers=None, headers_cb=None, ssl_details=None,
238- check_status=True, allow_redirects=True, exception_cb=None):
239+ check_status=True, allow_redirects=True, exception_cb=None,
240+ session=None):
241 url = _cleanurl(url)
242 req_args = {
243 'url': url,
244@@ -231,7 +232,12 @@ def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
245 LOG.debug("[%s/%s] open '%s' with %s configuration", i,
246 manual_tries, url, filtered_req_args)
247
248- r = requests.request(**req_args)
249+ if session is None:
250+ session = requests.Session()
251+
252+ with session as sess:
253+ r = sess.request(**req_args)
254+
255 if check_status:
256 r.raise_for_status()
257 LOG.debug("Read from %s (%s, %sb) after %s attempts", url,
258diff --git a/tests/unittests/test_datasource/test_scaleway.py b/tests/unittests/test_datasource/test_scaleway.py
259new file mode 100644
260index 0000000..65d83ad
261--- /dev/null
262+++ b/tests/unittests/test_datasource/test_scaleway.py
263@@ -0,0 +1,262 @@
264+# This file is part of cloud-init. See LICENSE file for license information.
265+
266+import json
267+
268+import httpretty
269+import requests
270+
271+from cloudinit import helpers
272+from cloudinit import settings
273+from cloudinit.sources import DataSourceScaleway
274+
275+from ..helpers import mock, HttprettyTestCase, TestCase
276+
277+
278+class DataResponses(object):
279+ """
280+ Possible responses of the API endpoint
281+ 169.254.42.42/user_data/cloud-init and
282+ 169.254.42.42/vendor_data/cloud-init.
283+ """
284+
285+ FAKE_USER_DATA = '#!/bin/bash\necho "user-data"'
286+
287+ @staticmethod
288+ def rate_limited(method, uri, headers):
289+ return 429, headers, ''
290+
291+ @staticmethod
292+ def api_error(method, uri, headers):
293+ return 500, headers, ''
294+
295+ @classmethod
296+ def get_ok(cls, method, uri, headers):
297+ return 200, headers, cls.FAKE_USER_DATA
298+
299+ @staticmethod
300+ def empty(method, uri, headers):
301+ """
302+ No user data for this server.
303+ """
304+ return 404, headers, ''
305+
306+
307+class MetadataResponses(object):
308+ """
309+ Possible responses of the metadata API.
310+ """
311+
312+ FAKE_METADATA = {
313+ 'id': '00000000-0000-0000-0000-000000000000',
314+ 'hostname': 'scaleway.host',
315+ 'ssh_public_keys': [{
316+ 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA',
317+ 'fingerprint': '2048 06:ae:... login (RSA)'
318+ }, {
319+ 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABCCCCC',
320+ 'fingerprint': '2048 06:ff:... login2 (RSA)'
321+ }]
322+ }
323+
324+ @classmethod
325+ def get_ok(cls, method, uri, headers):
326+ return 200, headers, json.dumps(cls.FAKE_METADATA)
327+
328+
329+class TestOnScaleway(TestCase):
330+
331+ def install_mocks(self, fake_dmi, fake_file_exists, fake_cmdline):
332+ mock, faked = fake_dmi
333+ mock.return_value = 'Scaleway' if faked else 'Whatever'
334+
335+ mock, faked = fake_file_exists
336+ mock.return_value = faked
337+
338+ mock, faked = fake_cmdline
339+ mock.return_value = \
340+ 'initrd=initrd showopts scaleway nousb' if faked \
341+ else 'BOOT_IMAGE=/vmlinuz-3.11.0-26-generic'
342+
343+ @mock.patch('cloudinit.util.get_cmdline')
344+ @mock.patch('os.path.exists')
345+ @mock.patch('cloudinit.util.read_dmi_data')
346+ def test_not_on_scaleway(self, m_read_dmi_data, m_file_exists,
347+ m_get_cmdline):
348+ self.install_mocks(
349+ fake_dmi=(m_read_dmi_data, False),
350+ fake_file_exists=(m_file_exists, False),
351+ fake_cmdline=(m_get_cmdline, False)
352+ )
353+ self.assertFalse(DataSourceScaleway.on_scaleway())
354+
355+ # When not on Scaleway, get_data() returns False.
356+ datasource = DataSourceScaleway.DataSourceScaleway(
357+ settings.CFG_BUILTIN, None, helpers.Paths({})
358+ )
359+ self.assertFalse(datasource.get_data())
360+
361+ @mock.patch('cloudinit.util.get_cmdline')
362+ @mock.patch('os.path.exists')
363+ @mock.patch('cloudinit.util.read_dmi_data')
364+ def test_on_scaleway_dmi(self, m_read_dmi_data, m_file_exists,
365+ m_get_cmdline):
366+ """
367+ dmidecode returns "Scaleway".
368+ """
369+ # dmidecode returns "Scaleway"
370+ self.install_mocks(
371+ fake_dmi=(m_read_dmi_data, True),
372+ fake_file_exists=(m_file_exists, False),
373+ fake_cmdline=(m_get_cmdline, False)
374+ )
375+ self.assertTrue(DataSourceScaleway.on_scaleway())
376+
377+ @mock.patch('cloudinit.util.get_cmdline')
378+ @mock.patch('os.path.exists')
379+ @mock.patch('cloudinit.util.read_dmi_data')
380+ def test_on_scaleway_var_run_scaleway(self, m_read_dmi_data, m_file_exists,
381+ m_get_cmdline):
382+ """
383+ /var/run/scaleway exists.
384+ """
385+ self.install_mocks(
386+ fake_dmi=(m_read_dmi_data, False),
387+ fake_file_exists=(m_file_exists, True),
388+ fake_cmdline=(m_get_cmdline, False)
389+ )
390+ self.assertTrue(DataSourceScaleway.on_scaleway())
391+
392+ @mock.patch('cloudinit.util.get_cmdline')
393+ @mock.patch('os.path.exists')
394+ @mock.patch('cloudinit.util.read_dmi_data')
395+ def test_on_scaleway_cmdline(self, m_read_dmi_data, m_file_exists,
396+ m_get_cmdline):
397+ """
398+ "scaleway" in /proc/cmdline.
399+ """
400+ self.install_mocks(
401+ fake_dmi=(m_read_dmi_data, False),
402+ fake_file_exists=(m_file_exists, False),
403+ fake_cmdline=(m_get_cmdline, True)
404+ )
405+ self.assertTrue(DataSourceScaleway.on_scaleway())
406+
407+
408+def get_source_address_adapter(*args, **kwargs):
409+ """
410+ Scaleway user/vendor data API requires to be called with a privileged port.
411+
412+ If the unittests are run as non-root, the user doesn't have the permission
413+ to bind on ports below 1024.
414+
415+ This function removes the bind on a privileged address, since anyway the
416+ HTTP call is mocked by httpretty.
417+ """
418+ kwargs.pop('source_address')
419+ return requests.adapters.HTTPAdapter(*args, **kwargs)
420+
421+
422+class TestDataSourceScaleway(HttprettyTestCase):
423+
424+ def setUp(self):
425+ self.datasource = DataSourceScaleway.DataSourceScaleway(
426+ settings.CFG_BUILTIN, None, helpers.Paths({})
427+ )
428+ super(TestDataSourceScaleway, self).setUp()
429+
430+ self.metadata_url = \
431+ DataSourceScaleway.BUILTIN_DS_CONFIG['metadata_url']
432+ self.userdata_url = \
433+ DataSourceScaleway.BUILTIN_DS_CONFIG['userdata_url']
434+ self.vendordata_url = \
435+ DataSourceScaleway.BUILTIN_DS_CONFIG['vendordata_url']
436+
437+ @httpretty.activate
438+ @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
439+ get_source_address_adapter)
440+ @mock.patch('cloudinit.util.get_cmdline')
441+ @mock.patch('time.sleep', return_value=None)
442+ def test_metadata_ok(self, sleep, m_get_cmdline):
443+ """
444+ get_data() returns metadata, user data and vendor data.
445+ """
446+ m_get_cmdline.return_value = 'scaleway'
447+
448+ # Make user data API return a valid response
449+ httpretty.register_uri(httpretty.GET, self.metadata_url,
450+ body=MetadataResponses.get_ok)
451+ httpretty.register_uri(httpretty.GET, self.userdata_url,
452+ body=DataResponses.get_ok)
453+ httpretty.register_uri(httpretty.GET, self.vendordata_url,
454+ body=DataResponses.get_ok)
455+ self.datasource.get_data()
456+
457+ self.assertEqual(self.datasource.get_instance_id(),
458+ MetadataResponses.FAKE_METADATA['id'])
459+ self.assertEqual(self.datasource.get_public_ssh_keys(), [
460+ elem['key'] for elem in
461+ MetadataResponses.FAKE_METADATA['ssh_public_keys']
462+ ])
463+ self.assertEqual(self.datasource.get_hostname(),
464+ MetadataResponses.FAKE_METADATA['hostname'])
465+ self.assertEqual(self.datasource.get_userdata_raw(),
466+ DataResponses.FAKE_USER_DATA)
467+ self.assertEqual(self.datasource.get_vendordata_raw(),
468+ DataResponses.FAKE_USER_DATA)
469+ self.assertIsNone(self.datasource.availability_zone)
470+ self.assertIsNone(self.datasource.region)
471+ self.assertEqual(sleep.call_count, 0)
472+
473+ @httpretty.activate
474+ @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
475+ get_source_address_adapter)
476+ @mock.patch('cloudinit.util.get_cmdline')
477+ @mock.patch('time.sleep', return_value=None)
478+ def test_metadata_404(self, sleep, m_get_cmdline):
479+ """
480+ get_data() returns metadata, but no user data nor vendor data.
481+ """
482+ m_get_cmdline.return_value = 'scaleway'
483+
484+ # Make user and vendor data APIs return HTTP/404, which means there is
485+ # no user / vendor data for the server.
486+ httpretty.register_uri(httpretty.GET, self.metadata_url,
487+ body=MetadataResponses.get_ok)
488+ httpretty.register_uri(httpretty.GET, self.userdata_url,
489+ body=DataResponses.empty)
490+ httpretty.register_uri(httpretty.GET, self.vendordata_url,
491+ body=DataResponses.empty)
492+ self.datasource.get_data()
493+ self.assertIsNone(self.datasource.get_userdata_raw())
494+ self.assertIsNone(self.datasource.get_vendordata_raw())
495+ self.assertEqual(sleep.call_count, 0)
496+
497+ @httpretty.activate
498+ @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
499+ get_source_address_adapter)
500+ @mock.patch('cloudinit.util.get_cmdline')
501+ @mock.patch('time.sleep', return_value=None)
502+ def test_metadata_rate_limit(self, sleep, m_get_cmdline):
503+ """
504+ get_data() is rate limited two times by the metadata API when fetching
505+ user data.
506+ """
507+ m_get_cmdline.return_value = 'scaleway'
508+
509+ httpretty.register_uri(httpretty.GET, self.metadata_url,
510+ body=MetadataResponses.get_ok)
511+ httpretty.register_uri(httpretty.GET, self.vendordata_url,
512+ body=DataResponses.empty)
513+
514+ httpretty.register_uri(
515+ httpretty.GET, self.userdata_url,
516+ responses=[
517+ httpretty.Response(body=DataResponses.rate_limited),
518+ httpretty.Response(body=DataResponses.rate_limited),
519+ httpretty.Response(body=DataResponses.get_ok),
520+ ]
521+ )
522+ self.datasource.get_data()
523+ self.assertEqual(self.datasource.get_userdata_raw(),
524+ DataResponses.FAKE_USER_DATA)
525+ self.assertEqual(sleep.call_count, 2)
526diff --git a/tools/ds-identify b/tools/ds-identify
527index 7c8b144..33bd299 100755
528--- a/tools/ds-identify
529+++ b/tools/ds-identify
530@@ -112,7 +112,7 @@ DI_DSNAME=""
531 # be searched if there is no setting found in config.
532 DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
533 CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
534-OVF SmartOS"
535+OVF SmartOS Scaleway"
536 DI_DSLIST=""
537 DI_MODE=""
538 DI_ON_FOUND=""
539@@ -896,6 +896,22 @@ dscheck_None() {
540 return ${DS_NOT_FOUND}
541 }
542
543+dscheck_Scaleway() {
544+ if [ "${DI_DMI_SYS_VENDOR}" = "Scaleway" ]; then
545+ return $DS_FOUND
546+ fi
547+
548+ case " ${DI_KERNEL_CMDLINE} " in
549+ *\ scaleway\ *) return ${DS_FOUND};;
550+ esac
551+
552+ if [ -f ${PATH_ROOT}/var/run/scaleway ]; then
553+ return ${DS_FOUND}
554+ fi
555+
556+ return ${DS_NOT_FOUND}
557+}
558+
559 collect_info() {
560 read_virt
561 read_pid1_product_name

Subscribers

People subscribed via source and target branches