Merge ~jcastets/cloud-init:scaleway-datasource into cloud-init:master

Proposed by Julien Castets
Status: Merged
Approved by: Scott Moser
Approved revision: f61323fb96a58688be73566eaa30c8f7c3b3adc6
Merged at revision: e80517ae6aea49c9ab3bd622a33fee44014f485f
Proposed branch: ~jcastets/cloud-init:scaleway-datasource
Merge into: cloud-init:master
Diff against target: 561 lines (+510/-3)
4 files modified
cloudinit/sources/DataSourceScaleway.py (+223/-0)
cloudinit/url_helper.py (+8/-2)
tests/unittests/test_datasource/test_scaleway.py (+262/-0)
tools/ds-identify (+17/-1)
Reviewer Review Type Date Requested Status
Chad Smith Approve
Scott Moser Approve
Server Team CI bot continuous-integration Approve
Review via email: mp+325740@code.launchpad.net

Description of the change

Implements Scaleway datasource with user and vendor data.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

Some things, and some content inline:
 * We'll need some unit tests for this. Otherwise it is at increased risk of being inadvertently broken.
 * We will need to add knowledge of the datasource to tools/ds-identify (without that your datasource will only ever be considered if it is a single entry in the configured list)
 * we really, *REALLY* want a positive non-network identification. Without such a thing, we can't enable the datasource by default, meaning Ubuntu or other images that would work elsewhere wont work on your platform.

Also, give a nicer commit message:
  Summary
  <blank line>
  More information
  ...
  <blank line>
  LP: #XXXXXXX

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

over all nice.
some comments.

thank you, Julien.

Revision history for this message
Scott Moser (smoser) wrote :

I'm going to move this to 'work in progress'
Please address the comments / tests and move it back to 'Needs Review'.

review: Needs Fixing
Revision history for this message
Julien Castets (jcastets) wrote :

Everything should be fixed.

* I updated the header of cloudinit/sources/DatasourceScaleway. Is it what you expect?
* on_scaleway doesn't rely on network anymore. It checks if "scaleway" is in /var/run/scaleway, in the commandline, or in DMI data.
* Made some unittests
* requests.requests creates a requests.Session object and calls session.request: https://github.com/requests/requests/blob/master/requests/api.py#L57 ; so my change should be backward compatible

If you need any precision or change, feel free to ask.

Since the datasource no longer makes network requests, is there a chance to enable it by default?

Thanks,

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

Some comments inline. mostly questions.
Would it be easier for you to use requests directly rather than going through urlhelper ?

c8201d5... by Julien Castets

Scaleway: fix typo

55eb390... by Julien Castets

Scaleway: add logging

c28b9e4... by Julien Castets

Scaleway: fix header copyright header

f52c323... by Julien Castets

Scaleway: remove inline pylint bypass

80be5ac... by Julien Castets

Scaleway: fix docstring format

3fddcf0... by Julien Castets

Scaleway: assert sleep is called in tests

Revision history for this message
Julien Castets (jcastets) wrote :

> Some comments inline. mostly questions.

Everything should be fixed. I can rebase my commits into one if you want me to.

> Would it be easier for you to use requests directly rather than going through
> urlhelper ?

I'd prefer not to. url_helper is doing some logging, sets the user-agent, gracefully handles SSL errors... even if it seems hackish, using url_helper.readurl is IMO the best way to do.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:3fddcf07e447cdc87abbb9a3a13744de17d889c0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/35/rebuild

review: Approve (continuous-integration)
f61323f... by Julien Castets

Scaleway: split unittests

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:f61323fb96a58688be73566eaa30c8f7c3b3adc6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/40/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/40/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

I approve of this at this point.
The use of urllib3 through requests could be a problem in the future, but
should not cause any regression potential here as this is a new
datasource.

I plan on adding the additional changes at
 http://paste.ubuntu.com/25114217/

Just for our future reference / context.

Revision history for this message
Scott Moser (smoser) wrote :

chad does this MP pluls above patch look ok to you ?

review: Approve
Revision history for this message
Chad Smith (chad.smith) wrote :

Unit test decomposition looks good; thank you for that. I'd avoid wrapper functions in the future like install_mocks just so we can see mocked return_values are seen local to the unit test instead of having to look up at another method defintion to find out what it really is set to. I get that in this case it gives and easy eye in the 3 unit tests toward reading the True,False flags for whether the mock is 'active'.

  Per the content of the merge proposal, the altered readurl in cloudinit.url_helper with the explicit session context manager doesn't adversely affect us on existing deployments +1 there. I also agree that any potential maintenance of the Scaleway datasource importing request.packages.urllib3 on other distributions will probably be minimal/infrequent, and if there is adverse impact, it will be limited to Scaleway cloud-init users.

Approved with Scott's comments for context and s/priviledged/privileged/.

review: Approve
Revision history for this message
Julien Castets (jcastets) wrote :

Thanks a lot for your help :)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/cloudinit/sources/DataSourceScaleway.py b/cloudinit/sources/DataSourceScaleway.py
0new file mode 1006440new file mode 100644
index 0000000..93b4be9
--- /dev/null
+++ b/cloudinit/sources/DataSourceScaleway.py
@@ -0,0 +1,223 @@
1# Author: Julien Castets <castets.j@gmail.com>
2#
3# This file is part of cloud-init. See LICENSE file for license information.
4
5# Scaleway API:
6# https://developer.scaleway.com/#metadata
7
8import json
9import os
10import socket
11import time
12
13import requests
14
15# pylint fails to import the two modules below.
16# pylint: disable=E0401
17from requests.packages.urllib3.connection import HTTPConnection
18from requests.packages.urllib3.poolmanager import PoolManager
19
20from cloudinit import log as logging
21from cloudinit import sources
22from cloudinit import url_helper
23from cloudinit import util
24
25
26LOG = logging.getLogger(__name__)
27
28DS_BASE_URL = 'http://169.254.42.42'
29
30BUILTIN_DS_CONFIG = {
31 'metadata_url': DS_BASE_URL + '/conf?format=json',
32 'userdata_url': DS_BASE_URL + '/user_data/cloud-init',
33 'vendordata_url': DS_BASE_URL + '/vendor_data/cloud-init'
34}
35
36DEF_MD_RETRIES = 5
37DEF_MD_TIMEOUT = 10
38
39
40def on_scaleway():
41 """
42 There are three ways to detect if you are on Scaleway:
43
44 * check DMI data: not yet implemented by Scaleway, but the check is made to
45 be future-proof.
46 * the initrd created the file /var/run/scaleway.
47 * "scaleway" is in the kernel cmdline.
48 """
49 vendor_name = util.read_dmi_data('system-manufacturer')
50 if vendor_name == 'Scaleway':
51 return True
52
53 if os.path.exists('/var/run/scaleway'):
54 return True
55
56 cmdline = util.get_cmdline()
57 if 'scaleway' in cmdline:
58 return True
59
60 return False
61
62
63class SourceAddressAdapter(requests.adapters.HTTPAdapter):
64 """
65 Adapter for requests to choose the local address to bind to.
66 """
67 def __init__(self, source_address, **kwargs):
68 self.source_address = source_address
69 super(SourceAddressAdapter, self).__init__(**kwargs)
70
71 def init_poolmanager(self, connections, maxsize, block=False):
72 socket_options = HTTPConnection.default_socket_options + [
73 (socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
74 ]
75 self.poolmanager = PoolManager(num_pools=connections,
76 maxsize=maxsize,
77 block=block,
78 source_address=self.source_address,
79 socket_options=socket_options)
80
81
82def query_data_api_once(api_address, timeout, requests_session):
83 """
84 Retrieve user data or vendor data.
85
86 Scaleway user/vendor data API returns HTTP/404 if user/vendor data is not
87 set.
88
89 This function calls `url_helper.readurl` but instead of considering
90 HTTP/404 as an error that requires a retry, it considers it as empty
91 user/vendor data.
92
93 Also, be aware the user data/vendor API requires the source port to be
94 below 1024 to ensure the client is root (since non-root users can't bind
95 ports below 1024). If requests raises ConnectionError (EADDRINUSE), the
96 caller should retry to call this function on an other port.
97 """
98 try:
99 resp = url_helper.readurl(
100 api_address,
101 data=None,
102 timeout=timeout,
103 # It's the caller's responsability to recall this function in case
104 # of exception. Don't let url_helper.readurl() retry by itself.
105 retries=0,
106 session=requests_session,
107 # If the error is a HTTP/404 or a ConnectionError, go into raise
108 # block below.
109 exception_cb=lambda _, exc: exc.code == 404 or (
110 isinstance(exc.cause, requests.exceptions.ConnectionError)
111 )
112 )
113 return util.decode_binary(resp.contents)
114 except url_helper.UrlError as exc:
115 # Empty user data.
116 if exc.code == 404:
117 return None
118 raise
119
120
121def query_data_api(api_type, api_address, retries, timeout):
122 """
123 Get user or vendor data.
124
125 Handle the retrying logic in case the source port is used.
126 """
127 # Query user/vendor data. Try to make a request on the first privileged
128 # port available.
129 for port in range(1, max(retries, 2)):
130 try:
131 LOG.debug(
132 'Trying to get %s data (bind on port %d)...',
133 api_type, port
134 )
135 requests_session = requests.Session()
136 requests_session.mount(
137 'http://',
138 SourceAddressAdapter(source_address=('0.0.0.0', port))
139 )
140 data = query_data_api_once(
141 api_address,
142 timeout=timeout,
143 requests_session=requests_session
144 )
145 LOG.debug('%s-data downloaded', api_type)
146 return data
147
148 except url_helper.UrlError as exc:
149 # Local port already in use or HTTP/429.
150 LOG.warning('Error while trying to get %s data: %s', api_type, exc)
151 time.sleep(5)
152 last_exc = exc
153 continue
154
155 # Max number of retries reached.
156 raise last_exc
157
158
159class DataSourceScaleway(sources.DataSource):
160
161 def __init__(self, sys_cfg, distro, paths):
162 super(DataSourceScaleway, self).__init__(sys_cfg, distro, paths)
163
164 self.ds_cfg = util.mergemanydict([
165 util.get_cfg_by_path(sys_cfg, ["datasource", "Scaleway"], {}),
166 BUILTIN_DS_CONFIG
167 ])
168
169 self.metadata_address = self.ds_cfg['metadata_url']
170 self.userdata_address = self.ds_cfg['userdata_url']
171 self.vendordata_address = self.ds_cfg['vendordata_url']
172
173 self.retries = int(self.ds_cfg.get('retries', DEF_MD_RETRIES))
174 self.timeout = int(self.ds_cfg.get('timeout', DEF_MD_TIMEOUT))
175
176 def get_data(self):
177 if not on_scaleway():
178 return False
179
180 resp = url_helper.readurl(self.metadata_address,
181 timeout=self.timeout,
182 retries=self.retries)
183 self.metadata = json.loads(util.decode_binary(resp.contents))
184
185 self.userdata_raw = query_data_api(
186 'user-data', self.userdata_address,
187 self.retries, self.timeout
188 )
189 self.vendordata_raw = query_data_api(
190 'vendor-data', self.vendordata_address,
191 self.retries, self.timeout
192 )
193 return True
194
195 @property
196 def launch_index(self):
197 return None
198
199 def get_instance_id(self):
200 return self.metadata['id']
201
202 def get_public_ssh_keys(self):
203 return [key['key'] for key in self.metadata['ssh_public_keys']]
204
205 def get_hostname(self, fqdn=False, resolve_ip=False):
206 return self.metadata['hostname']
207
208 @property
209 def availability_zone(self):
210 return None
211
212 @property
213 def region(self):
214 return None
215
216
217datasources = [
218 (DataSourceScaleway, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
219]
220
221
222def get_datasource_list(depends):
223 return sources.list_from_depends(depends, datasources)
diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
index d2b92e6..7cf76aa 100644
--- a/cloudinit/url_helper.py
+++ b/cloudinit/url_helper.py
@@ -172,7 +172,8 @@ def _get_ssl_args(url, ssl_details):
172172
173def readurl(url, data=None, timeout=None, retries=0, sec_between=1,173def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
174 headers=None, headers_cb=None, ssl_details=None,174 headers=None, headers_cb=None, ssl_details=None,
175 check_status=True, allow_redirects=True, exception_cb=None):175 check_status=True, allow_redirects=True, exception_cb=None,
176 session=None):
176 url = _cleanurl(url)177 url = _cleanurl(url)
177 req_args = {178 req_args = {
178 'url': url,179 'url': url,
@@ -231,7 +232,12 @@ def readurl(url, data=None, timeout=None, retries=0, sec_between=1,
231 LOG.debug("[%s/%s] open '%s' with %s configuration", i,232 LOG.debug("[%s/%s] open '%s' with %s configuration", i,
232 manual_tries, url, filtered_req_args)233 manual_tries, url, filtered_req_args)
233234
234 r = requests.request(**req_args)235 if session is None:
236 session = requests.Session()
237
238 with session as sess:
239 r = sess.request(**req_args)
240
235 if check_status:241 if check_status:
236 r.raise_for_status()242 r.raise_for_status()
237 LOG.debug("Read from %s (%s, %sb) after %s attempts", url,243 LOG.debug("Read from %s (%s, %sb) after %s attempts", url,
diff --git a/tests/unittests/test_datasource/test_scaleway.py b/tests/unittests/test_datasource/test_scaleway.py
238new file mode 100644244new file mode 100644
index 0000000..65d83ad
--- /dev/null
+++ b/tests/unittests/test_datasource/test_scaleway.py
@@ -0,0 +1,262 @@
1# This file is part of cloud-init. See LICENSE file for license information.
2
3import json
4
5import httpretty
6import requests
7
8from cloudinit import helpers
9from cloudinit import settings
10from cloudinit.sources import DataSourceScaleway
11
12from ..helpers import mock, HttprettyTestCase, TestCase
13
14
15class DataResponses(object):
16 """
17 Possible responses of the API endpoint
18 169.254.42.42/user_data/cloud-init and
19 169.254.42.42/vendor_data/cloud-init.
20 """
21
22 FAKE_USER_DATA = '#!/bin/bash\necho "user-data"'
23
24 @staticmethod
25 def rate_limited(method, uri, headers):
26 return 429, headers, ''
27
28 @staticmethod
29 def api_error(method, uri, headers):
30 return 500, headers, ''
31
32 @classmethod
33 def get_ok(cls, method, uri, headers):
34 return 200, headers, cls.FAKE_USER_DATA
35
36 @staticmethod
37 def empty(method, uri, headers):
38 """
39 No user data for this server.
40 """
41 return 404, headers, ''
42
43
44class MetadataResponses(object):
45 """
46 Possible responses of the metadata API.
47 """
48
49 FAKE_METADATA = {
50 'id': '00000000-0000-0000-0000-000000000000',
51 'hostname': 'scaleway.host',
52 'ssh_public_keys': [{
53 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABA',
54 'fingerprint': '2048 06:ae:... login (RSA)'
55 }, {
56 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABCCCCC',
57 'fingerprint': '2048 06:ff:... login2 (RSA)'
58 }]
59 }
60
61 @classmethod
62 def get_ok(cls, method, uri, headers):
63 return 200, headers, json.dumps(cls.FAKE_METADATA)
64
65
66class TestOnScaleway(TestCase):
67
68 def install_mocks(self, fake_dmi, fake_file_exists, fake_cmdline):
69 mock, faked = fake_dmi
70 mock.return_value = 'Scaleway' if faked else 'Whatever'
71
72 mock, faked = fake_file_exists
73 mock.return_value = faked
74
75 mock, faked = fake_cmdline
76 mock.return_value = \
77 'initrd=initrd showopts scaleway nousb' if faked \
78 else 'BOOT_IMAGE=/vmlinuz-3.11.0-26-generic'
79
80 @mock.patch('cloudinit.util.get_cmdline')
81 @mock.patch('os.path.exists')
82 @mock.patch('cloudinit.util.read_dmi_data')
83 def test_not_on_scaleway(self, m_read_dmi_data, m_file_exists,
84 m_get_cmdline):
85 self.install_mocks(
86 fake_dmi=(m_read_dmi_data, False),
87 fake_file_exists=(m_file_exists, False),
88 fake_cmdline=(m_get_cmdline, False)
89 )
90 self.assertFalse(DataSourceScaleway.on_scaleway())
91
92 # When not on Scaleway, get_data() returns False.
93 datasource = DataSourceScaleway.DataSourceScaleway(
94 settings.CFG_BUILTIN, None, helpers.Paths({})
95 )
96 self.assertFalse(datasource.get_data())
97
98 @mock.patch('cloudinit.util.get_cmdline')
99 @mock.patch('os.path.exists')
100 @mock.patch('cloudinit.util.read_dmi_data')
101 def test_on_scaleway_dmi(self, m_read_dmi_data, m_file_exists,
102 m_get_cmdline):
103 """
104 dmidecode returns "Scaleway".
105 """
106 # dmidecode returns "Scaleway"
107 self.install_mocks(
108 fake_dmi=(m_read_dmi_data, True),
109 fake_file_exists=(m_file_exists, False),
110 fake_cmdline=(m_get_cmdline, False)
111 )
112 self.assertTrue(DataSourceScaleway.on_scaleway())
113
114 @mock.patch('cloudinit.util.get_cmdline')
115 @mock.patch('os.path.exists')
116 @mock.patch('cloudinit.util.read_dmi_data')
117 def test_on_scaleway_var_run_scaleway(self, m_read_dmi_data, m_file_exists,
118 m_get_cmdline):
119 """
120 /var/run/scaleway exists.
121 """
122 self.install_mocks(
123 fake_dmi=(m_read_dmi_data, False),
124 fake_file_exists=(m_file_exists, True),
125 fake_cmdline=(m_get_cmdline, False)
126 )
127 self.assertTrue(DataSourceScaleway.on_scaleway())
128
129 @mock.patch('cloudinit.util.get_cmdline')
130 @mock.patch('os.path.exists')
131 @mock.patch('cloudinit.util.read_dmi_data')
132 def test_on_scaleway_cmdline(self, m_read_dmi_data, m_file_exists,
133 m_get_cmdline):
134 """
135 "scaleway" in /proc/cmdline.
136 """
137 self.install_mocks(
138 fake_dmi=(m_read_dmi_data, False),
139 fake_file_exists=(m_file_exists, False),
140 fake_cmdline=(m_get_cmdline, True)
141 )
142 self.assertTrue(DataSourceScaleway.on_scaleway())
143
144
145def get_source_address_adapter(*args, **kwargs):
146 """
147 Scaleway user/vendor data API requires to be called with a privileged port.
148
149 If the unittests are run as non-root, the user doesn't have the permission
150 to bind on ports below 1024.
151
152 This function removes the bind on a privileged address, since anyway the
153 HTTP call is mocked by httpretty.
154 """
155 kwargs.pop('source_address')
156 return requests.adapters.HTTPAdapter(*args, **kwargs)
157
158
159class TestDataSourceScaleway(HttprettyTestCase):
160
161 def setUp(self):
162 self.datasource = DataSourceScaleway.DataSourceScaleway(
163 settings.CFG_BUILTIN, None, helpers.Paths({})
164 )
165 super(TestDataSourceScaleway, self).setUp()
166
167 self.metadata_url = \
168 DataSourceScaleway.BUILTIN_DS_CONFIG['metadata_url']
169 self.userdata_url = \
170 DataSourceScaleway.BUILTIN_DS_CONFIG['userdata_url']
171 self.vendordata_url = \
172 DataSourceScaleway.BUILTIN_DS_CONFIG['vendordata_url']
173
174 @httpretty.activate
175 @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
176 get_source_address_adapter)
177 @mock.patch('cloudinit.util.get_cmdline')
178 @mock.patch('time.sleep', return_value=None)
179 def test_metadata_ok(self, sleep, m_get_cmdline):
180 """
181 get_data() returns metadata, user data and vendor data.
182 """
183 m_get_cmdline.return_value = 'scaleway'
184
185 # Make user data API return a valid response
186 httpretty.register_uri(httpretty.GET, self.metadata_url,
187 body=MetadataResponses.get_ok)
188 httpretty.register_uri(httpretty.GET, self.userdata_url,
189 body=DataResponses.get_ok)
190 httpretty.register_uri(httpretty.GET, self.vendordata_url,
191 body=DataResponses.get_ok)
192 self.datasource.get_data()
193
194 self.assertEqual(self.datasource.get_instance_id(),
195 MetadataResponses.FAKE_METADATA['id'])
196 self.assertEqual(self.datasource.get_public_ssh_keys(), [
197 elem['key'] for elem in
198 MetadataResponses.FAKE_METADATA['ssh_public_keys']
199 ])
200 self.assertEqual(self.datasource.get_hostname(),
201 MetadataResponses.FAKE_METADATA['hostname'])
202 self.assertEqual(self.datasource.get_userdata_raw(),
203 DataResponses.FAKE_USER_DATA)
204 self.assertEqual(self.datasource.get_vendordata_raw(),
205 DataResponses.FAKE_USER_DATA)
206 self.assertIsNone(self.datasource.availability_zone)
207 self.assertIsNone(self.datasource.region)
208 self.assertEqual(sleep.call_count, 0)
209
210 @httpretty.activate
211 @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
212 get_source_address_adapter)
213 @mock.patch('cloudinit.util.get_cmdline')
214 @mock.patch('time.sleep', return_value=None)
215 def test_metadata_404(self, sleep, m_get_cmdline):
216 """
217 get_data() returns metadata, but no user data nor vendor data.
218 """
219 m_get_cmdline.return_value = 'scaleway'
220
221 # Make user and vendor data APIs return HTTP/404, which means there is
222 # no user / vendor data for the server.
223 httpretty.register_uri(httpretty.GET, self.metadata_url,
224 body=MetadataResponses.get_ok)
225 httpretty.register_uri(httpretty.GET, self.userdata_url,
226 body=DataResponses.empty)
227 httpretty.register_uri(httpretty.GET, self.vendordata_url,
228 body=DataResponses.empty)
229 self.datasource.get_data()
230 self.assertIsNone(self.datasource.get_userdata_raw())
231 self.assertIsNone(self.datasource.get_vendordata_raw())
232 self.assertEqual(sleep.call_count, 0)
233
234 @httpretty.activate
235 @mock.patch('cloudinit.sources.DataSourceScaleway.SourceAddressAdapter',
236 get_source_address_adapter)
237 @mock.patch('cloudinit.util.get_cmdline')
238 @mock.patch('time.sleep', return_value=None)
239 def test_metadata_rate_limit(self, sleep, m_get_cmdline):
240 """
241 get_data() is rate limited two times by the metadata API when fetching
242 user data.
243 """
244 m_get_cmdline.return_value = 'scaleway'
245
246 httpretty.register_uri(httpretty.GET, self.metadata_url,
247 body=MetadataResponses.get_ok)
248 httpretty.register_uri(httpretty.GET, self.vendordata_url,
249 body=DataResponses.empty)
250
251 httpretty.register_uri(
252 httpretty.GET, self.userdata_url,
253 responses=[
254 httpretty.Response(body=DataResponses.rate_limited),
255 httpretty.Response(body=DataResponses.rate_limited),
256 httpretty.Response(body=DataResponses.get_ok),
257 ]
258 )
259 self.datasource.get_data()
260 self.assertEqual(self.datasource.get_userdata_raw(),
261 DataResponses.FAKE_USER_DATA)
262 self.assertEqual(sleep.call_count, 2)
diff --git a/tools/ds-identify b/tools/ds-identify
index 7c8b144..33bd299 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -112,7 +112,7 @@ DI_DSNAME=""
112# be searched if there is no setting found in config.112# be searched if there is no setting found in config.
113DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \113DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
114CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \114CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
115OVF SmartOS"115OVF SmartOS Scaleway"
116DI_DSLIST=""116DI_DSLIST=""
117DI_MODE=""117DI_MODE=""
118DI_ON_FOUND=""118DI_ON_FOUND=""
@@ -896,6 +896,22 @@ dscheck_None() {
896 return ${DS_NOT_FOUND}896 return ${DS_NOT_FOUND}
897}897}
898898
899dscheck_Scaleway() {
900 if [ "${DI_DMI_SYS_VENDOR}" = "Scaleway" ]; then
901 return $DS_FOUND
902 fi
903
904 case " ${DI_KERNEL_CMDLINE} " in
905 *\ scaleway\ *) return ${DS_FOUND};;
906 esac
907
908 if [ -f ${PATH_ROOT}/var/run/scaleway ]; then
909 return ${DS_FOUND}
910 fi
911
912 return ${DS_NOT_FOUND}
913}
914
899collect_info() {915collect_info() {
900 read_virt916 read_virt
901 read_pid1_product_name917 read_pid1_product_name

Subscribers

People subscribed via source and target branches