Merge ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master

Proposed by Chris Glass on 2019-07-01
Status: Merged
Approved by: Dan Watkins on 2019-08-08
Approved revision: afe3b3bd3f0de7f5364115ad17b4e585dcdb75e4
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~tribaal/cloud-init:feat/datasource-exoscale
Merge into: cloud-init:master
Diff against target: 628 lines (+540/-1)
8 files modified
cloudinit/apport.py (+1/-0)
cloudinit/settings.py (+1/-0)
cloudinit/sources/DataSourceExoscale.py (+258/-0)
doc/rtd/topics/datasources.rst (+1/-0)
doc/rtd/topics/datasources/exoscale.rst (+68/-0)
tests/unittests/test_datasource/test_common.py (+2/-0)
tests/unittests/test_datasource/test_exoscale.py (+203/-0)
tools/ds-identify (+6/-1)
Reviewer Review Type Date Requested Status
Dan Watkins Approve on 2019-08-08
Server Team CI bot continuous-integration Approve on 2019-08-06
Ryan Harper 2019-07-01 Approve on 2019-07-31
Review via email: mp+369516@code.launchpad.net

Commit message

New data source for the Exoscale.com cloud platform

- dsidentify switches to the new Exoscale datasource on matching DMI name
- New Exoscale datasource added

Author: Chris Glass <email address hidden>
Signed-off-by: Mathieu Corbin <email address hidden>

Description of the change

New data source for the Exoscale.com cloud platform

While the Exoscale cloud platform started as a Cloudstack fork, over the years
more and more of the codebase was rewritten, replaced, or otherwise
fundamentally changed.

Our recent feature development around networking made it clear that while we
intend for our cloud platform to remain as compatible with the Cloudstack
datasource in the forseeable future (once forced - see below), it is time for
us to bite the proverbial bullet and submit our own datasource for review.

Networking glitch:

The Cloudstack datasource doesn't work for us in the scenario where a user boots
a VM attached to the default network and one or more private network(s) running
DHCP (hosted by us, or customer-deployed): in some race condition the private
interface will be the lease on which the datasource will try to query for
metadata - which can't work. The submitted datasource fixes the problem by
forcing the datasource IP address to always be the same, so that we don't need
to depend on DHCP at all to determine it, fixing the race (we ensure routes to
that address are provided).

DMI product name change:

Along with this change, we intend to switch the DMI product name from
"CloudStack KVM Hypervisor" to "Exoscale Compute Platform" and the DMI
manufacturer string from "Apache Software Foundation" to "Exoscale".

Current templates situation:

We currently do not advertise custom templates to our customers, but intend to do
so more widely in the near future.

All of our current production templates force the Cloudstack datasource usage
(via configuration in /etc/cloud*).

Once this code lands, and our DMI product name change is performed, we will
document the requirements for (future) custom templates to be either:

- A recent enough cloud-init (containing the proposed code)
- Forcing the datasource to be "Cloudstack" for older cloud-init versions. This
  will trigger the above-mentioned race, but that is a specific, relatively rare
  use-case.
- Certain (future) features might only be available to customers running a
  recent-enough cloud-init version.

Testing this code:

Inclusion in the cloud-init test suite in addition to the submitted unit tests
would be welcome.

To post a comment you must log in.

FAILED: Continuous integration, rev:7279a5570bc87909022d9f849ac609d7acdbb3ca
https://jenkins.ubuntu.com/server/job/cloud-init-ci/751/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/751/rebuild

review: Needs Fixing (continuous-integration)

PASSED: Continuous integration, rev:7279a5570bc87909022d9f849ac609d7acdbb3ca
https://jenkins.ubuntu.com/server/job/cloud-init-ci/752/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/752/rebuild

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

Thanks for submitting this. Some inline comments and questions.

In your description you mention networking configuration issues. Have you looked at providing network-metadata and having your datasource provide a network-config property to allow cloud-init to render networking[1] for the instance?

1. https://cloudinit.readthedocs.io/en/latest/topics/network-config.html

Dan Watkins (daniel-thewatkins) wrote :

Quick drive-by review to ask if we can reuse the password fetching from the CloudStack DS.

FAILED: Continuous integration, rev:64974621d238bd2bb5bbbf79fc27cc318975b1d0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/758/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/758/rebuild

review: Needs Fixing (continuous-integration)
Chris Glass (tribaal) wrote :

A first pass fixing most review comments is done.

One open point here is the extra_config overwriting user-data - it seems to me that perhaps we should be using some other mechanism instead (my assumption was that extra_config would be overwritten by user data - but that doesn't seem to be the case).

Should we use "vendordata" instead for this particular case?

Ryan Harper (raharper) wrote :

Thanks for updating. The use of extra_config is fine for providing some default values when no-user-data is provided; You're correct that it will be overriden by user-data; I don't think you need to switch to vendor-data.

The jenkins failure was related to the switch to log_time and the time module import is no longer needed.

I've added some other questions inline. Thanks

review: Needs Information
Chris Glass (tribaal) wrote :

Thanks for the review(s).

I think all the points are once again addressed.

FAILED: Continuous integration, rev:bb9ca041b15d36a6b368ad6af09a99dbfef6295a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/760/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/760/rebuild

review: Needs Fixing (continuous-integration)
Ryan Harper (raharper) wrote :

Thanks for the updated, a couple more questions inline.

Chad Smith (chad.smith) :
Chad Smith (chad.smith) :
Chris Glass (tribaal) wrote :

Addressed all review comments.

FAILED: Continuous integration, rev:874cb3a6bfd158f9523cced13bd42f2dbaed0e4b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/763/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/763/rebuild

review: Needs Fixing (continuous-integration)

PASSED: Continuous integration, rev:5c01e89c8f5abcc6150ebbcc82fc520fad87fbbb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/764/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/764/rebuild

review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

Some doc in doc/rtd/topics/datasources would be good.
https://cloudinit.readthedocs.io/en/latest/topics/datasources.html has examples.... Just at very least a link to exoscale.com and a mention on how that differs from ec2.

Chris Glass (tribaal) wrote :

Thanks for your review.

I think all your points have been addressed. While adding a bit of documentation I realized the datasource was not actually configurable. This is now the case.

Scott Moser (smoser) wrote :

This looks fine to me. one comment on the doc. i'm fine with this, but think it'd be good to have someone else also review.

Chris Glass (tribaal) wrote :

Dropped comments. Thanks!

Once again ready for review(s)!

Chad Smith (chad.smith) wrote :

thank you for the updates here. Couple of minor nits that may involve documentation changes and a request for unit test coverage.

PASSED: Continuous integration, rev:2fc709c9b07cba1fc43cc0aa95c899fac4a8910d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/769/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/769/rebuild

review: Approve (continuous-integration)
Chris Glass (tribaal) wrote :

Should be ready for another pass once again.

Thanks a lot for your reviews so far!

PASSED: Continuous integration, rev:fca069fa63f8d541d6ec9ce1d25a1ffcab824bb7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/770/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/770/rebuild

review: Approve (continuous-integration)

PASSED: Continuous integration, rev:c5ea4ac4642feeb0deb35ec2f4117ee948491ebc
https://jenkins.ubuntu.com/server/job/cloud-init-ci/771/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/771/rebuild

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

A couple of minor changes in docs requested.

review: Needs Fixing
Chris Glass (tribaal) wrote :

All requested changes should be addressed.

Thanks!

PASSED: Continuous integration, rev:89c1eff973dcd6b8927a9835b0264fac498a94be
https://jenkins.ubuntu.com/server/job/cloud-init-ci/785/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/785/rebuild

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

One typo, and question about the templates vs built-in datasource config for the set_password frequency.

fe07f84... by Chris Glass on 2019-07-30

Datasource makes password module run "always"

Doing this here ensures the config is merged the proper way (explicit
user choices via user-data override this setting).

9f73b66... by Chris Glass on 2019-07-30

Fix typo.

20f0c6a... by Chris Glass on 2019-07-30

Add extra tests.

Chris Glass (tribaal) wrote :

This is now ready for another round of review.

Thanks a lot!

PASSED: Continuous integration, rev:20f0c6a1487572eaf5975b80665641098ba965b8
https://jenkins.ubuntu.com/server/job/cloud-init-ci/801/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/801/rebuild

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

Thanks Chris,

This looks really good.

review: Approve
Dan Watkins (daniel-thewatkins) wrote :

Hey Chris, Mathieu!

Thanks for this change, it's looking in pretty good shape! There are quite a few inline comments here _but_ there are only two that I'm considering a hard blocker for landing this[0]:

* the missing copyright header
* my comment on line 143; I think that's the only (potential!) bug I spotted, and fixing it might drive some other changes to the data source so I'd like us to understand the scope of those before merging this

Please do review my other comments, and let me know what you think. Provided you can commit to addressing them in a follow-up merge proposal[1], once we've resolved the above we can land this and SRU it back with 19.2.

Thanks!

Dan

[0] Terms and conditions apply.
[1] These are the terms and conditions. ;)

review: Needs Fixing
afe3b3b... by Chris Glass on 2019-08-06

Address review comments

- Copyright headers
- Various phrasing fixes in the documentation
- Code fixes
- Ensure a non-responding password server still lets the datasource use
  the scraped metadata/userdata information

Chris Glass (tribaal) wrote :

Hey Dan, thanks for the review!

All comments should be addressed.

PASSED: Continuous integration, rev:afe3b3bd3f0de7f5364115ad17b4e585dcdb75e4
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1027/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/1027//rebuild

review: Approve (continuous-integration)
Dan Watkins (daniel-thewatkins) wrote :

Hey Chris, thanks for the updates and the thoughtful responses, this looks great!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/apport.py b/cloudinit/apport.py
2index 22cb7fd..003ff1f 100644
3--- a/cloudinit/apport.py
4+++ b/cloudinit/apport.py
5@@ -23,6 +23,7 @@ KNOWN_CLOUD_NAMES = [
6 'CloudStack',
7 'DigitalOcean',
8 'GCE - Google Compute Engine',
9+ 'Exoscale',
10 'Hetzner Cloud',
11 'IBM - (aka SoftLayer or BlueMix)',
12 'LXD',
13diff --git a/cloudinit/settings.py b/cloudinit/settings.py
14index b1ebaad..2060d81 100644
15--- a/cloudinit/settings.py
16+++ b/cloudinit/settings.py
17@@ -39,6 +39,7 @@ CFG_BUILTIN = {
18 'Hetzner',
19 'IBMCloud',
20 'Oracle',
21+ 'Exoscale',
22 # At the end to act as a 'catch' when none of the above work...
23 'None',
24 ],
25diff --git a/cloudinit/sources/DataSourceExoscale.py b/cloudinit/sources/DataSourceExoscale.py
26new file mode 100644
27index 0000000..52e7f6f
28--- /dev/null
29+++ b/cloudinit/sources/DataSourceExoscale.py
30@@ -0,0 +1,258 @@
31+# Author: Mathieu Corbin <mathieu.corbin@exoscale.com>
32+# Author: Christopher Glass <christopher.glass@exoscale.com>
33+#
34+# This file is part of cloud-init. See LICENSE file for license information.
35+
36+from cloudinit import ec2_utils as ec2
37+from cloudinit import log as logging
38+from cloudinit import sources
39+from cloudinit import url_helper
40+from cloudinit import util
41+
42+LOG = logging.getLogger(__name__)
43+
44+METADATA_URL = "http://169.254.169.254"
45+API_VERSION = "1.0"
46+PASSWORD_SERVER_PORT = 8080
47+
48+URL_TIMEOUT = 10
49+URL_RETRIES = 6
50+
51+EXOSCALE_DMI_NAME = "Exoscale"
52+
53+BUILTIN_DS_CONFIG = {
54+ # We run the set password config module on every boot in order to enable
55+ # resetting the instance's password via the exoscale console (and a
56+ # subsequent instance reboot).
57+ 'cloud_config_modules': [["set-passwords", "always"]]
58+}
59+
60+
61+class DataSourceExoscale(sources.DataSource):
62+
63+ dsname = 'Exoscale'
64+
65+ def __init__(self, sys_cfg, distro, paths):
66+ super(DataSourceExoscale, self).__init__(sys_cfg, distro, paths)
67+ LOG.debug("Initializing the Exoscale datasource")
68+
69+ self.metadata_url = self.ds_cfg.get('metadata_url', METADATA_URL)
70+ self.api_version = self.ds_cfg.get('api_version', API_VERSION)
71+ self.password_server_port = int(
72+ self.ds_cfg.get('password_server_port', PASSWORD_SERVER_PORT))
73+ self.url_timeout = self.ds_cfg.get('timeout', URL_TIMEOUT)
74+ self.url_retries = self.ds_cfg.get('retries', URL_RETRIES)
75+
76+ self.extra_config = BUILTIN_DS_CONFIG
77+
78+ def wait_for_metadata_service(self):
79+ """Wait for the metadata service to be reachable."""
80+
81+ metadata_url = "{}/{}/meta-data/instance-id".format(
82+ self.metadata_url, self.api_version)
83+
84+ url = url_helper.wait_for_url(
85+ urls=[metadata_url],
86+ max_wait=self.url_max_wait,
87+ timeout=self.url_timeout,
88+ status_cb=LOG.critical)
89+
90+ return bool(url)
91+
92+ def crawl_metadata(self):
93+ """
94+ Crawl the metadata service when available.
95+
96+ @returns: Dictionary of crawled metadata content.
97+ """
98+ metadata_ready = util.log_time(
99+ logfunc=LOG.info,
100+ msg='waiting for the metadata service',
101+ func=self.wait_for_metadata_service)
102+
103+ if not metadata_ready:
104+ return {}
105+
106+ return read_metadata(self.metadata_url, self.api_version,
107+ self.password_server_port, self.url_timeout,
108+ self.url_retries)
109+
110+ def _get_data(self):
111+ """Fetch the user data, the metadata and the VM password
112+ from the metadata service.
113+
114+ Please refer to the datasource documentation for details on how the
115+ metadata server and password server are crawled.
116+ """
117+ if not self._is_platform_viable():
118+ return False
119+
120+ data = util.log_time(
121+ logfunc=LOG.debug,
122+ msg='Crawl of metadata service',
123+ func=self.crawl_metadata)
124+
125+ if not data:
126+ return False
127+
128+ self.userdata_raw = data['user-data']
129+ self.metadata = data['meta-data']
130+ password = data.get('password')
131+
132+ password_config = {}
133+ if password:
134+ # Since we have a password, let's make sure we are allowed to use
135+ # it by allowing ssh_pwauth.
136+ # The password module's default behavior is to leave the
137+ # configuration as-is in this regard, so that means it will either
138+ # leave the password always disabled if no password is ever set, or
139+ # leave the password login enabled if we set it once.
140+ password_config = {
141+ 'ssh_pwauth': True,
142+ 'password': password,
143+ 'chpasswd': {
144+ 'expire': False,
145+ },
146+ }
147+
148+ # builtin extra_config overrides password_config
149+ self.extra_config = util.mergemanydict(
150+ [self.extra_config, password_config])
151+
152+ return True
153+
154+ def get_config_obj(self):
155+ return self.extra_config
156+
157+ def _is_platform_viable(self):
158+ return util.read_dmi_data('system-product-name').startswith(
159+ EXOSCALE_DMI_NAME)
160+
161+
162+# Used to match classes to dependencies
163+datasources = [
164+ (DataSourceExoscale, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
165+]
166+
167+
168+# Return a list of data sources that match this set of dependencies
169+def get_datasource_list(depends):
170+ return sources.list_from_depends(depends, datasources)
171+
172+
173+def get_password(metadata_url=METADATA_URL,
174+ api_version=API_VERSION,
175+ password_server_port=PASSWORD_SERVER_PORT,
176+ url_timeout=URL_TIMEOUT,
177+ url_retries=URL_RETRIES):
178+ """Obtain the VM's password if set.
179+
180+ Once fetched the password is marked saved. Future calls to this method may
181+ return empty string or 'saved_password'."""
182+ password_url = "{}:{}/{}/".format(metadata_url, password_server_port,
183+ api_version)
184+ response = url_helper.read_file_or_url(
185+ password_url,
186+ ssl_details=None,
187+ headers={"DomU_Request": "send_my_password"},
188+ timeout=url_timeout,
189+ retries=url_retries)
190+ password = response.contents.decode('utf-8')
191+ # the password is empty or already saved
192+ # Note: the original metadata server would answer an additional
193+ # 'bad_request' status, but the Exoscale implementation does not.
194+ if password in ['', 'saved_password']:
195+ return None
196+ # save the password
197+ url_helper.read_file_or_url(
198+ password_url,
199+ ssl_details=None,
200+ headers={"DomU_Request": "saved_password"},
201+ timeout=url_timeout,
202+ retries=url_retries)
203+ return password
204+
205+
206+def read_metadata(metadata_url=METADATA_URL,
207+ api_version=API_VERSION,
208+ password_server_port=PASSWORD_SERVER_PORT,
209+ url_timeout=URL_TIMEOUT,
210+ url_retries=URL_RETRIES):
211+ """Query the metadata server and return the retrieved data."""
212+ crawled_metadata = {}
213+ crawled_metadata['_metadata_api_version'] = api_version
214+ try:
215+ crawled_metadata['user-data'] = ec2.get_instance_userdata(
216+ api_version,
217+ metadata_url,
218+ timeout=url_timeout,
219+ retries=url_retries)
220+ crawled_metadata['meta-data'] = ec2.get_instance_metadata(
221+ api_version,
222+ metadata_url,
223+ timeout=url_timeout,
224+ retries=url_retries)
225+ except Exception as e:
226+ util.logexc(LOG, "failed reading from metadata url %s (%s)",
227+ metadata_url, e)
228+ return {}
229+
230+ try:
231+ crawled_metadata['password'] = get_password(
232+ api_version=api_version,
233+ metadata_url=metadata_url,
234+ password_server_port=password_server_port,
235+ url_retries=url_retries,
236+ url_timeout=url_timeout)
237+ except Exception as e:
238+ util.logexc(LOG, "failed to read from password server url %s:%s (%s)",
239+ metadata_url, password_server_port, e)
240+
241+ return crawled_metadata
242+
243+
244+if __name__ == "__main__":
245+ import argparse
246+
247+ parser = argparse.ArgumentParser(description='Query Exoscale Metadata')
248+ parser.add_argument(
249+ "--endpoint",
250+ metavar="URL",
251+ help="The url of the metadata service.",
252+ default=METADATA_URL)
253+ parser.add_argument(
254+ "--version",
255+ metavar="VERSION",
256+ help="The version of the metadata endpoint to query.",
257+ default=API_VERSION)
258+ parser.add_argument(
259+ "--retries",
260+ metavar="NUM",
261+ type=int,
262+ help="The number of retries querying the endpoint.",
263+ default=URL_RETRIES)
264+ parser.add_argument(
265+ "--timeout",
266+ metavar="NUM",
267+ type=int,
268+ help="The time in seconds to wait before timing out.",
269+ default=URL_TIMEOUT)
270+ parser.add_argument(
271+ "--password-port",
272+ metavar="PORT",
273+ type=int,
274+ help="The port on which the password endpoint listens",
275+ default=PASSWORD_SERVER_PORT)
276+
277+ args = parser.parse_args()
278+
279+ data = read_metadata(
280+ metadata_url=args.endpoint,
281+ api_version=args.version,
282+ password_server_port=args.password_port,
283+ url_timeout=args.timeout,
284+ url_retries=args.retries)
285+
286+ print(util.json_dumps(data))
287+
288+# vi: ts=4 expandtab
289diff --git a/doc/rtd/topics/datasources.rst b/doc/rtd/topics/datasources.rst
290index 648c606..2148cd5 100644
291--- a/doc/rtd/topics/datasources.rst
292+++ b/doc/rtd/topics/datasources.rst
293@@ -155,6 +155,7 @@ Follow for more information.
294 datasources/configdrive.rst
295 datasources/digitalocean.rst
296 datasources/ec2.rst
297+ datasources/exoscale.rst
298 datasources/maas.rst
299 datasources/nocloud.rst
300 datasources/opennebula.rst
301diff --git a/doc/rtd/topics/datasources/exoscale.rst b/doc/rtd/topics/datasources/exoscale.rst
302new file mode 100644
303index 0000000..27aec9c
304--- /dev/null
305+++ b/doc/rtd/topics/datasources/exoscale.rst
306@@ -0,0 +1,68 @@
307+.. _datasource_exoscale:
308+
309+Exoscale
310+========
311+
312+This datasource supports reading from the metadata server used on the
313+`Exoscale platform <https://exoscale.com>`_.
314+
315+Use of the Exoscale datasource is recommended to benefit from new features of
316+the Exoscale platform.
317+
318+The datasource relies on the availability of a compatible metadata server
319+(``http://169.254.169.254`` is used by default) and its companion password
320+server, reachable at the same address (by default on port 8080).
321+
322+Crawling of metadata
323+--------------------
324+
325+The metadata service and password server are crawled slightly differently:
326+
327+ * The "metadata service" is crawled every boot.
328+ * The password server is also crawled every boot (the Exoscale datasource
329+ forces the password module to run with "frequency always").
330+
331+In the password server case, the following rules apply in order to enable the
332+"restore instance password" functionality:
333+
334+ * If a password is returned by the password server, it is then marked "saved"
335+ by the cloud-init datasource. Subsequent boots will skip setting the password
336+ (the password server will return "saved_password").
337+ * When the instance password is reset (via the Exoscale UI), the password
338+ server will return the non-empty password at next boot, therefore causing
339+ cloud-init to reset the instance's password.
340+
341+Configuration
342+-------------
343+
344+Users of this datasource are discouraged from changing the default settings
345+unless instructed to by Exoscale support.
346+
347+The following settings are available and can be set for the datasource in system
348+configuration (in `/etc/cloud/cloud.cfg.d/`).
349+
350+The settings available are:
351+
352+ * **metadata_url**: The URL for the metadata service (defaults to
353+ ``http://169.254.169.254``)
354+ * **api_version**: The API version path on which to query the instance metadata
355+ (defaults to ``1.0``)
356+ * **password_server_port**: The port (on the metadata server) on which the
357+ password server listens (defaults to ``8080``).
358+ * **timeout**: the timeout value provided to urlopen for each individual http
359+ request. (defaults to ``10``)
360+ * **retries**: The number of retries that should be done for an http request
361+ (defaults to ``6``)
362+
363+
364+An example configuration with the default values is provided below:
365+
366+.. sourcecode:: yaml
367+
368+ datasource:
369+ Exoscale:
370+ metadata_url: "http://169.254.169.254"
371+ api_version: "1.0"
372+ password_server_port: 8080
373+ timeout: 10
374+ retries: 6
375diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
376index 2a9cfb2..61a7a76 100644
377--- a/tests/unittests/test_datasource/test_common.py
378+++ b/tests/unittests/test_datasource/test_common.py
379@@ -13,6 +13,7 @@ from cloudinit.sources import (
380 DataSourceConfigDrive as ConfigDrive,
381 DataSourceDigitalOcean as DigitalOcean,
382 DataSourceEc2 as Ec2,
383+ DataSourceExoscale as Exoscale,
384 DataSourceGCE as GCE,
385 DataSourceHetzner as Hetzner,
386 DataSourceIBMCloud as IBMCloud,
387@@ -53,6 +54,7 @@ DEFAULT_NETWORK = [
388 CloudStack.DataSourceCloudStack,
389 DSNone.DataSourceNone,
390 Ec2.DataSourceEc2,
391+ Exoscale.DataSourceExoscale,
392 GCE.DataSourceGCE,
393 MAAS.DataSourceMAAS,
394 NoCloud.DataSourceNoCloudNet,
395diff --git a/tests/unittests/test_datasource/test_exoscale.py b/tests/unittests/test_datasource/test_exoscale.py
396new file mode 100644
397index 0000000..350c330
398--- /dev/null
399+++ b/tests/unittests/test_datasource/test_exoscale.py
400@@ -0,0 +1,203 @@
401+# Author: Mathieu Corbin <mathieu.corbin@exoscale.com>
402+# Author: Christopher Glass <christopher.glass@exoscale.com>
403+#
404+# This file is part of cloud-init. See LICENSE file for license information.
405+from cloudinit import helpers
406+from cloudinit.sources.DataSourceExoscale import (
407+ API_VERSION,
408+ DataSourceExoscale,
409+ METADATA_URL,
410+ get_password,
411+ PASSWORD_SERVER_PORT,
412+ read_metadata)
413+from cloudinit.tests.helpers import HttprettyTestCase, mock
414+
415+import httpretty
416+import requests
417+
418+
419+TEST_PASSWORD_URL = "{}:{}/{}/".format(METADATA_URL,
420+ PASSWORD_SERVER_PORT,
421+ API_VERSION)
422+
423+TEST_METADATA_URL = "{}/{}/meta-data/".format(METADATA_URL,
424+ API_VERSION)
425+
426+TEST_USERDATA_URL = "{}/{}/user-data".format(METADATA_URL,
427+ API_VERSION)
428+
429+
430+@httpretty.activate
431+class TestDatasourceExoscale(HttprettyTestCase):
432+
433+ def setUp(self):
434+ super(TestDatasourceExoscale, self).setUp()
435+ self.tmp = self.tmp_dir()
436+ self.password_url = TEST_PASSWORD_URL
437+ self.metadata_url = TEST_METADATA_URL
438+ self.userdata_url = TEST_USERDATA_URL
439+
440+ def test_password_saved(self):
441+ """The password is not set when it is not found
442+ in the metadata service."""
443+ httpretty.register_uri(httpretty.GET,
444+ self.password_url,
445+ body="saved_password")
446+ self.assertFalse(get_password())
447+
448+ def test_password_empty(self):
449+ """No password is set if the metadata service returns
450+ an empty string."""
451+ httpretty.register_uri(httpretty.GET,
452+ self.password_url,
453+ body="")
454+ self.assertFalse(get_password())
455+
456+ def test_password(self):
457+ """The password is set to what is found in the metadata
458+ service."""
459+ expected_password = "p@ssw0rd"
460+ httpretty.register_uri(httpretty.GET,
461+ self.password_url,
462+ body=expected_password)
463+ password = get_password()
464+ self.assertEqual(expected_password, password)
465+
466+ def test_get_data(self):
467+ """The datasource conforms to expected behavior when supplied
468+ full test data."""
469+ path = helpers.Paths({'run_dir': self.tmp})
470+ ds = DataSourceExoscale({}, None, path)
471+ ds._is_platform_viable = lambda: True
472+ expected_password = "p@ssw0rd"
473+ expected_id = "12345"
474+ expected_hostname = "myname"
475+ expected_userdata = "#cloud-config"
476+ httpretty.register_uri(httpretty.GET,
477+ self.userdata_url,
478+ body=expected_userdata)
479+ httpretty.register_uri(httpretty.GET,
480+ self.password_url,
481+ body=expected_password)
482+ httpretty.register_uri(httpretty.GET,
483+ self.metadata_url,
484+ body="instance-id\nlocal-hostname")
485+ httpretty.register_uri(httpretty.GET,
486+ "{}local-hostname".format(self.metadata_url),
487+ body=expected_hostname)
488+ httpretty.register_uri(httpretty.GET,
489+ "{}instance-id".format(self.metadata_url),
490+ body=expected_id)
491+ self.assertTrue(ds._get_data())
492+ self.assertEqual(ds.userdata_raw.decode("utf-8"), "#cloud-config")
493+ self.assertEqual(ds.metadata, {"instance-id": expected_id,
494+ "local-hostname": expected_hostname})
495+ self.assertEqual(ds.get_config_obj(),
496+ {'ssh_pwauth': True,
497+ 'password': expected_password,
498+ 'cloud_config_modules': [
499+ ["set-passwords", "always"]],
500+ 'chpasswd': {
501+ 'expire': False,
502+ }})
503+
504+ def test_get_data_saved_password(self):
505+ """The datasource conforms to expected behavior when saved_password is
506+ returned by the password server."""
507+ path = helpers.Paths({'run_dir': self.tmp})
508+ ds = DataSourceExoscale({}, None, path)
509+ ds._is_platform_viable = lambda: True
510+ expected_answer = "saved_password"
511+ expected_id = "12345"
512+ expected_hostname = "myname"
513+ expected_userdata = "#cloud-config"
514+ httpretty.register_uri(httpretty.GET,
515+ self.userdata_url,
516+ body=expected_userdata)
517+ httpretty.register_uri(httpretty.GET,
518+ self.password_url,
519+ body=expected_answer)
520+ httpretty.register_uri(httpretty.GET,
521+ self.metadata_url,
522+ body="instance-id\nlocal-hostname")
523+ httpretty.register_uri(httpretty.GET,
524+ "{}local-hostname".format(self.metadata_url),
525+ body=expected_hostname)
526+ httpretty.register_uri(httpretty.GET,
527+ "{}instance-id".format(self.metadata_url),
528+ body=expected_id)
529+ self.assertTrue(ds._get_data())
530+ self.assertEqual(ds.userdata_raw.decode("utf-8"), "#cloud-config")
531+ self.assertEqual(ds.metadata, {"instance-id": expected_id,
532+ "local-hostname": expected_hostname})
533+ self.assertEqual(ds.get_config_obj(),
534+ {'cloud_config_modules': [
535+ ["set-passwords", "always"]]})
536+
537+ def test_get_data_no_password(self):
538+ """The datasource conforms to expected behavior when no password is
539+ returned by the password server."""
540+ path = helpers.Paths({'run_dir': self.tmp})
541+ ds = DataSourceExoscale({}, None, path)
542+ ds._is_platform_viable = lambda: True
543+ expected_answer = ""
544+ expected_id = "12345"
545+ expected_hostname = "myname"
546+ expected_userdata = "#cloud-config"
547+ httpretty.register_uri(httpretty.GET,
548+ self.userdata_url,
549+ body=expected_userdata)
550+ httpretty.register_uri(httpretty.GET,
551+ self.password_url,
552+ body=expected_answer)
553+ httpretty.register_uri(httpretty.GET,
554+ self.metadata_url,
555+ body="instance-id\nlocal-hostname")
556+ httpretty.register_uri(httpretty.GET,
557+ "{}local-hostname".format(self.metadata_url),
558+ body=expected_hostname)
559+ httpretty.register_uri(httpretty.GET,
560+ "{}instance-id".format(self.metadata_url),
561+ body=expected_id)
562+ self.assertTrue(ds._get_data())
563+ self.assertEqual(ds.userdata_raw.decode("utf-8"), "#cloud-config")
564+ self.assertEqual(ds.metadata, {"instance-id": expected_id,
565+ "local-hostname": expected_hostname})
566+ self.assertEqual(ds.get_config_obj(),
567+ {'cloud_config_modules': [
568+ ["set-passwords", "always"]]})
569+
570+ @mock.patch('cloudinit.sources.DataSourceExoscale.get_password')
571+ def test_read_metadata_when_password_server_unreachable(self, m_password):
572+ """The read_metadata function returns partial results in case the
573+ password server (only) is unreachable."""
574+ expected_id = "12345"
575+ expected_hostname = "myname"
576+ expected_userdata = "#cloud-config"
577+
578+ m_password.side_effect = requests.Timeout('Fake Connection Timeout')
579+ httpretty.register_uri(httpretty.GET,
580+ self.userdata_url,
581+ body=expected_userdata)
582+ httpretty.register_uri(httpretty.GET,
583+ self.metadata_url,
584+ body="instance-id\nlocal-hostname")
585+ httpretty.register_uri(httpretty.GET,
586+ "{}local-hostname".format(self.metadata_url),
587+ body=expected_hostname)
588+ httpretty.register_uri(httpretty.GET,
589+ "{}instance-id".format(self.metadata_url),
590+ body=expected_id)
591+
592+ result = read_metadata()
593+
594+ self.assertIsNone(result.get("password"))
595+ self.assertEqual(result.get("user-data").decode("utf-8"),
596+ expected_userdata)
597+
598+ def test_non_viable_platform(self):
599+ """The datasource fails fast when the platform is not viable."""
600+ path = helpers.Paths({'run_dir': self.tmp})
601+ ds = DataSourceExoscale({}, None, path)
602+ ds._is_platform_viable = lambda: False
603+ self.assertFalse(ds._get_data())
604diff --git a/tools/ds-identify b/tools/ds-identify
605index 0305e36..e0d4865 100755
606--- a/tools/ds-identify
607+++ b/tools/ds-identify
608@@ -124,7 +124,7 @@ DI_DSNAME=""
609 # be searched if there is no setting found in config.
610 DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
611 CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
612-OVF SmartOS Scaleway Hetzner IBMCloud Oracle"
613+OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale"
614 DI_DSLIST=""
615 DI_MODE=""
616 DI_ON_FOUND=""
617@@ -553,6 +553,11 @@ dscheck_CloudStack() {
618 return $DS_NOT_FOUND
619 }
620
621+dscheck_Exoscale() {
622+ dmi_product_name_matches "Exoscale*" && return $DS_FOUND
623+ return $DS_NOT_FOUND
624+}
625+
626 dscheck_CloudSigma() {
627 # http://paste.ubuntu.com/23624795/
628 dmi_product_name_matches "CloudSigma" && return $DS_FOUND

Subscribers

People subscribed via source and target branches