Merge ~illfelder/cloud-init:master into cloud-init:master
- Git
- lp:~illfelder/cloud-init
- master
- Merge into master
Status: | Merged | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Approved by: | Chad Smith | ||||||||||||
Approved revision: | 2baaf285df5725f6ed42cb49facbddeb0b67b211 | ||||||||||||
Merge reported by: | Chad Smith | ||||||||||||
Merged at revision: | 2d781c6a3e27433b7fa993cd54b269ceb74e10b2 | ||||||||||||
Proposed branch: | ~illfelder/cloud-init:master | ||||||||||||
Merge into: | cloud-init:master | ||||||||||||
Diff against target: |
548 lines (+267/-60) 2 files modified
cloudinit/sources/DataSourceGCE.py (+95/-39) tests/unittests/test_datasource/test_gce.py (+172/-21) |
||||||||||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Scott Moser | Approve | ||
Server Team CI bot | continuous-integration | Approve | |
Dan Watkins | Pending | ||
Review via email: mp+334777@code.launchpad.net |
Commit message
GCE: Improvements and changes to ssh key behavior for default user.
The behavior changes and improvements include:
- Only import keys into the default user that contain the name of the
default user ('ubuntu', or 'centos') or that contain 'cloudinit'.
- Use instance or project level keys based on GCE convention.
- Respect expiration time when keys are set.
Do not import expired keys.
- Support ssh-keys in project level metadata (the GCE default).
As part of this change, we also update the request header when talking
to the metadata server based on the documentation:
https:/
Description of the change
Server Team CI bot (server-team-bot) wrote : | # |
- a7f3ead... by Max Illfelder
-
Fix linter errors.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:a7f3eadf33e
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
It is not the 'ubuntu' user that gets keys.
It is the system configured default user.
The proposal here changes behavior for all Ubuntu instances launched on GCE.
While some users may expect Ubuntu to act consistently with other platforms
when running on GCE other users expect Ubuntu to act consistently across
all platforms.
So some more thought needs to be done here on what happens by default.
There are some specific nits inline below. Definitely bug 1707037 is
the tough portion of this.
- ae79eaa... by Max Illfelder
-
Add documentation for the GCE SSH key format.
Max Illfelder (illfelder) wrote : | # |
I updated the PR for the nits. With regard to the ubuntu user, what logic determines the name of the user account? The user account name should be deterministic (potentially based off of metadata). I don't have a problem adjusting the function's logic to determine the appropriate name in place of "ubuntu".
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:ae79eaa0e29
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
givin config 'cfg' and distro 'distro'
(users, _groups) = ug_util.
(default_user, _user_config) = ug_util.
default_user is then the use name.
I think there may be an issue though in that the cfg that the datasource gets
is only config that is present in /etc/cloud/
the user's provided cloud-config is not yet present (it has not yet been read).
I think that can be made acceptable though, see comment inline.
- 06cd805... by Max Illfelder
-
Include "cloudinit" SSH keys for the default user.
Small style fixes and remove print statements.
- 7885d1b... by Max Illfelder
-
Fix capitalization in comment.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:7885d1bd6e8
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
- 76a2c16... by Max Illfelder
-
Fix broken SSH key test.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:76a2c167c0a
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Scott Moser (smoser) wrote : | # |
I'm not opposed to the name 'cloudinit' as the key.
However, i think we need to also match the default user configured in the system.
- 88054e4... by Max Illfelder
-
Add SSH keys for the default user.
The default user is only available if the distro config exists.
Max Illfelder (illfelder) wrote : | # |
Apologies for the delay. I updated the change based on your suggestions, but I am not able to validate the default user portion works; when I run the tests locally, distro is not set.
Server Team CI bot (server-team-bot) wrote : | # |
FAILED: Continuous integration, rev:88054e42715
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild:
https:/
- 5b73ba7... by Max Illfelder
-
Fix a linter error.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:5b73ba7bcb3
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Max Illfelder (illfelder) wrote : | # |
Friendly ping?
Scott Moser (smoser) wrote : | # |
Max,
thanks for following up. sorry for slow reply.
I spent some time looking at this, and put together:
http://
What is there is
- store the un-filtered public keys in an actual list
named metadata[
as 'public-keys' is confusing since it is used by other datasources.
- store a list in public-keys-all rather than a newline separated string.
- update tests to support building the datasource with a distro and
also a default user.
- add a test that verifies the default user is read.
I'd like for you to add some tests of _parse_public_keys for different
content, including expired keys.
You can just add a class that tests that one function.
Let me know if you disagree with any of the changes I've mde there, or pull
them into your branch.
- 6923403... by Max Illfelder
-
Improve testing and code review comment updates
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:6923403a135
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- 2baaf28... by Max Illfelder
-
Improve testing for key expiration and parsing.
Max Illfelder (illfelder) wrote : | # |
The most recent update should address each of the items you previously enumerated. Please take another look.
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:2baaf285df5
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
SUCCESS: MAAS Compatability Testing
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
Max Illfelder (illfelder) wrote : | # |
Hi Scott, are any other changes needed?
Max Illfelder (illfelder) wrote : | # |
Any update?
Scott Moser (smoser) wrote : | # |
Max, thanks for the work.
Looks good.
Ryan Harper (raharper) wrote : | # |
Rebased and updated to address feedback
Preview Diff
1 | diff --git a/cloudinit/sources/DataSourceGCE.py b/cloudinit/sources/DataSourceGCE.py |
2 | index ad6dae3..2da34a9 100644 |
3 | --- a/cloudinit/sources/DataSourceGCE.py |
4 | +++ b/cloudinit/sources/DataSourceGCE.py |
5 | @@ -2,8 +2,12 @@ |
6 | # |
7 | # This file is part of cloud-init. See LICENSE file for license information. |
8 | |
9 | +import datetime |
10 | +import json |
11 | + |
12 | from base64 import b64decode |
13 | |
14 | +from cloudinit.distros import ug_util |
15 | from cloudinit import log as logging |
16 | from cloudinit import sources |
17 | from cloudinit import url_helper |
18 | @@ -17,16 +21,18 @@ REQUIRED_FIELDS = ('instance-id', 'availability-zone', 'local-hostname') |
19 | |
20 | |
21 | class GoogleMetadataFetcher(object): |
22 | - headers = {'X-Google-Metadata-Request': 'True'} |
23 | + headers = {'Metadata-Flavor': 'Google'} |
24 | |
25 | def __init__(self, metadata_address): |
26 | self.metadata_address = metadata_address |
27 | |
28 | - def get_value(self, path, is_text): |
29 | + def get_value(self, path, is_text, is_recursive=False): |
30 | value = None |
31 | try: |
32 | - resp = url_helper.readurl(url=self.metadata_address + path, |
33 | - headers=self.headers) |
34 | + url = self.metadata_address + path |
35 | + if is_recursive: |
36 | + url += '/?recursive=True' |
37 | + resp = url_helper.readurl(url=url, headers=self.headers) |
38 | except url_helper.UrlError as exc: |
39 | msg = "url %s raised exception %s" |
40 | LOG.debug(msg, path, exc) |
41 | @@ -35,7 +41,7 @@ class GoogleMetadataFetcher(object): |
42 | if is_text: |
43 | value = util.decode_binary(resp.contents) |
44 | else: |
45 | - value = resp.contents |
46 | + value = resp.contents.decode('utf-8') |
47 | else: |
48 | LOG.debug("url %s returned code %s", path, resp.code) |
49 | return value |
50 | @@ -47,6 +53,10 @@ class DataSourceGCE(sources.DataSource): |
51 | |
52 | def __init__(self, sys_cfg, distro, paths): |
53 | sources.DataSource.__init__(self, sys_cfg, distro, paths) |
54 | + self.default_user = None |
55 | + if distro: |
56 | + (users, _groups) = ug_util.normalize_users_groups(sys_cfg, distro) |
57 | + (self.default_user, _user_config) = ug_util.extract_default(users) |
58 | self.metadata = dict() |
59 | self.ds_cfg = util.mergemanydict([ |
60 | util.get_cfg_by_path(sys_cfg, ["datasource", "GCE"], {}), |
61 | @@ -70,17 +80,18 @@ class DataSourceGCE(sources.DataSource): |
62 | |
63 | @property |
64 | def launch_index(self): |
65 | - # GCE does not provide lauch_index property |
66 | + # GCE does not provide lauch_index property. |
67 | return None |
68 | |
69 | def get_instance_id(self): |
70 | return self.metadata['instance-id'] |
71 | |
72 | def get_public_ssh_keys(self): |
73 | - return self.metadata['public-keys'] |
74 | + public_keys_data = self.metadata['public-keys-data'] |
75 | + return _parse_public_keys(public_keys_data, self.default_user) |
76 | |
77 | def get_hostname(self, fqdn=False, resolve_ip=False): |
78 | - # GCE has long FDQN's and has asked for short hostnames |
79 | + # GCE has long FDQN's and has asked for short hostnames. |
80 | return self.metadata['local-hostname'].split('.')[0] |
81 | |
82 | @property |
83 | @@ -92,15 +103,58 @@ class DataSourceGCE(sources.DataSource): |
84 | return self.availability_zone.rsplit('-', 1)[0] |
85 | |
86 | |
87 | -def _trim_key(public_key): |
88 | - # GCE takes sshKeys attribute in the format of '<user>:<public_key>' |
89 | - # so we have to trim each key to remove the username part |
90 | +def _has_expired(public_key): |
91 | + # Check whether an SSH key is expired. Public key input is a single SSH |
92 | + # public key in the GCE specific key format documented here: |
93 | + # https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#sshkeyformat |
94 | + try: |
95 | + # Check for the Google-specific schema identifier. |
96 | + schema, json_str = public_key.split(None, 3)[2:] |
97 | + except (ValueError, AttributeError): |
98 | + return False |
99 | + |
100 | + # Do not expire keys if they do not have the expected schema identifier. |
101 | + if schema != 'google-ssh': |
102 | + return False |
103 | + |
104 | + try: |
105 | + json_obj = json.loads(json_str) |
106 | + except ValueError: |
107 | + return False |
108 | + |
109 | + # Do not expire keys if there is no expriation timestamp. |
110 | + if 'expireOn' not in json_obj: |
111 | + return False |
112 | + |
113 | + expire_str = json_obj['expireOn'] |
114 | + format_str = '%Y-%m-%dT%H:%M:%S+0000' |
115 | try: |
116 | - index = public_key.index(':') |
117 | - if index > 0: |
118 | - return public_key[(index + 1):] |
119 | - except Exception: |
120 | - return public_key |
121 | + expire_time = datetime.datetime.strptime(expire_str, format_str) |
122 | + except ValueError: |
123 | + return False |
124 | + |
125 | + # Expire the key if and only if we have exceeded the expiration timestamp. |
126 | + return datetime.datetime.utcnow() > expire_time |
127 | + |
128 | + |
129 | +def _parse_public_keys(public_keys_data, default_user=None): |
130 | + # Parse the SSH key data for the default user account. Public keys input is |
131 | + # a list containing SSH public keys in the GCE specific key format |
132 | + # documented here: |
133 | + # https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys#sshkeyformat |
134 | + public_keys = [] |
135 | + if not public_keys_data: |
136 | + return public_keys |
137 | + for public_key in public_keys_data: |
138 | + if not public_key or not all(ord(c) < 128 for c in public_key): |
139 | + continue |
140 | + split_public_key = public_key.split(':', 1) |
141 | + if len(split_public_key) != 2: |
142 | + continue |
143 | + user, key = split_public_key |
144 | + if user in ('cloudinit', default_user) and not _has_expired(key): |
145 | + public_keys.append(key) |
146 | + return public_keys |
147 | |
148 | |
149 | def read_md(address=None, platform_check=True): |
150 | @@ -116,31 +170,28 @@ def read_md(address=None, platform_check=True): |
151 | ret['reason'] = "Not running on GCE." |
152 | return ret |
153 | |
154 | - # if we cannot resolve the metadata server, then no point in trying |
155 | + # If we cannot resolve the metadata server, then no point in trying. |
156 | if not util.is_resolvable_url(address): |
157 | LOG.debug("%s is not resolvable", address) |
158 | ret['reason'] = 'address "%s" is not resolvable' % address |
159 | return ret |
160 | |
161 | - # url_map: (our-key, path, required, is_text) |
162 | + # url_map: (our-key, path, required, is_text, is_recursive) |
163 | url_map = [ |
164 | - ('instance-id', ('instance/id',), True, True), |
165 | - ('availability-zone', ('instance/zone',), True, True), |
166 | - ('local-hostname', ('instance/hostname',), True, True), |
167 | - ('public-keys', ('project/attributes/sshKeys', |
168 | - 'instance/attributes/ssh-keys'), False, True), |
169 | - ('user-data', ('instance/attributes/user-data',), False, False), |
170 | - ('user-data-encoding', ('instance/attributes/user-data-encoding',), |
171 | - False, True), |
172 | + ('instance-id', ('instance/id',), True, True, False), |
173 | + ('availability-zone', ('instance/zone',), True, True, False), |
174 | + ('local-hostname', ('instance/hostname',), True, True, False), |
175 | + ('instance-data', ('instance/attributes',), False, False, True), |
176 | + ('project-data', ('project/attributes',), False, False, True), |
177 | ] |
178 | |
179 | metadata_fetcher = GoogleMetadataFetcher(address) |
180 | md = {} |
181 | - # iterate over url_map keys to get metadata items |
182 | - for (mkey, paths, required, is_text) in url_map: |
183 | + # Iterate over url_map keys to get metadata items. |
184 | + for (mkey, paths, required, is_text, is_recursive) in url_map: |
185 | value = None |
186 | for path in paths: |
187 | - new_value = metadata_fetcher.get_value(path, is_text) |
188 | + new_value = metadata_fetcher.get_value(path, is_text, is_recursive) |
189 | if new_value is not None: |
190 | value = new_value |
191 | if required and value is None: |
192 | @@ -149,17 +200,23 @@ def read_md(address=None, platform_check=True): |
193 | return ret |
194 | md[mkey] = value |
195 | |
196 | - if md['public-keys']: |
197 | - lines = md['public-keys'].splitlines() |
198 | - md['public-keys'] = [_trim_key(k) for k in lines] |
199 | + instance_data = json.loads(md['instance-data'] or '{}') |
200 | + project_data = json.loads(md['project-data'] or '{}') |
201 | + valid_keys = [instance_data.get('sshKeys'), instance_data.get('ssh-keys')] |
202 | + block_project = instance_data.get('block-project-ssh-keys', '').lower() |
203 | + if block_project != 'true' and not instance_data.get('sshKeys'): |
204 | + valid_keys.append(project_data.get('ssh-keys')) |
205 | + valid_keys.append(project_data.get('sshKeys')) |
206 | + public_keys_data = '\n'.join([key for key in valid_keys if key]) |
207 | + md['public-keys-data'] = public_keys_data.splitlines() |
208 | |
209 | if md['availability-zone']: |
210 | md['availability-zone'] = md['availability-zone'].split('/')[-1] |
211 | |
212 | - encoding = md.get('user-data-encoding') |
213 | + encoding = instance_data.get('user-data-encoding') |
214 | if encoding: |
215 | if encoding == 'base64': |
216 | - md['user-data'] = b64decode(md['user-data']) |
217 | + md['user-data'] = b64decode(instance_data.get('user-data')) |
218 | else: |
219 | LOG.warning('unknown user-data-encoding: %s, ignoring', encoding) |
220 | |
221 | @@ -188,20 +245,19 @@ def platform_reports_gce(): |
222 | return False |
223 | |
224 | |
225 | -# Used to match classes to dependencies |
226 | +# Used to match classes to dependencies. |
227 | datasources = [ |
228 | (DataSourceGCE, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)), |
229 | ] |
230 | |
231 | |
232 | -# Return a list of data sources that match this set of dependencies |
233 | +# Return a list of data sources that match this set of dependencies. |
234 | def get_datasource_list(depends): |
235 | return sources.list_from_depends(depends, datasources) |
236 | |
237 | |
238 | if __name__ == "__main__": |
239 | import argparse |
240 | - import json |
241 | import sys |
242 | |
243 | from base64 import b64encode |
244 | @@ -217,7 +273,7 @@ if __name__ == "__main__": |
245 | data = read_md(address=args.endpoint, platform_check=args.platform_check) |
246 | if 'user-data' in data: |
247 | # user-data is bytes not string like other things. Handle it specially. |
248 | - # if it can be represented as utf-8 then do so. Otherwise print base64 |
249 | + # If it can be represented as utf-8 then do so. Otherwise print base64 |
250 | # encoded value in the key user-data-b64. |
251 | try: |
252 | data['user-data'] = data['user-data'].decode() |
253 | @@ -225,7 +281,7 @@ if __name__ == "__main__": |
254 | sys.stderr.write("User-data cannot be decoded. " |
255 | "Writing as base64\n") |
256 | del data['user-data'] |
257 | - # b64encode returns a bytes value. decode to get the string. |
258 | + # b64encode returns a bytes value. Decode to get the string. |
259 | data['user-data-b64'] = b64encode(data['user-data']).decode() |
260 | |
261 | print(json.dumps(data, indent=1, sort_keys=True, separators=(',', ': '))) |
262 | diff --git a/tests/unittests/test_datasource/test_gce.py b/tests/unittests/test_datasource/test_gce.py |
263 | index 82c788d..12d6800 100644 |
264 | --- a/tests/unittests/test_datasource/test_gce.py |
265 | +++ b/tests/unittests/test_datasource/test_gce.py |
266 | @@ -4,13 +4,16 @@ |
267 | # |
268 | # This file is part of cloud-init. See LICENSE file for license information. |
269 | |
270 | +import datetime |
271 | import httpretty |
272 | +import json |
273 | import mock |
274 | import re |
275 | |
276 | from base64 import b64encode, b64decode |
277 | from six.moves.urllib_parse import urlparse |
278 | |
279 | +from cloudinit import distros |
280 | from cloudinit import helpers |
281 | from cloudinit import settings |
282 | from cloudinit.sources import DataSourceGCE |
283 | @@ -21,10 +24,7 @@ from cloudinit.tests import helpers as test_helpers |
284 | GCE_META = { |
285 | 'instance/id': '123', |
286 | 'instance/zone': 'foo/bar', |
287 | - 'project/attributes/sshKeys': 'user:ssh-rsa AA2..+aRD0fyVw== root@server', |
288 | 'instance/hostname': 'server.project-foo.local', |
289 | - # UnicodeDecodeError below if set to ds.userdata instead of userdata_raw |
290 | - 'instance/attributes/user-data': b'/bin/echo \xff\n', |
291 | } |
292 | |
293 | GCE_META_PARTIAL = { |
294 | @@ -37,11 +37,13 @@ GCE_META_ENCODING = { |
295 | 'instance/id': '12345', |
296 | 'instance/hostname': 'server.project-baz.local', |
297 | 'instance/zone': 'baz/bang', |
298 | - 'instance/attributes/user-data': b64encode(b'/bin/echo baz\n'), |
299 | - 'instance/attributes/user-data-encoding': 'base64', |
300 | + 'instance/attributes': { |
301 | + 'user-data': b64encode(b'/bin/echo baz\n').decode('utf-8'), |
302 | + 'user-data-encoding': 'base64', |
303 | + } |
304 | } |
305 | |
306 | -HEADERS = {'X-Google-Metadata-Request': 'True'} |
307 | +HEADERS = {'Metadata-Flavor': 'Google'} |
308 | MD_URL_RE = re.compile( |
309 | r'http://metadata.google.internal/computeMetadata/v1/.*') |
310 | |
311 | @@ -54,10 +56,15 @@ def _set_mock_metadata(gce_meta=None): |
312 | url_path = urlparse(uri).path |
313 | if url_path.startswith('/computeMetadata/v1/'): |
314 | path = url_path.split('/computeMetadata/v1/')[1:][0] |
315 | + recursive = path.endswith('/') |
316 | + path = path.rstrip('/') |
317 | else: |
318 | path = None |
319 | if path in gce_meta: |
320 | - return (200, headers, gce_meta.get(path)) |
321 | + response = gce_meta.get(path) |
322 | + if recursive: |
323 | + response = json.dumps(response) |
324 | + return (200, headers, response) |
325 | else: |
326 | return (404, headers, '') |
327 | |
328 | @@ -69,6 +76,16 @@ def _set_mock_metadata(gce_meta=None): |
329 | @httpretty.activate |
330 | class TestDataSourceGCE(test_helpers.HttprettyTestCase): |
331 | |
332 | + def _make_distro(self, dtype, def_user=None): |
333 | + cfg = dict(settings.CFG_BUILTIN) |
334 | + cfg['system_info']['distro'] = dtype |
335 | + paths = helpers.Paths(cfg['system_info']['paths']) |
336 | + distro_cls = distros.fetch(dtype) |
337 | + if def_user: |
338 | + cfg['system_info']['default_user'] = def_user.copy() |
339 | + distro = distro_cls(dtype, cfg['system_info'], paths) |
340 | + return distro |
341 | + |
342 | def setUp(self): |
343 | tmp = self.tmp_dir() |
344 | self.ds = DataSourceGCE.DataSourceGCE( |
345 | @@ -90,6 +107,10 @@ class TestDataSourceGCE(test_helpers.HttprettyTestCase): |
346 | self.assertDictContainsSubset(HEADERS, req_header) |
347 | |
348 | def test_metadata(self): |
349 | + # UnicodeDecodeError if set to ds.userdata instead of userdata_raw |
350 | + meta = GCE_META.copy() |
351 | + meta['instance/attributes/user-data'] = b'/bin/echo \xff\n' |
352 | + |
353 | _set_mock_metadata() |
354 | self.ds.get_data() |
355 | |
356 | @@ -118,8 +139,8 @@ class TestDataSourceGCE(test_helpers.HttprettyTestCase): |
357 | _set_mock_metadata(GCE_META_ENCODING) |
358 | self.ds.get_data() |
359 | |
360 | - decoded = b64decode( |
361 | - GCE_META_ENCODING.get('instance/attributes/user-data')) |
362 | + instance_data = GCE_META_ENCODING.get('instance/attributes') |
363 | + decoded = b64decode(instance_data.get('user-data')) |
364 | self.assertEqual(decoded, self.ds.get_userdata_raw()) |
365 | |
366 | def test_missing_required_keys_return_false(self): |
367 | @@ -131,33 +152,124 @@ class TestDataSourceGCE(test_helpers.HttprettyTestCase): |
368 | self.assertEqual(False, self.ds.get_data()) |
369 | httpretty.reset() |
370 | |
371 | - def test_project_level_ssh_keys_are_used(self): |
372 | + def test_no_ssh_keys_metadata(self): |
373 | _set_mock_metadata() |
374 | self.ds.get_data() |
375 | + self.assertEqual([], self.ds.get_public_ssh_keys()) |
376 | + |
377 | + def test_cloudinit_ssh_keys(self): |
378 | + valid_key = 'ssh-rsa VALID {0}' |
379 | + invalid_key = 'ssh-rsa INVALID {0}' |
380 | + project_attributes = { |
381 | + 'sshKeys': '\n'.join([ |
382 | + 'cloudinit:{0}'.format(valid_key.format(0)), |
383 | + 'user:{0}'.format(invalid_key.format(0)), |
384 | + ]), |
385 | + 'ssh-keys': '\n'.join([ |
386 | + 'cloudinit:{0}'.format(valid_key.format(1)), |
387 | + 'user:{0}'.format(invalid_key.format(1)), |
388 | + ]), |
389 | + } |
390 | + instance_attributes = { |
391 | + 'ssh-keys': '\n'.join([ |
392 | + 'cloudinit:{0}'.format(valid_key.format(2)), |
393 | + 'user:{0}'.format(invalid_key.format(2)), |
394 | + ]), |
395 | + 'block-project-ssh-keys': 'False', |
396 | + } |
397 | + |
398 | + meta = GCE_META.copy() |
399 | + meta['project/attributes'] = project_attributes |
400 | + meta['instance/attributes'] = instance_attributes |
401 | + |
402 | + _set_mock_metadata(meta) |
403 | + self.ds.get_data() |
404 | + |
405 | + expected = [valid_key.format(key) for key in range(3)] |
406 | + self.assertEqual(set(expected), set(self.ds.get_public_ssh_keys())) |
407 | + |
408 | + @mock.patch("cloudinit.sources.DataSourceGCE.ug_util") |
409 | + def test_default_user_ssh_keys(self, mock_ug_util): |
410 | + mock_ug_util.normalize_users_groups.return_value = None, None |
411 | + mock_ug_util.extract_default.return_value = 'ubuntu', None |
412 | + ubuntu_ds = DataSourceGCE.DataSourceGCE( |
413 | + settings.CFG_BUILTIN, self._make_distro('ubuntu'), |
414 | + helpers.Paths({})) |
415 | + |
416 | + valid_key = 'ssh-rsa VALID {0}' |
417 | + invalid_key = 'ssh-rsa INVALID {0}' |
418 | + project_attributes = { |
419 | + 'sshKeys': '\n'.join([ |
420 | + 'ubuntu:{0}'.format(valid_key.format(0)), |
421 | + 'user:{0}'.format(invalid_key.format(0)), |
422 | + ]), |
423 | + 'ssh-keys': '\n'.join([ |
424 | + 'ubuntu:{0}'.format(valid_key.format(1)), |
425 | + 'user:{0}'.format(invalid_key.format(1)), |
426 | + ]), |
427 | + } |
428 | + instance_attributes = { |
429 | + 'ssh-keys': '\n'.join([ |
430 | + 'ubuntu:{0}'.format(valid_key.format(2)), |
431 | + 'user:{0}'.format(invalid_key.format(2)), |
432 | + ]), |
433 | + 'block-project-ssh-keys': 'False', |
434 | + } |
435 | |
436 | - # we expect a list of public ssh keys with user names stripped |
437 | - self.assertEqual(['ssh-rsa AA2..+aRD0fyVw== root@server'], |
438 | - self.ds.get_public_ssh_keys()) |
439 | + meta = GCE_META.copy() |
440 | + meta['project/attributes'] = project_attributes |
441 | + meta['instance/attributes'] = instance_attributes |
442 | + |
443 | + _set_mock_metadata(meta) |
444 | + ubuntu_ds.get_data() |
445 | + |
446 | + expected = [valid_key.format(key) for key in range(3)] |
447 | + self.assertEqual(set(expected), set(ubuntu_ds.get_public_ssh_keys())) |
448 | + |
449 | + def test_instance_ssh_keys_override(self): |
450 | + valid_key = 'ssh-rsa VALID {0}' |
451 | + invalid_key = 'ssh-rsa INVALID {0}' |
452 | + project_attributes = { |
453 | + 'sshKeys': 'cloudinit:{0}'.format(invalid_key.format(0)), |
454 | + 'ssh-keys': 'cloudinit:{0}'.format(invalid_key.format(1)), |
455 | + } |
456 | + instance_attributes = { |
457 | + 'sshKeys': 'cloudinit:{0}'.format(valid_key.format(0)), |
458 | + 'ssh-keys': 'cloudinit:{0}'.format(valid_key.format(1)), |
459 | + 'block-project-ssh-keys': 'False', |
460 | + } |
461 | |
462 | - def test_instance_level_ssh_keys_are_used(self): |
463 | - key_content = 'ssh-rsa JustAUser root@server' |
464 | meta = GCE_META.copy() |
465 | - meta['instance/attributes/ssh-keys'] = 'user:{0}'.format(key_content) |
466 | + meta['project/attributes'] = project_attributes |
467 | + meta['instance/attributes'] = instance_attributes |
468 | |
469 | _set_mock_metadata(meta) |
470 | self.ds.get_data() |
471 | |
472 | - self.assertIn(key_content, self.ds.get_public_ssh_keys()) |
473 | + expected = [valid_key.format(key) for key in range(2)] |
474 | + self.assertEqual(set(expected), set(self.ds.get_public_ssh_keys())) |
475 | + |
476 | + def test_block_project_ssh_keys_override(self): |
477 | + valid_key = 'ssh-rsa VALID {0}' |
478 | + invalid_key = 'ssh-rsa INVALID {0}' |
479 | + project_attributes = { |
480 | + 'sshKeys': 'cloudinit:{0}'.format(invalid_key.format(0)), |
481 | + 'ssh-keys': 'cloudinit:{0}'.format(invalid_key.format(1)), |
482 | + } |
483 | + instance_attributes = { |
484 | + 'ssh-keys': 'cloudinit:{0}'.format(valid_key.format(0)), |
485 | + 'block-project-ssh-keys': 'True', |
486 | + } |
487 | |
488 | - def test_instance_level_keys_replace_project_level_keys(self): |
489 | - key_content = 'ssh-rsa JustAUser root@server' |
490 | meta = GCE_META.copy() |
491 | - meta['instance/attributes/ssh-keys'] = 'user:{0}'.format(key_content) |
492 | + meta['project/attributes'] = project_attributes |
493 | + meta['instance/attributes'] = instance_attributes |
494 | |
495 | _set_mock_metadata(meta) |
496 | self.ds.get_data() |
497 | |
498 | - self.assertEqual([key_content], self.ds.get_public_ssh_keys()) |
499 | + expected = [valid_key.format(0)] |
500 | + self.assertEqual(set(expected), set(self.ds.get_public_ssh_keys())) |
501 | |
502 | def test_only_last_part_of_zone_used_for_availability_zone(self): |
503 | _set_mock_metadata() |
504 | @@ -172,5 +284,44 @@ class TestDataSourceGCE(test_helpers.HttprettyTestCase): |
505 | self.assertEqual(False, ret) |
506 | m_fetcher.assert_not_called() |
507 | |
508 | + def test_has_expired(self): |
509 | + |
510 | + def _get_timestamp(days): |
511 | + format_str = '%Y-%m-%dT%H:%M:%S+0000' |
512 | + today = datetime.datetime.now() |
513 | + timestamp = today + datetime.timedelta(days=days) |
514 | + return timestamp.strftime(format_str) |
515 | + |
516 | + past = _get_timestamp(-1) |
517 | + future = _get_timestamp(1) |
518 | + ssh_keys = { |
519 | + None: False, |
520 | + '': False, |
521 | + 'Invalid': False, |
522 | + 'user:ssh-rsa key user@domain.com': False, |
523 | + 'user:ssh-rsa key google {"expireOn":"%s"}' % past: False, |
524 | + 'user:ssh-rsa key google-ssh': False, |
525 | + 'user:ssh-rsa key google-ssh {invalid:json}': False, |
526 | + 'user:ssh-rsa key google-ssh {"userName":"user"}': False, |
527 | + 'user:ssh-rsa key google-ssh {"expireOn":"invalid"}': False, |
528 | + 'user:xyz key google-ssh {"expireOn":"%s"}' % future: False, |
529 | + 'user:xyz key google-ssh {"expireOn":"%s"}' % past: True, |
530 | + } |
531 | + |
532 | + for key, expired in ssh_keys.items(): |
533 | + self.assertEqual(DataSourceGCE._has_expired(key), expired) |
534 | + |
535 | + def test_parse_public_keys_non_ascii(self): |
536 | + public_key_data = [ |
537 | + 'cloudinit:rsa ssh-ke%s invalid' % chr(165), |
538 | + 'use%sname:rsa ssh-key' % chr(174), |
539 | + 'cloudinit:test 1', |
540 | + 'default:test 2', |
541 | + 'user:test 3', |
542 | + ] |
543 | + expected = ['test 1', 'test 2'] |
544 | + found = DataSourceGCE._parse_public_keys( |
545 | + public_key_data, default_user='default') |
546 | + self.assertEqual(sorted(found), sorted(expected)) |
547 | |
548 | # vi: ts=4 expandtab |
FAILED: Continuous integration, rev:977d55d946c 01e8f571273b624 69a278886077b4 /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 581/
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Unit & Style Tests
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 581/rebuild
https:/