Merge lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host into lp:~openstack-charmers-archive/charms/trusty/ceilometer/next
- Trusty Tahr (14.04)
- public-endpoint-host
- Merge into next
Status: | Merged |
---|---|
Merged at revision: | 85 |
Proposed branch: | lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host |
Merge into: | lp:~openstack-charmers-archive/charms/trusty/ceilometer/next |
Diff against target: |
924 lines (+404/-103) 12 files modified
config.yaml (+12/-0) hooks/charmhelpers/contrib/hahelpers/cluster.py (+25/-0) hooks/charmhelpers/contrib/openstack/ip.py (+49/-44) hooks/charmhelpers/contrib/openstack/neutron.py (+10/-5) hooks/charmhelpers/contrib/openstack/utils.py (+65/-18) hooks/charmhelpers/contrib/python/packages.py (+28/-5) hooks/charmhelpers/core/hookenv.py (+147/-10) hooks/charmhelpers/core/host.py (+1/-1) hooks/charmhelpers/core/services/base.py (+32/-11) hooks/charmhelpers/fetch/__init__.py (+1/-1) hooks/charmhelpers/fetch/giturl.py (+7/-5) unit_tests/test_ceilometer_hooks.py (+27/-3) |
To merge this branch: | bzr merge lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
OpenStack Charmers | Pending | ||
Review via email: mp+261004@code.launchpad.net |
Commit message
Description of the change
Provides a config option which allows the user to specify the public hostname used to advertise to keystone when creating endpoints.
uosci-testing-bot (uosci-testing-bot) wrote : | # |
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #4722 ceilometer-next for billy-olsen mp261004
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #4448 ceilometer-next for billy-olsen mp261004
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
ERROR subprocess encountered error code 1
ERROR:root:Make target returned non-zero.
Full amulet test output: http://
Build: http://
Ryan Beisner (1chb1n) wrote : | # |
Amulet test also fails for trusty-icehouse ceilometer/next. It looks like either mongodb or ceilometer have had a change in expected behaviors and/or relations since the 15.04 release, and the functional test didn't get updated to reflect that.
So this failure is not specific to your proposal, but until I can track down the change that broke the test, your proposal can't be validated via the functional tests.
00:28:14.389 unexpected relation data in mongodb database - key 'replset' does not exist
Ryan Beisner (1chb1n) wrote : | # |
FYI, tracking bug for the amulet test. I'll work on that. https:/
Ryan Beisner (1chb1n) wrote : | # |
2015-06-04 11:19:46,514 _validate_dict_data DEBUG: actual: {u'private-
2015-06-04 11:19:46,514 _validate_dict_data DEBUG: expected: {'private-address': <bound method OpenStackAmulet
unexpected relation data in mongodb database - key 'replset' does not exist
Ryan Beisner (1chb1n) wrote : | # |
FYI, ceilometer/next amulet tests fixed and landed re: bug 1461911.
You may want to pull those changes into your branch.
- 87. By Billy Olsen
-
Merge with /next
- 88. By Billy Olsen
-
c-h sync
- 89. By Billy Olsen
-
Fix broken unit test from c-h sync
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #4745 ceilometer-next for billy-olsen mp261004
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #5066 ceilometer-next for billy-olsen mp261004
LINT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #5067 ceilometer-next for billy-olsen mp261004
LINT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #4746 ceilometer-next for billy-olsen mp261004
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #4473 ceilometer-next for billy-olsen mp261004
AMULET OK: passed
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #4474 ceilometer-next for billy-olsen mp261004
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.
Full amulet test output: http://
Build: http://
Corey Bryant (corey.bryant) wrote : | # |
Looks good, but are amulet tests still failing?
Billy Olsen (billy-olsen) wrote : | # |
Amulet tests were due to timeout unfortunately
Corey Bryant (corey.bryant) wrote : | # |
Fyi, I ran the amulet tests manually and they were successful:
Results: 6 passed, 0 failed, 0 errored
Preview Diff
1 | === modified file 'config.yaml' |
2 | --- config.yaml 2015-02-19 02:03:06 +0000 |
3 | +++ config.yaml 2015-06-04 23:04:28 +0000 |
4 | @@ -100,6 +100,18 @@ |
5 | 192.168.0.0/24) |
6 | . |
7 | This network will be used for public endpoints. |
8 | + os-public-hostname: |
9 | + type: string |
10 | + default: |
11 | + description: | |
12 | + The hostname or address of the public endpoints created for ceilometer |
13 | + in the keystone identity provider. |
14 | + . |
15 | + This value will be used for public endpoints. For example, an |
16 | + os-public-hostname set to 'ceilometer.example.com' with ssl enabled will |
17 | + create the following public endpoints for ceilometer. |
18 | + . |
19 | + https://ceilometer.example.com:8777/ |
20 | # HA configuration settings |
21 | vip: |
22 | type: string |
23 | |
24 | === modified file 'hooks/charmhelpers/contrib/hahelpers/cluster.py' |
25 | --- hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-03-13 12:57:45 +0000 |
26 | +++ hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-06-04 23:04:28 +0000 |
27 | @@ -52,6 +52,8 @@ |
28 | bool_from_string, |
29 | ) |
30 | |
31 | +DC_RESOURCE_NAME = 'DC' |
32 | + |
33 | |
34 | class HAIncompleteConfig(Exception): |
35 | pass |
36 | @@ -95,6 +97,27 @@ |
37 | return False |
38 | |
39 | |
40 | +def is_crm_dc(): |
41 | + """ |
42 | + Determine leadership by querying the pacemaker Designated Controller |
43 | + """ |
44 | + cmd = ['crm', 'status'] |
45 | + try: |
46 | + status = subprocess.check_output(cmd, stderr=subprocess.STDOUT) |
47 | + if not isinstance(status, six.text_type): |
48 | + status = six.text_type(status, "utf-8") |
49 | + except subprocess.CalledProcessError: |
50 | + return False |
51 | + current_dc = '' |
52 | + for line in status.split('\n'): |
53 | + if line.startswith('Current DC'): |
54 | + # Current DC: juju-lytrusty-machine-2 (168108163) - partition with quorum |
55 | + current_dc = line.split(':')[1].split()[0] |
56 | + if current_dc == get_unit_hostname(): |
57 | + return True |
58 | + return False |
59 | + |
60 | + |
61 | @retry_on_exception(5, base_delay=2, exc_type=CRMResourceNotFound) |
62 | def is_crm_leader(resource, retry=False): |
63 | """ |
64 | @@ -104,6 +127,8 @@ |
65 | We allow this operation to be retried to avoid the possibility of getting a |
66 | false negative. See LP #1396246 for more info. |
67 | """ |
68 | + if resource == DC_RESOURCE_NAME: |
69 | + return is_crm_dc() |
70 | cmd = ['crm', 'resource', 'show', resource] |
71 | try: |
72 | status = subprocess.check_output(cmd, stderr=subprocess.STDOUT) |
73 | |
74 | === modified file 'hooks/charmhelpers/contrib/openstack/ip.py' |
75 | --- hooks/charmhelpers/contrib/openstack/ip.py 2015-03-13 12:57:45 +0000 |
76 | +++ hooks/charmhelpers/contrib/openstack/ip.py 2015-06-04 23:04:28 +0000 |
77 | @@ -17,6 +17,7 @@ |
78 | from charmhelpers.core.hookenv import ( |
79 | config, |
80 | unit_get, |
81 | + service_name, |
82 | ) |
83 | from charmhelpers.contrib.network.ip import ( |
84 | get_address_in_network, |
85 | @@ -26,8 +27,6 @@ |
86 | ) |
87 | from charmhelpers.contrib.hahelpers.cluster import is_clustered |
88 | |
89 | -from functools import partial |
90 | - |
91 | PUBLIC = 'public' |
92 | INTERNAL = 'int' |
93 | ADMIN = 'admin' |
94 | @@ -35,15 +34,18 @@ |
95 | ADDRESS_MAP = { |
96 | PUBLIC: { |
97 | 'config': 'os-public-network', |
98 | - 'fallback': 'public-address' |
99 | + 'fallback': 'public-address', |
100 | + 'override': 'os-public-hostname', |
101 | }, |
102 | INTERNAL: { |
103 | 'config': 'os-internal-network', |
104 | - 'fallback': 'private-address' |
105 | + 'fallback': 'private-address', |
106 | + 'override': 'os-internal-hostname', |
107 | }, |
108 | ADMIN: { |
109 | 'config': 'os-admin-network', |
110 | - 'fallback': 'private-address' |
111 | + 'fallback': 'private-address', |
112 | + 'override': 'os-admin-hostname', |
113 | } |
114 | } |
115 | |
116 | @@ -57,15 +59,50 @@ |
117 | :param endpoint_type: str endpoint type to resolve. |
118 | :param returns: str base URL for services on the current service unit. |
119 | """ |
120 | - scheme = 'http' |
121 | - if 'https' in configs.complete_contexts(): |
122 | - scheme = 'https' |
123 | + scheme = _get_scheme(configs) |
124 | + |
125 | address = resolve_address(endpoint_type) |
126 | if is_ipv6(address): |
127 | address = "[{}]".format(address) |
128 | + |
129 | return '%s://%s' % (scheme, address) |
130 | |
131 | |
132 | +def _get_scheme(configs): |
133 | + """Returns the scheme to use for the url (either http or https) |
134 | + depending upon whether https is in the configs value. |
135 | + |
136 | + :param configs: OSTemplateRenderer config templating object to inspect |
137 | + for a complete https context. |
138 | + :returns: either 'http' or 'https' depending on whether https is |
139 | + configured within the configs context. |
140 | + """ |
141 | + scheme = 'http' |
142 | + if configs and 'https' in configs.complete_contexts(): |
143 | + scheme = 'https' |
144 | + return scheme |
145 | + |
146 | + |
147 | +def _get_address_override(endpoint_type=PUBLIC): |
148 | + """Returns any address overrides that the user has defined based on the |
149 | + endpoint type. |
150 | + |
151 | + Note: this function allows for the service name to be inserted into the |
152 | + address if the user specifies {service_name}.somehost.org. |
153 | + |
154 | + :param endpoint_type: the type of endpoint to retrieve the override |
155 | + value for. |
156 | + :returns: any endpoint address or hostname that the user has overridden |
157 | + or None if an override is not present. |
158 | + """ |
159 | + override_key = ADDRESS_MAP[endpoint_type]['override'] |
160 | + addr_override = config(override_key) |
161 | + if not addr_override: |
162 | + return None |
163 | + else: |
164 | + return addr_override.format(service_name=service_name()) |
165 | + |
166 | + |
167 | def resolve_address(endpoint_type=PUBLIC): |
168 | """Return unit address depending on net config. |
169 | |
170 | @@ -77,7 +114,10 @@ |
171 | |
172 | :param endpoint_type: Network endpoing type |
173 | """ |
174 | - resolved_address = None |
175 | + resolved_address = _get_address_override(endpoint_type) |
176 | + if resolved_address: |
177 | + return resolved_address |
178 | + |
179 | vips = config('vip') |
180 | if vips: |
181 | vips = vips.split() |
182 | @@ -109,38 +149,3 @@ |
183 | "clustered=%s)" % (net_type, clustered)) |
184 | |
185 | return resolved_address |
186 | - |
187 | - |
188 | -def endpoint_url(configs, url_template, port, endpoint_type=PUBLIC, |
189 | - override=None): |
190 | - """Returns the correct endpoint URL to advertise to Keystone. |
191 | - |
192 | - This method provides the correct endpoint URL which should be advertised to |
193 | - the keystone charm for endpoint creation. This method allows for the url to |
194 | - be overridden to force a keystone endpoint to have specific URL for any of |
195 | - the defined scopes (admin, internal, public). |
196 | - |
197 | - :param configs: OSTemplateRenderer config templating object to inspect |
198 | - for a complete https context. |
199 | - :param url_template: str format string for creating the url template. Only |
200 | - two values will be passed - the scheme+hostname |
201 | - returned by the canonical_url and the port. |
202 | - :param endpoint_type: str endpoint type to resolve. |
203 | - :param override: str the name of the config option which overrides the |
204 | - endpoint URL defined by the charm itself. None will |
205 | - disable any overrides (default). |
206 | - """ |
207 | - if override: |
208 | - # Return any user-defined overrides for the keystone endpoint URL. |
209 | - user_value = config(override) |
210 | - if user_value: |
211 | - return user_value.strip() |
212 | - |
213 | - return url_template % (canonical_url(configs, endpoint_type), port) |
214 | - |
215 | - |
216 | -public_endpoint = partial(endpoint_url, endpoint_type=PUBLIC) |
217 | - |
218 | -internal_endpoint = partial(endpoint_url, endpoint_type=INTERNAL) |
219 | - |
220 | -admin_endpoint = partial(endpoint_url, endpoint_type=ADMIN) |
221 | |
222 | === modified file 'hooks/charmhelpers/contrib/openstack/neutron.py' |
223 | --- hooks/charmhelpers/contrib/openstack/neutron.py 2015-04-16 10:29:08 +0000 |
224 | +++ hooks/charmhelpers/contrib/openstack/neutron.py 2015-06-04 23:04:28 +0000 |
225 | @@ -256,11 +256,14 @@ |
226 | def parse_mappings(mappings): |
227 | parsed = {} |
228 | if mappings: |
229 | - mappings = mappings.split(' ') |
230 | + mappings = mappings.split() |
231 | for m in mappings: |
232 | p = m.partition(':') |
233 | - if p[1] == ':': |
234 | - parsed[p[0].strip()] = p[2].strip() |
235 | + key = p[0].strip() |
236 | + if p[1]: |
237 | + parsed[key] = p[2].strip() |
238 | + else: |
239 | + parsed[key] = '' |
240 | |
241 | return parsed |
242 | |
243 | @@ -283,13 +286,13 @@ |
244 | Returns dict of the form {bridge:port}. |
245 | """ |
246 | _mappings = parse_mappings(mappings) |
247 | - if not _mappings: |
248 | + if not _mappings or list(_mappings.values()) == ['']: |
249 | if not mappings: |
250 | return {} |
251 | |
252 | # For backwards-compatibility we need to support port-only provided in |
253 | # config. |
254 | - _mappings = {default_bridge: mappings.split(' ')[0]} |
255 | + _mappings = {default_bridge: mappings.split()[0]} |
256 | |
257 | bridges = _mappings.keys() |
258 | ports = _mappings.values() |
259 | @@ -309,6 +312,8 @@ |
260 | |
261 | Mappings must be a space-delimited list of provider:start:end mappings. |
262 | |
263 | + The start:end range is optional and may be omitted. |
264 | + |
265 | Returns dict of the form {provider: (start, end)}. |
266 | """ |
267 | _mappings = parse_mappings(mappings) |
268 | |
269 | === modified file 'hooks/charmhelpers/contrib/openstack/utils.py' |
270 | --- hooks/charmhelpers/contrib/openstack/utils.py 2015-04-16 21:32:20 +0000 |
271 | +++ hooks/charmhelpers/contrib/openstack/utils.py 2015-06-04 23:04:28 +0000 |
272 | @@ -53,9 +53,13 @@ |
273 | get_ipv6_addr |
274 | ) |
275 | |
276 | +from charmhelpers.contrib.python.packages import ( |
277 | + pip_create_virtualenv, |
278 | + pip_install, |
279 | +) |
280 | + |
281 | from charmhelpers.core.host import lsb_release, mounts, umount |
282 | from charmhelpers.fetch import apt_install, apt_cache, install_remote |
283 | -from charmhelpers.contrib.python.packages import pip_install |
284 | from charmhelpers.contrib.storage.linux.utils import is_block_device, zap_disk |
285 | from charmhelpers.contrib.storage.linux.loopback import ensure_loopback_device |
286 | |
287 | @@ -497,7 +501,17 @@ |
288 | requirements_dir = None |
289 | |
290 | |
291 | -def git_clone_and_install(projects_yaml, core_project): |
292 | +def _git_yaml_load(projects_yaml): |
293 | + """ |
294 | + Load the specified yaml into a dictionary. |
295 | + """ |
296 | + if not projects_yaml: |
297 | + return None |
298 | + |
299 | + return yaml.load(projects_yaml) |
300 | + |
301 | + |
302 | +def git_clone_and_install(projects_yaml, core_project, depth=1): |
303 | """ |
304 | Clone/install all specified OpenStack repositories. |
305 | |
306 | @@ -510,23 +524,22 @@ |
307 | repository: 'git://git.openstack.org/openstack/requirements.git', |
308 | branch: 'stable/icehouse'} |
309 | directory: /mnt/openstack-git |
310 | - http_proxy: http://squid.internal:3128 |
311 | - https_proxy: https://squid.internal:3128 |
312 | + http_proxy: squid-proxy-url |
313 | + https_proxy: squid-proxy-url |
314 | |
315 | The directory, http_proxy, and https_proxy keys are optional. |
316 | """ |
317 | global requirements_dir |
318 | parent_dir = '/mnt/openstack-git' |
319 | - |
320 | - if not projects_yaml: |
321 | - return |
322 | - |
323 | - projects = yaml.load(projects_yaml) |
324 | + http_proxy = None |
325 | + |
326 | + projects = _git_yaml_load(projects_yaml) |
327 | _git_validate_projects_yaml(projects, core_project) |
328 | |
329 | old_environ = dict(os.environ) |
330 | |
331 | if 'http_proxy' in projects.keys(): |
332 | + http_proxy = projects['http_proxy'] |
333 | os.environ['http_proxy'] = projects['http_proxy'] |
334 | if 'https_proxy' in projects.keys(): |
335 | os.environ['https_proxy'] = projects['https_proxy'] |
336 | @@ -534,15 +547,19 @@ |
337 | if 'directory' in projects.keys(): |
338 | parent_dir = projects['directory'] |
339 | |
340 | + pip_create_virtualenv(os.path.join(parent_dir, 'venv')) |
341 | + |
342 | for p in projects['repositories']: |
343 | repo = p['repository'] |
344 | branch = p['branch'] |
345 | if p['name'] == 'requirements': |
346 | - repo_dir = _git_clone_and_install_single(repo, branch, parent_dir, |
347 | + repo_dir = _git_clone_and_install_single(repo, branch, depth, |
348 | + parent_dir, http_proxy, |
349 | update_requirements=False) |
350 | requirements_dir = repo_dir |
351 | else: |
352 | - repo_dir = _git_clone_and_install_single(repo, branch, parent_dir, |
353 | + repo_dir = _git_clone_and_install_single(repo, branch, depth, |
354 | + parent_dir, http_proxy, |
355 | update_requirements=True) |
356 | |
357 | os.environ = old_environ |
358 | @@ -574,7 +591,8 @@ |
359 | error_out('openstack-origin-git key \'{}\' is missing'.format(key)) |
360 | |
361 | |
362 | -def _git_clone_and_install_single(repo, branch, parent_dir, update_requirements): |
363 | +def _git_clone_and_install_single(repo, branch, depth, parent_dir, http_proxy, |
364 | + update_requirements): |
365 | """ |
366 | Clone and install a single git repository. |
367 | """ |
368 | @@ -587,7 +605,8 @@ |
369 | |
370 | if not os.path.exists(dest_dir): |
371 | juju_log('Cloning git repo: {}, branch: {}'.format(repo, branch)) |
372 | - repo_dir = install_remote(repo, dest=parent_dir, branch=branch) |
373 | + repo_dir = install_remote(repo, dest=parent_dir, branch=branch, |
374 | + depth=depth) |
375 | else: |
376 | repo_dir = dest_dir |
377 | |
378 | @@ -598,7 +617,12 @@ |
379 | _git_update_requirements(repo_dir, requirements_dir) |
380 | |
381 | juju_log('Installing git repo from dir: {}'.format(repo_dir)) |
382 | - pip_install(repo_dir) |
383 | + if http_proxy: |
384 | + pip_install(repo_dir, proxy=http_proxy, |
385 | + venv=os.path.join(parent_dir, 'venv')) |
386 | + else: |
387 | + pip_install(repo_dir, |
388 | + venv=os.path.join(parent_dir, 'venv')) |
389 | |
390 | return repo_dir |
391 | |
392 | @@ -621,16 +645,27 @@ |
393 | os.chdir(orig_dir) |
394 | |
395 | |
396 | +def git_pip_venv_dir(projects_yaml): |
397 | + """ |
398 | + Return the pip virtualenv path. |
399 | + """ |
400 | + parent_dir = '/mnt/openstack-git' |
401 | + |
402 | + projects = _git_yaml_load(projects_yaml) |
403 | + |
404 | + if 'directory' in projects.keys(): |
405 | + parent_dir = projects['directory'] |
406 | + |
407 | + return os.path.join(parent_dir, 'venv') |
408 | + |
409 | + |
410 | def git_src_dir(projects_yaml, project): |
411 | """ |
412 | Return the directory where the specified project's source is located. |
413 | """ |
414 | parent_dir = '/mnt/openstack-git' |
415 | |
416 | - if not projects_yaml: |
417 | - return |
418 | - |
419 | - projects = yaml.load(projects_yaml) |
420 | + projects = _git_yaml_load(projects_yaml) |
421 | |
422 | if 'directory' in projects.keys(): |
423 | parent_dir = projects['directory'] |
424 | @@ -640,3 +675,15 @@ |
425 | return os.path.join(parent_dir, os.path.basename(p['repository'])) |
426 | |
427 | return None |
428 | + |
429 | + |
430 | +def git_yaml_value(projects_yaml, key): |
431 | + """ |
432 | + Return the value in projects_yaml for the specified key. |
433 | + """ |
434 | + projects = _git_yaml_load(projects_yaml) |
435 | + |
436 | + if key in projects.keys(): |
437 | + return projects[key] |
438 | + |
439 | + return None |
440 | |
441 | === modified file 'hooks/charmhelpers/contrib/python/packages.py' |
442 | --- hooks/charmhelpers/contrib/python/packages.py 2015-03-13 12:57:45 +0000 |
443 | +++ hooks/charmhelpers/contrib/python/packages.py 2015-06-04 23:04:28 +0000 |
444 | @@ -17,8 +17,11 @@ |
445 | # You should have received a copy of the GNU Lesser General Public License |
446 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
447 | |
448 | +import os |
449 | +import subprocess |
450 | + |
451 | from charmhelpers.fetch import apt_install, apt_update |
452 | -from charmhelpers.core.hookenv import log |
453 | +from charmhelpers.core.hookenv import charm_dir, log |
454 | |
455 | try: |
456 | from pip import main as pip_execute |
457 | @@ -51,11 +54,15 @@ |
458 | pip_execute(command) |
459 | |
460 | |
461 | -def pip_install(package, fatal=False, upgrade=False, **options): |
462 | +def pip_install(package, fatal=False, upgrade=False, venv=None, **options): |
463 | """Install a python package""" |
464 | - command = ["install"] |
465 | + if venv: |
466 | + venv_python = os.path.join(venv, 'bin/pip') |
467 | + command = [venv_python, "install"] |
468 | + else: |
469 | + command = ["install"] |
470 | |
471 | - available_options = ('proxy', 'src', 'log', "index-url", ) |
472 | + available_options = ('proxy', 'src', 'log', 'index-url', ) |
473 | for option in parse_options(options, available_options): |
474 | command.append(option) |
475 | |
476 | @@ -69,7 +76,10 @@ |
477 | |
478 | log("Installing {} package with options: {}".format(package, |
479 | command)) |
480 | - pip_execute(command) |
481 | + if venv: |
482 | + subprocess.check_call(command) |
483 | + else: |
484 | + pip_execute(command) |
485 | |
486 | |
487 | def pip_uninstall(package, **options): |
488 | @@ -94,3 +104,16 @@ |
489 | """Returns the list of current python installed packages |
490 | """ |
491 | return pip_execute(["list"]) |
492 | + |
493 | + |
494 | +def pip_create_virtualenv(path=None): |
495 | + """Create an isolated Python environment.""" |
496 | + apt_install('python-virtualenv') |
497 | + |
498 | + if path: |
499 | + venv_path = path |
500 | + else: |
501 | + venv_path = os.path.join(charm_dir(), 'venv') |
502 | + |
503 | + if not os.path.exists(venv_path): |
504 | + subprocess.check_call(['virtualenv', venv_path]) |
505 | |
506 | === modified file 'hooks/charmhelpers/core/hookenv.py' |
507 | --- hooks/charmhelpers/core/hookenv.py 2015-04-16 11:21:16 +0000 |
508 | +++ hooks/charmhelpers/core/hookenv.py 2015-06-04 23:04:28 +0000 |
509 | @@ -21,12 +21,14 @@ |
510 | # Charm Helpers Developers <juju@lists.ubuntu.com> |
511 | |
512 | from __future__ import print_function |
513 | +from functools import wraps |
514 | import os |
515 | import json |
516 | import yaml |
517 | import subprocess |
518 | import sys |
519 | import errno |
520 | +import tempfile |
521 | from subprocess import CalledProcessError |
522 | |
523 | import six |
524 | @@ -58,15 +60,17 @@ |
525 | |
526 | will cache the result of unit_get + 'test' for future calls. |
527 | """ |
528 | + @wraps(func) |
529 | def wrapper(*args, **kwargs): |
530 | global cache |
531 | key = str((func, args, kwargs)) |
532 | try: |
533 | return cache[key] |
534 | except KeyError: |
535 | - res = func(*args, **kwargs) |
536 | - cache[key] = res |
537 | - return res |
538 | + pass # Drop out of the exception handler scope. |
539 | + res = func(*args, **kwargs) |
540 | + cache[key] = res |
541 | + return res |
542 | return wrapper |
543 | |
544 | |
545 | @@ -178,7 +182,7 @@ |
546 | |
547 | def remote_unit(): |
548 | """The remote unit for the current relation hook""" |
549 | - return os.environ['JUJU_REMOTE_UNIT'] |
550 | + return os.environ.get('JUJU_REMOTE_UNIT', None) |
551 | |
552 | |
553 | def service_name(): |
554 | @@ -250,6 +254,12 @@ |
555 | except KeyError: |
556 | return (self._prev_dict or {})[key] |
557 | |
558 | + def get(self, key, default=None): |
559 | + try: |
560 | + return self[key] |
561 | + except KeyError: |
562 | + return default |
563 | + |
564 | def keys(self): |
565 | prev_keys = [] |
566 | if self._prev_dict is not None: |
567 | @@ -353,18 +363,49 @@ |
568 | """Set relation information for the current unit""" |
569 | relation_settings = relation_settings if relation_settings else {} |
570 | relation_cmd_line = ['relation-set'] |
571 | + accepts_file = "--file" in subprocess.check_output( |
572 | + relation_cmd_line + ["--help"], universal_newlines=True) |
573 | if relation_id is not None: |
574 | relation_cmd_line.extend(('-r', relation_id)) |
575 | - for k, v in (list(relation_settings.items()) + list(kwargs.items())): |
576 | - if v is None: |
577 | - relation_cmd_line.append('{}='.format(k)) |
578 | - else: |
579 | - relation_cmd_line.append('{}={}'.format(k, v)) |
580 | - subprocess.check_call(relation_cmd_line) |
581 | + settings = relation_settings.copy() |
582 | + settings.update(kwargs) |
583 | + for key, value in settings.items(): |
584 | + # Force value to be a string: it always should, but some call |
585 | + # sites pass in things like dicts or numbers. |
586 | + if value is not None: |
587 | + settings[key] = "{}".format(value) |
588 | + if accepts_file: |
589 | + # --file was introduced in Juju 1.23.2. Use it by default if |
590 | + # available, since otherwise we'll break if the relation data is |
591 | + # too big. Ideally we should tell relation-set to read the data from |
592 | + # stdin, but that feature is broken in 1.23.2: Bug #1454678. |
593 | + with tempfile.NamedTemporaryFile(delete=False) as settings_file: |
594 | + settings_file.write(yaml.safe_dump(settings).encode("utf-8")) |
595 | + subprocess.check_call( |
596 | + relation_cmd_line + ["--file", settings_file.name]) |
597 | + os.remove(settings_file.name) |
598 | + else: |
599 | + for key, value in settings.items(): |
600 | + if value is None: |
601 | + relation_cmd_line.append('{}='.format(key)) |
602 | + else: |
603 | + relation_cmd_line.append('{}={}'.format(key, value)) |
604 | + subprocess.check_call(relation_cmd_line) |
605 | # Flush cache of any relation-gets for local unit |
606 | flush(local_unit()) |
607 | |
608 | |
609 | +def relation_clear(r_id=None): |
610 | + ''' Clears any relation data already set on relation r_id ''' |
611 | + settings = relation_get(rid=r_id, |
612 | + unit=local_unit()) |
613 | + for setting in settings: |
614 | + if setting not in ['public-address', 'private-address']: |
615 | + settings[setting] = None |
616 | + relation_set(relation_id=r_id, |
617 | + **settings) |
618 | + |
619 | + |
620 | @cached |
621 | def relation_ids(reltype=None): |
622 | """A list of relation_ids""" |
623 | @@ -509,6 +550,11 @@ |
624 | return None |
625 | |
626 | |
627 | +def unit_public_ip(): |
628 | + """Get this unit's public IP address""" |
629 | + return unit_get('public-address') |
630 | + |
631 | + |
632 | def unit_private_ip(): |
633 | """Get this unit's private IP address""" |
634 | return unit_get('private-address') |
635 | @@ -605,3 +651,94 @@ |
636 | |
637 | The results set by action_set are preserved.""" |
638 | subprocess.check_call(['action-fail', message]) |
639 | + |
640 | + |
641 | +def status_set(workload_state, message): |
642 | + """Set the workload state with a message |
643 | + |
644 | + Use status-set to set the workload state with a message which is visible |
645 | + to the user via juju status. If the status-set command is not found then |
646 | + assume this is juju < 1.23 and juju-log the message unstead. |
647 | + |
648 | + workload_state -- valid juju workload state. |
649 | + message -- status update message |
650 | + """ |
651 | + valid_states = ['maintenance', 'blocked', 'waiting', 'active'] |
652 | + if workload_state not in valid_states: |
653 | + raise ValueError( |
654 | + '{!r} is not a valid workload state'.format(workload_state) |
655 | + ) |
656 | + cmd = ['status-set', workload_state, message] |
657 | + try: |
658 | + ret = subprocess.call(cmd) |
659 | + if ret == 0: |
660 | + return |
661 | + except OSError as e: |
662 | + if e.errno != errno.ENOENT: |
663 | + raise |
664 | + log_message = 'status-set failed: {} {}'.format(workload_state, |
665 | + message) |
666 | + log(log_message, level='INFO') |
667 | + |
668 | + |
669 | +def status_get(): |
670 | + """Retrieve the previously set juju workload state |
671 | + |
672 | + If the status-set command is not found then assume this is juju < 1.23 and |
673 | + return 'unknown' |
674 | + """ |
675 | + cmd = ['status-get'] |
676 | + try: |
677 | + raw_status = subprocess.check_output(cmd, universal_newlines=True) |
678 | + status = raw_status.rstrip() |
679 | + return status |
680 | + except OSError as e: |
681 | + if e.errno == errno.ENOENT: |
682 | + return 'unknown' |
683 | + else: |
684 | + raise |
685 | + |
686 | + |
687 | +def translate_exc(from_exc, to_exc): |
688 | + def inner_translate_exc1(f): |
689 | + def inner_translate_exc2(*args, **kwargs): |
690 | + try: |
691 | + return f(*args, **kwargs) |
692 | + except from_exc: |
693 | + raise to_exc |
694 | + |
695 | + return inner_translate_exc2 |
696 | + |
697 | + return inner_translate_exc1 |
698 | + |
699 | + |
700 | +@translate_exc(from_exc=OSError, to_exc=NotImplementedError) |
701 | +def is_leader(): |
702 | + """Does the current unit hold the juju leadership |
703 | + |
704 | + Uses juju to determine whether the current unit is the leader of its peers |
705 | + """ |
706 | + cmd = ['is-leader', '--format=json'] |
707 | + return json.loads(subprocess.check_output(cmd).decode('UTF-8')) |
708 | + |
709 | + |
710 | +@translate_exc(from_exc=OSError, to_exc=NotImplementedError) |
711 | +def leader_get(attribute=None): |
712 | + """Juju leader get value(s)""" |
713 | + cmd = ['leader-get', '--format=json'] + [attribute or '-'] |
714 | + return json.loads(subprocess.check_output(cmd).decode('UTF-8')) |
715 | + |
716 | + |
717 | +@translate_exc(from_exc=OSError, to_exc=NotImplementedError) |
718 | +def leader_set(settings=None, **kwargs): |
719 | + """Juju leader set value(s)""" |
720 | + log("Juju leader-set '%s'" % (settings), level=DEBUG) |
721 | + cmd = ['leader-set'] |
722 | + settings = settings or {} |
723 | + settings.update(kwargs) |
724 | + for k, v in settings.iteritems(): |
725 | + if v is None: |
726 | + cmd.append('{}='.format(k)) |
727 | + else: |
728 | + cmd.append('{}={}'.format(k, v)) |
729 | + subprocess.check_call(cmd) |
730 | |
731 | === modified file 'hooks/charmhelpers/core/host.py' |
732 | --- hooks/charmhelpers/core/host.py 2015-03-18 12:31:20 +0000 |
733 | +++ hooks/charmhelpers/core/host.py 2015-06-04 23:04:28 +0000 |
734 | @@ -90,7 +90,7 @@ |
735 | ['service', service_name, 'status'], |
736 | stderr=subprocess.STDOUT).decode('UTF-8') |
737 | except subprocess.CalledProcessError as e: |
738 | - return 'unrecognized service' not in e.output |
739 | + return b'unrecognized service' not in e.output |
740 | else: |
741 | return True |
742 | |
743 | |
744 | === modified file 'hooks/charmhelpers/core/services/base.py' |
745 | --- hooks/charmhelpers/core/services/base.py 2015-01-26 09:48:14 +0000 |
746 | +++ hooks/charmhelpers/core/services/base.py 2015-06-04 23:04:28 +0000 |
747 | @@ -15,9 +15,9 @@ |
748 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
749 | |
750 | import os |
751 | -import re |
752 | import json |
753 | -from collections import Iterable |
754 | +from inspect import getargspec |
755 | +from collections import Iterable, OrderedDict |
756 | |
757 | from charmhelpers.core import host |
758 | from charmhelpers.core import hookenv |
759 | @@ -119,7 +119,7 @@ |
760 | """ |
761 | self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json') |
762 | self._ready = None |
763 | - self.services = {} |
764 | + self.services = OrderedDict() |
765 | for service in services or []: |
766 | service_name = service['service'] |
767 | self.services[service_name] = service |
768 | @@ -132,8 +132,8 @@ |
769 | if hook_name == 'stop': |
770 | self.stop_services() |
771 | else: |
772 | + self.reconfigure_services() |
773 | self.provide_data() |
774 | - self.reconfigure_services() |
775 | cfg = hookenv.config() |
776 | if cfg.implicit_save: |
777 | cfg.save() |
778 | @@ -145,15 +145,36 @@ |
779 | A provider must have a `name` attribute, which indicates which relation |
780 | to set data on, and a `provide_data()` method, which returns a dict of |
781 | data to set. |
782 | + |
783 | + The `provide_data()` method can optionally accept two parameters: |
784 | + |
785 | + * ``remote_service`` The name of the remote service that the data will |
786 | + be provided to. The `provide_data()` method will be called once |
787 | + for each connected service (not unit). This allows the method to |
788 | + tailor its data to the given service. |
789 | + * ``service_ready`` Whether or not the service definition had all of |
790 | + its requirements met, and thus the ``data_ready`` callbacks run. |
791 | + |
792 | + Note that the ``provided_data`` methods are now called **after** the |
793 | + ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks |
794 | + a chance to generate any data necessary for the providing to the remote |
795 | + services. |
796 | """ |
797 | - hook_name = hookenv.hook_name() |
798 | - for service in self.services.values(): |
799 | + for service_name, service in self.services.items(): |
800 | + service_ready = self.is_ready(service_name) |
801 | for provider in service.get('provided_data', []): |
802 | - if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name): |
803 | - data = provider.provide_data() |
804 | - _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data |
805 | - if _ready: |
806 | - hookenv.relation_set(None, data) |
807 | + for relid in hookenv.relation_ids(provider.name): |
808 | + units = hookenv.related_units(relid) |
809 | + if not units: |
810 | + continue |
811 | + remote_service = units[0].split('/')[0] |
812 | + argspec = getargspec(provider.provide_data) |
813 | + if len(argspec.args) > 1: |
814 | + data = provider.provide_data(remote_service, service_ready) |
815 | + else: |
816 | + data = provider.provide_data() |
817 | + if data: |
818 | + hookenv.relation_set(relid, data) |
819 | |
820 | def reconfigure_services(self, *service_names): |
821 | """ |
822 | |
823 | === modified file 'hooks/charmhelpers/fetch/__init__.py' |
824 | --- hooks/charmhelpers/fetch/__init__.py 2015-01-26 09:48:14 +0000 |
825 | +++ hooks/charmhelpers/fetch/__init__.py 2015-06-04 23:04:28 +0000 |
826 | @@ -158,7 +158,7 @@ |
827 | |
828 | def apt_cache(in_memory=True): |
829 | """Build and return an apt cache""" |
830 | - import apt_pkg |
831 | + from apt import apt_pkg |
832 | apt_pkg.init() |
833 | if in_memory: |
834 | apt_pkg.config.set("Dir::Cache::pkgcache", "") |
835 | |
836 | === modified file 'hooks/charmhelpers/fetch/giturl.py' |
837 | --- hooks/charmhelpers/fetch/giturl.py 2015-03-13 12:57:45 +0000 |
838 | +++ hooks/charmhelpers/fetch/giturl.py 2015-06-04 23:04:28 +0000 |
839 | @@ -45,14 +45,16 @@ |
840 | else: |
841 | return True |
842 | |
843 | - def clone(self, source, dest, branch): |
844 | + def clone(self, source, dest, branch, depth=None): |
845 | if not self.can_handle(source): |
846 | raise UnhandledSource("Cannot handle {}".format(source)) |
847 | |
848 | - repo = Repo.clone_from(source, dest) |
849 | - repo.git.checkout(branch) |
850 | + if depth: |
851 | + Repo.clone_from(source, dest, branch=branch, depth=depth) |
852 | + else: |
853 | + Repo.clone_from(source, dest, branch=branch) |
854 | |
855 | - def install(self, source, branch="master", dest=None): |
856 | + def install(self, source, branch="master", dest=None, depth=None): |
857 | url_parts = self.parse_url(source) |
858 | branch_name = url_parts.path.strip("/").split("/")[-1] |
859 | if dest: |
860 | @@ -63,7 +65,7 @@ |
861 | if not os.path.exists(dest_dir): |
862 | mkdir(dest_dir, perms=0o755) |
863 | try: |
864 | - self.clone(source, dest_dir, branch) |
865 | + self.clone(source, dest_dir, branch, depth) |
866 | except GitCommandError as e: |
867 | raise UnhandledSource(e.message) |
868 | except OSError as e: |
869 | |
870 | === modified file 'unit_tests/test_ceilometer_hooks.py' |
871 | --- unit_tests/test_ceilometer_hooks.py 2015-04-08 15:49:34 +0000 |
872 | +++ unit_tests/test_ceilometer_hooks.py 2015-06-04 23:04:28 +0000 |
873 | @@ -31,7 +31,6 @@ |
874 | 'get_ceilometer_context', |
875 | 'lsb_release', |
876 | 'get_packages', |
877 | - 'canonical_url', |
878 | 'service_restart', |
879 | 'update_nrpe_config', |
880 | 'configure_https', |
881 | @@ -136,9 +135,10 @@ |
882 | self.assertTrue(self.CONFIGS.write_all.called) |
883 | self.assertTrue(joined.called) |
884 | |
885 | + @patch.object(hooks, 'canonical_url') |
886 | @patch('charmhelpers.core.hookenv.config') |
887 | - def test_keystone_joined(self, mock_config): |
888 | - self.canonical_url.return_value = "http://thishost" |
889 | + def test_keystone_joined(self, mock_config, _canonical_url): |
890 | + _canonical_url.return_value = "http://thishost" |
891 | self.test_config.set('region', 'myregion') |
892 | hooks.hooks.execute(['hooks/identity-service-relation-joined']) |
893 | url = "http://{}:{}".format('thishost', hooks.CEILOMETER_PORT) |
894 | @@ -148,6 +148,30 @@ |
895 | requested_roles=hooks.CEILOMETER_ROLE, |
896 | region='myregion', relation_id=None) |
897 | |
898 | + @patch('charmhelpers.contrib.openstack.ip.service_name', |
899 | + lambda *args: 'ceilometer') |
900 | + @patch('charmhelpers.contrib.openstack.ip.unit_get') |
901 | + @patch('charmhelpers.contrib.openstack.ip.is_clustered') |
902 | + @patch('charmhelpers.core.hookenv.config') |
903 | + @patch('charmhelpers.contrib.openstack.ip.config') |
904 | + def test_keystone_joined_url_override(self, _config, mock_config, |
905 | + _is_clustered, _unit_get): |
906 | + _unit_get.return_value = "thishost" |
907 | + _is_clustered.return_value = False |
908 | + _config.side_effect = self.test_config.get |
909 | + mock_config.side_effect = self.test_config.get |
910 | + self.test_config.set('region', 'myregion') |
911 | + self.test_config.set('os-public-hostname', 'ceilometer.example.com') |
912 | + hooks.keystone_joined(None) |
913 | + url = "http://{}:{}".format('thishost', hooks.CEILOMETER_PORT) |
914 | + public_url = "http://{}:{}".format('ceilometer.example.com', |
915 | + hooks.CEILOMETER_PORT) |
916 | + self.relation_set.assert_called_with( |
917 | + service=hooks.CEILOMETER_SERVICE, |
918 | + public_url=public_url, admin_url=url, internal_url=url, |
919 | + requested_roles=hooks.CEILOMETER_ROLE, |
920 | + region='myregion', relation_id=None) |
921 | + |
922 | @patch('charmhelpers.core.hookenv.config') |
923 | def test_ceilometer_joined(self, mock_config): |
924 | self.relation_ids.return_value = ['ceilometer:0'] |
charm_lint_check #5042 ceilometer-next for billy-olsen mp261004
LINT OK: passed
Build: http:// 10.245. 162.77: 8080/job/ charm_lint_ check/5042/