Merge lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host into lp:~openstack-charmers-archive/charms/trusty/ceilometer/next

Proposed by Billy Olsen
Status: Merged
Merged at revision: 85
Proposed branch: lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host
Merge into: lp:~openstack-charmers-archive/charms/trusty/ceilometer/next
Diff against target: 924 lines (+404/-103)
12 files modified
config.yaml (+12/-0)
hooks/charmhelpers/contrib/hahelpers/cluster.py (+25/-0)
hooks/charmhelpers/contrib/openstack/ip.py (+49/-44)
hooks/charmhelpers/contrib/openstack/neutron.py (+10/-5)
hooks/charmhelpers/contrib/openstack/utils.py (+65/-18)
hooks/charmhelpers/contrib/python/packages.py (+28/-5)
hooks/charmhelpers/core/hookenv.py (+147/-10)
hooks/charmhelpers/core/host.py (+1/-1)
hooks/charmhelpers/core/services/base.py (+32/-11)
hooks/charmhelpers/fetch/__init__.py (+1/-1)
hooks/charmhelpers/fetch/giturl.py (+7/-5)
unit_tests/test_ceilometer_hooks.py (+27/-3)
To merge this branch: bzr merge lp:~billy-olsen/charms/trusty/ceilometer/public-endpoint-host
Reviewer Review Type Date Requested Status
OpenStack Charmers Pending
Review via email: mp+261004@code.launchpad.net

Description of the change

Provides a config option which allows the user to specify the public hostname used to advertise to keystone when creating endpoints.

To post a comment you must log in.
Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #5042 ceilometer-next for billy-olsen mp261004
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5042/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #4722 ceilometer-next for billy-olsen mp261004
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/4722/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #4448 ceilometer-next for billy-olsen mp261004
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
ERROR subprocess encountered error code 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11563701/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4448/

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Amulet test also fails for trusty-icehouse ceilometer/next. It looks like either mongodb or ceilometer have had a change in expected behaviors and/or relations since the 15.04 release, and the functional test didn't get updated to reflect that.

So this failure is not specific to your proposal, but until I can track down the change that broke the test, your proposal can't be validated via the functional tests.

00:28:14.389 unexpected relation data in mongodb database - key 'replset' does not exist

Revision history for this message
Ryan Beisner (1chb1n) wrote :

FYI, tracking bug for the amulet test. I'll work on that. https://bugs.launchpad.net/charms/+source/ceilometer/+bug/1461911

Revision history for this message
Ryan Beisner (1chb1n) wrote :

2015-06-04 11:19:46,514 _validate_dict_data DEBUG: actual: {u'private-address': u'172.17.105.198', u'hostname': u'172.17.105.198', u'type': u'database', u'port': u'27017'}

2015-06-04 11:19:46,514 _validate_dict_data DEBUG: expected: {'private-address': <bound method OpenStackAmuletUtils.valid_ip of <charmhelpers.contrib.openstack.amulet.utils.OpenStackAmuletUtils object at 0x2b613b2be7d0>>, 'hostname': <bound method OpenStackAmuletUtils.valid_ip of <charmhelpers.contrib.openstack.amulet.utils.OpenStackAmuletUtils object at 0x2b613b2be7d0>>, 'type': 'database', 'port': '27017', 'replset': 'myset'}

unexpected relation data in mongodb database - key 'replset' does not exist

Revision history for this message
Ryan Beisner (1chb1n) wrote :

FYI, ceilometer/next amulet tests fixed and landed re: bug 1461911.

You may want to pull those changes into your branch.

87. By Billy Olsen

Merge with /next

88. By Billy Olsen

c-h sync

89. By Billy Olsen

Fix broken unit test from c-h sync

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #4745 ceilometer-next for billy-olsen mp261004
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/4745/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #5066 ceilometer-next for billy-olsen mp261004
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5066/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #5067 ceilometer-next for billy-olsen mp261004
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5067/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #4746 ceilometer-next for billy-olsen mp261004
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/4746/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #4473 ceilometer-next for billy-olsen mp261004
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4473/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #4474 ceilometer-next for billy-olsen mp261004
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11577130/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4474/

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Looks good, but are amulet tests still failing?

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Amulet tests were due to timeout unfortunately

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Fyi, I ran the amulet tests manually and they were successful:

Results: 6 passed, 0 failed, 0 errored

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'config.yaml'
--- config.yaml 2015-02-19 02:03:06 +0000
+++ config.yaml 2015-06-04 23:04:28 +0000
@@ -100,6 +100,18 @@
100 192.168.0.0/24)100 192.168.0.0/24)
101 .101 .
102 This network will be used for public endpoints.102 This network will be used for public endpoints.
103 os-public-hostname:
104 type: string
105 default:
106 description: |
107 The hostname or address of the public endpoints created for ceilometer
108 in the keystone identity provider.
109 .
110 This value will be used for public endpoints. For example, an
111 os-public-hostname set to 'ceilometer.example.com' with ssl enabled will
112 create the following public endpoints for ceilometer.
113 .
114 https://ceilometer.example.com:8777/
103 # HA configuration settings115 # HA configuration settings
104 vip:116 vip:
105 type: string117 type: string
106118
=== modified file 'hooks/charmhelpers/contrib/hahelpers/cluster.py'
--- hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-03-13 12:57:45 +0000
+++ hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-06-04 23:04:28 +0000
@@ -52,6 +52,8 @@
52 bool_from_string,52 bool_from_string,
53)53)
5454
55DC_RESOURCE_NAME = 'DC'
56
5557
56class HAIncompleteConfig(Exception):58class HAIncompleteConfig(Exception):
57 pass59 pass
@@ -95,6 +97,27 @@
95 return False97 return False
9698
9799
100def is_crm_dc():
101 """
102 Determine leadership by querying the pacemaker Designated Controller
103 """
104 cmd = ['crm', 'status']
105 try:
106 status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
107 if not isinstance(status, six.text_type):
108 status = six.text_type(status, "utf-8")
109 except subprocess.CalledProcessError:
110 return False
111 current_dc = ''
112 for line in status.split('\n'):
113 if line.startswith('Current DC'):
114 # Current DC: juju-lytrusty-machine-2 (168108163) - partition with quorum
115 current_dc = line.split(':')[1].split()[0]
116 if current_dc == get_unit_hostname():
117 return True
118 return False
119
120
98@retry_on_exception(5, base_delay=2, exc_type=CRMResourceNotFound)121@retry_on_exception(5, base_delay=2, exc_type=CRMResourceNotFound)
99def is_crm_leader(resource, retry=False):122def is_crm_leader(resource, retry=False):
100 """123 """
@@ -104,6 +127,8 @@
104 We allow this operation to be retried to avoid the possibility of getting a127 We allow this operation to be retried to avoid the possibility of getting a
105 false negative. See LP #1396246 for more info.128 false negative. See LP #1396246 for more info.
106 """129 """
130 if resource == DC_RESOURCE_NAME:
131 return is_crm_dc()
107 cmd = ['crm', 'resource', 'show', resource]132 cmd = ['crm', 'resource', 'show', resource]
108 try:133 try:
109 status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)134 status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
110135
=== modified file 'hooks/charmhelpers/contrib/openstack/ip.py'
--- hooks/charmhelpers/contrib/openstack/ip.py 2015-03-13 12:57:45 +0000
+++ hooks/charmhelpers/contrib/openstack/ip.py 2015-06-04 23:04:28 +0000
@@ -17,6 +17,7 @@
17from charmhelpers.core.hookenv import (17from charmhelpers.core.hookenv import (
18 config,18 config,
19 unit_get,19 unit_get,
20 service_name,
20)21)
21from charmhelpers.contrib.network.ip import (22from charmhelpers.contrib.network.ip import (
22 get_address_in_network,23 get_address_in_network,
@@ -26,8 +27,6 @@
26)27)
27from charmhelpers.contrib.hahelpers.cluster import is_clustered28from charmhelpers.contrib.hahelpers.cluster import is_clustered
2829
29from functools import partial
30
31PUBLIC = 'public'30PUBLIC = 'public'
32INTERNAL = 'int'31INTERNAL = 'int'
33ADMIN = 'admin'32ADMIN = 'admin'
@@ -35,15 +34,18 @@
35ADDRESS_MAP = {34ADDRESS_MAP = {
36 PUBLIC: {35 PUBLIC: {
37 'config': 'os-public-network',36 'config': 'os-public-network',
38 'fallback': 'public-address'37 'fallback': 'public-address',
38 'override': 'os-public-hostname',
39 },39 },
40 INTERNAL: {40 INTERNAL: {
41 'config': 'os-internal-network',41 'config': 'os-internal-network',
42 'fallback': 'private-address'42 'fallback': 'private-address',
43 'override': 'os-internal-hostname',
43 },44 },
44 ADMIN: {45 ADMIN: {
45 'config': 'os-admin-network',46 'config': 'os-admin-network',
46 'fallback': 'private-address'47 'fallback': 'private-address',
48 'override': 'os-admin-hostname',
47 }49 }
48}50}
4951
@@ -57,15 +59,50 @@
57 :param endpoint_type: str endpoint type to resolve.59 :param endpoint_type: str endpoint type to resolve.
58 :param returns: str base URL for services on the current service unit.60 :param returns: str base URL for services on the current service unit.
59 """61 """
60 scheme = 'http'62 scheme = _get_scheme(configs)
61 if 'https' in configs.complete_contexts():63
62 scheme = 'https'
63 address = resolve_address(endpoint_type)64 address = resolve_address(endpoint_type)
64 if is_ipv6(address):65 if is_ipv6(address):
65 address = "[{}]".format(address)66 address = "[{}]".format(address)
67
66 return '%s://%s' % (scheme, address)68 return '%s://%s' % (scheme, address)
6769
6870
71def _get_scheme(configs):
72 """Returns the scheme to use for the url (either http or https)
73 depending upon whether https is in the configs value.
74
75 :param configs: OSTemplateRenderer config templating object to inspect
76 for a complete https context.
77 :returns: either 'http' or 'https' depending on whether https is
78 configured within the configs context.
79 """
80 scheme = 'http'
81 if configs and 'https' in configs.complete_contexts():
82 scheme = 'https'
83 return scheme
84
85
86def _get_address_override(endpoint_type=PUBLIC):
87 """Returns any address overrides that the user has defined based on the
88 endpoint type.
89
90 Note: this function allows for the service name to be inserted into the
91 address if the user specifies {service_name}.somehost.org.
92
93 :param endpoint_type: the type of endpoint to retrieve the override
94 value for.
95 :returns: any endpoint address or hostname that the user has overridden
96 or None if an override is not present.
97 """
98 override_key = ADDRESS_MAP[endpoint_type]['override']
99 addr_override = config(override_key)
100 if not addr_override:
101 return None
102 else:
103 return addr_override.format(service_name=service_name())
104
105
69def resolve_address(endpoint_type=PUBLIC):106def resolve_address(endpoint_type=PUBLIC):
70 """Return unit address depending on net config.107 """Return unit address depending on net config.
71108
@@ -77,7 +114,10 @@
77114
78 :param endpoint_type: Network endpoing type115 :param endpoint_type: Network endpoing type
79 """116 """
80 resolved_address = None117 resolved_address = _get_address_override(endpoint_type)
118 if resolved_address:
119 return resolved_address
120
81 vips = config('vip')121 vips = config('vip')
82 if vips:122 if vips:
83 vips = vips.split()123 vips = vips.split()
@@ -109,38 +149,3 @@
109 "clustered=%s)" % (net_type, clustered))149 "clustered=%s)" % (net_type, clustered))
110150
111 return resolved_address151 return resolved_address
112
113
114def endpoint_url(configs, url_template, port, endpoint_type=PUBLIC,
115 override=None):
116 """Returns the correct endpoint URL to advertise to Keystone.
117
118 This method provides the correct endpoint URL which should be advertised to
119 the keystone charm for endpoint creation. This method allows for the url to
120 be overridden to force a keystone endpoint to have specific URL for any of
121 the defined scopes (admin, internal, public).
122
123 :param configs: OSTemplateRenderer config templating object to inspect
124 for a complete https context.
125 :param url_template: str format string for creating the url template. Only
126 two values will be passed - the scheme+hostname
127 returned by the canonical_url and the port.
128 :param endpoint_type: str endpoint type to resolve.
129 :param override: str the name of the config option which overrides the
130 endpoint URL defined by the charm itself. None will
131 disable any overrides (default).
132 """
133 if override:
134 # Return any user-defined overrides for the keystone endpoint URL.
135 user_value = config(override)
136 if user_value:
137 return user_value.strip()
138
139 return url_template % (canonical_url(configs, endpoint_type), port)
140
141
142public_endpoint = partial(endpoint_url, endpoint_type=PUBLIC)
143
144internal_endpoint = partial(endpoint_url, endpoint_type=INTERNAL)
145
146admin_endpoint = partial(endpoint_url, endpoint_type=ADMIN)
147152
=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py 2015-04-16 10:29:08 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py 2015-06-04 23:04:28 +0000
@@ -256,11 +256,14 @@
256def parse_mappings(mappings):256def parse_mappings(mappings):
257 parsed = {}257 parsed = {}
258 if mappings:258 if mappings:
259 mappings = mappings.split(' ')259 mappings = mappings.split()
260 for m in mappings:260 for m in mappings:
261 p = m.partition(':')261 p = m.partition(':')
262 if p[1] == ':':262 key = p[0].strip()
263 parsed[p[0].strip()] = p[2].strip()263 if p[1]:
264 parsed[key] = p[2].strip()
265 else:
266 parsed[key] = ''
264267
265 return parsed268 return parsed
266269
@@ -283,13 +286,13 @@
283 Returns dict of the form {bridge:port}.286 Returns dict of the form {bridge:port}.
284 """287 """
285 _mappings = parse_mappings(mappings)288 _mappings = parse_mappings(mappings)
286 if not _mappings:289 if not _mappings or list(_mappings.values()) == ['']:
287 if not mappings:290 if not mappings:
288 return {}291 return {}
289292
290 # For backwards-compatibility we need to support port-only provided in293 # For backwards-compatibility we need to support port-only provided in
291 # config.294 # config.
292 _mappings = {default_bridge: mappings.split(' ')[0]}295 _mappings = {default_bridge: mappings.split()[0]}
293296
294 bridges = _mappings.keys()297 bridges = _mappings.keys()
295 ports = _mappings.values()298 ports = _mappings.values()
@@ -309,6 +312,8 @@
309312
310 Mappings must be a space-delimited list of provider:start:end mappings.313 Mappings must be a space-delimited list of provider:start:end mappings.
311314
315 The start:end range is optional and may be omitted.
316
312 Returns dict of the form {provider: (start, end)}.317 Returns dict of the form {provider: (start, end)}.
313 """318 """
314 _mappings = parse_mappings(mappings)319 _mappings = parse_mappings(mappings)
315320
=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
--- hooks/charmhelpers/contrib/openstack/utils.py 2015-04-16 21:32:20 +0000
+++ hooks/charmhelpers/contrib/openstack/utils.py 2015-06-04 23:04:28 +0000
@@ -53,9 +53,13 @@
53 get_ipv6_addr53 get_ipv6_addr
54)54)
5555
56from charmhelpers.contrib.python.packages import (
57 pip_create_virtualenv,
58 pip_install,
59)
60
56from charmhelpers.core.host import lsb_release, mounts, umount61from charmhelpers.core.host import lsb_release, mounts, umount
57from charmhelpers.fetch import apt_install, apt_cache, install_remote62from charmhelpers.fetch import apt_install, apt_cache, install_remote
58from charmhelpers.contrib.python.packages import pip_install
59from charmhelpers.contrib.storage.linux.utils import is_block_device, zap_disk63from charmhelpers.contrib.storage.linux.utils import is_block_device, zap_disk
60from charmhelpers.contrib.storage.linux.loopback import ensure_loopback_device64from charmhelpers.contrib.storage.linux.loopback import ensure_loopback_device
6165
@@ -497,7 +501,17 @@
497requirements_dir = None501requirements_dir = None
498502
499503
500def git_clone_and_install(projects_yaml, core_project):504def _git_yaml_load(projects_yaml):
505 """
506 Load the specified yaml into a dictionary.
507 """
508 if not projects_yaml:
509 return None
510
511 return yaml.load(projects_yaml)
512
513
514def git_clone_and_install(projects_yaml, core_project, depth=1):
501 """515 """
502 Clone/install all specified OpenStack repositories.516 Clone/install all specified OpenStack repositories.
503517
@@ -510,23 +524,22 @@
510 repository: 'git://git.openstack.org/openstack/requirements.git',524 repository: 'git://git.openstack.org/openstack/requirements.git',
511 branch: 'stable/icehouse'}525 branch: 'stable/icehouse'}
512 directory: /mnt/openstack-git526 directory: /mnt/openstack-git
513 http_proxy: http://squid.internal:3128527 http_proxy: squid-proxy-url
514 https_proxy: https://squid.internal:3128528 https_proxy: squid-proxy-url
515529
516 The directory, http_proxy, and https_proxy keys are optional.530 The directory, http_proxy, and https_proxy keys are optional.
517 """531 """
518 global requirements_dir532 global requirements_dir
519 parent_dir = '/mnt/openstack-git'533 parent_dir = '/mnt/openstack-git'
520534 http_proxy = None
521 if not projects_yaml:535
522 return536 projects = _git_yaml_load(projects_yaml)
523
524 projects = yaml.load(projects_yaml)
525 _git_validate_projects_yaml(projects, core_project)537 _git_validate_projects_yaml(projects, core_project)
526538
527 old_environ = dict(os.environ)539 old_environ = dict(os.environ)
528540
529 if 'http_proxy' in projects.keys():541 if 'http_proxy' in projects.keys():
542 http_proxy = projects['http_proxy']
530 os.environ['http_proxy'] = projects['http_proxy']543 os.environ['http_proxy'] = projects['http_proxy']
531 if 'https_proxy' in projects.keys():544 if 'https_proxy' in projects.keys():
532 os.environ['https_proxy'] = projects['https_proxy']545 os.environ['https_proxy'] = projects['https_proxy']
@@ -534,15 +547,19 @@
534 if 'directory' in projects.keys():547 if 'directory' in projects.keys():
535 parent_dir = projects['directory']548 parent_dir = projects['directory']
536549
550 pip_create_virtualenv(os.path.join(parent_dir, 'venv'))
551
537 for p in projects['repositories']:552 for p in projects['repositories']:
538 repo = p['repository']553 repo = p['repository']
539 branch = p['branch']554 branch = p['branch']
540 if p['name'] == 'requirements':555 if p['name'] == 'requirements':
541 repo_dir = _git_clone_and_install_single(repo, branch, parent_dir,556 repo_dir = _git_clone_and_install_single(repo, branch, depth,
557 parent_dir, http_proxy,
542 update_requirements=False)558 update_requirements=False)
543 requirements_dir = repo_dir559 requirements_dir = repo_dir
544 else:560 else:
545 repo_dir = _git_clone_and_install_single(repo, branch, parent_dir,561 repo_dir = _git_clone_and_install_single(repo, branch, depth,
562 parent_dir, http_proxy,
546 update_requirements=True)563 update_requirements=True)
547564
548 os.environ = old_environ565 os.environ = old_environ
@@ -574,7 +591,8 @@
574 error_out('openstack-origin-git key \'{}\' is missing'.format(key))591 error_out('openstack-origin-git key \'{}\' is missing'.format(key))
575592
576593
577def _git_clone_and_install_single(repo, branch, parent_dir, update_requirements):594def _git_clone_and_install_single(repo, branch, depth, parent_dir, http_proxy,
595 update_requirements):
578 """596 """
579 Clone and install a single git repository.597 Clone and install a single git repository.
580 """598 """
@@ -587,7 +605,8 @@
587605
588 if not os.path.exists(dest_dir):606 if not os.path.exists(dest_dir):
589 juju_log('Cloning git repo: {}, branch: {}'.format(repo, branch))607 juju_log('Cloning git repo: {}, branch: {}'.format(repo, branch))
590 repo_dir = install_remote(repo, dest=parent_dir, branch=branch)608 repo_dir = install_remote(repo, dest=parent_dir, branch=branch,
609 depth=depth)
591 else:610 else:
592 repo_dir = dest_dir611 repo_dir = dest_dir
593612
@@ -598,7 +617,12 @@
598 _git_update_requirements(repo_dir, requirements_dir)617 _git_update_requirements(repo_dir, requirements_dir)
599618
600 juju_log('Installing git repo from dir: {}'.format(repo_dir))619 juju_log('Installing git repo from dir: {}'.format(repo_dir))
601 pip_install(repo_dir)620 if http_proxy:
621 pip_install(repo_dir, proxy=http_proxy,
622 venv=os.path.join(parent_dir, 'venv'))
623 else:
624 pip_install(repo_dir,
625 venv=os.path.join(parent_dir, 'venv'))
602626
603 return repo_dir627 return repo_dir
604628
@@ -621,16 +645,27 @@
621 os.chdir(orig_dir)645 os.chdir(orig_dir)
622646
623647
648def git_pip_venv_dir(projects_yaml):
649 """
650 Return the pip virtualenv path.
651 """
652 parent_dir = '/mnt/openstack-git'
653
654 projects = _git_yaml_load(projects_yaml)
655
656 if 'directory' in projects.keys():
657 parent_dir = projects['directory']
658
659 return os.path.join(parent_dir, 'venv')
660
661
624def git_src_dir(projects_yaml, project):662def git_src_dir(projects_yaml, project):
625 """663 """
626 Return the directory where the specified project's source is located.664 Return the directory where the specified project's source is located.
627 """665 """
628 parent_dir = '/mnt/openstack-git'666 parent_dir = '/mnt/openstack-git'
629667
630 if not projects_yaml:668 projects = _git_yaml_load(projects_yaml)
631 return
632
633 projects = yaml.load(projects_yaml)
634669
635 if 'directory' in projects.keys():670 if 'directory' in projects.keys():
636 parent_dir = projects['directory']671 parent_dir = projects['directory']
@@ -640,3 +675,15 @@
640 return os.path.join(parent_dir, os.path.basename(p['repository']))675 return os.path.join(parent_dir, os.path.basename(p['repository']))
641676
642 return None677 return None
678
679
680def git_yaml_value(projects_yaml, key):
681 """
682 Return the value in projects_yaml for the specified key.
683 """
684 projects = _git_yaml_load(projects_yaml)
685
686 if key in projects.keys():
687 return projects[key]
688
689 return None
643690
=== modified file 'hooks/charmhelpers/contrib/python/packages.py'
--- hooks/charmhelpers/contrib/python/packages.py 2015-03-13 12:57:45 +0000
+++ hooks/charmhelpers/contrib/python/packages.py 2015-06-04 23:04:28 +0000
@@ -17,8 +17,11 @@
17# You should have received a copy of the GNU Lesser General Public License17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1919
20import os
21import subprocess
22
20from charmhelpers.fetch import apt_install, apt_update23from charmhelpers.fetch import apt_install, apt_update
21from charmhelpers.core.hookenv import log24from charmhelpers.core.hookenv import charm_dir, log
2225
23try:26try:
24 from pip import main as pip_execute27 from pip import main as pip_execute
@@ -51,11 +54,15 @@
51 pip_execute(command)54 pip_execute(command)
5255
5356
54def pip_install(package, fatal=False, upgrade=False, **options):57def pip_install(package, fatal=False, upgrade=False, venv=None, **options):
55 """Install a python package"""58 """Install a python package"""
56 command = ["install"]59 if venv:
60 venv_python = os.path.join(venv, 'bin/pip')
61 command = [venv_python, "install"]
62 else:
63 command = ["install"]
5764
58 available_options = ('proxy', 'src', 'log', "index-url", )65 available_options = ('proxy', 'src', 'log', 'index-url', )
59 for option in parse_options(options, available_options):66 for option in parse_options(options, available_options):
60 command.append(option)67 command.append(option)
6168
@@ -69,7 +76,10 @@
6976
70 log("Installing {} package with options: {}".format(package,77 log("Installing {} package with options: {}".format(package,
71 command))78 command))
72 pip_execute(command)79 if venv:
80 subprocess.check_call(command)
81 else:
82 pip_execute(command)
7383
7484
75def pip_uninstall(package, **options):85def pip_uninstall(package, **options):
@@ -94,3 +104,16 @@
94 """Returns the list of current python installed packages104 """Returns the list of current python installed packages
95 """105 """
96 return pip_execute(["list"])106 return pip_execute(["list"])
107
108
109def pip_create_virtualenv(path=None):
110 """Create an isolated Python environment."""
111 apt_install('python-virtualenv')
112
113 if path:
114 venv_path = path
115 else:
116 venv_path = os.path.join(charm_dir(), 'venv')
117
118 if not os.path.exists(venv_path):
119 subprocess.check_call(['virtualenv', venv_path])
97120
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py 2015-04-16 11:21:16 +0000
+++ hooks/charmhelpers/core/hookenv.py 2015-06-04 23:04:28 +0000
@@ -21,12 +21,14 @@
21# Charm Helpers Developers <juju@lists.ubuntu.com>21# Charm Helpers Developers <juju@lists.ubuntu.com>
2222
23from __future__ import print_function23from __future__ import print_function
24from functools import wraps
24import os25import os
25import json26import json
26import yaml27import yaml
27import subprocess28import subprocess
28import sys29import sys
29import errno30import errno
31import tempfile
30from subprocess import CalledProcessError32from subprocess import CalledProcessError
3133
32import six34import six
@@ -58,15 +60,17 @@
5860
59 will cache the result of unit_get + 'test' for future calls.61 will cache the result of unit_get + 'test' for future calls.
60 """62 """
63 @wraps(func)
61 def wrapper(*args, **kwargs):64 def wrapper(*args, **kwargs):
62 global cache65 global cache
63 key = str((func, args, kwargs))66 key = str((func, args, kwargs))
64 try:67 try:
65 return cache[key]68 return cache[key]
66 except KeyError:69 except KeyError:
67 res = func(*args, **kwargs)70 pass # Drop out of the exception handler scope.
68 cache[key] = res71 res = func(*args, **kwargs)
69 return res72 cache[key] = res
73 return res
70 return wrapper74 return wrapper
7175
7276
@@ -178,7 +182,7 @@
178182
179def remote_unit():183def remote_unit():
180 """The remote unit for the current relation hook"""184 """The remote unit for the current relation hook"""
181 return os.environ['JUJU_REMOTE_UNIT']185 return os.environ.get('JUJU_REMOTE_UNIT', None)
182186
183187
184def service_name():188def service_name():
@@ -250,6 +254,12 @@
250 except KeyError:254 except KeyError:
251 return (self._prev_dict or {})[key]255 return (self._prev_dict or {})[key]
252256
257 def get(self, key, default=None):
258 try:
259 return self[key]
260 except KeyError:
261 return default
262
253 def keys(self):263 def keys(self):
254 prev_keys = []264 prev_keys = []
255 if self._prev_dict is not None:265 if self._prev_dict is not None:
@@ -353,18 +363,49 @@
353 """Set relation information for the current unit"""363 """Set relation information for the current unit"""
354 relation_settings = relation_settings if relation_settings else {}364 relation_settings = relation_settings if relation_settings else {}
355 relation_cmd_line = ['relation-set']365 relation_cmd_line = ['relation-set']
366 accepts_file = "--file" in subprocess.check_output(
367 relation_cmd_line + ["--help"], universal_newlines=True)
356 if relation_id is not None:368 if relation_id is not None:
357 relation_cmd_line.extend(('-r', relation_id))369 relation_cmd_line.extend(('-r', relation_id))
358 for k, v in (list(relation_settings.items()) + list(kwargs.items())):370 settings = relation_settings.copy()
359 if v is None:371 settings.update(kwargs)
360 relation_cmd_line.append('{}='.format(k))372 for key, value in settings.items():
361 else:373 # Force value to be a string: it always should, but some call
362 relation_cmd_line.append('{}={}'.format(k, v))374 # sites pass in things like dicts or numbers.
363 subprocess.check_call(relation_cmd_line)375 if value is not None:
376 settings[key] = "{}".format(value)
377 if accepts_file:
378 # --file was introduced in Juju 1.23.2. Use it by default if
379 # available, since otherwise we'll break if the relation data is
380 # too big. Ideally we should tell relation-set to read the data from
381 # stdin, but that feature is broken in 1.23.2: Bug #1454678.
382 with tempfile.NamedTemporaryFile(delete=False) as settings_file:
383 settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
384 subprocess.check_call(
385 relation_cmd_line + ["--file", settings_file.name])
386 os.remove(settings_file.name)
387 else:
388 for key, value in settings.items():
389 if value is None:
390 relation_cmd_line.append('{}='.format(key))
391 else:
392 relation_cmd_line.append('{}={}'.format(key, value))
393 subprocess.check_call(relation_cmd_line)
364 # Flush cache of any relation-gets for local unit394 # Flush cache of any relation-gets for local unit
365 flush(local_unit())395 flush(local_unit())
366396
367397
398def relation_clear(r_id=None):
399 ''' Clears any relation data already set on relation r_id '''
400 settings = relation_get(rid=r_id,
401 unit=local_unit())
402 for setting in settings:
403 if setting not in ['public-address', 'private-address']:
404 settings[setting] = None
405 relation_set(relation_id=r_id,
406 **settings)
407
408
368@cached409@cached
369def relation_ids(reltype=None):410def relation_ids(reltype=None):
370 """A list of relation_ids"""411 """A list of relation_ids"""
@@ -509,6 +550,11 @@
509 return None550 return None
510551
511552
553def unit_public_ip():
554 """Get this unit's public IP address"""
555 return unit_get('public-address')
556
557
512def unit_private_ip():558def unit_private_ip():
513 """Get this unit's private IP address"""559 """Get this unit's private IP address"""
514 return unit_get('private-address')560 return unit_get('private-address')
@@ -605,3 +651,94 @@
605651
606 The results set by action_set are preserved."""652 The results set by action_set are preserved."""
607 subprocess.check_call(['action-fail', message])653 subprocess.check_call(['action-fail', message])
654
655
656def status_set(workload_state, message):
657 """Set the workload state with a message
658
659 Use status-set to set the workload state with a message which is visible
660 to the user via juju status. If the status-set command is not found then
661 assume this is juju < 1.23 and juju-log the message unstead.
662
663 workload_state -- valid juju workload state.
664 message -- status update message
665 """
666 valid_states = ['maintenance', 'blocked', 'waiting', 'active']
667 if workload_state not in valid_states:
668 raise ValueError(
669 '{!r} is not a valid workload state'.format(workload_state)
670 )
671 cmd = ['status-set', workload_state, message]
672 try:
673 ret = subprocess.call(cmd)
674 if ret == 0:
675 return
676 except OSError as e:
677 if e.errno != errno.ENOENT:
678 raise
679 log_message = 'status-set failed: {} {}'.format(workload_state,
680 message)
681 log(log_message, level='INFO')
682
683
684def status_get():
685 """Retrieve the previously set juju workload state
686
687 If the status-set command is not found then assume this is juju < 1.23 and
688 return 'unknown'
689 """
690 cmd = ['status-get']
691 try:
692 raw_status = subprocess.check_output(cmd, universal_newlines=True)
693 status = raw_status.rstrip()
694 return status
695 except OSError as e:
696 if e.errno == errno.ENOENT:
697 return 'unknown'
698 else:
699 raise
700
701
702def translate_exc(from_exc, to_exc):
703 def inner_translate_exc1(f):
704 def inner_translate_exc2(*args, **kwargs):
705 try:
706 return f(*args, **kwargs)
707 except from_exc:
708 raise to_exc
709
710 return inner_translate_exc2
711
712 return inner_translate_exc1
713
714
715@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
716def is_leader():
717 """Does the current unit hold the juju leadership
718
719 Uses juju to determine whether the current unit is the leader of its peers
720 """
721 cmd = ['is-leader', '--format=json']
722 return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
723
724
725@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
726def leader_get(attribute=None):
727 """Juju leader get value(s)"""
728 cmd = ['leader-get', '--format=json'] + [attribute or '-']
729 return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
730
731
732@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
733def leader_set(settings=None, **kwargs):
734 """Juju leader set value(s)"""
735 log("Juju leader-set '%s'" % (settings), level=DEBUG)
736 cmd = ['leader-set']
737 settings = settings or {}
738 settings.update(kwargs)
739 for k, v in settings.iteritems():
740 if v is None:
741 cmd.append('{}='.format(k))
742 else:
743 cmd.append('{}={}'.format(k, v))
744 subprocess.check_call(cmd)
608745
=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py 2015-03-18 12:31:20 +0000
+++ hooks/charmhelpers/core/host.py 2015-06-04 23:04:28 +0000
@@ -90,7 +90,7 @@
90 ['service', service_name, 'status'],90 ['service', service_name, 'status'],
91 stderr=subprocess.STDOUT).decode('UTF-8')91 stderr=subprocess.STDOUT).decode('UTF-8')
92 except subprocess.CalledProcessError as e:92 except subprocess.CalledProcessError as e:
93 return 'unrecognized service' not in e.output93 return b'unrecognized service' not in e.output
94 else:94 else:
95 return True95 return True
9696
9797
=== modified file 'hooks/charmhelpers/core/services/base.py'
--- hooks/charmhelpers/core/services/base.py 2015-01-26 09:48:14 +0000
+++ hooks/charmhelpers/core/services/base.py 2015-06-04 23:04:28 +0000
@@ -15,9 +15,9 @@
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1616
17import os17import os
18import re
19import json18import json
20from collections import Iterable19from inspect import getargspec
20from collections import Iterable, OrderedDict
2121
22from charmhelpers.core import host22from charmhelpers.core import host
23from charmhelpers.core import hookenv23from charmhelpers.core import hookenv
@@ -119,7 +119,7 @@
119 """119 """
120 self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')120 self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
121 self._ready = None121 self._ready = None
122 self.services = {}122 self.services = OrderedDict()
123 for service in services or []:123 for service in services or []:
124 service_name = service['service']124 service_name = service['service']
125 self.services[service_name] = service125 self.services[service_name] = service
@@ -132,8 +132,8 @@
132 if hook_name == 'stop':132 if hook_name == 'stop':
133 self.stop_services()133 self.stop_services()
134 else:134 else:
135 self.reconfigure_services()
135 self.provide_data()136 self.provide_data()
136 self.reconfigure_services()
137 cfg = hookenv.config()137 cfg = hookenv.config()
138 if cfg.implicit_save:138 if cfg.implicit_save:
139 cfg.save()139 cfg.save()
@@ -145,15 +145,36 @@
145 A provider must have a `name` attribute, which indicates which relation145 A provider must have a `name` attribute, which indicates which relation
146 to set data on, and a `provide_data()` method, which returns a dict of146 to set data on, and a `provide_data()` method, which returns a dict of
147 data to set.147 data to set.
148
149 The `provide_data()` method can optionally accept two parameters:
150
151 * ``remote_service`` The name of the remote service that the data will
152 be provided to. The `provide_data()` method will be called once
153 for each connected service (not unit). This allows the method to
154 tailor its data to the given service.
155 * ``service_ready`` Whether or not the service definition had all of
156 its requirements met, and thus the ``data_ready`` callbacks run.
157
158 Note that the ``provided_data`` methods are now called **after** the
159 ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks
160 a chance to generate any data necessary for the providing to the remote
161 services.
148 """162 """
149 hook_name = hookenv.hook_name()163 for service_name, service in self.services.items():
150 for service in self.services.values():164 service_ready = self.is_ready(service_name)
151 for provider in service.get('provided_data', []):165 for provider in service.get('provided_data', []):
152 if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):166 for relid in hookenv.relation_ids(provider.name):
153 data = provider.provide_data()167 units = hookenv.related_units(relid)
154 _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data168 if not units:
155 if _ready:169 continue
156 hookenv.relation_set(None, data)170 remote_service = units[0].split('/')[0]
171 argspec = getargspec(provider.provide_data)
172 if len(argspec.args) > 1:
173 data = provider.provide_data(remote_service, service_ready)
174 else:
175 data = provider.provide_data()
176 if data:
177 hookenv.relation_set(relid, data)
157178
158 def reconfigure_services(self, *service_names):179 def reconfigure_services(self, *service_names):
159 """180 """
160181
=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py 2015-01-26 09:48:14 +0000
+++ hooks/charmhelpers/fetch/__init__.py 2015-06-04 23:04:28 +0000
@@ -158,7 +158,7 @@
158158
159def apt_cache(in_memory=True):159def apt_cache(in_memory=True):
160 """Build and return an apt cache"""160 """Build and return an apt cache"""
161 import apt_pkg161 from apt import apt_pkg
162 apt_pkg.init()162 apt_pkg.init()
163 if in_memory:163 if in_memory:
164 apt_pkg.config.set("Dir::Cache::pkgcache", "")164 apt_pkg.config.set("Dir::Cache::pkgcache", "")
165165
=== modified file 'hooks/charmhelpers/fetch/giturl.py'
--- hooks/charmhelpers/fetch/giturl.py 2015-03-13 12:57:45 +0000
+++ hooks/charmhelpers/fetch/giturl.py 2015-06-04 23:04:28 +0000
@@ -45,14 +45,16 @@
45 else:45 else:
46 return True46 return True
4747
48 def clone(self, source, dest, branch):48 def clone(self, source, dest, branch, depth=None):
49 if not self.can_handle(source):49 if not self.can_handle(source):
50 raise UnhandledSource("Cannot handle {}".format(source))50 raise UnhandledSource("Cannot handle {}".format(source))
5151
52 repo = Repo.clone_from(source, dest)52 if depth:
53 repo.git.checkout(branch)53 Repo.clone_from(source, dest, branch=branch, depth=depth)
54 else:
55 Repo.clone_from(source, dest, branch=branch)
5456
55 def install(self, source, branch="master", dest=None):57 def install(self, source, branch="master", dest=None, depth=None):
56 url_parts = self.parse_url(source)58 url_parts = self.parse_url(source)
57 branch_name = url_parts.path.strip("/").split("/")[-1]59 branch_name = url_parts.path.strip("/").split("/")[-1]
58 if dest:60 if dest:
@@ -63,7 +65,7 @@
63 if not os.path.exists(dest_dir):65 if not os.path.exists(dest_dir):
64 mkdir(dest_dir, perms=0o755)66 mkdir(dest_dir, perms=0o755)
65 try:67 try:
66 self.clone(source, dest_dir, branch)68 self.clone(source, dest_dir, branch, depth)
67 except GitCommandError as e:69 except GitCommandError as e:
68 raise UnhandledSource(e.message)70 raise UnhandledSource(e.message)
69 except OSError as e:71 except OSError as e:
7072
=== modified file 'unit_tests/test_ceilometer_hooks.py'
--- unit_tests/test_ceilometer_hooks.py 2015-04-08 15:49:34 +0000
+++ unit_tests/test_ceilometer_hooks.py 2015-06-04 23:04:28 +0000
@@ -31,7 +31,6 @@
31 'get_ceilometer_context',31 'get_ceilometer_context',
32 'lsb_release',32 'lsb_release',
33 'get_packages',33 'get_packages',
34 'canonical_url',
35 'service_restart',34 'service_restart',
36 'update_nrpe_config',35 'update_nrpe_config',
37 'configure_https',36 'configure_https',
@@ -136,9 +135,10 @@
136 self.assertTrue(self.CONFIGS.write_all.called)135 self.assertTrue(self.CONFIGS.write_all.called)
137 self.assertTrue(joined.called)136 self.assertTrue(joined.called)
138137
138 @patch.object(hooks, 'canonical_url')
139 @patch('charmhelpers.core.hookenv.config')139 @patch('charmhelpers.core.hookenv.config')
140 def test_keystone_joined(self, mock_config):140 def test_keystone_joined(self, mock_config, _canonical_url):
141 self.canonical_url.return_value = "http://thishost"141 _canonical_url.return_value = "http://thishost"
142 self.test_config.set('region', 'myregion')142 self.test_config.set('region', 'myregion')
143 hooks.hooks.execute(['hooks/identity-service-relation-joined'])143 hooks.hooks.execute(['hooks/identity-service-relation-joined'])
144 url = "http://{}:{}".format('thishost', hooks.CEILOMETER_PORT)144 url = "http://{}:{}".format('thishost', hooks.CEILOMETER_PORT)
@@ -148,6 +148,30 @@
148 requested_roles=hooks.CEILOMETER_ROLE,148 requested_roles=hooks.CEILOMETER_ROLE,
149 region='myregion', relation_id=None)149 region='myregion', relation_id=None)
150150
151 @patch('charmhelpers.contrib.openstack.ip.service_name',
152 lambda *args: 'ceilometer')
153 @patch('charmhelpers.contrib.openstack.ip.unit_get')
154 @patch('charmhelpers.contrib.openstack.ip.is_clustered')
155 @patch('charmhelpers.core.hookenv.config')
156 @patch('charmhelpers.contrib.openstack.ip.config')
157 def test_keystone_joined_url_override(self, _config, mock_config,
158 _is_clustered, _unit_get):
159 _unit_get.return_value = "thishost"
160 _is_clustered.return_value = False
161 _config.side_effect = self.test_config.get
162 mock_config.side_effect = self.test_config.get
163 self.test_config.set('region', 'myregion')
164 self.test_config.set('os-public-hostname', 'ceilometer.example.com')
165 hooks.keystone_joined(None)
166 url = "http://{}:{}".format('thishost', hooks.CEILOMETER_PORT)
167 public_url = "http://{}:{}".format('ceilometer.example.com',
168 hooks.CEILOMETER_PORT)
169 self.relation_set.assert_called_with(
170 service=hooks.CEILOMETER_SERVICE,
171 public_url=public_url, admin_url=url, internal_url=url,
172 requested_roles=hooks.CEILOMETER_ROLE,
173 region='myregion', relation_id=None)
174
151 @patch('charmhelpers.core.hookenv.config')175 @patch('charmhelpers.core.hookenv.config')
152 def test_ceilometer_joined(self, mock_config):176 def test_ceilometer_joined(self, mock_config):
153 self.relation_ids.return_value = ['ceilometer:0']177 self.relation_ids.return_value = ['ceilometer:0']

Subscribers

People subscribed via source and target branches