Merge lp:~johnsca/charms/trusty/cf-go-router/refactor into lp:~cf-charmers/charms/trusty/cf-go-router/trunk

Proposed by Cory Johns
Status: Merged
Merged at revision: 27
Proposed branch: lp:~johnsca/charms/trusty/cf-go-router/refactor
Merge into: lp:~cf-charmers/charms/trusty/cf-go-router/trunk
Diff against target: 1628 lines (+720/-430)
19 files modified
hooks/charmhelpers/contrib/cloudfoundry/common.py (+3/-62)
hooks/charmhelpers/contrib/cloudfoundry/config_helper.py (+0/-11)
hooks/charmhelpers/contrib/cloudfoundry/contexts.py (+26/-54)
hooks/charmhelpers/contrib/cloudfoundry/install.py (+0/-35)
hooks/charmhelpers/contrib/cloudfoundry/services.py (+0/-118)
hooks/charmhelpers/contrib/cloudfoundry/upstart_helper.py (+0/-14)
hooks/charmhelpers/contrib/hahelpers/apache.py (+9/-8)
hooks/charmhelpers/contrib/openstack/context.py (+107/-26)
hooks/charmhelpers/contrib/openstack/neutron.py (+31/-5)
hooks/charmhelpers/contrib/openstack/utils.py (+9/-1)
hooks/charmhelpers/contrib/storage/linux/lvm.py (+1/-1)
hooks/charmhelpers/contrib/storage/linux/utils.py (+28/-5)
hooks/charmhelpers/core/hookenv.py (+98/-1)
hooks/charmhelpers/core/host.py (+47/-0)
hooks/charmhelpers/core/services.py (+84/-0)
hooks/charmhelpers/core/templating.py (+158/-0)
hooks/charmhelpers/fetch/__init__.py (+97/-65)
hooks/hooks.py (+16/-14)
hooks/install (+6/-10)
To merge this branch: bzr merge lp:~johnsca/charms/trusty/cf-go-router/refactor
Reviewer Review Type Date Requested Status
Cloud Foundry Charmers Pending
Review via email: mp+219914@code.launchpad.net

Description of the change

Refactored to use refactored charm-helpers

https://codereview.appspot.com/96360049/

To post a comment you must log in.
Revision history for this message
Cory Johns (johnsca) wrote :

Reviewers: mp+219914_code.launchpad.net,

Message:
Please take a look.

Description:
Refactored to use refactored charm-helpers

https://code.launchpad.net/~johnsca/charms/trusty/cf-go-router/refactor/+merge/219914

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/96360049/

Affected files (+462, -357 lines):
   A [revision details]
   M hooks/charmhelpers/contrib/cloudfoundry/common.py
   D hooks/charmhelpers/contrib/cloudfoundry/config_helper.py
   M hooks/charmhelpers/contrib/cloudfoundry/contexts.py
   D hooks/charmhelpers/contrib/cloudfoundry/install.py
   D hooks/charmhelpers/contrib/cloudfoundry/services.py
   D hooks/charmhelpers/contrib/cloudfoundry/upstart_helper.py
   M hooks/charmhelpers/contrib/hahelpers/apache.py
   M hooks/charmhelpers/contrib/openstack/context.py
   M hooks/charmhelpers/contrib/openstack/neutron.py
   M hooks/charmhelpers/contrib/openstack/utils.py
   M hooks/charmhelpers/contrib/storage/linux/utils.py
   M hooks/charmhelpers/core/host.py
   A hooks/charmhelpers/core/services.py
   A hooks/charmhelpers/core/templating.py
   M hooks/charmhelpers/fetch/__init__.py
   M hooks/hooks.py
   M hooks/install
   M templates/gorouter.conf

Revision history for this message
Benjamin Saller (bcsaller) wrote :
28. By Cory Johns

Merged :parent

29. By Cory Johns

Resynced charm-helpers

30. By Cory Johns

Removed unnecessary prefix from config var

31. By Cory Johns

Slight addendum to previous commit

Revision history for this message
Cory Johns (johnsca) wrote :

*** Submitted:

Refactored to use refactored charm-helpers

R=benjamin.saller
CC=
https://codereview.appspot.com/96360049

https://codereview.appspot.com/96360049/

Revision history for this message
Alex Lomov (lomov-as) wrote :

I think that it would be better to rename gorouter.conf upstart job to
cf-gorouter.conf or cf-go-router.conf (I mean this change
https://codereview.appspot.com/96360049/patch/20001/30021),
just to do it in the same way with other charms.

On 21 May 2014 01:03, Cory Johns <email address hidden> wrote:

> *** Submitted:
>
> Refactored to use refactored charm-helpers
>
> R=benjamin.saller
> CC=
> https://codereview.appspot.com/96360049
>
>
> https://codereview.appspot.com/96360049/
>
> --
>
> https://code.launchpad.net/~johnsca/charms/trusty/cf-go-router/refactor/+merge/219914
> Your team Cloud Foundry Charmers is requested to review the proposed merge
> of lp:~johnsca/charms/trusty/cf-go-router/refactor into
> lp:~cf-charmers/charms/trusty/cf-go-router/trunk.
>
> --
> Mailing list: https://launchpad.net/~cf-charmers
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~cf-charmers
> More help : https://help.launchpad.net/ListHelp
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== removed directory 'files'
=== removed directory 'files/upstart'
=== modified file 'hooks/charmhelpers/contrib/cloudfoundry/common.py'
--- hooks/charmhelpers/contrib/cloudfoundry/common.py 2014-05-07 16:31:23 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/common.py 2014-05-20 21:57:55 +0000
@@ -1,11 +1,3 @@
1import sys
2import os
3import pwd
4import grp
5import subprocess
6
7from contextlib import contextmanager
8from charmhelpers.core.hookenv import log, ERROR, DEBUG
9from charmhelpers.core import host1from charmhelpers.core import host
102
11from charmhelpers.fetch import (3from charmhelpers.fetch import (
@@ -13,59 +5,8 @@
13)5)
146
157
16def run(command, exit_on_error=True, quiet=False):
17 '''Run a command and return the output.'''
18 if not quiet:
19 log("Running {!r}".format(command), DEBUG)
20 p = subprocess.Popen(
21 command, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
22 shell=isinstance(command, basestring))
23 p.stdin.close()
24 lines = []
25 for line in p.stdout:
26 if line:
27 if not quiet:
28 print line
29 lines.append(line)
30 elif p.poll() is not None:
31 break
32
33 p.wait()
34
35 if p.returncode == 0:
36 return '\n'.join(lines)
37
38 if p.returncode != 0 and exit_on_error:
39 log("ERROR: {}".format(p.returncode), ERROR)
40 sys.exit(p.returncode)
41
42 raise subprocess.CalledProcessError(
43 p.returncode, command, '\n'.join(lines))
44
45
46def chownr(path, owner, group):
47 uid = pwd.getpwnam(owner).pw_uid
48 gid = grp.getgrnam(group).gr_gid
49 for root, dirs, files in os.walk(path):
50 for momo in dirs:
51 os.chown(os.path.join(root, momo), uid, gid)
52 for momo in files:
53 os.chown(os.path.join(root, momo), uid, gid)
54
55
56@contextmanager
57def chdir(d):
58 cur = os.getcwd()
59 try:
60 yield os.chdir(d)
61 finally:
62 os.chdir(cur)
63
64
65def prepare_cloudfoundry_environment(config_data, packages):8def prepare_cloudfoundry_environment(config_data, packages):
66 if 'source' in config_data:9 add_source(config_data['source'], config_data.get('key'))
67 add_source(config_data['source'], config_data.get('key'))10 apt_update(fatal=True)
68 apt_update(fatal=True)11 apt_install(packages=filter_installed_packages(packages), fatal=True)
69 if packages:
70 apt_install(packages=filter_installed_packages(packages), fatal=True)
71 host.adduser('vcap')12 host.adduser('vcap')
7213
=== removed file 'hooks/charmhelpers/contrib/cloudfoundry/config_helper.py'
--- hooks/charmhelpers/contrib/cloudfoundry/config_helper.py 2014-04-05 16:51:21 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/config_helper.py 1970-01-01 00:00:00 +0000
@@ -1,11 +0,0 @@
1import jinja2
2
3TEMPLATES_DIR = 'templates'
4
5def render_template(template_name, context, template_dir=TEMPLATES_DIR):
6 templates = jinja2.Environment(
7 loader=jinja2.FileSystemLoader(template_dir))
8 template = templates.get_template(template_name)
9 return template.render(context)
10
11
120
=== modified file 'hooks/charmhelpers/contrib/cloudfoundry/contexts.py'
--- hooks/charmhelpers/contrib/cloudfoundry/contexts.py 2014-05-07 16:31:23 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/contexts.py 2014-05-20 21:57:55 +0000
@@ -1,54 +1,16 @@
1import os1import os
2import yaml2
33from charmhelpers.core.templating import (
4from charmhelpers.core import hookenv4 ContextGenerator,
5from charmhelpers.contrib.openstack.context import OSContextGenerator5 RelationContext,
66 StorableContext,
77)
8class RelationContext(OSContextGenerator):
9 def __call__(self):
10 if not hookenv.relation_ids(self.interface):
11 return {}
12
13 ctx = {}
14 for rid in hookenv.relation_ids(self.interface):
15 for unit in hookenv.related_units(rid):
16 reldata = hookenv.relation_get(rid=rid, unit=unit)
17 required = set(self.required_keys)
18 if set(reldata.keys()).issuperset(required):
19 ns = ctx.setdefault(self.interface, {})
20 for k, v in reldata.items():
21 ns[k] = v
22 return ctx
23
24 return {}
25
26
27class ConfigContext(OSContextGenerator):
28 def __call__(self):
29 return hookenv.config()
30
31
32class StorableContext(object):
33
34 def store_context(self, file_name, config_data):
35 with open(file_name, 'w') as file_stream:
36 yaml.dump(config_data, file_stream)
37
38 def read_context(self, file_name):
39 with open(file_name, 'r') as file_stream:
40 data = yaml.load(file_stream)
41 if not data:
42 raise OSError("%s is empty" % file_name)
43 return data
448
459
46# Stores `config_data` hash into yaml file with `file_name` as a name10# Stores `config_data` hash into yaml file with `file_name` as a name
47# if `file_name` already exists, then it loads data from `file_name`.11# if `file_name` already exists, then it loads data from `file_name`.
48class StoredContext(OSContextGenerator, StorableContext):12class StoredContext(ContextGenerator, StorableContext):
49
50 def __init__(self, file_name, config_data):13 def __init__(self, file_name, config_data):
51 self.data = config_data
52 if os.path.exists(file_name):14 if os.path.exists(file_name):
53 self.data = self.read_context(file_name)15 self.data = self.read_context(file_name)
54 else:16 else:
@@ -59,25 +21,35 @@
59 return self.data21 return self.data
6022
6123
62class StaticContext(OSContextGenerator):
63 def __init__(self, data):
64 self.data = data
65
66 def __call__(self):
67 return self.data
68
69
70class NatsContext(RelationContext):24class NatsContext(RelationContext):
71 interface = 'nats'25 interface = 'nats'
72 required_keys = ['nats_port', 'nats_address', 'nats_user', 'nats_password']26 required_keys = ['nats_port', 'nats_address', 'nats_user', 'nats_password']
7327
7428
29class MysqlDSNContext(RelationContext):
30 interface = 'db'
31 required_keys = ['user', 'password', 'host', 'database']
32 dsn_template = "mysql2://{user}:{password}@{host}:{port}/{database}"
33
34 def __call__(self):
35 ctx = RelationContext.__call__(self)
36 if ctx:
37 if 'port' not in ctx:
38 ctx['db']['port'] = '3306'
39 ctx['db']['dsn'] = self.dsn_template.format(**ctx['db'])
40 return ctx
41
42
75class RouterContext(RelationContext):43class RouterContext(RelationContext):
76 interface = 'router'44 interface = 'router'
77 required_keys = ['domain']45 required_keys = ['domain']
7846
7947
48class LogRouterContext(RelationContext):
49 interface = 'logrouter'
50 required_keys = ['shared-secret', 'logrouter-address']
51
52
80class LoggregatorContext(RelationContext):53class LoggregatorContext(RelationContext):
81 interface = 'loggregator'54 interface = 'loggregator'
82 required_keys = ['shared_secret', 'loggregator_address']55 required_keys = ['shared_secret', 'loggregator_address']
83
8456
=== removed file 'hooks/charmhelpers/contrib/cloudfoundry/install.py'
--- hooks/charmhelpers/contrib/cloudfoundry/install.py 2014-04-01 07:20:45 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/install.py 1970-01-01 00:00:00 +0000
@@ -1,35 +0,0 @@
1import os
2import subprocess
3
4
5def install(src, dest, fileprops=None, sudo=False):
6 """Install a file from src to dest. Dest can be a complete filename
7 or a target directory. fileprops is a dict with 'owner' (username of owner)
8 and mode (octal string) as keys, the defaults are 'ubuntu' and '400'
9
10 When owner is passed or when access requires it sudo can be set to True and
11 sudo will be used to install the file.
12 """
13 if not fileprops:
14 fileprops = {}
15 mode = fileprops.get('mode', '400')
16 owner = fileprops.get('owner')
17 cmd = ['install']
18
19 if not os.path.exists(src):
20 raise OSError(src)
21
22 if not os.path.exists(dest) and not os.path.exists(os.path.dirname(dest)):
23 # create all but the last component as path
24 cmd.append('-D')
25
26 if mode:
27 cmd.extend(['-m', mode])
28
29 if owner:
30 cmd.extend(['-o', owner])
31
32 if sudo:
33 cmd.insert(0, 'sudo')
34 cmd.extend([src, dest])
35 subprocess.check_call(cmd)
360
=== removed file 'hooks/charmhelpers/contrib/cloudfoundry/services.py'
--- hooks/charmhelpers/contrib/cloudfoundry/services.py 2014-04-05 16:51:21 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/services.py 1970-01-01 00:00:00 +0000
@@ -1,118 +0,0 @@
1import os
2import tempfile
3from charmhelpers.core import host
4
5from charmhelpers.contrib.cloudfoundry.install import install
6from charmhelpers.core.hookenv import log
7from jinja2 import Environment, FileSystemLoader
8
9SERVICE_CONFIG = []
10TEMPLATE_LOADER = None
11
12
13def render_template(template_name, context):
14 """Render template to a tempfile returning the name"""
15 _, fn = tempfile.mkstemp()
16 template = load_template(template_name)
17 output = template.render(context)
18 with open(fn, "w") as fp:
19 fp.write(output)
20 return fn
21
22
23def collect_contexts(context_providers):
24 ctx = {}
25 for provider in context_providers:
26 c = provider()
27 if not c:
28 return {}
29 ctx.update(c)
30 return ctx
31
32
33def load_template(name):
34 return TEMPLATE_LOADER.get_template(name)
35
36
37def configure_templates(template_dir):
38 global TEMPLATE_LOADER
39 TEMPLATE_LOADER = Environment(loader=FileSystemLoader(template_dir))
40
41
42def register(service_configs, template_dir):
43 """Register a list of service configs.
44
45 Service Configs are dicts in the following formats:
46
47 {
48 "service": <service name>,
49 "templates": [ {
50 'target': <render target of template>,
51 'source': <optional name of template in passed in template_dir>
52 'file_properties': <optional dict taking owner and octal mode>
53 'contexts': [ context generators, see contexts.py ]
54 }
55 ] }
56
57 If 'source' is not provided for a template the template_dir will
58 be consulted for ``basename(target).j2``.
59 """
60 global SERVICE_CONFIG
61 if template_dir:
62 configure_templates(template_dir)
63 SERVICE_CONFIG.extend(service_configs)
64
65
66def reset():
67 global SERVICE_CONFIG
68 SERVICE_CONFIG = []
69
70
71# def service_context(name):
72# contexts = collect_contexts(template['contexts'])
73
74def reconfigure_service(service_name, restart=True):
75 global SERVICE_CONFIG
76 service = None
77 for service in SERVICE_CONFIG:
78 if service['service'] == service_name:
79 break
80 if not service or service['service'] != service_name:
81 raise KeyError('Service not registered: %s' % service_name)
82
83 templates = service['templates']
84 for template in templates:
85 contexts = collect_contexts(template['contexts'])
86 if contexts:
87 template_target = template['target']
88 default_template = "%s.j2" % os.path.basename(template_target)
89 template_name = template.get('source', default_template)
90 output_file = render_template(template_name, contexts)
91 file_properties = template.get('file_properties')
92 install(output_file, template_target, file_properties)
93 os.unlink(output_file)
94 else:
95 restart = False
96
97 if restart:
98 host.service_restart(service_name)
99
100
101def stop_services():
102 global SERVICE_CONFIG
103 for service in SERVICE_CONFIG:
104 if host.service_running(service['service']):
105 host.service_stop(service['service'])
106
107
108def get_service(service_name):
109 global SERVICE_CONFIG
110 for service in SERVICE_CONFIG:
111 if service_name == service['service']:
112 return service
113 return None
114
115
116def reconfigure_services(restart=True):
117 for service in SERVICE_CONFIG:
118 reconfigure_service(service['service'], restart=restart)
1190
=== removed file 'hooks/charmhelpers/contrib/cloudfoundry/upstart_helper.py'
--- hooks/charmhelpers/contrib/cloudfoundry/upstart_helper.py 2014-04-05 16:51:21 +0000
+++ hooks/charmhelpers/contrib/cloudfoundry/upstart_helper.py 1970-01-01 00:00:00 +0000
@@ -1,14 +0,0 @@
1import os
2import glob
3from charmhelpers.core import hookenv
4from charmhelpers.core.hookenv import charm_dir
5from charmhelpers.contrib.cloudfoundry.install import install
6
7
8def install_upstart_scripts(dirname=os.path.join(hookenv.charm_dir(),
9 'files/upstart'),
10 pattern='*.conf'):
11 for script in glob.glob("%s/%s" % (dirname, pattern)):
12 filename = os.path.join(dirname, script)
13 hookenv.log('Installing upstart job:' + filename, hookenv.DEBUG)
14 install(filename, '/etc/init')
150
=== modified file 'hooks/charmhelpers/contrib/hahelpers/apache.py'
--- hooks/charmhelpers/contrib/hahelpers/apache.py 2014-04-01 09:50:03 +0000
+++ hooks/charmhelpers/contrib/hahelpers/apache.py 2014-05-20 21:57:55 +0000
@@ -39,14 +39,15 @@
3939
4040
41def get_ca_cert():41def get_ca_cert():
42 ca_cert = None42 ca_cert = config_get('ssl_ca')
43 log("Inspecting identity-service relations for CA SSL certificate.",43 if ca_cert is None:
44 level=INFO)44 log("Inspecting identity-service relations for CA SSL certificate.",
45 for r_id in relation_ids('identity-service'):45 level=INFO)
46 for unit in relation_list(r_id):46 for r_id in relation_ids('identity-service'):
47 if not ca_cert:47 for unit in relation_list(r_id):
48 ca_cert = relation_get('ca_cert',48 if ca_cert is None:
49 rid=r_id, unit=unit)49 ca_cert = relation_get('ca_cert',
50 rid=r_id, unit=unit)
50 return ca_cert51 return ca_cert
5152
5253
5354
=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
--- hooks/charmhelpers/contrib/openstack/context.py 2014-04-01 07:20:45 +0000
+++ hooks/charmhelpers/contrib/openstack/context.py 2014-05-20 21:57:55 +0000
@@ -1,5 +1,6 @@
1import json1import json
2import os2import os
3import time
34
4from base64 import b64decode5from base64 import b64decode
56
@@ -113,7 +114,8 @@
113class SharedDBContext(OSContextGenerator):114class SharedDBContext(OSContextGenerator):
114 interfaces = ['shared-db']115 interfaces = ['shared-db']
115116
116 def __init__(self, database=None, user=None, relation_prefix=None):117 def __init__(self,
118 database=None, user=None, relation_prefix=None, ssl_dir=None):
117 '''119 '''
118 Allows inspecting relation for settings prefixed with relation_prefix.120 Allows inspecting relation for settings prefixed with relation_prefix.
119 This is useful for parsing access for multiple databases returned via121 This is useful for parsing access for multiple databases returned via
@@ -122,6 +124,7 @@
122 self.relation_prefix = relation_prefix124 self.relation_prefix = relation_prefix
123 self.database = database125 self.database = database
124 self.user = user126 self.user = user
127 self.ssl_dir = ssl_dir
125128
126 def __call__(self):129 def __call__(self):
127 self.database = self.database or config('database')130 self.database = self.database or config('database')
@@ -139,17 +142,72 @@
139142
140 for rid in relation_ids('shared-db'):143 for rid in relation_ids('shared-db'):
141 for unit in related_units(rid):144 for unit in related_units(rid):
142 passwd = relation_get(password_setting, rid=rid, unit=unit)145 rdata = relation_get(rid=rid, unit=unit)
143 ctxt = {146 ctxt = {
144 'database_host': relation_get('db_host', rid=rid,147 'database_host': rdata.get('db_host'),
145 unit=unit),
146 'database': self.database,148 'database': self.database,
147 'database_user': self.user,149 'database_user': self.user,
148 'database_password': passwd,150 'database_password': rdata.get(password_setting),
149 }151 'database_type': 'mysql'
150 if context_complete(ctxt):152 }
151 return ctxt153 if context_complete(ctxt):
152 return {}154 db_ssl(rdata, ctxt, self.ssl_dir)
155 return ctxt
156 return {}
157
158
159class PostgresqlDBContext(OSContextGenerator):
160 interfaces = ['pgsql-db']
161
162 def __init__(self, database=None):
163 self.database = database
164
165 def __call__(self):
166 self.database = self.database or config('database')
167 if self.database is None:
168 log('Could not generate postgresql_db context. '
169 'Missing required charm config options. '
170 '(database name)')
171 raise OSContextError
172 ctxt = {}
173
174 for rid in relation_ids(self.interfaces[0]):
175 for unit in related_units(rid):
176 ctxt = {
177 'database_host': relation_get('host', rid=rid, unit=unit),
178 'database': self.database,
179 'database_user': relation_get('user', rid=rid, unit=unit),
180 'database_password': relation_get('password', rid=rid, unit=unit),
181 'database_type': 'postgresql',
182 }
183 if context_complete(ctxt):
184 return ctxt
185 return {}
186
187
188def db_ssl(rdata, ctxt, ssl_dir):
189 if 'ssl_ca' in rdata and ssl_dir:
190 ca_path = os.path.join(ssl_dir, 'db-client.ca')
191 with open(ca_path, 'w') as fh:
192 fh.write(b64decode(rdata['ssl_ca']))
193 ctxt['database_ssl_ca'] = ca_path
194 elif 'ssl_ca' in rdata:
195 log("Charm not setup for ssl support but ssl ca found")
196 return ctxt
197 if 'ssl_cert' in rdata:
198 cert_path = os.path.join(
199 ssl_dir, 'db-client.cert')
200 if not os.path.exists(cert_path):
201 log("Waiting 1m for ssl client cert validity")
202 time.sleep(60)
203 with open(cert_path, 'w') as fh:
204 fh.write(b64decode(rdata['ssl_cert']))
205 ctxt['database_ssl_cert'] = cert_path
206 key_path = os.path.join(ssl_dir, 'db-client.key')
207 with open(key_path, 'w') as fh:
208 fh.write(b64decode(rdata['ssl_key']))
209 ctxt['database_ssl_key'] = key_path
210 return ctxt
153211
154212
155class IdentityServiceContext(OSContextGenerator):213class IdentityServiceContext(OSContextGenerator):
@@ -161,24 +219,25 @@
161219
162 for rid in relation_ids('identity-service'):220 for rid in relation_ids('identity-service'):
163 for unit in related_units(rid):221 for unit in related_units(rid):
222 rdata = relation_get(rid=rid, unit=unit)
164 ctxt = {223 ctxt = {
165 'service_port': relation_get('service_port', rid=rid,224 'service_port': rdata.get('service_port'),
166 unit=unit),225 'service_host': rdata.get('service_host'),
167 'service_host': relation_get('service_host', rid=rid,226 'auth_host': rdata.get('auth_host'),
168 unit=unit),227 'auth_port': rdata.get('auth_port'),
169 'auth_host': relation_get('auth_host', rid=rid, unit=unit),228 'admin_tenant_name': rdata.get('service_tenant'),
170 'auth_port': relation_get('auth_port', rid=rid, unit=unit),229 'admin_user': rdata.get('service_username'),
171 'admin_tenant_name': relation_get('service_tenant',230 'admin_password': rdata.get('service_password'),
172 rid=rid, unit=unit),231 'service_protocol':
173 'admin_user': relation_get('service_username', rid=rid,232 rdata.get('service_protocol') or 'http',
174 unit=unit),233 'auth_protocol':
175 'admin_password': relation_get('service_password', rid=rid,234 rdata.get('auth_protocol') or 'http',
176 unit=unit),
177 # XXX: Hard-coded http.
178 'service_protocol': 'http',
179 'auth_protocol': 'http',
180 }235 }
181 if context_complete(ctxt):236 if context_complete(ctxt):
237 # NOTE(jamespage) this is required for >= icehouse
238 # so a missing value just indicates keystone needs
239 # upgrading
240 ctxt['admin_tenant_id'] = rdata.get('service_tenant_id')
182 return ctxt241 return ctxt
183 return {}242 return {}
184243
@@ -186,6 +245,9 @@
186class AMQPContext(OSContextGenerator):245class AMQPContext(OSContextGenerator):
187 interfaces = ['amqp']246 interfaces = ['amqp']
188247
248 def __init__(self, ssl_dir=None):
249 self.ssl_dir = ssl_dir
250
189 def __call__(self):251 def __call__(self):
190 log('Generating template context for amqp')252 log('Generating template context for amqp')
191 conf = config()253 conf = config()
@@ -196,7 +258,6 @@
196 log('Could not generate shared_db context. '258 log('Could not generate shared_db context. '
197 'Missing required charm config options: %s.' % e)259 'Missing required charm config options: %s.' % e)
198 raise OSContextError260 raise OSContextError
199
200 ctxt = {}261 ctxt = {}
201 for rid in relation_ids('amqp'):262 for rid in relation_ids('amqp'):
202 ha_vip_only = False263 ha_vip_only = False
@@ -214,6 +275,14 @@
214 unit=unit),275 unit=unit),
215 'rabbitmq_virtual_host': vhost,276 'rabbitmq_virtual_host': vhost,
216 })277 })
278
279 ssl_port = relation_get('ssl_port', rid=rid, unit=unit)
280 if ssl_port:
281 ctxt['rabbit_ssl_port'] = ssl_port
282 ssl_ca = relation_get('ssl_ca', rid=rid, unit=unit)
283 if ssl_ca:
284 ctxt['rabbit_ssl_ca'] = ssl_ca
285
217 if relation_get('ha_queues', rid=rid, unit=unit) is not None:286 if relation_get('ha_queues', rid=rid, unit=unit) is not None:
218 ctxt['rabbitmq_ha_queues'] = True287 ctxt['rabbitmq_ha_queues'] = True
219288
@@ -221,6 +290,16 @@
221 rid=rid, unit=unit) is not None290 rid=rid, unit=unit) is not None
222291
223 if context_complete(ctxt):292 if context_complete(ctxt):
293 if 'rabbit_ssl_ca' in ctxt:
294 if not self.ssl_dir:
295 log(("Charm not setup for ssl support "
296 "but ssl ca found"))
297 break
298 ca_path = os.path.join(
299 self.ssl_dir, 'rabbit-client-ca.pem')
300 with open(ca_path, 'w') as fh:
301 fh.write(b64decode(ctxt['rabbit_ssl_ca']))
302 ctxt['rabbit_ssl_ca'] = ca_path
224 # Sufficient information found = break out!303 # Sufficient information found = break out!
225 break304 break
226 # Used for active/active rabbitmq >= grizzly305 # Used for active/active rabbitmq >= grizzly
@@ -391,6 +470,8 @@
391 'private_address': unit_get('private-address'),470 'private_address': unit_get('private-address'),
392 'endpoints': []471 'endpoints': []
393 }472 }
473 if is_clustered():
474 ctxt['private_address'] = config('vip')
394 for api_port in self.external_ports:475 for api_port in self.external_ports:
395 ext_port = determine_apache_port(api_port)476 ext_port = determine_apache_port(api_port)
396 int_port = determine_api_port(api_port)477 int_port = determine_api_port(api_port)
@@ -489,7 +570,7 @@
489570
490 if self.plugin == 'ovs':571 if self.plugin == 'ovs':
491 ctxt.update(self.ovs_ctxt())572 ctxt.update(self.ovs_ctxt())
492 elif self.plugin == 'nvp':573 elif self.plugin in ['nvp', 'nsx']:
493 ctxt.update(self.nvp_ctxt())574 ctxt.update(self.nvp_ctxt())
494575
495 alchemy_flags = config('neutron-alchemy-flags')576 alchemy_flags = config('neutron-alchemy-flags')
496577
=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py 2014-04-01 07:20:45 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py 2014-05-20 21:57:55 +0000
@@ -17,6 +17,8 @@
17 kver = check_output(['uname', '-r']).strip()17 kver = check_output(['uname', '-r']).strip()
18 return 'linux-headers-%s' % kver18 return 'linux-headers-%s' % kver
1919
20QUANTUM_CONF_DIR = '/etc/quantum'
21
2022
21def kernel_version():23def kernel_version():
22 """ Retrieve the current major kernel version as a tuple e.g. (3, 13) """24 """ Retrieve the current major kernel version as a tuple e.g. (3, 13) """
@@ -35,6 +37,8 @@
3537
3638
37# legacy39# legacy
40
41
38def quantum_plugins():42def quantum_plugins():
39 from charmhelpers.contrib.openstack import context43 from charmhelpers.contrib.openstack import context
40 return {44 return {
@@ -46,7 +50,8 @@
46 'contexts': [50 'contexts': [
47 context.SharedDBContext(user=config('neutron-database-user'),51 context.SharedDBContext(user=config('neutron-database-user'),
48 database=config('neutron-database'),52 database=config('neutron-database'),
49 relation_prefix='neutron')],53 relation_prefix='neutron',
54 ssl_dir=QUANTUM_CONF_DIR)],
50 'services': ['quantum-plugin-openvswitch-agent'],55 'services': ['quantum-plugin-openvswitch-agent'],
51 'packages': [[headers_package()] + determine_dkms_package(),56 'packages': [[headers_package()] + determine_dkms_package(),
52 ['quantum-plugin-openvswitch-agent']],57 ['quantum-plugin-openvswitch-agent']],
@@ -61,7 +66,8 @@
61 'contexts': [66 'contexts': [
62 context.SharedDBContext(user=config('neutron-database-user'),67 context.SharedDBContext(user=config('neutron-database-user'),
63 database=config('neutron-database'),68 database=config('neutron-database'),
64 relation_prefix='neutron')],69 relation_prefix='neutron',
70 ssl_dir=QUANTUM_CONF_DIR)],
65 'services': [],71 'services': [],
66 'packages': [],72 'packages': [],
67 'server_packages': ['quantum-server',73 'server_packages': ['quantum-server',
@@ -70,6 +76,8 @@
70 }76 }
71 }77 }
7278
79NEUTRON_CONF_DIR = '/etc/neutron'
80
7381
74def neutron_plugins():82def neutron_plugins():
75 from charmhelpers.contrib.openstack import context83 from charmhelpers.contrib.openstack import context
@@ -83,7 +91,8 @@
83 'contexts': [91 'contexts': [
84 context.SharedDBContext(user=config('neutron-database-user'),92 context.SharedDBContext(user=config('neutron-database-user'),
85 database=config('neutron-database'),93 database=config('neutron-database'),
86 relation_prefix='neutron')],94 relation_prefix='neutron',
95 ssl_dir=NEUTRON_CONF_DIR)],
87 'services': ['neutron-plugin-openvswitch-agent'],96 'services': ['neutron-plugin-openvswitch-agent'],
88 'packages': [[headers_package()] + determine_dkms_package(),97 'packages': [[headers_package()] + determine_dkms_package(),
89 ['neutron-plugin-openvswitch-agent']],98 ['neutron-plugin-openvswitch-agent']],
@@ -98,20 +107,37 @@
98 'contexts': [107 'contexts': [
99 context.SharedDBContext(user=config('neutron-database-user'),108 context.SharedDBContext(user=config('neutron-database-user'),
100 database=config('neutron-database'),109 database=config('neutron-database'),
101 relation_prefix='neutron')],110 relation_prefix='neutron',
111 ssl_dir=NEUTRON_CONF_DIR)],
102 'services': [],112 'services': [],
103 'packages': [],113 'packages': [],
104 'server_packages': ['neutron-server',114 'server_packages': ['neutron-server',
105 'neutron-plugin-nicira'],115 'neutron-plugin-nicira'],
106 'server_services': ['neutron-server']116 'server_services': ['neutron-server']
117 },
118 'nsx': {
119 'config': '/etc/neutron/plugins/vmware/nsx.ini',
120 'driver': 'vmware',
121 'contexts': [
122 context.SharedDBContext(user=config('neutron-database-user'),
123 database=config('neutron-database'),
124 relation_prefix='neutron',
125 ssl_dir=NEUTRON_CONF_DIR)],
126 'services': [],
127 'packages': [],
128 'server_packages': ['neutron-server',
129 'neutron-plugin-vmware'],
130 'server_services': ['neutron-server']
107 }131 }
108 }132 }
109 # NOTE: patch in ml2 plugin for icehouse onwards
110 if release >= 'icehouse':133 if release >= 'icehouse':
134 # NOTE: patch in ml2 plugin for icehouse onwards
111 plugins['ovs']['config'] = '/etc/neutron/plugins/ml2/ml2_conf.ini'135 plugins['ovs']['config'] = '/etc/neutron/plugins/ml2/ml2_conf.ini'
112 plugins['ovs']['driver'] = 'neutron.plugins.ml2.plugin.Ml2Plugin'136 plugins['ovs']['driver'] = 'neutron.plugins.ml2.plugin.Ml2Plugin'
113 plugins['ovs']['server_packages'] = ['neutron-server',137 plugins['ovs']['server_packages'] = ['neutron-server',
114 'neutron-plugin-ml2']138 'neutron-plugin-ml2']
139 # NOTE: patch in vmware renames nvp->nsx for icehouse onwards
140 plugins['nvp'] = plugins['nsx']
115 return plugins141 return plugins
116142
117143
118144
=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
--- hooks/charmhelpers/contrib/openstack/utils.py 2014-04-01 07:20:45 +0000
+++ hooks/charmhelpers/contrib/openstack/utils.py 2014-05-20 21:57:55 +0000
@@ -65,6 +65,7 @@
65 ('1.10.0', 'havana'),65 ('1.10.0', 'havana'),
66 ('1.9.1', 'havana'),66 ('1.9.1', 'havana'),
67 ('1.9.0', 'havana'),67 ('1.9.0', 'havana'),
68 ('1.13.1', 'icehouse'),
68 ('1.13.0', 'icehouse'),69 ('1.13.0', 'icehouse'),
69 ('1.12.0', 'icehouse'),70 ('1.12.0', 'icehouse'),
70 ('1.11.0', 'icehouse'),71 ('1.11.0', 'icehouse'),
@@ -130,6 +131,11 @@
130def get_os_codename_package(package, fatal=True):131def get_os_codename_package(package, fatal=True):
131 '''Derive OpenStack release codename from an installed package.'''132 '''Derive OpenStack release codename from an installed package.'''
132 apt.init()133 apt.init()
134
135 # Tell apt to build an in-memory cache to prevent race conditions (if
136 # another process is already building the cache).
137 apt.config.set("Dir::Cache::pkgcache", "")
138
133 cache = apt.Cache()139 cache = apt.Cache()
134140
135 try:141 try:
@@ -182,7 +188,7 @@
182 if cname == codename:188 if cname == codename:
183 return version189 return version
184 #e = "Could not determine OpenStack version for package: %s" % pkg190 #e = "Could not determine OpenStack version for package: %s" % pkg
185 #error_out(e)191 # error_out(e)
186192
187193
188os_rel = None194os_rel = None
@@ -400,6 +406,8 @@
400 rtype = 'PTR'406 rtype = 'PTR'
401 elif isinstance(address, basestring):407 elif isinstance(address, basestring):
402 rtype = 'A'408 rtype = 'A'
409 else:
410 return None
403411
404 answers = dns.resolver.query(address, rtype)412 answers = dns.resolver.query(address, rtype)
405 if answers:413 if answers:
406414
=== modified file 'hooks/charmhelpers/contrib/storage/linux/lvm.py'
--- hooks/charmhelpers/contrib/storage/linux/lvm.py 2014-04-01 09:50:03 +0000
+++ hooks/charmhelpers/contrib/storage/linux/lvm.py 2014-05-20 21:57:55 +0000
@@ -62,7 +62,7 @@
62 pvd = check_output(['pvdisplay', block_device]).splitlines()62 pvd = check_output(['pvdisplay', block_device]).splitlines()
63 for l in pvd:63 for l in pvd:
64 if l.strip().startswith('VG Name'):64 if l.strip().startswith('VG Name'):
65 vg = ' '.join(l.split()).split(' ').pop()65 vg = ' '.join(l.strip().split()[2:])
66 return vg66 return vg
6767
6868
6969
=== modified file 'hooks/charmhelpers/contrib/storage/linux/utils.py'
--- hooks/charmhelpers/contrib/storage/linux/utils.py 2014-04-01 09:50:03 +0000
+++ hooks/charmhelpers/contrib/storage/linux/utils.py 2014-05-20 21:57:55 +0000
@@ -1,8 +1,11 @@
1from os import stat1import os
2import re
2from stat import S_ISBLK3from stat import S_ISBLK
34
4from subprocess import (5from subprocess import (
5 check_call6 check_call,
7 check_output,
8 call
6)9)
710
811
@@ -12,7 +15,9 @@
1215
13 :returns: boolean: True if path is a block device, False if not.16 :returns: boolean: True if path is a block device, False if not.
14 '''17 '''
15 return S_ISBLK(stat(path).st_mode)18 if not os.path.exists(path):
19 return False
20 return S_ISBLK(os.stat(path).st_mode)
1621
1722
18def zap_disk(block_device):23def zap_disk(block_device):
@@ -22,5 +27,23 @@
2227
23 :param block_device: str: Full path of block device to clean.28 :param block_device: str: Full path of block device to clean.
24 '''29 '''
25 check_call(['sgdisk', '--zap-all', '--clear',30 # sometimes sgdisk exits non-zero; this is OK, dd will clean up
26 '--mbrtogpt', block_device])31 call(['sgdisk', '--zap-all', '--mbrtogpt',
32 '--clear', block_device])
33 dev_end = check_output(['blockdev', '--getsz', block_device])
34 gpt_end = int(dev_end.split()[0]) - 100
35 check_call(['dd', 'if=/dev/zero', 'of=%s' % (block_device),
36 'bs=1M', 'count=1'])
37 check_call(['dd', 'if=/dev/zero', 'of=%s' % (block_device),
38 'bs=512', 'count=100', 'seek=%s' % (gpt_end)])
39
40def is_device_mounted(device):
41 '''Given a device path, return True if that device is mounted, and False
42 if it isn't.
43
44 :param device: str: Full path of the device to check.
45 :returns: boolean: True if the path represents a mounted device, False if
46 it doesn't.
47 '''
48 out = check_output(['mount'])
49 return bool(re.search(device + r"[0-9]+\b", out))
2750
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py 2014-03-26 17:44:40 +0000
+++ hooks/charmhelpers/core/hookenv.py 2014-05-20 21:57:55 +0000
@@ -155,6 +155,100 @@
155 return os.path.basename(sys.argv[0])155 return os.path.basename(sys.argv[0])
156156
157157
158class Config(dict):
159 """A Juju charm config dictionary that can write itself to
160 disk (as json) and track which values have changed since
161 the previous hook invocation.
162
163 Do not instantiate this object directly - instead call
164 ``hookenv.config()``
165
166 Example usage::
167
168 >>> # inside a hook
169 >>> from charmhelpers.core import hookenv
170 >>> config = hookenv.config()
171 >>> config['foo']
172 'bar'
173 >>> config['mykey'] = 'myval'
174 >>> config.save()
175
176
177 >>> # user runs `juju set mycharm foo=baz`
178 >>> # now we're inside subsequent config-changed hook
179 >>> config = hookenv.config()
180 >>> config['foo']
181 'baz'
182 >>> # test to see if this val has changed since last hook
183 >>> config.changed('foo')
184 True
185 >>> # what was the previous value?
186 >>> config.previous('foo')
187 'bar'
188 >>> # keys/values that we add are preserved across hooks
189 >>> config['mykey']
190 'myval'
191 >>> # don't forget to save at the end of hook!
192 >>> config.save()
193
194 """
195 CONFIG_FILE_NAME = '.juju-persistent-config'
196
197 def __init__(self, *args, **kw):
198 super(Config, self).__init__(*args, **kw)
199 self._prev_dict = None
200 self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
201 if os.path.exists(self.path):
202 self.load_previous()
203
204 def load_previous(self, path=None):
205 """Load previous copy of config from disk so that current values
206 can be compared to previous values.
207
208 :param path:
209
210 File path from which to load the previous config. If `None`,
211 config is loaded from the default location. If `path` is
212 specified, subsequent `save()` calls will write to the same
213 path.
214
215 """
216 self.path = path or self.path
217 with open(self.path) as f:
218 self._prev_dict = json.load(f)
219
220 def changed(self, key):
221 """Return true if the value for this key has changed since
222 the last save.
223
224 """
225 if self._prev_dict is None:
226 return True
227 return self.previous(key) != self.get(key)
228
229 def previous(self, key):
230 """Return previous value for this key, or None if there
231 is no "previous" value.
232
233 """
234 if self._prev_dict:
235 return self._prev_dict.get(key)
236 return None
237
238 def save(self):
239 """Save this config to disk.
240
241 Preserves items in _prev_dict that do not exist in self.
242
243 """
244 if self._prev_dict:
245 for k, v in self._prev_dict.iteritems():
246 if k not in self:
247 self[k] = v
248 with open(self.path, 'w') as f:
249 json.dump(self, f)
250
251
158@cached252@cached
159def config(scope=None):253def config(scope=None):
160 """Juju charm configuration"""254 """Juju charm configuration"""
@@ -163,7 +257,10 @@
163 config_cmd_line.append(scope)257 config_cmd_line.append(scope)
164 config_cmd_line.append('--format=json')258 config_cmd_line.append('--format=json')
165 try:259 try:
166 return json.loads(subprocess.check_output(config_cmd_line))260 config_data = json.loads(subprocess.check_output(config_cmd_line))
261 if scope is not None:
262 return config_data
263 return Config(config_data)
167 except ValueError:264 except ValueError:
168 return None265 return None
169266
170267
=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py 2014-03-26 17:44:40 +0000
+++ hooks/charmhelpers/core/host.py 2014-05-20 21:57:55 +0000
@@ -12,6 +12,9 @@
12import string12import string
13import subprocess13import subprocess
14import hashlib14import hashlib
15import shutil
16import apt_pkg
17from contextlib import contextmanager
1518
16from collections import OrderedDict19from collections import OrderedDict
1720
@@ -143,6 +146,16 @@
143 target.write(content)146 target.write(content)
144147
145148
149def copy_file(src, dst, owner='root', group='root', perms=0444):
150 """Create or overwrite a file with the contents of another file"""
151 log("Writing file {} {}:{} {:o} from {}".format(dst, owner, group, perms, src))
152 uid = pwd.getpwnam(owner).pw_uid
153 gid = grp.getgrnam(group).gr_gid
154 shutil.copyfile(src, dst)
155 os.chown(dst, uid, gid)
156 os.chmod(dst, perms)
157
158
146def mount(device, mountpoint, options=None, persist=False):159def mount(device, mountpoint, options=None, persist=False):
147 """Mount a filesystem at a particular mountpoint"""160 """Mount a filesystem at a particular mountpoint"""
148 cmd_args = ['mount']161 cmd_args = ['mount']
@@ -295,3 +308,37 @@
295 if 'link/ether' in words:308 if 'link/ether' in words:
296 hwaddr = words[words.index('link/ether') + 1]309 hwaddr = words[words.index('link/ether') + 1]
297 return hwaddr310 return hwaddr
311
312
313def cmp_pkgrevno(package, revno, pkgcache=None):
314 '''Compare supplied revno with the revno of the installed package
315 1 => Installed revno is greater than supplied arg
316 0 => Installed revno is the same as supplied arg
317 -1 => Installed revno is less than supplied arg
318 '''
319 if not pkgcache:
320 apt_pkg.init()
321 pkgcache = apt_pkg.Cache()
322 pkg = pkgcache[package]
323 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
324
325
326@contextmanager
327def chdir(d):
328 cur = os.getcwd()
329 try:
330 yield os.chdir(d)
331 finally:
332 os.chdir(cur)
333
334
335def chownr(path, owner, group):
336 uid = pwd.getpwnam(owner).pw_uid
337 gid = grp.getgrnam(group).gr_gid
338
339 for root, dirs, files in os.walk(path):
340 for name in dirs + files:
341 full = os.path.join(root, name)
342 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
343 if not broken_symlink:
344 os.chown(full, uid, gid)
298345
=== added file 'hooks/charmhelpers/core/services.py'
--- hooks/charmhelpers/core/services.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/services.py 2014-05-20 21:57:55 +0000
@@ -0,0 +1,84 @@
1from charmhelpers.core import templating
2from charmhelpers.core import host
3
4
5SERVICES = {}
6
7
8def register(services, templates_dir=None):
9 """
10 Register a list of service configs.
11
12 Service Configs are dicts in the following formats:
13
14 {
15 "service": <service name>,
16 "templates": [ {
17 'target': <render target of template>,
18 'source': <optional name of template in passed in templates_dir>
19 'file_properties': <optional dict taking owner and octal mode>
20 'contexts': [ context generators, see contexts.py ]
21 }
22 ] }
23
24 Either `source` or `target` must be provided.
25
26 If 'source' is not provided for a template the templates_dir will
27 be consulted for ``basename(target).j2``.
28
29 If `target` is not provided, it will be assumed to be
30 ``/etc/init/<service name>.conf``.
31 """
32 for service in services:
33 service.setdefault('templates_dir', templates_dir)
34 SERVICES[service['service']] = service
35
36
37def reconfigure_services(restart=True):
38 """
39 Update all files for all services and optionally restart them, if ready.
40 """
41 for service_name in SERVICES.keys():
42 reconfigure_service(service_name, restart=restart)
43
44
45def reconfigure_service(service_name, restart=True):
46 """
47 Update all files for a single service and optionally restart it, if ready.
48 """
49 service = SERVICES.get(service_name)
50 if not service or service['service'] != service_name:
51 raise KeyError('Service not registered: %s' % service_name)
52
53 manager_type = service.get('type', UpstartService)
54 manager_type(service).reconfigure(restart)
55
56
57def stop_services():
58 for service_name in SERVICES.keys():
59 if host.service_running(service_name):
60 host.service_stop(service_name)
61
62
63class ServiceTypeManager(object):
64 def __init__(self, service_definition):
65 self.service_name = service_definition['service']
66 self.templates = service_definition['templates']
67 self.templates_dir = service_definition['templates_dir']
68
69 def reconfigure(self, restart=True):
70 raise NotImplementedError()
71
72
73class UpstartService(ServiceTypeManager):
74 def __init__(self, service_definition):
75 super(UpstartService, self).__init__(service_definition)
76 for tmpl in self.templates:
77 if 'target' not in tmpl:
78 tmpl['target'] = '/etc/init/%s.conf' % self.service_name
79
80 def reconfigure(self, restart):
81 complete = templating.render(self.templates, self.templates_dir)
82
83 if restart and complete:
84 host.service_restart(self.service_name)
085
=== added file 'hooks/charmhelpers/core/templating.py'
--- hooks/charmhelpers/core/templating.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/templating.py 2014-05-20 21:57:55 +0000
@@ -0,0 +1,158 @@
1import os
2import yaml
3
4from charmhelpers.core import host
5from charmhelpers.core import hookenv
6
7
8class ContextGenerator(object):
9 """
10 Base interface for template context container generators.
11
12 A template context is a dictionary that contains data needed to populate
13 the template. The generator instance should produce the context when
14 called (without arguments) by collecting information from juju (config-get,
15 relation-get, etc), the system, or whatever other sources are appropriate.
16
17 A context generator should only return any values if it has enough information
18 to provide all of its values. Any context that is missing data is considered
19 incomplete and will cause that template to not render until it has all of its
20 necessary data.
21
22 The template may receive several contexts, which will be merged together,
23 so care should be taken in the key names.
24 """
25 def __call__(self):
26 raise NotImplementedError
27
28
29class StorableContext(object):
30 """
31 A mixin for persisting a context to disk.
32 """
33 def store_context(self, file_name, config_data):
34 with open(file_name, 'w') as file_stream:
35 yaml.dump(config_data, file_stream)
36
37 def read_context(self, file_name):
38 with open(file_name, 'r') as file_stream:
39 data = yaml.load(file_stream)
40 if not data:
41 raise OSError("%s is empty" % file_name)
42 return data
43
44
45class ConfigContext(ContextGenerator):
46 """
47 A context generator that generates a context containing all of the
48 juju config values.
49 """
50 def __call__(self):
51 return hookenv.config()
52
53
54class RelationContext(ContextGenerator):
55 """
56 Base class for a context generator that gets relation data from juju.
57
58 Subclasses must provide `interface`, which is the interface type of interest,
59 and `required_keys`, which is the set of keys required for the relation to
60 be considered complete. The first relation for the interface that is complete
61 will be used to populate the data for template.
62
63 The generated context will be namespaced under the interface type, to prevent
64 potential naming conflicts.
65 """
66 interface = None
67 required_keys = []
68
69 def __call__(self):
70 if not hookenv.relation_ids(self.interface):
71 return {}
72
73 ctx = {}
74 for rid in hookenv.relation_ids(self.interface):
75 for unit in hookenv.related_units(rid):
76 reldata = hookenv.relation_get(rid=rid, unit=unit)
77 required = set(self.required_keys)
78 if set(reldata.keys()).issuperset(required):
79 ns = ctx.setdefault(self.interface, {})
80 for k, v in reldata.items():
81 ns[k] = v
82 return ctx
83
84 return {}
85
86
87class StaticContext(ContextGenerator):
88 def __init__(self, data):
89 self.data = data
90
91 def __call__(self):
92 return self.data
93
94
95def _collect_contexts(context_providers):
96 """
97 Helper function to collect and merge contexts from a list of providers.
98
99 If any of the contexts are incomplete (i.e., they return an empty dict),
100 the template is considered incomplete and will not render.
101 """
102 ctx = {}
103 for provider in context_providers:
104 c = provider()
105 if not c:
106 return False
107 ctx.update(c)
108 return ctx
109
110
111def render(template_definitions, templates_dir=None):
112 """
113 Render one or more templates, given a list of template definitions.
114
115 The template definitions should be dicts with the keys: `source`, `target`,
116 `file_properties`, and `contexts`.
117
118 The `source` path, if not absolute, is relative to the `templates_dir`
119 given when the rendered was created. If `source` is not provided
120 for a template the `template_dir` will be consulted for
121 ``basename(target).j2``.
122
123 The `target` path should be absolute.
124
125 The `file_properties` should be a dict optionally containing
126 `owner`, `group`, or `perms` options, to be passed to `write_file`.
127
128 The `contexts` should be a list containing zero or more ContextGenerators.
129
130 The `template_dir` defaults to `$CHARM_DIR/templates`
131
132 Returns True if all of the templates were "complete" (i.e., the context
133 generators were able to collect the information needed to render the
134 template) and were rendered.
135 """
136 # lazy import jinja2 in case templating is needed in install hook
137 from jinja2 import FileSystemLoader, Environment, exceptions
138 all_complete = True
139 if templates_dir is None:
140 templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
141 loader = Environment(loader=FileSystemLoader(templates_dir))
142 for tmpl in template_definitions:
143 ctx = _collect_contexts(tmpl.get('contexts', []))
144 if ctx is False:
145 all_complete = False
146 continue
147 try:
148 source = tmpl.get('source', os.path.basename(tmpl['target'])+'.j2')
149 template = loader.get_template(source)
150 except exceptions.TemplateNotFound as e:
151 hookenv.log('Could not load template %s from %s.' %
152 (tmpl['source'], templates_dir),
153 level=hookenv.ERROR)
154 raise e
155 content = template.render(ctx)
156 host.mkdir(os.path.dirname(tmpl['target']))
157 host.write_file(tmpl['target'], content, **tmpl.get('file_properties', {}))
158 return all_complete
0159
=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py 2014-03-27 04:30:27 +0000
+++ hooks/charmhelpers/fetch/__init__.py 2014-05-20 21:57:55 +0000
@@ -1,4 +1,5 @@
1import importlib1import importlib
2import time
2from yaml import safe_load3from yaml import safe_load
3from charmhelpers.core.host import (4from charmhelpers.core.host import (
4 lsb_release5 lsb_release
@@ -15,6 +16,7 @@
15import apt_pkg16import apt_pkg
16import os17import os
1718
19
18CLOUD_ARCHIVE = """# Ubuntu Cloud Archive20CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
19deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main21deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
20"""22"""
@@ -56,10 +58,62 @@
56 'precise-proposed/icehouse': 'precise-proposed/icehouse',58 'precise-proposed/icehouse': 'precise-proposed/icehouse',
57}59}
5860
61# The order of this list is very important. Handlers should be listed in from
62# least- to most-specific URL matching.
63FETCH_HANDLERS = (
64 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
65 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
66)
67
68APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
69APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
70APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
71
72
73class SourceConfigError(Exception):
74 pass
75
76
77class UnhandledSource(Exception):
78 pass
79
80
81class AptLockError(Exception):
82 pass
83
84
85class BaseFetchHandler(object):
86
87 """Base class for FetchHandler implementations in fetch plugins"""
88
89 def can_handle(self, source):
90 """Returns True if the source can be handled. Otherwise returns
91 a string explaining why it cannot"""
92 return "Wrong source type"
93
94 def install(self, source):
95 """Try to download and unpack the source. Return the path to the
96 unpacked files or raise UnhandledSource."""
97 raise UnhandledSource("Wrong source type {}".format(source))
98
99 def parse_url(self, url):
100 return urlparse(url)
101
102 def base_url(self, url):
103 """Return url without querystring or fragment"""
104 parts = list(self.parse_url(url))
105 parts[4:] = ['' for i in parts[4:]]
106 return urlunparse(parts)
107
59108
60def filter_installed_packages(packages):109def filter_installed_packages(packages):
61 """Returns a list of packages that require installation"""110 """Returns a list of packages that require installation"""
62 apt_pkg.init()111 apt_pkg.init()
112
113 # Tell apt to build an in-memory cache to prevent race conditions (if
114 # another process is already building the cache).
115 apt_pkg.config.set("Dir::Cache::pkgcache", "")
116
63 cache = apt_pkg.Cache()117 cache = apt_pkg.Cache()
64 _pkgs = []118 _pkgs = []
65 for package in packages:119 for package in packages:
@@ -87,14 +141,7 @@
87 cmd.extend(packages)141 cmd.extend(packages)
88 log("Installing {} with options: {}".format(packages,142 log("Installing {} with options: {}".format(packages,
89 options))143 options))
90 env = os.environ.copy()144 _run_apt_command(cmd, fatal)
91 if 'DEBIAN_FRONTEND' not in env:
92 env['DEBIAN_FRONTEND'] = 'noninteractive'
93
94 if fatal:
95 subprocess.check_call(cmd, env=env)
96 else:
97 subprocess.call(cmd, env=env)
98145
99146
100def apt_upgrade(options=None, fatal=False, dist=False):147def apt_upgrade(options=None, fatal=False, dist=False):
@@ -109,24 +156,13 @@
109 else:156 else:
110 cmd.append('upgrade')157 cmd.append('upgrade')
111 log("Upgrading with options: {}".format(options))158 log("Upgrading with options: {}".format(options))
112159 _run_apt_command(cmd, fatal)
113 env = os.environ.copy()
114 if 'DEBIAN_FRONTEND' not in env:
115 env['DEBIAN_FRONTEND'] = 'noninteractive'
116
117 if fatal:
118 subprocess.check_call(cmd, env=env)
119 else:
120 subprocess.call(cmd, env=env)
121160
122161
123def apt_update(fatal=False):162def apt_update(fatal=False):
124 """Update local apt cache"""163 """Update local apt cache"""
125 cmd = ['apt-get', 'update']164 cmd = ['apt-get', 'update']
126 if fatal:165 _run_apt_command(cmd, fatal)
127 subprocess.check_call(cmd)
128 else:
129 subprocess.call(cmd)
130166
131167
132def apt_purge(packages, fatal=False):168def apt_purge(packages, fatal=False):
@@ -137,10 +173,7 @@
137 else:173 else:
138 cmd.extend(packages)174 cmd.extend(packages)
139 log("Purging {}".format(packages))175 log("Purging {}".format(packages))
140 if fatal:176 _run_apt_command(cmd, fatal)
141 subprocess.check_call(cmd)
142 else:
143 subprocess.call(cmd)
144177
145178
146def apt_hold(packages, fatal=False):179def apt_hold(packages, fatal=False):
@@ -151,6 +184,7 @@
151 else:184 else:
152 cmd.extend(packages)185 cmd.extend(packages)
153 log("Holding {}".format(packages))186 log("Holding {}".format(packages))
187
154 if fatal:188 if fatal:
155 subprocess.check_call(cmd)189 subprocess.check_call(cmd)
156 else:190 else:
@@ -184,14 +218,10 @@
184 apt.write(PROPOSED_POCKET.format(release))218 apt.write(PROPOSED_POCKET.format(release))
185 if key:219 if key:
186 subprocess.check_call(['apt-key', 'adv', '--keyserver',220 subprocess.check_call(['apt-key', 'adv', '--keyserver',
187 'keyserver.ubuntu.com', '--recv',221 'hkp://keyserver.ubuntu.com:80', '--recv',
188 key])222 key])
189223
190224
191class SourceConfigError(Exception):
192 pass
193
194
195def configure_sources(update=False,225def configure_sources(update=False,
196 sources_var='install_sources',226 sources_var='install_sources',
197 keys_var='install_keys'):227 keys_var='install_keys'):
@@ -224,17 +254,6 @@
224 if update:254 if update:
225 apt_update(fatal=True)255 apt_update(fatal=True)
226256
227# The order of this list is very important. Handlers should be listed in from
228# least- to most-specific URL matching.
229FETCH_HANDLERS = (
230 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
231 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
232)
233
234
235class UnhandledSource(Exception):
236 pass
237
238257
239def install_remote(source):258def install_remote(source):
240 """259 """
@@ -265,30 +284,6 @@
265 return install_remote(source)284 return install_remote(source)
266285
267286
268class BaseFetchHandler(object):
269
270 """Base class for FetchHandler implementations in fetch plugins"""
271
272 def can_handle(self, source):
273 """Returns True if the source can be handled. Otherwise returns
274 a string explaining why it cannot"""
275 return "Wrong source type"
276
277 def install(self, source):
278 """Try to download and unpack the source. Return the path to the
279 unpacked files or raise UnhandledSource."""
280 raise UnhandledSource("Wrong source type {}".format(source))
281
282 def parse_url(self, url):
283 return urlparse(url)
284
285 def base_url(self, url):
286 """Return url without querystring or fragment"""
287 parts = list(self.parse_url(url))
288 parts[4:] = ['' for i in parts[4:]]
289 return urlunparse(parts)
290
291
292def plugins(fetch_handlers=None):287def plugins(fetch_handlers=None):
293 if not fetch_handlers:288 if not fetch_handlers:
294 fetch_handlers = FETCH_HANDLERS289 fetch_handlers = FETCH_HANDLERS
@@ -306,3 +301,40 @@
306 log("FetchHandler {} not found, skipping plugin".format(301 log("FetchHandler {} not found, skipping plugin".format(
307 handler_name))302 handler_name))
308 return plugin_list303 return plugin_list
304
305
306def _run_apt_command(cmd, fatal=False):
307 """
308 Run an APT command, checking output and retrying if the fatal flag is set
309 to True.
310
311 :param: cmd: str: The apt command to run.
312 :param: fatal: bool: Whether the command's output should be checked and
313 retried.
314 """
315 env = os.environ.copy()
316
317 if 'DEBIAN_FRONTEND' not in env:
318 env['DEBIAN_FRONTEND'] = 'noninteractive'
319
320 if fatal:
321 retry_count = 0
322 result = None
323
324 # If the command is considered "fatal", we need to retry if the apt
325 # lock was not acquired.
326
327 while result is None or result == APT_NO_LOCK:
328 try:
329 result = subprocess.check_call(cmd, env=env)
330 except subprocess.CalledProcessError, e:
331 retry_count = retry_count + 1
332 if retry_count > APT_NO_LOCK_RETRY_COUNT:
333 raise
334 result = e.returncode
335 log("Couldn't acquire DPKG lock. Will retry in {} seconds."
336 "".format(APT_NO_LOCK_RETRY_DELAY))
337 time.sleep(APT_NO_LOCK_RETRY_DELAY)
338
339 else:
340 subprocess.call(cmd, env=env)
309341
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py 2014-04-09 12:21:03 +0000
+++ hooks/hooks.py 2014-05-20 21:57:55 +0000
@@ -7,9 +7,10 @@
77
8from charmhelpers.core import hookenv8from charmhelpers.core import hookenv
9from charmhelpers.core.hookenv import log9from charmhelpers.core.hookenv import log
10from charmhelpers.core import services
11from charmhelpers.core import templating
10from charmhelpers.contrib.cloudfoundry import contexts12from charmhelpers.contrib.cloudfoundry import contexts
11from charmhelpers.contrib.cloudfoundry import services13import config
12from config import *
1314
1415
15def default_domain():16def default_domain():
@@ -31,23 +32,24 @@
3132
3233
33hooks = hookenv.Hooks()34hooks = hookenv.Hooks()
34fileproperties = {'owner': 'vcap', 'mode': '644'}35fileproperties = {'owner': 'vcap', 'perms': 0644}
3536
36services.register([37services.register([
37 {38 {
38 'service': 'gorouter',39 'service': 'gorouter',
39 'templates': [{40 'templates': [
40 'source': 'gorouter.yml',41 {'source': 'gorouter.conf'},
41 'target': ROUTER_CONFIG_FILE,42 {'source': 'gorouter.yml',
42 'file_properties': fileproperties,43 'target': config.ROUTER_CONFIG_FILE,
43 'contexts': [44 'file_properties': fileproperties,
44 contexts.StaticContext({'domain': default_domain()}),45 'contexts': [
45 contexts.ConfigContext(),46 templating.StaticContext({'domain': default_domain()}),
46 contexts.NatsContext()47 templating.ConfigContext(),
47 ]48 contexts.NatsContext()
48 }]49 ]},
50 ]
49 }51 }
50], os.path.join(hookenv.charm_dir(), 'templates'))52])
5153
5254
53@hooks.hook()55@hooks.hook()
5456
=== modified file 'hooks/install'
--- hooks/install 2014-04-08 07:39:30 +0000
+++ hooks/install 2014-05-20 21:57:55 +0000
@@ -1,26 +1,22 @@
1#!/usr/bin/env python1#!/usr/bin/env python
2# vim: et ai ts=4 sw=4:2# vim: et ai ts=4 sw=4:
33
4from config import *
5# from charmhelpers.fetch import apt_install, apt_update, add_source
6from charmhelpers.core import hookenv, host4from charmhelpers.core import hookenv, host
7from charmhelpers.contrib.cloudfoundry.upstart_helper import (
8 install_upstart_scripts
9)
10from charmhelpers.contrib.cloudfoundry.common import (5from charmhelpers.contrib.cloudfoundry.common import (
11 chownr, prepare_cloudfoundry_environment6 prepare_cloudfoundry_environment
12)7)
138
9import config
10
1411
15def install():12def install():
16 prepare_cloudfoundry_environment(hookenv.config(), ROUTER_PACKAGES)13 prepare_cloudfoundry_environment(hookenv.config(), config.ROUTER_PACKAGES)
17 install_upstart_scripts()14 dirs = [config.CF_DIR, config.ROUTER_DIR, config.ROUTER_CONFIG_DIR,
18 dirs = [CF_DIR, ROUTER_DIR, ROUTER_CONFIG_DIR,
19 "/var/vcap/sys/log/gorouter",15 "/var/vcap/sys/log/gorouter",
20 "/var/vcap/sys/log/gorouter"]16 "/var/vcap/sys/log/gorouter"]
21 for dir in dirs:17 for dir in dirs:
22 host.mkdir(dir, owner='vcap', group='vcap', perms=0775)18 host.mkdir(dir, owner='vcap', group='vcap', perms=0775)
23 chownr(CF_DIR, 'vcap', 'vcap')19 host.chownr(config.CF_DIR, 'vcap', 'vcap')
2420
2521
26if __name__ == '__main__':22if __name__ == '__main__':
2723
=== renamed file 'files/upstart/gorouter.conf' => 'templates/gorouter.conf'

Subscribers

People subscribed via source and target branches