Merge lp:~1chb1n/charms/trusty/swift-proxy/vivid-kilo-ch-sync into lp:~openstack-charmers-archive/charms/trusty/swift-proxy/next
- Trusty Tahr (14.04)
- vivid-kilo-ch-sync
- Merge into next
Status: | Merged |
---|---|
Approved by: | Billy Olsen |
Approved revision: | 81 |
Merged at revision: | 80 |
Proposed branch: | lp:~1chb1n/charms/trusty/swift-proxy/vivid-kilo-ch-sync |
Merge into: | lp:~openstack-charmers-archive/charms/trusty/swift-proxy/next |
Diff against target: |
1291 lines (+863/-46) 20 files modified
config.yaml (+10/-0) hooks/charmhelpers/contrib/charmsupport/nrpe.py (+41/-7) hooks/charmhelpers/contrib/hahelpers/cluster.py (+5/-1) hooks/charmhelpers/contrib/openstack/amulet/deployment.py (+5/-2) hooks/charmhelpers/contrib/openstack/files/__init__.py (+18/-0) hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh (+32/-0) hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh (+30/-0) hooks/charmhelpers/contrib/openstack/ip.py (+37/-0) hooks/charmhelpers/contrib/openstack/utils.py (+1/-0) hooks/charmhelpers/contrib/python/packages.py (+2/-2) hooks/charmhelpers/core/fstab.py (+4/-4) hooks/charmhelpers/core/host.py (+5/-5) hooks/charmhelpers/core/strutils.py (+42/-0) hooks/charmhelpers/core/sysctl.py (+13/-7) hooks/charmhelpers/core/templating.py (+3/-3) hooks/charmhelpers/core/unitdata.py (+477/-0) hooks/charmhelpers/fetch/archiveurl.py (+10/-10) hooks/charmhelpers/fetch/giturl.py (+1/-1) tests/charmhelpers/contrib/amulet/utils.py (+122/-2) tests/charmhelpers/contrib/openstack/amulet/deployment.py (+5/-2) |
To merge this branch: | bzr merge lp:~1chb1n/charms/trusty/swift-proxy/vivid-kilo-ch-sync |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Corey Bryant (community) | Approve | ||
Billy Olsen | Approve | ||
Review via email: mp+250377@code.launchpad.net |
This proposal supersedes a proposal from 2015-02-19.
Commit message
Description of the change
Charmhelper sync to resolve swift version table issue.
https:/
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
charm_unit_test #1911 swift-proxy for 1chb1n mp250374
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
charm_amulet_test #2030 swift-proxy for 1chb1n mp250374
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
ERROR subprocess encountered error code 1
make: *** [test] Error 1
Full amulet test output: http://
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
charm_lint_check #2122 swift-proxy for 1chb1n mp250374
LINT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
charm_lint_check #2125 swift-proxy for 1chb1n mp250374
LINT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : Posted in a previous version of this proposal | # |
charm_unit_test #1914 swift-proxy for 1chb1n mp250374
UNIT OK: passed
Ryan Beisner (1chb1n) wrote : | # |
Re-targeted merge to the next branch.
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #2126 swift-proxy-next for 1chb1n mp250377
LINT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #1915 swift-proxy-next for 1chb1n mp250377
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #2034 swift-proxy-next for 1chb1n mp250377
AMULET OK: passed
Build: http://
Ryan Beisner (1chb1n) wrote : | # |
PS. Also resolves charm proof warnings in config.yaml.
Billy Olsen (billy-olsen) wrote : | # |
LGTM, Approved
Corey Bryant (corey.bryant) wrote : | # |
LGTM too, thanks!
Preview Diff
1 | === modified file 'config.yaml' |
2 | --- config.yaml 2015-01-09 10:36:01 +0000 |
3 | +++ config.yaml 2015-02-19 20:35:53 +0000 |
4 | @@ -63,6 +63,7 @@ |
5 | # User provided SSL cert and key |
6 | ssl_cert: |
7 | type: string |
8 | + default: |
9 | description: | |
10 | Base64 encoded SSL certificate to install and use for API ports. |
11 | . |
12 | @@ -75,6 +76,7 @@ |
13 | do so). |
14 | ssl_key: |
15 | type: string |
16 | + default: |
17 | description: | |
18 | Base64 encoded SSL key to use with certificate specified as ssl_cert. |
19 | # General Swift Proxy configuration |
20 | @@ -117,6 +119,7 @@ |
21 | # Manual Keystone configuration. |
22 | keystone-auth-host: |
23 | type: string |
24 | + default: |
25 | description: Keystone authentication host |
26 | keystone-auth-port: |
27 | default: 35357 |
28 | @@ -132,16 +135,20 @@ |
29 | description: Keystone admin tenant name |
30 | keystone-admin-user: |
31 | type: string |
32 | + default: |
33 | description: Keystone admin username |
34 | keystone-admin-password: |
35 | type: string |
36 | + default: |
37 | description: Keystone admin password |
38 | # HA configuration settings |
39 | swift-hash: |
40 | type: string |
41 | + default: |
42 | description: Hash to use across all swift-proxy servers - don't loose |
43 | vip: |
44 | type: string |
45 | + default: |
46 | description: | |
47 | Virtual IP(s) to use to front API services in HA configuration. |
48 | . |
49 | @@ -163,6 +170,7 @@ |
50 | # by default all access is over 'private-address' |
51 | os-admin-network: |
52 | type: string |
53 | + default: |
54 | description: | |
55 | The IP address and netmask of the OpenStack Admin network (e.g., |
56 | 192.168.0.0/24) |
57 | @@ -170,6 +178,7 @@ |
58 | This network will be used for admin endpoints. |
59 | os-internal-network: |
60 | type: string |
61 | + default: |
62 | description: | |
63 | The IP address and netmask of the OpenStack Internal network (e.g., |
64 | 192.168.0.0/24) |
65 | @@ -177,6 +186,7 @@ |
66 | This network will be used for internal endpoints. |
67 | os-public-network: |
68 | type: string |
69 | + default: |
70 | description: | |
71 | The IP address and netmask of the OpenStack Public network (e.g., |
72 | 192.168.0.0/24) |
73 | |
74 | === modified file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py' |
75 | --- hooks/charmhelpers/contrib/charmsupport/nrpe.py 2015-01-26 09:45:43 +0000 |
76 | +++ hooks/charmhelpers/contrib/charmsupport/nrpe.py 2015-02-19 20:35:53 +0000 |
77 | @@ -24,6 +24,8 @@ |
78 | import pwd |
79 | import grp |
80 | import os |
81 | +import glob |
82 | +import shutil |
83 | import re |
84 | import shlex |
85 | import yaml |
86 | @@ -161,7 +163,7 @@ |
87 | log('Check command not found: {}'.format(parts[0])) |
88 | return '' |
89 | |
90 | - def write(self, nagios_context, hostname, nagios_servicegroups=None): |
91 | + def write(self, nagios_context, hostname, nagios_servicegroups): |
92 | nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format( |
93 | self.command) |
94 | with open(nrpe_check_file, 'w') as nrpe_check_config: |
95 | @@ -177,14 +179,11 @@ |
96 | nagios_servicegroups) |
97 | |
98 | def write_service_config(self, nagios_context, hostname, |
99 | - nagios_servicegroups=None): |
100 | + nagios_servicegroups): |
101 | for f in os.listdir(NRPE.nagios_exportdir): |
102 | if re.search('.*{}.cfg'.format(self.command), f): |
103 | os.remove(os.path.join(NRPE.nagios_exportdir, f)) |
104 | |
105 | - if not nagios_servicegroups: |
106 | - nagios_servicegroups = nagios_context |
107 | - |
108 | templ_vars = { |
109 | 'nagios_hostname': hostname, |
110 | 'nagios_servicegroup': nagios_servicegroups, |
111 | @@ -211,10 +210,10 @@ |
112 | super(NRPE, self).__init__() |
113 | self.config = config() |
114 | self.nagios_context = self.config['nagios_context'] |
115 | - if 'nagios_servicegroups' in self.config: |
116 | + if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']: |
117 | self.nagios_servicegroups = self.config['nagios_servicegroups'] |
118 | else: |
119 | - self.nagios_servicegroups = 'juju' |
120 | + self.nagios_servicegroups = self.nagios_context |
121 | self.unit_name = local_unit().replace('/', '-') |
122 | if hostname: |
123 | self.hostname = hostname |
124 | @@ -322,3 +321,38 @@ |
125 | check_cmd='check_status_file.py -f ' |
126 | '/var/lib/nagios/service-check-%s.txt' % svc, |
127 | ) |
128 | + |
129 | + |
130 | +def copy_nrpe_checks(): |
131 | + """ |
132 | + Copy the nrpe checks into place |
133 | + |
134 | + """ |
135 | + NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins' |
136 | + nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks', |
137 | + 'charmhelpers', 'contrib', 'openstack', |
138 | + 'files') |
139 | + |
140 | + if not os.path.exists(NAGIOS_PLUGINS): |
141 | + os.makedirs(NAGIOS_PLUGINS) |
142 | + for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")): |
143 | + if os.path.isfile(fname): |
144 | + shutil.copy2(fname, |
145 | + os.path.join(NAGIOS_PLUGINS, os.path.basename(fname))) |
146 | + |
147 | + |
148 | +def add_haproxy_checks(nrpe, unit_name): |
149 | + """ |
150 | + Add checks for each service in list |
151 | + |
152 | + :param NRPE nrpe: NRPE object to add check to |
153 | + :param str unit_name: Unit name to use in check description |
154 | + """ |
155 | + nrpe.add_check( |
156 | + shortname='haproxy_servers', |
157 | + description='Check HAProxy {%s}' % unit_name, |
158 | + check_cmd='check_haproxy.sh') |
159 | + nrpe.add_check( |
160 | + shortname='haproxy_queue', |
161 | + description='Check HAProxy queue depth {%s}' % unit_name, |
162 | + check_cmd='check_haproxy_queue_depth.sh') |
163 | |
164 | === modified file 'hooks/charmhelpers/contrib/hahelpers/cluster.py' |
165 | --- hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-01-26 09:45:43 +0000 |
166 | +++ hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-02-19 20:35:53 +0000 |
167 | @@ -48,6 +48,9 @@ |
168 | from charmhelpers.core.decorators import ( |
169 | retry_on_exception, |
170 | ) |
171 | +from charmhelpers.core.strutils import ( |
172 | + bool_from_string, |
173 | +) |
174 | |
175 | |
176 | class HAIncompleteConfig(Exception): |
177 | @@ -164,7 +167,8 @@ |
178 | . |
179 | returns: boolean |
180 | ''' |
181 | - if config_get('use-https') == "yes": |
182 | + use_https = config_get('use-https') |
183 | + if use_https and bool_from_string(use_https): |
184 | return True |
185 | if config_get('ssl_cert') and config_get('ssl_key'): |
186 | return True |
187 | |
188 | === modified file 'hooks/charmhelpers/contrib/openstack/amulet/deployment.py' |
189 | --- hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-01-26 09:45:43 +0000 |
190 | +++ hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-02-19 20:35:53 +0000 |
191 | @@ -71,16 +71,19 @@ |
192 | services.append(this_service) |
193 | use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph', |
194 | 'ceph-osd', 'ceph-radosgw'] |
195 | + # Openstack subordinate charms do not expose an origin option as that |
196 | + # is controlled by the principle |
197 | + ignore = ['neutron-openvswitch'] |
198 | |
199 | if self.openstack: |
200 | for svc in services: |
201 | - if svc['name'] not in use_source: |
202 | + if svc['name'] not in use_source + ignore: |
203 | config = {'openstack-origin': self.openstack} |
204 | self.d.configure(svc['name'], config) |
205 | |
206 | if self.source: |
207 | for svc in services: |
208 | - if svc['name'] in use_source: |
209 | + if svc['name'] in use_source and svc['name'] not in ignore: |
210 | config = {'source': self.source} |
211 | self.d.configure(svc['name'], config) |
212 | |
213 | |
214 | === added directory 'hooks/charmhelpers/contrib/openstack/files' |
215 | === added file 'hooks/charmhelpers/contrib/openstack/files/__init__.py' |
216 | --- hooks/charmhelpers/contrib/openstack/files/__init__.py 1970-01-01 00:00:00 +0000 |
217 | +++ hooks/charmhelpers/contrib/openstack/files/__init__.py 2015-02-19 20:35:53 +0000 |
218 | @@ -0,0 +1,18 @@ |
219 | +# Copyright 2014-2015 Canonical Limited. |
220 | +# |
221 | +# This file is part of charm-helpers. |
222 | +# |
223 | +# charm-helpers is free software: you can redistribute it and/or modify |
224 | +# it under the terms of the GNU Lesser General Public License version 3 as |
225 | +# published by the Free Software Foundation. |
226 | +# |
227 | +# charm-helpers is distributed in the hope that it will be useful, |
228 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
229 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
230 | +# GNU Lesser General Public License for more details. |
231 | +# |
232 | +# You should have received a copy of the GNU Lesser General Public License |
233 | +# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
234 | + |
235 | +# dummy __init__.py to fool syncer into thinking this is a syncable python |
236 | +# module |
237 | |
238 | === added file 'hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh' |
239 | --- hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh 1970-01-01 00:00:00 +0000 |
240 | +++ hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh 2015-02-19 20:35:53 +0000 |
241 | @@ -0,0 +1,32 @@ |
242 | +#!/bin/bash |
243 | +#-------------------------------------------- |
244 | +# This file is managed by Juju |
245 | +#-------------------------------------------- |
246 | +# |
247 | +# Copyright 2009,2012 Canonical Ltd. |
248 | +# Author: Tom Haddon |
249 | + |
250 | +CRITICAL=0 |
251 | +NOTACTIVE='' |
252 | +LOGFILE=/var/log/nagios/check_haproxy.log |
253 | +AUTH=$(grep -r "stats auth" /etc/haproxy | head -1 | awk '{print $4}') |
254 | + |
255 | +for appserver in $(grep ' server' /etc/haproxy/haproxy.cfg | awk '{print $2'}); |
256 | +do |
257 | + output=$(/usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 --regex="class=\"(active|backup)(2|3).*${appserver}" -e ' 200 OK') |
258 | + if [ $? != 0 ]; then |
259 | + date >> $LOGFILE |
260 | + echo $output >> $LOGFILE |
261 | + /usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 -v | grep $appserver >> $LOGFILE 2>&1 |
262 | + CRITICAL=1 |
263 | + NOTACTIVE="${NOTACTIVE} $appserver" |
264 | + fi |
265 | +done |
266 | + |
267 | +if [ $CRITICAL = 1 ]; then |
268 | + echo "CRITICAL:${NOTACTIVE}" |
269 | + exit 2 |
270 | +fi |
271 | + |
272 | +echo "OK: All haproxy instances looking good" |
273 | +exit 0 |
274 | |
275 | === added file 'hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh' |
276 | --- hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh 1970-01-01 00:00:00 +0000 |
277 | +++ hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh 2015-02-19 20:35:53 +0000 |
278 | @@ -0,0 +1,30 @@ |
279 | +#!/bin/bash |
280 | +#-------------------------------------------- |
281 | +# This file is managed by Juju |
282 | +#-------------------------------------------- |
283 | +# |
284 | +# Copyright 2009,2012 Canonical Ltd. |
285 | +# Author: Tom Haddon |
286 | + |
287 | +# These should be config options at some stage |
288 | +CURRQthrsh=0 |
289 | +MAXQthrsh=100 |
290 | + |
291 | +AUTH=$(grep -r "stats auth" /etc/haproxy | head -1 | awk '{print $4}') |
292 | + |
293 | +HAPROXYSTATS=$(/usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 -u '/;csv' -v) |
294 | + |
295 | +for BACKEND in $(echo $HAPROXYSTATS| xargs -n1 | grep BACKEND | awk -F , '{print $1}') |
296 | +do |
297 | + CURRQ=$(echo "$HAPROXYSTATS" | grep $BACKEND | grep BACKEND | cut -d , -f 3) |
298 | + MAXQ=$(echo "$HAPROXYSTATS" | grep $BACKEND | grep BACKEND | cut -d , -f 4) |
299 | + |
300 | + if [[ $CURRQ -gt $CURRQthrsh || $MAXQ -gt $MAXQthrsh ]] ; then |
301 | + echo "CRITICAL: queue depth for $BACKEND - CURRENT:$CURRQ MAX:$MAXQ" |
302 | + exit 2 |
303 | + fi |
304 | +done |
305 | + |
306 | +echo "OK: All haproxy queue depths looking good" |
307 | +exit 0 |
308 | + |
309 | |
310 | === modified file 'hooks/charmhelpers/contrib/openstack/ip.py' |
311 | --- hooks/charmhelpers/contrib/openstack/ip.py 2015-01-26 09:45:43 +0000 |
312 | +++ hooks/charmhelpers/contrib/openstack/ip.py 2015-02-19 20:35:53 +0000 |
313 | @@ -26,6 +26,8 @@ |
314 | ) |
315 | from charmhelpers.contrib.hahelpers.cluster import is_clustered |
316 | |
317 | +from functools import partial |
318 | + |
319 | PUBLIC = 'public' |
320 | INTERNAL = 'int' |
321 | ADMIN = 'admin' |
322 | @@ -107,3 +109,38 @@ |
323 | "clustered=%s)" % (net_type, clustered)) |
324 | |
325 | return resolved_address |
326 | + |
327 | + |
328 | +def endpoint_url(configs, url_template, port, endpoint_type=PUBLIC, |
329 | + override=None): |
330 | + """Returns the correct endpoint URL to advertise to Keystone. |
331 | + |
332 | + This method provides the correct endpoint URL which should be advertised to |
333 | + the keystone charm for endpoint creation. This method allows for the url to |
334 | + be overridden to force a keystone endpoint to have specific URL for any of |
335 | + the defined scopes (admin, internal, public). |
336 | + |
337 | + :param configs: OSTemplateRenderer config templating object to inspect |
338 | + for a complete https context. |
339 | + :param url_template: str format string for creating the url template. Only |
340 | + two values will be passed - the scheme+hostname |
341 | + returned by the canonical_url and the port. |
342 | + :param endpoint_type: str endpoint type to resolve. |
343 | + :param override: str the name of the config option which overrides the |
344 | + endpoint URL defined by the charm itself. None will |
345 | + disable any overrides (default). |
346 | + """ |
347 | + if override: |
348 | + # Return any user-defined overrides for the keystone endpoint URL. |
349 | + user_value = config(override) |
350 | + if user_value: |
351 | + return user_value.strip() |
352 | + |
353 | + return url_template % (canonical_url(configs, endpoint_type), port) |
354 | + |
355 | + |
356 | +public_endpoint = partial(endpoint_url, endpoint_type=PUBLIC) |
357 | + |
358 | +internal_endpoint = partial(endpoint_url, endpoint_type=INTERNAL) |
359 | + |
360 | +admin_endpoint = partial(endpoint_url, endpoint_type=ADMIN) |
361 | |
362 | === modified file 'hooks/charmhelpers/contrib/openstack/utils.py' |
363 | --- hooks/charmhelpers/contrib/openstack/utils.py 2015-01-26 09:45:43 +0000 |
364 | +++ hooks/charmhelpers/contrib/openstack/utils.py 2015-02-19 20:35:53 +0000 |
365 | @@ -103,6 +103,7 @@ |
366 | ('2.1.0', 'juno'), |
367 | ('2.2.0', 'juno'), |
368 | ('2.2.1', 'kilo'), |
369 | + ('2.2.2', 'kilo'), |
370 | ]) |
371 | |
372 | DEFAULT_LOOPBACK_SIZE = '5G' |
373 | |
374 | === modified file 'hooks/charmhelpers/contrib/python/packages.py' |
375 | --- hooks/charmhelpers/contrib/python/packages.py 2015-01-26 09:45:43 +0000 |
376 | +++ hooks/charmhelpers/contrib/python/packages.py 2015-02-19 20:35:53 +0000 |
377 | @@ -17,8 +17,6 @@ |
378 | # You should have received a copy of the GNU Lesser General Public License |
379 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
380 | |
381 | -__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>" |
382 | - |
383 | from charmhelpers.fetch import apt_install, apt_update |
384 | from charmhelpers.core.hookenv import log |
385 | |
386 | @@ -29,6 +27,8 @@ |
387 | apt_install('python-pip') |
388 | from pip import main as pip_execute |
389 | |
390 | +__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>" |
391 | + |
392 | |
393 | def parse_options(given, available): |
394 | """Given a set of options, check if available""" |
395 | |
396 | === modified file 'hooks/charmhelpers/core/fstab.py' |
397 | --- hooks/charmhelpers/core/fstab.py 2015-01-26 09:45:43 +0000 |
398 | +++ hooks/charmhelpers/core/fstab.py 2015-02-19 20:35:53 +0000 |
399 | @@ -17,11 +17,11 @@ |
400 | # You should have received a copy of the GNU Lesser General Public License |
401 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
402 | |
403 | -__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
404 | - |
405 | import io |
406 | import os |
407 | |
408 | +__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
409 | + |
410 | |
411 | class Fstab(io.FileIO): |
412 | """This class extends file in order to implement a file reader/writer |
413 | @@ -77,7 +77,7 @@ |
414 | for line in self.readlines(): |
415 | line = line.decode('us-ascii') |
416 | try: |
417 | - if line.strip() and not line.startswith("#"): |
418 | + if line.strip() and not line.strip().startswith("#"): |
419 | yield self._hydrate_entry(line) |
420 | except ValueError: |
421 | pass |
422 | @@ -104,7 +104,7 @@ |
423 | |
424 | found = False |
425 | for index, line in enumerate(lines): |
426 | - if not line.startswith("#"): |
427 | + if line.strip() and not line.strip().startswith("#"): |
428 | if self._hydrate_entry(line) == entry: |
429 | found = True |
430 | break |
431 | |
432 | === modified file 'hooks/charmhelpers/core/host.py' |
433 | --- hooks/charmhelpers/core/host.py 2015-01-26 09:45:43 +0000 |
434 | +++ hooks/charmhelpers/core/host.py 2015-02-19 20:35:53 +0000 |
435 | @@ -191,11 +191,11 @@ |
436 | |
437 | |
438 | def write_file(path, content, owner='root', group='root', perms=0o444): |
439 | - """Create or overwrite a file with the contents of a string""" |
440 | + """Create or overwrite a file with the contents of a byte string.""" |
441 | log("Writing file {} {}:{} {:o}".format(path, owner, group, perms)) |
442 | uid = pwd.getpwnam(owner).pw_uid |
443 | gid = grp.getgrnam(group).gr_gid |
444 | - with open(path, 'w') as target: |
445 | + with open(path, 'wb') as target: |
446 | os.fchown(target.fileno(), uid, gid) |
447 | os.fchmod(target.fileno(), perms) |
448 | target.write(content) |
449 | @@ -305,11 +305,11 @@ |
450 | ceph_client_changed function. |
451 | """ |
452 | def wrap(f): |
453 | - def wrapped_f(*args): |
454 | + def wrapped_f(*args, **kwargs): |
455 | checksums = {} |
456 | for path in restart_map: |
457 | checksums[path] = file_hash(path) |
458 | - f(*args) |
459 | + f(*args, **kwargs) |
460 | restarts = [] |
461 | for path in restart_map: |
462 | if checksums[path] != file_hash(path): |
463 | @@ -361,7 +361,7 @@ |
464 | ip_output = (line for line in ip_output if line) |
465 | for line in ip_output: |
466 | if line.split()[1].startswith(int_type): |
467 | - matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line) |
468 | + matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line) |
469 | if matched: |
470 | interface = matched.groups()[0] |
471 | else: |
472 | |
473 | === added file 'hooks/charmhelpers/core/strutils.py' |
474 | --- hooks/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000 |
475 | +++ hooks/charmhelpers/core/strutils.py 2015-02-19 20:35:53 +0000 |
476 | @@ -0,0 +1,42 @@ |
477 | +#!/usr/bin/env python |
478 | +# -*- coding: utf-8 -*- |
479 | + |
480 | +# Copyright 2014-2015 Canonical Limited. |
481 | +# |
482 | +# This file is part of charm-helpers. |
483 | +# |
484 | +# charm-helpers is free software: you can redistribute it and/or modify |
485 | +# it under the terms of the GNU Lesser General Public License version 3 as |
486 | +# published by the Free Software Foundation. |
487 | +# |
488 | +# charm-helpers is distributed in the hope that it will be useful, |
489 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
490 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
491 | +# GNU Lesser General Public License for more details. |
492 | +# |
493 | +# You should have received a copy of the GNU Lesser General Public License |
494 | +# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
495 | + |
496 | +import six |
497 | + |
498 | + |
499 | +def bool_from_string(value): |
500 | + """Interpret string value as boolean. |
501 | + |
502 | + Returns True if value translates to True otherwise False. |
503 | + """ |
504 | + if isinstance(value, six.string_types): |
505 | + value = six.text_type(value) |
506 | + else: |
507 | + msg = "Unable to interpret non-string value '%s' as boolean" % (value) |
508 | + raise ValueError(msg) |
509 | + |
510 | + value = value.strip().lower() |
511 | + |
512 | + if value in ['y', 'yes', 'true', 't']: |
513 | + return True |
514 | + elif value in ['n', 'no', 'false', 'f']: |
515 | + return False |
516 | + |
517 | + msg = "Unable to interpret string value '%s' as boolean" % (value) |
518 | + raise ValueError(msg) |
519 | |
520 | === modified file 'hooks/charmhelpers/core/sysctl.py' |
521 | --- hooks/charmhelpers/core/sysctl.py 2015-01-26 09:45:43 +0000 |
522 | +++ hooks/charmhelpers/core/sysctl.py 2015-02-19 20:35:53 +0000 |
523 | @@ -17,8 +17,6 @@ |
524 | # You should have received a copy of the GNU Lesser General Public License |
525 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
526 | |
527 | -__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
528 | - |
529 | import yaml |
530 | |
531 | from subprocess import check_call |
532 | @@ -26,25 +24,33 @@ |
533 | from charmhelpers.core.hookenv import ( |
534 | log, |
535 | DEBUG, |
536 | + ERROR, |
537 | ) |
538 | |
539 | +__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
540 | + |
541 | |
542 | def create(sysctl_dict, sysctl_file): |
543 | """Creates a sysctl.conf file from a YAML associative array |
544 | |
545 | - :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 } |
546 | - :type sysctl_dict: dict |
547 | + :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }" |
548 | + :type sysctl_dict: str |
549 | :param sysctl_file: path to the sysctl file to be saved |
550 | :type sysctl_file: str or unicode |
551 | :returns: None |
552 | """ |
553 | - sysctl_dict = yaml.load(sysctl_dict) |
554 | + try: |
555 | + sysctl_dict_parsed = yaml.safe_load(sysctl_dict) |
556 | + except yaml.YAMLError: |
557 | + log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict), |
558 | + level=ERROR) |
559 | + return |
560 | |
561 | with open(sysctl_file, "w") as fd: |
562 | - for key, value in sysctl_dict.items(): |
563 | + for key, value in sysctl_dict_parsed.items(): |
564 | fd.write("{}={}\n".format(key, value)) |
565 | |
566 | - log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict), |
567 | + log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed), |
568 | level=DEBUG) |
569 | |
570 | check_call(["sysctl", "-p", sysctl_file]) |
571 | |
572 | === modified file 'hooks/charmhelpers/core/templating.py' |
573 | --- hooks/charmhelpers/core/templating.py 2015-01-26 09:45:43 +0000 |
574 | +++ hooks/charmhelpers/core/templating.py 2015-02-19 20:35:53 +0000 |
575 | @@ -21,7 +21,7 @@ |
576 | |
577 | |
578 | def render(source, target, context, owner='root', group='root', |
579 | - perms=0o444, templates_dir=None): |
580 | + perms=0o444, templates_dir=None, encoding='UTF-8'): |
581 | """ |
582 | Render a template. |
583 | |
584 | @@ -64,5 +64,5 @@ |
585 | level=hookenv.ERROR) |
586 | raise e |
587 | content = template.render(context) |
588 | - host.mkdir(os.path.dirname(target), owner, group) |
589 | - host.write_file(target, content, owner, group, perms) |
590 | + host.mkdir(os.path.dirname(target), owner, group, perms=0o755) |
591 | + host.write_file(target, content.encode(encoding), owner, group, perms) |
592 | |
593 | === added file 'hooks/charmhelpers/core/unitdata.py' |
594 | --- hooks/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000 |
595 | +++ hooks/charmhelpers/core/unitdata.py 2015-02-19 20:35:53 +0000 |
596 | @@ -0,0 +1,477 @@ |
597 | +#!/usr/bin/env python |
598 | +# -*- coding: utf-8 -*- |
599 | +# |
600 | +# Copyright 2014-2015 Canonical Limited. |
601 | +# |
602 | +# This file is part of charm-helpers. |
603 | +# |
604 | +# charm-helpers is free software: you can redistribute it and/or modify |
605 | +# it under the terms of the GNU Lesser General Public License version 3 as |
606 | +# published by the Free Software Foundation. |
607 | +# |
608 | +# charm-helpers is distributed in the hope that it will be useful, |
609 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
610 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
611 | +# GNU Lesser General Public License for more details. |
612 | +# |
613 | +# You should have received a copy of the GNU Lesser General Public License |
614 | +# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
615 | +# |
616 | +# |
617 | +# Authors: |
618 | +# Kapil Thangavelu <kapil.foss@gmail.com> |
619 | +# |
620 | +""" |
621 | +Intro |
622 | +----- |
623 | + |
624 | +A simple way to store state in units. This provides a key value |
625 | +storage with support for versioned, transactional operation, |
626 | +and can calculate deltas from previous values to simplify unit logic |
627 | +when processing changes. |
628 | + |
629 | + |
630 | +Hook Integration |
631 | +---------------- |
632 | + |
633 | +There are several extant frameworks for hook execution, including |
634 | + |
635 | + - charmhelpers.core.hookenv.Hooks |
636 | + - charmhelpers.core.services.ServiceManager |
637 | + |
638 | +The storage classes are framework agnostic, one simple integration is |
639 | +via the HookData contextmanager. It will record the current hook |
640 | +execution environment (including relation data, config data, etc.), |
641 | +setup a transaction and allow easy access to the changes from |
642 | +previously seen values. One consequence of the integration is the |
643 | +reservation of particular keys ('rels', 'unit', 'env', 'config', |
644 | +'charm_revisions') for their respective values. |
645 | + |
646 | +Here's a fully worked integration example using hookenv.Hooks:: |
647 | + |
648 | + from charmhelper.core import hookenv, unitdata |
649 | + |
650 | + hook_data = unitdata.HookData() |
651 | + db = unitdata.kv() |
652 | + hooks = hookenv.Hooks() |
653 | + |
654 | + @hooks.hook |
655 | + def config_changed(): |
656 | + # Print all changes to configuration from previously seen |
657 | + # values. |
658 | + for changed, (prev, cur) in hook_data.conf.items(): |
659 | + print('config changed', changed, |
660 | + 'previous value', prev, |
661 | + 'current value', cur) |
662 | + |
663 | + # Get some unit specific bookeeping |
664 | + if not db.get('pkg_key'): |
665 | + key = urllib.urlopen('https://example.com/pkg_key').read() |
666 | + db.set('pkg_key', key) |
667 | + |
668 | + # Directly access all charm config as a mapping. |
669 | + conf = db.getrange('config', True) |
670 | + |
671 | + # Directly access all relation data as a mapping |
672 | + rels = db.getrange('rels', True) |
673 | + |
674 | + if __name__ == '__main__': |
675 | + with hook_data(): |
676 | + hook.execute() |
677 | + |
678 | + |
679 | +A more basic integration is via the hook_scope context manager which simply |
680 | +manages transaction scope (and records hook name, and timestamp):: |
681 | + |
682 | + >>> from unitdata import kv |
683 | + >>> db = kv() |
684 | + >>> with db.hook_scope('install'): |
685 | + ... # do work, in transactional scope. |
686 | + ... db.set('x', 1) |
687 | + >>> db.get('x') |
688 | + 1 |
689 | + |
690 | + |
691 | +Usage |
692 | +----- |
693 | + |
694 | +Values are automatically json de/serialized to preserve basic typing |
695 | +and complex data struct capabilities (dicts, lists, ints, booleans, etc). |
696 | + |
697 | +Individual values can be manipulated via get/set:: |
698 | + |
699 | + >>> kv.set('y', True) |
700 | + >>> kv.get('y') |
701 | + True |
702 | + |
703 | + # We can set complex values (dicts, lists) as a single key. |
704 | + >>> kv.set('config', {'a': 1, 'b': True'}) |
705 | + |
706 | + # Also supports returning dictionaries as a record which |
707 | + # provides attribute access. |
708 | + >>> config = kv.get('config', record=True) |
709 | + >>> config.b |
710 | + True |
711 | + |
712 | + |
713 | +Groups of keys can be manipulated with update/getrange:: |
714 | + |
715 | + >>> kv.update({'z': 1, 'y': 2}, prefix="gui.") |
716 | + >>> kv.getrange('gui.', strip=True) |
717 | + {'z': 1, 'y': 2} |
718 | + |
719 | +When updating values, its very helpful to understand which values |
720 | +have actually changed and how have they changed. The storage |
721 | +provides a delta method to provide for this:: |
722 | + |
723 | + >>> data = {'debug': True, 'option': 2} |
724 | + >>> delta = kv.delta(data, 'config.') |
725 | + >>> delta.debug.previous |
726 | + None |
727 | + >>> delta.debug.current |
728 | + True |
729 | + >>> delta |
730 | + {'debug': (None, True), 'option': (None, 2)} |
731 | + |
732 | +Note the delta method does not persist the actual change, it needs to |
733 | +be explicitly saved via 'update' method:: |
734 | + |
735 | + >>> kv.update(data, 'config.') |
736 | + |
737 | +Values modified in the context of a hook scope retain historical values |
738 | +associated to the hookname. |
739 | + |
740 | + >>> with db.hook_scope('config-changed'): |
741 | + ... db.set('x', 42) |
742 | + >>> db.gethistory('x') |
743 | + [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'), |
744 | + (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')] |
745 | + |
746 | +""" |
747 | + |
748 | +import collections |
749 | +import contextlib |
750 | +import datetime |
751 | +import json |
752 | +import os |
753 | +import pprint |
754 | +import sqlite3 |
755 | +import sys |
756 | + |
757 | +__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>' |
758 | + |
759 | + |
760 | +class Storage(object): |
761 | + """Simple key value database for local unit state within charms. |
762 | + |
763 | + Modifications are automatically committed at hook exit. That's |
764 | + currently regardless of exit code. |
765 | + |
766 | + To support dicts, lists, integer, floats, and booleans values |
767 | + are automatically json encoded/decoded. |
768 | + """ |
769 | + def __init__(self, path=None): |
770 | + self.db_path = path |
771 | + if path is None: |
772 | + self.db_path = os.path.join( |
773 | + os.environ.get('CHARM_DIR', ''), '.unit-state.db') |
774 | + self.conn = sqlite3.connect('%s' % self.db_path) |
775 | + self.cursor = self.conn.cursor() |
776 | + self.revision = None |
777 | + self._closed = False |
778 | + self._init() |
779 | + |
780 | + def close(self): |
781 | + if self._closed: |
782 | + return |
783 | + self.flush(False) |
784 | + self.cursor.close() |
785 | + self.conn.close() |
786 | + self._closed = True |
787 | + |
788 | + def _scoped_query(self, stmt, params=None): |
789 | + if params is None: |
790 | + params = [] |
791 | + return stmt, params |
792 | + |
793 | + def get(self, key, default=None, record=False): |
794 | + self.cursor.execute( |
795 | + *self._scoped_query( |
796 | + 'select data from kv where key=?', [key])) |
797 | + result = self.cursor.fetchone() |
798 | + if not result: |
799 | + return default |
800 | + if record: |
801 | + return Record(json.loads(result[0])) |
802 | + return json.loads(result[0]) |
803 | + |
804 | + def getrange(self, key_prefix, strip=False): |
805 | + stmt = "select key, data from kv where key like '%s%%'" % key_prefix |
806 | + self.cursor.execute(*self._scoped_query(stmt)) |
807 | + result = self.cursor.fetchall() |
808 | + |
809 | + if not result: |
810 | + return None |
811 | + if not strip: |
812 | + key_prefix = '' |
813 | + return dict([ |
814 | + (k[len(key_prefix):], json.loads(v)) for k, v in result]) |
815 | + |
816 | + def update(self, mapping, prefix=""): |
817 | + for k, v in mapping.items(): |
818 | + self.set("%s%s" % (prefix, k), v) |
819 | + |
820 | + def unset(self, key): |
821 | + self.cursor.execute('delete from kv where key=?', [key]) |
822 | + if self.revision and self.cursor.rowcount: |
823 | + self.cursor.execute( |
824 | + 'insert into kv_revisions values (?, ?, ?)', |
825 | + [key, self.revision, json.dumps('DELETED')]) |
826 | + |
827 | + def set(self, key, value): |
828 | + serialized = json.dumps(value) |
829 | + |
830 | + self.cursor.execute( |
831 | + 'select data from kv where key=?', [key]) |
832 | + exists = self.cursor.fetchone() |
833 | + |
834 | + # Skip mutations to the same value |
835 | + if exists: |
836 | + if exists[0] == serialized: |
837 | + return value |
838 | + |
839 | + if not exists: |
840 | + self.cursor.execute( |
841 | + 'insert into kv (key, data) values (?, ?)', |
842 | + (key, serialized)) |
843 | + else: |
844 | + self.cursor.execute(''' |
845 | + update kv |
846 | + set data = ? |
847 | + where key = ?''', [serialized, key]) |
848 | + |
849 | + # Save |
850 | + if not self.revision: |
851 | + return value |
852 | + |
853 | + self.cursor.execute( |
854 | + 'select 1 from kv_revisions where key=? and revision=?', |
855 | + [key, self.revision]) |
856 | + exists = self.cursor.fetchone() |
857 | + |
858 | + if not exists: |
859 | + self.cursor.execute( |
860 | + '''insert into kv_revisions ( |
861 | + revision, key, data) values (?, ?, ?)''', |
862 | + (self.revision, key, serialized)) |
863 | + else: |
864 | + self.cursor.execute( |
865 | + ''' |
866 | + update kv_revisions |
867 | + set data = ? |
868 | + where key = ? |
869 | + and revision = ?''', |
870 | + [serialized, key, self.revision]) |
871 | + |
872 | + return value |
873 | + |
874 | + def delta(self, mapping, prefix): |
875 | + """ |
876 | + return a delta containing values that have changed. |
877 | + """ |
878 | + previous = self.getrange(prefix, strip=True) |
879 | + if not previous: |
880 | + pk = set() |
881 | + else: |
882 | + pk = set(previous.keys()) |
883 | + ck = set(mapping.keys()) |
884 | + delta = DeltaSet() |
885 | + |
886 | + # added |
887 | + for k in ck.difference(pk): |
888 | + delta[k] = Delta(None, mapping[k]) |
889 | + |
890 | + # removed |
891 | + for k in pk.difference(ck): |
892 | + delta[k] = Delta(previous[k], None) |
893 | + |
894 | + # changed |
895 | + for k in pk.intersection(ck): |
896 | + c = mapping[k] |
897 | + p = previous[k] |
898 | + if c != p: |
899 | + delta[k] = Delta(p, c) |
900 | + |
901 | + return delta |
902 | + |
903 | + @contextlib.contextmanager |
904 | + def hook_scope(self, name=""): |
905 | + """Scope all future interactions to the current hook execution |
906 | + revision.""" |
907 | + assert not self.revision |
908 | + self.cursor.execute( |
909 | + 'insert into hooks (hook, date) values (?, ?)', |
910 | + (name or sys.argv[0], |
911 | + datetime.datetime.utcnow().isoformat())) |
912 | + self.revision = self.cursor.lastrowid |
913 | + try: |
914 | + yield self.revision |
915 | + self.revision = None |
916 | + except: |
917 | + self.flush(False) |
918 | + self.revision = None |
919 | + raise |
920 | + else: |
921 | + self.flush() |
922 | + |
923 | + def flush(self, save=True): |
924 | + if save: |
925 | + self.conn.commit() |
926 | + elif self._closed: |
927 | + return |
928 | + else: |
929 | + self.conn.rollback() |
930 | + |
931 | + def _init(self): |
932 | + self.cursor.execute(''' |
933 | + create table if not exists kv ( |
934 | + key text, |
935 | + data text, |
936 | + primary key (key) |
937 | + )''') |
938 | + self.cursor.execute(''' |
939 | + create table if not exists kv_revisions ( |
940 | + key text, |
941 | + revision integer, |
942 | + data text, |
943 | + primary key (key, revision) |
944 | + )''') |
945 | + self.cursor.execute(''' |
946 | + create table if not exists hooks ( |
947 | + version integer primary key autoincrement, |
948 | + hook text, |
949 | + date text |
950 | + )''') |
951 | + self.conn.commit() |
952 | + |
953 | + def gethistory(self, key, deserialize=False): |
954 | + self.cursor.execute( |
955 | + ''' |
956 | + select kv.revision, kv.key, kv.data, h.hook, h.date |
957 | + from kv_revisions kv, |
958 | + hooks h |
959 | + where kv.key=? |
960 | + and kv.revision = h.version |
961 | + ''', [key]) |
962 | + if deserialize is False: |
963 | + return self.cursor.fetchall() |
964 | + return map(_parse_history, self.cursor.fetchall()) |
965 | + |
966 | + def debug(self, fh=sys.stderr): |
967 | + self.cursor.execute('select * from kv') |
968 | + pprint.pprint(self.cursor.fetchall(), stream=fh) |
969 | + self.cursor.execute('select * from kv_revisions') |
970 | + pprint.pprint(self.cursor.fetchall(), stream=fh) |
971 | + |
972 | + |
973 | +def _parse_history(d): |
974 | + return (d[0], d[1], json.loads(d[2]), d[3], |
975 | + datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f")) |
976 | + |
977 | + |
978 | +class HookData(object): |
979 | + """Simple integration for existing hook exec frameworks. |
980 | + |
981 | + Records all unit information, and stores deltas for processing |
982 | + by the hook. |
983 | + |
984 | + Sample:: |
985 | + |
986 | + from charmhelper.core import hookenv, unitdata |
987 | + |
988 | + changes = unitdata.HookData() |
989 | + db = unitdata.kv() |
990 | + hooks = hookenv.Hooks() |
991 | + |
992 | + @hooks.hook |
993 | + def config_changed(): |
994 | + # View all changes to configuration |
995 | + for changed, (prev, cur) in changes.conf.items(): |
996 | + print('config changed', changed, |
997 | + 'previous value', prev, |
998 | + 'current value', cur) |
999 | + |
1000 | + # Get some unit specific bookeeping |
1001 | + if not db.get('pkg_key'): |
1002 | + key = urllib.urlopen('https://example.com/pkg_key').read() |
1003 | + db.set('pkg_key', key) |
1004 | + |
1005 | + if __name__ == '__main__': |
1006 | + with changes(): |
1007 | + hook.execute() |
1008 | + |
1009 | + """ |
1010 | + def __init__(self): |
1011 | + self.kv = kv() |
1012 | + self.conf = None |
1013 | + self.rels = None |
1014 | + |
1015 | + @contextlib.contextmanager |
1016 | + def __call__(self): |
1017 | + from charmhelpers.core import hookenv |
1018 | + hook_name = hookenv.hook_name() |
1019 | + |
1020 | + with self.kv.hook_scope(hook_name): |
1021 | + self._record_charm_version(hookenv.charm_dir()) |
1022 | + delta_config, delta_relation = self._record_hook(hookenv) |
1023 | + yield self.kv, delta_config, delta_relation |
1024 | + |
1025 | + def _record_charm_version(self, charm_dir): |
1026 | + # Record revisions.. charm revisions are meaningless |
1027 | + # to charm authors as they don't control the revision. |
1028 | + # so logic dependnent on revision is not particularly |
1029 | + # useful, however it is useful for debugging analysis. |
1030 | + charm_rev = open( |
1031 | + os.path.join(charm_dir, 'revision')).read().strip() |
1032 | + charm_rev = charm_rev or '0' |
1033 | + revs = self.kv.get('charm_revisions', []) |
1034 | + if charm_rev not in revs: |
1035 | + revs.append(charm_rev.strip() or '0') |
1036 | + self.kv.set('charm_revisions', revs) |
1037 | + |
1038 | + def _record_hook(self, hookenv): |
1039 | + data = hookenv.execution_environment() |
1040 | + self.conf = conf_delta = self.kv.delta(data['conf'], 'config') |
1041 | + self.rels = rels_delta = self.kv.delta(data['rels'], 'rels') |
1042 | + self.kv.set('env', data['env']) |
1043 | + self.kv.set('unit', data['unit']) |
1044 | + self.kv.set('relid', data.get('relid')) |
1045 | + return conf_delta, rels_delta |
1046 | + |
1047 | + |
1048 | +class Record(dict): |
1049 | + |
1050 | + __slots__ = () |
1051 | + |
1052 | + def __getattr__(self, k): |
1053 | + if k in self: |
1054 | + return self[k] |
1055 | + raise AttributeError(k) |
1056 | + |
1057 | + |
1058 | +class DeltaSet(Record): |
1059 | + |
1060 | + __slots__ = () |
1061 | + |
1062 | + |
1063 | +Delta = collections.namedtuple('Delta', ['previous', 'current']) |
1064 | + |
1065 | + |
1066 | +_KV = None |
1067 | + |
1068 | + |
1069 | +def kv(): |
1070 | + global _KV |
1071 | + if _KV is None: |
1072 | + _KV = Storage() |
1073 | + return _KV |
1074 | |
1075 | === modified file 'hooks/charmhelpers/fetch/archiveurl.py' |
1076 | --- hooks/charmhelpers/fetch/archiveurl.py 2015-01-26 09:45:43 +0000 |
1077 | +++ hooks/charmhelpers/fetch/archiveurl.py 2015-02-19 20:35:53 +0000 |
1078 | @@ -18,6 +18,16 @@ |
1079 | import hashlib |
1080 | import re |
1081 | |
1082 | +from charmhelpers.fetch import ( |
1083 | + BaseFetchHandler, |
1084 | + UnhandledSource |
1085 | +) |
1086 | +from charmhelpers.payload.archive import ( |
1087 | + get_archive_handler, |
1088 | + extract, |
1089 | +) |
1090 | +from charmhelpers.core.host import mkdir, check_hash |
1091 | + |
1092 | import six |
1093 | if six.PY3: |
1094 | from urllib.request import ( |
1095 | @@ -35,16 +45,6 @@ |
1096 | ) |
1097 | from urlparse import urlparse, urlunparse, parse_qs |
1098 | |
1099 | -from charmhelpers.fetch import ( |
1100 | - BaseFetchHandler, |
1101 | - UnhandledSource |
1102 | -) |
1103 | -from charmhelpers.payload.archive import ( |
1104 | - get_archive_handler, |
1105 | - extract, |
1106 | -) |
1107 | -from charmhelpers.core.host import mkdir, check_hash |
1108 | - |
1109 | |
1110 | def splituser(host): |
1111 | '''urllib.splituser(), but six's support of this seems broken''' |
1112 | |
1113 | === modified file 'hooks/charmhelpers/fetch/giturl.py' |
1114 | --- hooks/charmhelpers/fetch/giturl.py 2015-01-26 09:45:43 +0000 |
1115 | +++ hooks/charmhelpers/fetch/giturl.py 2015-02-19 20:35:53 +0000 |
1116 | @@ -32,7 +32,7 @@ |
1117 | apt_install("python-git") |
1118 | from git import Repo |
1119 | |
1120 | -from git.exc import GitCommandError |
1121 | +from git.exc import GitCommandError # noqa E402 |
1122 | |
1123 | |
1124 | class GitUrlFetchHandler(BaseFetchHandler): |
1125 | |
1126 | === modified file 'tests/charmhelpers/contrib/amulet/utils.py' |
1127 | --- tests/charmhelpers/contrib/amulet/utils.py 2015-01-26 09:45:43 +0000 |
1128 | +++ tests/charmhelpers/contrib/amulet/utils.py 2015-02-19 20:35:53 +0000 |
1129 | @@ -169,8 +169,13 @@ |
1130 | cmd = 'pgrep -o -f {}'.format(service) |
1131 | else: |
1132 | cmd = 'pgrep -o {}'.format(service) |
1133 | - proc_dir = '/proc/{}'.format(sentry_unit.run(cmd)[0].strip()) |
1134 | - return self._get_dir_mtime(sentry_unit, proc_dir) |
1135 | + cmd = cmd + ' | grep -v pgrep || exit 0' |
1136 | + cmd_out = sentry_unit.run(cmd) |
1137 | + self.log.debug('CMDout: ' + str(cmd_out)) |
1138 | + if cmd_out[0]: |
1139 | + self.log.debug('Pid for %s %s' % (service, str(cmd_out[0]))) |
1140 | + proc_dir = '/proc/{}'.format(cmd_out[0].strip()) |
1141 | + return self._get_dir_mtime(sentry_unit, proc_dir) |
1142 | |
1143 | def service_restarted(self, sentry_unit, service, filename, |
1144 | pgrep_full=False, sleep_time=20): |
1145 | @@ -187,6 +192,121 @@ |
1146 | else: |
1147 | return False |
1148 | |
1149 | + def service_restarted_since(self, sentry_unit, mtime, service, |
1150 | + pgrep_full=False, sleep_time=20, |
1151 | + retry_count=2): |
1152 | + """Check if service was been started after a given time. |
1153 | + |
1154 | + Args: |
1155 | + sentry_unit (sentry): The sentry unit to check for the service on |
1156 | + mtime (float): The epoch time to check against |
1157 | + service (string): service name to look for in process table |
1158 | + pgrep_full (boolean): Use full command line search mode with pgrep |
1159 | + sleep_time (int): Seconds to sleep before looking for process |
1160 | + retry_count (int): If service is not found, how many times to retry |
1161 | + |
1162 | + Returns: |
1163 | + bool: True if service found and its start time it newer than mtime, |
1164 | + False if service is older than mtime or if service was |
1165 | + not found. |
1166 | + """ |
1167 | + self.log.debug('Checking %s restarted since %s' % (service, mtime)) |
1168 | + time.sleep(sleep_time) |
1169 | + proc_start_time = self._get_proc_start_time(sentry_unit, service, |
1170 | + pgrep_full) |
1171 | + while retry_count > 0 and not proc_start_time: |
1172 | + self.log.debug('No pid file found for service %s, will retry %i ' |
1173 | + 'more times' % (service, retry_count)) |
1174 | + time.sleep(30) |
1175 | + proc_start_time = self._get_proc_start_time(sentry_unit, service, |
1176 | + pgrep_full) |
1177 | + retry_count = retry_count - 1 |
1178 | + |
1179 | + if not proc_start_time: |
1180 | + self.log.warn('No proc start time found, assuming service did ' |
1181 | + 'not start') |
1182 | + return False |
1183 | + if proc_start_time >= mtime: |
1184 | + self.log.debug('proc start time is newer than provided mtime' |
1185 | + '(%s >= %s)' % (proc_start_time, mtime)) |
1186 | + return True |
1187 | + else: |
1188 | + self.log.warn('proc start time (%s) is older than provided mtime ' |
1189 | + '(%s), service did not restart' % (proc_start_time, |
1190 | + mtime)) |
1191 | + return False |
1192 | + |
1193 | + def config_updated_since(self, sentry_unit, filename, mtime, |
1194 | + sleep_time=20): |
1195 | + """Check if file was modified after a given time. |
1196 | + |
1197 | + Args: |
1198 | + sentry_unit (sentry): The sentry unit to check the file mtime on |
1199 | + filename (string): The file to check mtime of |
1200 | + mtime (float): The epoch time to check against |
1201 | + sleep_time (int): Seconds to sleep before looking for process |
1202 | + |
1203 | + Returns: |
1204 | + bool: True if file was modified more recently than mtime, False if |
1205 | + file was modified before mtime, |
1206 | + """ |
1207 | + self.log.debug('Checking %s updated since %s' % (filename, mtime)) |
1208 | + time.sleep(sleep_time) |
1209 | + file_mtime = self._get_file_mtime(sentry_unit, filename) |
1210 | + if file_mtime >= mtime: |
1211 | + self.log.debug('File mtime is newer than provided mtime ' |
1212 | + '(%s >= %s)' % (file_mtime, mtime)) |
1213 | + return True |
1214 | + else: |
1215 | + self.log.warn('File mtime %s is older than provided mtime %s' |
1216 | + % (file_mtime, mtime)) |
1217 | + return False |
1218 | + |
1219 | + def validate_service_config_changed(self, sentry_unit, mtime, service, |
1220 | + filename, pgrep_full=False, |
1221 | + sleep_time=20, retry_count=2): |
1222 | + """Check service and file were updated after mtime |
1223 | + |
1224 | + Args: |
1225 | + sentry_unit (sentry): The sentry unit to check for the service on |
1226 | + mtime (float): The epoch time to check against |
1227 | + service (string): service name to look for in process table |
1228 | + filename (string): The file to check mtime of |
1229 | + pgrep_full (boolean): Use full command line search mode with pgrep |
1230 | + sleep_time (int): Seconds to sleep before looking for process |
1231 | + retry_count (int): If service is not found, how many times to retry |
1232 | + |
1233 | + Typical Usage: |
1234 | + u = OpenStackAmuletUtils(ERROR) |
1235 | + ... |
1236 | + mtime = u.get_sentry_time(self.cinder_sentry) |
1237 | + self.d.configure('cinder', {'verbose': 'True', 'debug': 'True'}) |
1238 | + if not u.validate_service_config_changed(self.cinder_sentry, |
1239 | + mtime, |
1240 | + 'cinder-api', |
1241 | + '/etc/cinder/cinder.conf') |
1242 | + amulet.raise_status(amulet.FAIL, msg='update failed') |
1243 | + Returns: |
1244 | + bool: True if both service and file where updated/restarted after |
1245 | + mtime, False if service is older than mtime or if service was |
1246 | + not found or if filename was modified before mtime. |
1247 | + """ |
1248 | + self.log.debug('Checking %s restarted since %s' % (service, mtime)) |
1249 | + time.sleep(sleep_time) |
1250 | + service_restart = self.service_restarted_since(sentry_unit, mtime, |
1251 | + service, |
1252 | + pgrep_full=pgrep_full, |
1253 | + sleep_time=0, |
1254 | + retry_count=retry_count) |
1255 | + config_update = self.config_updated_since(sentry_unit, filename, mtime, |
1256 | + sleep_time=0) |
1257 | + return service_restart and config_update |
1258 | + |
1259 | + def get_sentry_time(self, sentry_unit): |
1260 | + """Return current epoch time on a sentry""" |
1261 | + cmd = "date +'%s'" |
1262 | + return float(sentry_unit.run(cmd)[0]) |
1263 | + |
1264 | def relation_error(self, name, data): |
1265 | return 'unexpected relation data in {} - {}'.format(name, data) |
1266 | |
1267 | |
1268 | === modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py' |
1269 | --- tests/charmhelpers/contrib/openstack/amulet/deployment.py 2015-01-26 09:45:43 +0000 |
1270 | +++ tests/charmhelpers/contrib/openstack/amulet/deployment.py 2015-02-19 20:35:53 +0000 |
1271 | @@ -71,16 +71,19 @@ |
1272 | services.append(this_service) |
1273 | use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph', |
1274 | 'ceph-osd', 'ceph-radosgw'] |
1275 | + # Openstack subordinate charms do not expose an origin option as that |
1276 | + # is controlled by the principle |
1277 | + ignore = ['neutron-openvswitch'] |
1278 | |
1279 | if self.openstack: |
1280 | for svc in services: |
1281 | - if svc['name'] not in use_source: |
1282 | + if svc['name'] not in use_source + ignore: |
1283 | config = {'openstack-origin': self.openstack} |
1284 | self.d.configure(svc['name'], config) |
1285 | |
1286 | if self.source: |
1287 | for svc in services: |
1288 | - if svc['name'] in use_source: |
1289 | + if svc['name'] in use_source and svc['name'] not in ignore: |
1290 | config = {'source': self.source} |
1291 | self.d.configure(svc['name'], config) |
1292 |
charm_lint_check #2121 swift-proxy for 1chb1n mp250374
LINT FAIL: lint-test failed
LINT FAIL: charm-proof failed
LINT Results (max last 2 lines): admin-password does not have the keys: default
W: config.yaml: option keystone-
make: *** [lint] Error 100
Full lint test output: http:// paste.ubuntu. com/10313191/ 10.245. 162.77: 8080/job/ charm_lint_ check/2121/
Build: http://