Merge lp:~canonical-is-sa/charms/trusty/logstash/trunk into lp:~lazypower/charms/trusty/logstash/trunk
- Trusty Tahr (14.04)
- trunk
- Merge into trunk
Status: | Needs review |
---|---|
Proposed branch: | lp:~canonical-is-sa/charms/trusty/logstash/trunk |
Merge into: | lp:~lazypower/charms/trusty/logstash/trunk |
Diff against target: |
2990 lines (+1996/-154) 31 files modified
README.md (+4/-5) charm-helpers.yaml (+1/-0) config.yaml (+32/-0) hooks/client-relation-changed (+20/-14) hooks/config-changed (+14/-2) hooks/nrpe-external-master-relation-changed (+87/-0) lib/charmhelpers/contrib/charmsupport/__init__.py (+15/-0) lib/charmhelpers/contrib/charmsupport/nrpe.py (+360/-0) lib/charmhelpers/contrib/charmsupport/volumes.py (+175/-0) lib/charmhelpers/contrib/templating/__init__.py (+15/-0) lib/charmhelpers/contrib/templating/jinja.py (+25/-9) lib/charmhelpers/core/__init__.py (+15/-0) lib/charmhelpers/core/decorators.py (+57/-0) lib/charmhelpers/core/fstab.py (+30/-12) lib/charmhelpers/core/hookenv.py (+105/-22) lib/charmhelpers/core/host.py (+103/-38) lib/charmhelpers/core/services/__init__.py (+18/-2) lib/charmhelpers/core/services/base.py (+16/-0) lib/charmhelpers/core/services/helpers.py (+37/-9) lib/charmhelpers/core/strutils.py (+42/-0) lib/charmhelpers/core/sysctl.py (+56/-0) lib/charmhelpers/core/templating.py (+20/-3) lib/charmhelpers/core/unitdata.py (+477/-0) lib/charmhelpers/fetch/__init__.py (+44/-14) lib/charmhelpers/fetch/archiveurl.py (+76/-22) lib/charmhelpers/fetch/bzrurl.py (+30/-2) lib/charmhelpers/fetch/giturl.py (+71/-0) lib/charmhelpers/payload/__init__.py (+16/-0) lib/charmhelpers/payload/archive.py (+16/-0) lib/charmhelpers/payload/execd.py (+16/-0) metadata.yaml (+3/-0) |
To merge this branch: | bzr merge lp:~canonical-is-sa/charms/trusty/logstash/trunk |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Charles Butler | Pending | ||
Review via email: mp+258828@code.launchpad.net |
Commit message
Description of the change
Have added nagios checks for logstash and fixed client-
Also updated charmhelpers.
- 53. By Michael Foley
-
[foli] adjust nagios checks
- 54. By Michael Foley
-
[michael.
nelson, r=foli] Add a nagios cert check to avoid cert expiry. Add extra_config to enable juju setting extra filters. RT#85874
Unmerged revisions
- 54. By Michael Foley
-
[michael.
nelson, r=foli] Add a nagios cert check to avoid cert expiry. Add extra_config to enable juju setting extra filters. RT#85874 - 53. By Michael Foley
-
[foli] adjust nagios checks
- 52. By Michael Foley
-
[foli] fix nrpe-external-
master relation, don't add lumberjack port check if not configured - 51. By Michael Foley
-
[foli] initial add of nrpe-external-
master relation and custom checks - 50. By Michael Foley
-
[foli] addedh charmhelpers contrib.
charmsupport - 49. By Michael Foley
-
[foli] updated charmhelpers from upstream
- 48. By Michael Foley
-
[foli] Fix client-
relation- changed to use most recently joined elasticsearch unit
Preview Diff
1 | === modified file 'README.md' | |||
2 | --- README.md 2014-07-24 13:26:45 +0000 | |||
3 | +++ README.md 2015-11-01 20:32:57 +0000 | |||
4 | @@ -28,7 +28,7 @@ | |||
5 | 28 | http://ip-of-kibana | 28 | http://ip-of-kibana |
6 | 29 | 29 | ||
7 | 30 | example 2 - Indexer + 2 x ElasticSearch + Kibana | 30 | example 2 - Indexer + 2 x ElasticSearch + Kibana |
9 | 31 | ============================================ | 31 | ================================================ |
10 | 32 | 32 | ||
11 | 33 | juju deploy cs:trusty/elasticsearch | 33 | juju deploy cs:trusty/elasticsearch |
12 | 34 | juju add-unit elasticsearch | 34 | juju add-unit elasticsearch |
13 | @@ -41,7 +41,7 @@ | |||
14 | 41 | http://ip-of-kibana | 41 | http://ip-of-kibana |
15 | 42 | 42 | ||
16 | 43 | example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana | 43 | example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana |
18 | 44 | ============================================= | 44 | ========================================================= |
19 | 45 | 45 | ||
20 | 46 | juju deploy cs:trusty/elasticsearch | 46 | juju deploy cs:trusty/elasticsearch |
21 | 47 | juju add-unit elasticsearch | 47 | juju add-unit elasticsearch |
22 | @@ -54,7 +54,6 @@ | |||
23 | 54 | juju add-relation logstash-agent logstash-indexer:input | 54 | juju add-relation logstash-agent logstash-indexer:input |
24 | 55 | 55 | ||
25 | 56 | 56 | ||
26 | 57 | |||
27 | 58 | ### Caveats | 57 | ### Caveats |
28 | 59 | 58 | ||
29 | 60 | The charm will fetch the logstash complete archive every time. | 59 | The charm will fetch the logstash complete archive every time. |
30 | @@ -63,13 +62,13 @@ | |||
31 | 63 | 62 | ||
32 | 64 | # Configuration | 63 | # Configuration |
33 | 65 | 64 | ||
35 | 66 | The charm supports installation from anywhere that python requeusts can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options | 65 | The charm supports installation from anywhere that python requests can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options |
36 | 67 | 66 | ||
37 | 68 | juju set logstash logstash-source="https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz" logstash-sum="d59ef579c7614c5df9bd69cfdce20ed371f728ff" | 67 | juju set logstash logstash-source="https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz" logstash-sum="d59ef579c7614c5df9bd69cfdce20ed371f728ff" |
38 | 69 | 68 | ||
39 | 70 | There is also a configuration option to add arbitrary packages pre-installation of logstash. The format is a space separated list. | 69 | There is also a configuration option to add arbitrary packages pre-installation of logstash. The format is a space separated list. |
40 | 71 | 70 | ||
42 | 72 | juju set logstash extrapackages='vim byobu' | 71 | juju set logstash extra-packages='vim byobu' |
43 | 73 | 72 | ||
44 | 74 | # Contact Information | 73 | # Contact Information |
45 | 75 | 74 | ||
46 | 76 | 75 | ||
47 | === modified file 'charm-helpers.yaml' | |||
48 | --- charm-helpers.yaml 2014-09-23 12:11:39 +0000 | |||
49 | +++ charm-helpers.yaml 2015-11-01 20:32:57 +0000 | |||
50 | @@ -5,3 +5,4 @@ | |||
51 | 5 | - fetch | 5 | - fetch |
52 | 6 | - payload | 6 | - payload |
53 | 7 | - contrib.templating.jinja | 7 | - contrib.templating.jinja |
54 | 8 | - contrib.charmsupport | ||
55 | 8 | 9 | ||
56 | === modified file 'config.yaml' | |||
57 | --- config.yaml 2015-03-31 09:37:21 +0000 | |||
58 | +++ config.yaml 2015-11-01 20:32:57 +0000 | |||
59 | @@ -19,3 +19,35 @@ | |||
60 | 19 | default: "" | 19 | default: "" |
61 | 20 | type: string | 20 | type: string |
62 | 21 | description: "Base64-encoded SSL key" | 21 | description: "Base64-encoded SSL key" |
63 | 22 | extra-config: | ||
64 | 23 | type: string | ||
65 | 24 | default: '' | ||
66 | 25 | description: "Base64-encoded custom configuration content." | ||
67 | 26 | nagios_context: | ||
68 | 27 | default: "juju" | ||
69 | 28 | type: string | ||
70 | 29 | description: | | ||
71 | 30 | Used by the nrpe subordinate charms. | ||
72 | 31 | A string that will be prepended to instance name to set the host name | ||
73 | 32 | in nagios. So for instance the hostname would be something like: | ||
74 | 33 | juju-myservice-0 | ||
75 | 34 | If you're running multiple environments with the same services in them | ||
76 | 35 | this allows you to differentiate between them. | ||
77 | 36 | nagios_servicegroups: | ||
78 | 37 | default: "" | ||
79 | 38 | type: string | ||
80 | 39 | description: | | ||
81 | 40 | A comma-separated list of nagios servicegroups. | ||
82 | 41 | If left empty, the nagios_context will be used as the servicegroup | ||
83 | 42 | nagios_check_procs_params: | ||
84 | 43 | default: "-a /opt/logstash/lib/logstash/runner.rb -c 1:1" | ||
85 | 44 | type: string | ||
86 | 45 | description: The parameters to pass to the nrpe plugin check_procs. | ||
87 | 46 | nagios_check_tcp_params: | ||
88 | 47 | default: "--ssl -H localhost -p 5043 -c 0.3" | ||
89 | 48 | type: string | ||
90 | 49 | description: The parameters to pass to the nrpe plugin check_tcp. | ||
91 | 50 | nagios_check_cert_params: | ||
92 | 51 | default: "-D 30,14 -H 127.0.0.1 -p 5043" | ||
93 | 52 | type: string | ||
94 | 53 | description: The parameters to pass to the nrpe plugin "check_tcp --ssl" to check certificate expiration date. | ||
95 | 22 | 54 | ||
96 | === modified file 'hooks/client-relation-changed' | |||
97 | --- hooks/client-relation-changed 2014-09-23 16:59:13 +0000 | |||
98 | +++ hooks/client-relation-changed 2015-11-01 20:32:57 +0000 | |||
99 | @@ -29,29 +29,35 @@ | |||
100 | 29 | def write_config(): | 29 | def write_config(): |
101 | 30 | with open('host_cache', 'r') as f: | 30 | with open('host_cache', 'r') as f: |
102 | 31 | hosts = f.readlines() | 31 | hosts = f.readlines() |
103 | 32 | if not hosts: | ||
104 | 33 | sys.exit(0) | ||
105 | 32 | 34 | ||
107 | 33 | opts = {'hosts': hosts[0].rstrip()} | 35 | # Use last host in list as it will be the most recently added |
108 | 36 | # and first host in list may not exist anymore! TODO fix that. | ||
109 | 37 | opts = {'hosts': hosts[-1].rstrip()} | ||
110 | 34 | 38 | ||
111 | 35 | out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf') | 39 | out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf') |
112 | 36 | with open(out, 'w') as p: | 40 | with open(out, 'w') as p: |
115 | 37 | p.write(render(os.path.basename(out), opts)) | 41 | p.write(render(os.path.basename(out), opts)) |
114 | 38 | |||
116 | 39 | 42 | ||
117 | 40 | 43 | ||
118 | 41 | def cache_hosts(): | 44 | def cache_hosts(): |
124 | 42 | host = hookenv.relation_get('host') | 45 | rels = hookenv.relations_of_type("client") |
125 | 43 | if not host: | 46 | if not rels: |
126 | 44 | log('No host received. Assuming nothing to do.') | 47 | log('No client relations. Assuming nothing to do.') |
127 | 45 | sys.exit(0) | 48 | sys.exit(0) |
123 | 46 | |||
128 | 47 | if not os.path.exists('host_cache'): | 49 | if not os.path.exists('host_cache'): |
129 | 48 | open('host_cache', 'a').close() | 50 | open('host_cache', 'a').close() |
136 | 49 | 51 | for rel in rels: | |
137 | 50 | with open('host_cache', 'r') as f: | 52 | host = rel.get('host') |
138 | 51 | hosts = f.readlines() | 53 | if not host: |
139 | 52 | if not host in hosts: | 54 | log('No host received for relation: {}.'.format(rel)) |
140 | 53 | with open('host_cache', 'a') as f: | 55 | continue |
141 | 54 | f.write('{}\n'.format(host)) | 56 | with open('host_cache', 'r') as f: |
142 | 57 | hosts = f.readlines() | ||
143 | 58 | if host not in hosts: | ||
144 | 59 | with open('host_cache', 'a') as f: | ||
145 | 60 | f.write('{}\n'.format(host)) | ||
146 | 55 | 61 | ||
147 | 56 | 62 | ||
148 | 57 | if __name__ == "__main__": | 63 | if __name__ == "__main__": |
149 | 58 | 64 | ||
150 | === modified file 'hooks/config-changed' | |||
151 | --- hooks/config-changed 2015-04-07 00:25:13 +0000 | |||
152 | +++ hooks/config-changed 2015-11-01 20:32:57 +0000 | |||
153 | @@ -39,8 +39,10 @@ | |||
154 | 39 | # This only actually opens the port if we've exposed the service in juju | 39 | # This only actually opens the port if we've exposed the service in juju |
155 | 40 | hookenv.open_port(5043) | 40 | hookenv.open_port(5043) |
156 | 41 | 41 | ||
159 | 42 | # The install hook is idempotent, so re-run it. | 42 | # Restart the service when configuration has changed. |
160 | 43 | subprocess.check_output(shlex.split('hooks/install')) | 43 | subprocess.check_output(shlex.split('hooks/start')) |
161 | 44 | |||
162 | 45 | subprocess.check_output(shlex.split('hooks/nrpe-external-master-relation-changed')) | ||
163 | 44 | 46 | ||
164 | 45 | 47 | ||
165 | 46 | def copy_config(): | 48 | def copy_config(): |
166 | @@ -52,11 +54,21 @@ | |||
167 | 52 | key_file = os.path.join(cert_dir, 'logstash.key') | 54 | key_file = os.path.join(cert_dir, 'logstash.key') |
168 | 53 | 55 | ||
169 | 54 | for f in files: | 56 | for f in files: |
170 | 57 | # skip output-elasticsearch.conf, is managed by | ||
171 | 58 | # hooks/client-relation-changed | ||
172 | 59 | if os.path.basename(f) == "output-elasticsearch.conf": | ||
173 | 60 | continue | ||
174 | 55 | if os.path.basename(f) != lumberjack_template: | 61 | if os.path.basename(f) != lumberjack_template: |
175 | 56 | with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p: | 62 | with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p: |
176 | 57 | p.write(render(os.path.basename(f), opts)) | 63 | p.write(render(os.path.basename(f), opts)) |
177 | 58 | 64 | ||
178 | 59 | config_data = hookenv.config() | 65 | config_data = hookenv.config() |
179 | 66 | |||
180 | 67 | # Write custom configuration if set. | ||
181 | 68 | if config_data['extra-config']: | ||
182 | 69 | with open(os.path.join(BASEPATH, 'conf.d', 'extra.conf'), 'w') as f: | ||
183 | 70 | f.write(str(base64.b64decode(config_data['extra-config']))) | ||
184 | 71 | |||
185 | 60 | # Only setup lumberjack protocol if ssl cert and key are configured | 72 | # Only setup lumberjack protocol if ssl cert and key are configured |
186 | 61 | if config_data['ssl_cert'] and config_data['ssl_key']: | 73 | if config_data['ssl_cert'] and config_data['ssl_key']: |
187 | 62 | if not os.path.exists(cert_dir): | 74 | if not os.path.exists(cert_dir): |
188 | 63 | 75 | ||
189 | === modified file 'hooks/lumberjack-relation-changed' (properties changed: -x to +x) | |||
190 | === added file 'hooks/nrpe-external-master-relation-changed' | |||
191 | --- hooks/nrpe-external-master-relation-changed 1970-01-01 00:00:00 +0000 | |||
192 | +++ hooks/nrpe-external-master-relation-changed 2015-11-01 20:32:57 +0000 | |||
193 | @@ -0,0 +1,87 @@ | |||
194 | 1 | #!/usr/bin/python | ||
195 | 2 | |||
196 | 3 | import os | ||
197 | 4 | import sys | ||
198 | 5 | |||
199 | 6 | sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib')) | ||
200 | 7 | |||
201 | 8 | from charmhelpers.core import hookenv | ||
202 | 9 | from charmhelpers.contrib.charmsupport import nrpe | ||
203 | 10 | from charmhelpers.contrib.charmsupport.nrpe import NRPE | ||
204 | 11 | |||
205 | 12 | hooks = hookenv.Hooks() | ||
206 | 13 | log = hookenv.log | ||
207 | 14 | |||
208 | 15 | |||
209 | 16 | class CustomIntervalCheck(nrpe.Check): | ||
210 | 17 | |||
211 | 18 | service_template = (""" | ||
212 | 19 | #--------------------------------------------------- | ||
213 | 20 | # This file is Juju managed | ||
214 | 21 | #--------------------------------------------------- | ||
215 | 22 | define service {{ | ||
216 | 23 | use active-service | ||
217 | 24 | host_name {nagios_hostname} | ||
218 | 25 | service_description {nagios_hostname}[{shortname}] """ | ||
219 | 26 | """{description} | ||
220 | 27 | check_command check_nrpe!{command} | ||
221 | 28 | servicegroups {nagios_servicegroup} | ||
222 | 29 | %s | ||
223 | 30 | }} | ||
224 | 31 | """) | ||
225 | 32 | intervals_template = " {} {}\n" | ||
226 | 33 | |||
227 | 34 | def __init__(self, shortname, description, check_cmd, normal_check_interval=None, | ||
228 | 35 | retry_check_interval=None, notification_interval=None): | ||
229 | 36 | super(CustomIntervalCheck, self).__init__(shortname, description, check_cmd) | ||
230 | 37 | intervals = {} | ||
231 | 38 | if normal_check_interval: | ||
232 | 39 | intervals['normal_check_interval'] = normal_check_interval | ||
233 | 40 | if retry_check_interval: | ||
234 | 41 | intervals['retry_check_interval'] = retry_check_interval | ||
235 | 42 | if notification_interval: | ||
236 | 43 | intervals['notification_interval'] = notification_interval | ||
237 | 44 | intervals_config = "" | ||
238 | 45 | for k, v in intervals.items(): | ||
239 | 46 | intervals_config += self.intervals_template.format(k, v) | ||
240 | 47 | self.service_template = CustomIntervalCheck.service_template % intervals_config | ||
241 | 48 | |||
242 | 49 | |||
243 | 50 | @hooks.hook('nrpe-external-master-relation-changed') | ||
244 | 51 | def update_nrpe_checks(): | ||
245 | 52 | nrpe_compat = nrpe.NRPE() | ||
246 | 53 | conf = nrpe_compat.config | ||
247 | 54 | check_procs_params = conf.get('nagios_check_procs_params') | ||
248 | 55 | if check_procs_params: | ||
249 | 56 | nrpe_compat.add_check( | ||
250 | 57 | shortname='logstash_process', | ||
251 | 58 | description='Check logstash java process running', | ||
252 | 59 | check_cmd='check_procs %s' % check_procs_params | ||
253 | 60 | ) | ||
254 | 61 | check_tcp_params = conf.get('nagios_check_tcp_params') | ||
255 | 62 | check_cert_params = conf.get('nagios_check_cert_params') | ||
256 | 63 | config_data = hookenv.config() | ||
257 | 64 | # Only setup lumberjack protocol if ssl cert and key are configured | ||
258 | 65 | if config_data['ssl_cert'] and config_data['ssl_key']: | ||
259 | 66 | if check_tcp_params: | ||
260 | 67 | nrpe_compat.add_check( | ||
261 | 68 | shortname='lumberjack_tcp', | ||
262 | 69 | description='Check logstash lumberjack input tcp port', | ||
263 | 70 | check_cmd='check_tcp %s' % check_tcp_params | ||
264 | 71 | ) | ||
265 | 72 | if check_cert_params: | ||
266 | 73 | # check certificate expiry date, daily and retry every 2 hs | ||
267 | 74 | cert_check = CustomIntervalCheck( | ||
268 | 75 | shortname='lumberjack_ssl_check', | ||
269 | 76 | description='Check logstash ssl certificate expiry date', | ||
270 | 77 | check_cmd='check_tcp --ssl {}'.format(check_cert_params), | ||
271 | 78 | normal_check_interval=1440, # minutes | ||
272 | 79 | retry_check_interval=120, # minutes | ||
273 | 80 | ) | ||
274 | 81 | nrpe_compat.checks.append(cert_check) | ||
275 | 82 | |||
276 | 83 | nrpe_compat.write() | ||
277 | 84 | |||
278 | 85 | if __name__ == "__main__": | ||
279 | 86 | # execute a hook based on the name the program is called by | ||
280 | 87 | hooks.execute(sys.argv) | ||
281 | 0 | 88 | ||
282 | === added directory 'lib/charmhelpers/contrib/charmsupport' | |||
283 | === added file 'lib/charmhelpers/contrib/charmsupport/__init__.py' | |||
284 | --- lib/charmhelpers/contrib/charmsupport/__init__.py 1970-01-01 00:00:00 +0000 | |||
285 | +++ lib/charmhelpers/contrib/charmsupport/__init__.py 2015-11-01 20:32:57 +0000 | |||
286 | @@ -0,0 +1,15 @@ | |||
287 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
288 | 2 | # | ||
289 | 3 | # This file is part of charm-helpers. | ||
290 | 4 | # | ||
291 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
292 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
293 | 7 | # published by the Free Software Foundation. | ||
294 | 8 | # | ||
295 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
296 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
297 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
298 | 12 | # GNU Lesser General Public License for more details. | ||
299 | 13 | # | ||
300 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
301 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
302 | 0 | 16 | ||
303 | === added file 'lib/charmhelpers/contrib/charmsupport/nrpe.py' | |||
304 | --- lib/charmhelpers/contrib/charmsupport/nrpe.py 1970-01-01 00:00:00 +0000 | |||
305 | +++ lib/charmhelpers/contrib/charmsupport/nrpe.py 2015-11-01 20:32:57 +0000 | |||
306 | @@ -0,0 +1,360 @@ | |||
307 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
308 | 2 | # | ||
309 | 3 | # This file is part of charm-helpers. | ||
310 | 4 | # | ||
311 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
312 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
313 | 7 | # published by the Free Software Foundation. | ||
314 | 8 | # | ||
315 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
316 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
317 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
318 | 12 | # GNU Lesser General Public License for more details. | ||
319 | 13 | # | ||
320 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
321 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
322 | 16 | |||
323 | 17 | """Compatibility with the nrpe-external-master charm""" | ||
324 | 18 | # Copyright 2012 Canonical Ltd. | ||
325 | 19 | # | ||
326 | 20 | # Authors: | ||
327 | 21 | # Matthew Wedgwood <matthew.wedgwood@canonical.com> | ||
328 | 22 | |||
329 | 23 | import subprocess | ||
330 | 24 | import pwd | ||
331 | 25 | import grp | ||
332 | 26 | import os | ||
333 | 27 | import glob | ||
334 | 28 | import shutil | ||
335 | 29 | import re | ||
336 | 30 | import shlex | ||
337 | 31 | import yaml | ||
338 | 32 | |||
339 | 33 | from charmhelpers.core.hookenv import ( | ||
340 | 34 | config, | ||
341 | 35 | local_unit, | ||
342 | 36 | log, | ||
343 | 37 | relation_ids, | ||
344 | 38 | relation_set, | ||
345 | 39 | relations_of_type, | ||
346 | 40 | ) | ||
347 | 41 | |||
348 | 42 | from charmhelpers.core.host import service | ||
349 | 43 | |||
350 | 44 | # This module adds compatibility with the nrpe-external-master and plain nrpe | ||
351 | 45 | # subordinate charms. To use it in your charm: | ||
352 | 46 | # | ||
353 | 47 | # 1. Update metadata.yaml | ||
354 | 48 | # | ||
355 | 49 | # provides: | ||
356 | 50 | # (...) | ||
357 | 51 | # nrpe-external-master: | ||
358 | 52 | # interface: nrpe-external-master | ||
359 | 53 | # scope: container | ||
360 | 54 | # | ||
361 | 55 | # and/or | ||
362 | 56 | # | ||
363 | 57 | # provides: | ||
364 | 58 | # (...) | ||
365 | 59 | # local-monitors: | ||
366 | 60 | # interface: local-monitors | ||
367 | 61 | # scope: container | ||
368 | 62 | |||
369 | 63 | # | ||
370 | 64 | # 2. Add the following to config.yaml | ||
371 | 65 | # | ||
372 | 66 | # nagios_context: | ||
373 | 67 | # default: "juju" | ||
374 | 68 | # type: string | ||
375 | 69 | # description: | | ||
376 | 70 | # Used by the nrpe subordinate charms. | ||
377 | 71 | # A string that will be prepended to instance name to set the host name | ||
378 | 72 | # in nagios. So for instance the hostname would be something like: | ||
379 | 73 | # juju-myservice-0 | ||
380 | 74 | # If you're running multiple environments with the same services in them | ||
381 | 75 | # this allows you to differentiate between them. | ||
382 | 76 | # nagios_servicegroups: | ||
383 | 77 | # default: "" | ||
384 | 78 | # type: string | ||
385 | 79 | # description: | | ||
386 | 80 | # A comma-separated list of nagios servicegroups. | ||
387 | 81 | # If left empty, the nagios_context will be used as the servicegroup | ||
388 | 82 | # | ||
389 | 83 | # 3. Add custom checks (Nagios plugins) to files/nrpe-external-master | ||
390 | 84 | # | ||
391 | 85 | # 4. Update your hooks.py with something like this: | ||
392 | 86 | # | ||
393 | 87 | # from charmsupport.nrpe import NRPE | ||
394 | 88 | # (...) | ||
395 | 89 | # def update_nrpe_config(): | ||
396 | 90 | # nrpe_compat = NRPE() | ||
397 | 91 | # nrpe_compat.add_check( | ||
398 | 92 | # shortname = "myservice", | ||
399 | 93 | # description = "Check MyService", | ||
400 | 94 | # check_cmd = "check_http -w 2 -c 10 http://localhost" | ||
401 | 95 | # ) | ||
402 | 96 | # nrpe_compat.add_check( | ||
403 | 97 | # "myservice_other", | ||
404 | 98 | # "Check for widget failures", | ||
405 | 99 | # check_cmd = "/srv/myapp/scripts/widget_check" | ||
406 | 100 | # ) | ||
407 | 101 | # nrpe_compat.write() | ||
408 | 102 | # | ||
409 | 103 | # def config_changed(): | ||
410 | 104 | # (...) | ||
411 | 105 | # update_nrpe_config() | ||
412 | 106 | # | ||
413 | 107 | # def nrpe_external_master_relation_changed(): | ||
414 | 108 | # update_nrpe_config() | ||
415 | 109 | # | ||
416 | 110 | # def local_monitors_relation_changed(): | ||
417 | 111 | # update_nrpe_config() | ||
418 | 112 | # | ||
419 | 113 | # 5. ln -s hooks.py nrpe-external-master-relation-changed | ||
420 | 114 | # ln -s hooks.py local-monitors-relation-changed | ||
421 | 115 | |||
422 | 116 | |||
423 | 117 | class CheckException(Exception): | ||
424 | 118 | pass | ||
425 | 119 | |||
426 | 120 | |||
427 | 121 | class Check(object): | ||
428 | 122 | shortname_re = '[A-Za-z0-9-_]+$' | ||
429 | 123 | service_template = (""" | ||
430 | 124 | #--------------------------------------------------- | ||
431 | 125 | # This file is Juju managed | ||
432 | 126 | #--------------------------------------------------- | ||
433 | 127 | define service {{ | ||
434 | 128 | use active-service | ||
435 | 129 | host_name {nagios_hostname} | ||
436 | 130 | service_description {nagios_hostname}[{shortname}] """ | ||
437 | 131 | """{description} | ||
438 | 132 | check_command check_nrpe!{command} | ||
439 | 133 | servicegroups {nagios_servicegroup} | ||
440 | 134 | }} | ||
441 | 135 | """) | ||
442 | 136 | |||
443 | 137 | def __init__(self, shortname, description, check_cmd): | ||
444 | 138 | super(Check, self).__init__() | ||
445 | 139 | # XXX: could be better to calculate this from the service name | ||
446 | 140 | if not re.match(self.shortname_re, shortname): | ||
447 | 141 | raise CheckException("shortname must match {}".format( | ||
448 | 142 | Check.shortname_re)) | ||
449 | 143 | self.shortname = shortname | ||
450 | 144 | self.command = "check_{}".format(shortname) | ||
451 | 145 | # Note: a set of invalid characters is defined by the | ||
452 | 146 | # Nagios server config | ||
453 | 147 | # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()= | ||
454 | 148 | self.description = description | ||
455 | 149 | self.check_cmd = self._locate_cmd(check_cmd) | ||
456 | 150 | |||
457 | 151 | def _locate_cmd(self, check_cmd): | ||
458 | 152 | search_path = ( | ||
459 | 153 | '/usr/lib/nagios/plugins', | ||
460 | 154 | '/usr/local/lib/nagios/plugins', | ||
461 | 155 | ) | ||
462 | 156 | parts = shlex.split(check_cmd) | ||
463 | 157 | for path in search_path: | ||
464 | 158 | if os.path.exists(os.path.join(path, parts[0])): | ||
465 | 159 | command = os.path.join(path, parts[0]) | ||
466 | 160 | if len(parts) > 1: | ||
467 | 161 | command += " " + " ".join(parts[1:]) | ||
468 | 162 | return command | ||
469 | 163 | log('Check command not found: {}'.format(parts[0])) | ||
470 | 164 | return '' | ||
471 | 165 | |||
472 | 166 | def write(self, nagios_context, hostname, nagios_servicegroups): | ||
473 | 167 | nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format( | ||
474 | 168 | self.command) | ||
475 | 169 | with open(nrpe_check_file, 'w') as nrpe_check_config: | ||
476 | 170 | nrpe_check_config.write("# check {}\n".format(self.shortname)) | ||
477 | 171 | nrpe_check_config.write("command[{}]={}\n".format( | ||
478 | 172 | self.command, self.check_cmd)) | ||
479 | 173 | |||
480 | 174 | if not os.path.exists(NRPE.nagios_exportdir): | ||
481 | 175 | log('Not writing service config as {} is not accessible'.format( | ||
482 | 176 | NRPE.nagios_exportdir)) | ||
483 | 177 | else: | ||
484 | 178 | self.write_service_config(nagios_context, hostname, | ||
485 | 179 | nagios_servicegroups) | ||
486 | 180 | |||
487 | 181 | def write_service_config(self, nagios_context, hostname, | ||
488 | 182 | nagios_servicegroups): | ||
489 | 183 | for f in os.listdir(NRPE.nagios_exportdir): | ||
490 | 184 | if re.search('.*{}.cfg'.format(self.command), f): | ||
491 | 185 | os.remove(os.path.join(NRPE.nagios_exportdir, f)) | ||
492 | 186 | |||
493 | 187 | templ_vars = { | ||
494 | 188 | 'nagios_hostname': hostname, | ||
495 | 189 | 'nagios_servicegroup': nagios_servicegroups, | ||
496 | 190 | 'description': self.description, | ||
497 | 191 | 'shortname': self.shortname, | ||
498 | 192 | 'command': self.command, | ||
499 | 193 | } | ||
500 | 194 | nrpe_service_text = Check.service_template.format(**templ_vars) | ||
501 | 195 | nrpe_service_file = '{}/service__{}_{}.cfg'.format( | ||
502 | 196 | NRPE.nagios_exportdir, hostname, self.command) | ||
503 | 197 | with open(nrpe_service_file, 'w') as nrpe_service_config: | ||
504 | 198 | nrpe_service_config.write(str(nrpe_service_text)) | ||
505 | 199 | |||
506 | 200 | def run(self): | ||
507 | 201 | subprocess.call(self.check_cmd) | ||
508 | 202 | |||
509 | 203 | |||
510 | 204 | class NRPE(object): | ||
511 | 205 | nagios_logdir = '/var/log/nagios' | ||
512 | 206 | nagios_exportdir = '/var/lib/nagios/export' | ||
513 | 207 | nrpe_confdir = '/etc/nagios/nrpe.d' | ||
514 | 208 | |||
515 | 209 | def __init__(self, hostname=None): | ||
516 | 210 | super(NRPE, self).__init__() | ||
517 | 211 | self.config = config() | ||
518 | 212 | self.nagios_context = self.config['nagios_context'] | ||
519 | 213 | if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']: | ||
520 | 214 | self.nagios_servicegroups = self.config['nagios_servicegroups'] | ||
521 | 215 | else: | ||
522 | 216 | self.nagios_servicegroups = self.nagios_context | ||
523 | 217 | self.unit_name = local_unit().replace('/', '-') | ||
524 | 218 | if hostname: | ||
525 | 219 | self.hostname = hostname | ||
526 | 220 | else: | ||
527 | 221 | self.hostname = "{}-{}".format(self.nagios_context, self.unit_name) | ||
528 | 222 | self.checks = [] | ||
529 | 223 | |||
530 | 224 | def add_check(self, *args, **kwargs): | ||
531 | 225 | self.checks.append(Check(*args, **kwargs)) | ||
532 | 226 | |||
533 | 227 | def write(self): | ||
534 | 228 | try: | ||
535 | 229 | nagios_uid = pwd.getpwnam('nagios').pw_uid | ||
536 | 230 | nagios_gid = grp.getgrnam('nagios').gr_gid | ||
537 | 231 | except: | ||
538 | 232 | log("Nagios user not set up, nrpe checks not updated") | ||
539 | 233 | return | ||
540 | 234 | |||
541 | 235 | if not os.path.exists(NRPE.nagios_logdir): | ||
542 | 236 | os.mkdir(NRPE.nagios_logdir) | ||
543 | 237 | os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid) | ||
544 | 238 | |||
545 | 239 | nrpe_monitors = {} | ||
546 | 240 | monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}} | ||
547 | 241 | for nrpecheck in self.checks: | ||
548 | 242 | nrpecheck.write(self.nagios_context, self.hostname, | ||
549 | 243 | self.nagios_servicegroups) | ||
550 | 244 | nrpe_monitors[nrpecheck.shortname] = { | ||
551 | 245 | "command": nrpecheck.command, | ||
552 | 246 | } | ||
553 | 247 | |||
554 | 248 | service('restart', 'nagios-nrpe-server') | ||
555 | 249 | |||
556 | 250 | monitor_ids = relation_ids("local-monitors") + \ | ||
557 | 251 | relation_ids("nrpe-external-master") | ||
558 | 252 | for rid in monitor_ids: | ||
559 | 253 | relation_set(relation_id=rid, monitors=yaml.dump(monitors)) | ||
560 | 254 | |||
561 | 255 | |||
562 | 256 | def get_nagios_hostcontext(relation_name='nrpe-external-master'): | ||
563 | 257 | """ | ||
564 | 258 | Query relation with nrpe subordinate, return the nagios_host_context | ||
565 | 259 | |||
566 | 260 | :param str relation_name: Name of relation nrpe sub joined to | ||
567 | 261 | """ | ||
568 | 262 | for rel in relations_of_type(relation_name): | ||
569 | 263 | if 'nagios_hostname' in rel: | ||
570 | 264 | return rel['nagios_host_context'] | ||
571 | 265 | |||
572 | 266 | |||
573 | 267 | def get_nagios_hostname(relation_name='nrpe-external-master'): | ||
574 | 268 | """ | ||
575 | 269 | Query relation with nrpe subordinate, return the nagios_hostname | ||
576 | 270 | |||
577 | 271 | :param str relation_name: Name of relation nrpe sub joined to | ||
578 | 272 | """ | ||
579 | 273 | for rel in relations_of_type(relation_name): | ||
580 | 274 | if 'nagios_hostname' in rel: | ||
581 | 275 | return rel['nagios_hostname'] | ||
582 | 276 | |||
583 | 277 | |||
584 | 278 | def get_nagios_unit_name(relation_name='nrpe-external-master'): | ||
585 | 279 | """ | ||
586 | 280 | Return the nagios unit name prepended with host_context if needed | ||
587 | 281 | |||
588 | 282 | :param str relation_name: Name of relation nrpe sub joined to | ||
589 | 283 | """ | ||
590 | 284 | host_context = get_nagios_hostcontext(relation_name) | ||
591 | 285 | if host_context: | ||
592 | 286 | unit = "%s:%s" % (host_context, local_unit()) | ||
593 | 287 | else: | ||
594 | 288 | unit = local_unit() | ||
595 | 289 | return unit | ||
596 | 290 | |||
597 | 291 | |||
598 | 292 | def add_init_service_checks(nrpe, services, unit_name): | ||
599 | 293 | """ | ||
600 | 294 | Add checks for each service in list | ||
601 | 295 | |||
602 | 296 | :param NRPE nrpe: NRPE object to add check to | ||
603 | 297 | :param list services: List of services to check | ||
604 | 298 | :param str unit_name: Unit name to use in check description | ||
605 | 299 | """ | ||
606 | 300 | for svc in services: | ||
607 | 301 | upstart_init = '/etc/init/%s.conf' % svc | ||
608 | 302 | sysv_init = '/etc/init.d/%s' % svc | ||
609 | 303 | if os.path.exists(upstart_init): | ||
610 | 304 | nrpe.add_check( | ||
611 | 305 | shortname=svc, | ||
612 | 306 | description='process check {%s}' % unit_name, | ||
613 | 307 | check_cmd='check_upstart_job %s' % svc | ||
614 | 308 | ) | ||
615 | 309 | elif os.path.exists(sysv_init): | ||
616 | 310 | cronpath = '/etc/cron.d/nagios-service-check-%s' % svc | ||
617 | 311 | cron_file = ('*/5 * * * * root ' | ||
618 | 312 | '/usr/local/lib/nagios/plugins/check_exit_status.pl ' | ||
619 | 313 | '-s /etc/init.d/%s status > ' | ||
620 | 314 | '/var/lib/nagios/service-check-%s.txt\n' % (svc, | ||
621 | 315 | svc) | ||
622 | 316 | ) | ||
623 | 317 | f = open(cronpath, 'w') | ||
624 | 318 | f.write(cron_file) | ||
625 | 319 | f.close() | ||
626 | 320 | nrpe.add_check( | ||
627 | 321 | shortname=svc, | ||
628 | 322 | description='process check {%s}' % unit_name, | ||
629 | 323 | check_cmd='check_status_file.py -f ' | ||
630 | 324 | '/var/lib/nagios/service-check-%s.txt' % svc, | ||
631 | 325 | ) | ||
632 | 326 | |||
633 | 327 | |||
634 | 328 | def copy_nrpe_checks(): | ||
635 | 329 | """ | ||
636 | 330 | Copy the nrpe checks into place | ||
637 | 331 | |||
638 | 332 | """ | ||
639 | 333 | NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins' | ||
640 | 334 | nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks', | ||
641 | 335 | 'charmhelpers', 'contrib', 'openstack', | ||
642 | 336 | 'files') | ||
643 | 337 | |||
644 | 338 | if not os.path.exists(NAGIOS_PLUGINS): | ||
645 | 339 | os.makedirs(NAGIOS_PLUGINS) | ||
646 | 340 | for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")): | ||
647 | 341 | if os.path.isfile(fname): | ||
648 | 342 | shutil.copy2(fname, | ||
649 | 343 | os.path.join(NAGIOS_PLUGINS, os.path.basename(fname))) | ||
650 | 344 | |||
651 | 345 | |||
652 | 346 | def add_haproxy_checks(nrpe, unit_name): | ||
653 | 347 | """ | ||
654 | 348 | Add checks for each service in list | ||
655 | 349 | |||
656 | 350 | :param NRPE nrpe: NRPE object to add check to | ||
657 | 351 | :param str unit_name: Unit name to use in check description | ||
658 | 352 | """ | ||
659 | 353 | nrpe.add_check( | ||
660 | 354 | shortname='haproxy_servers', | ||
661 | 355 | description='Check HAProxy {%s}' % unit_name, | ||
662 | 356 | check_cmd='check_haproxy.sh') | ||
663 | 357 | nrpe.add_check( | ||
664 | 358 | shortname='haproxy_queue', | ||
665 | 359 | description='Check HAProxy queue depth {%s}' % unit_name, | ||
666 | 360 | check_cmd='check_haproxy_queue_depth.sh') | ||
667 | 0 | 361 | ||
668 | === added file 'lib/charmhelpers/contrib/charmsupport/volumes.py' | |||
669 | --- lib/charmhelpers/contrib/charmsupport/volumes.py 1970-01-01 00:00:00 +0000 | |||
670 | +++ lib/charmhelpers/contrib/charmsupport/volumes.py 2015-11-01 20:32:57 +0000 | |||
671 | @@ -0,0 +1,175 @@ | |||
672 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
673 | 2 | # | ||
674 | 3 | # This file is part of charm-helpers. | ||
675 | 4 | # | ||
676 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
677 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
678 | 7 | # published by the Free Software Foundation. | ||
679 | 8 | # | ||
680 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
681 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
682 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
683 | 12 | # GNU Lesser General Public License for more details. | ||
684 | 13 | # | ||
685 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
686 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
687 | 16 | |||
688 | 17 | ''' | ||
689 | 18 | Functions for managing volumes in juju units. One volume is supported per unit. | ||
690 | 19 | Subordinates may have their own storage, provided it is on its own partition. | ||
691 | 20 | |||
692 | 21 | Configuration stanzas:: | ||
693 | 22 | |||
694 | 23 | volume-ephemeral: | ||
695 | 24 | type: boolean | ||
696 | 25 | default: true | ||
697 | 26 | description: > | ||
698 | 27 | If false, a volume is mounted as sepecified in "volume-map" | ||
699 | 28 | If true, ephemeral storage will be used, meaning that log data | ||
700 | 29 | will only exist as long as the machine. YOU HAVE BEEN WARNED. | ||
701 | 30 | volume-map: | ||
702 | 31 | type: string | ||
703 | 32 | default: {} | ||
704 | 33 | description: > | ||
705 | 34 | YAML map of units to device names, e.g: | ||
706 | 35 | "{ rsyslog/0: /dev/vdb, rsyslog/1: /dev/vdb }" | ||
707 | 36 | Service units will raise a configure-error if volume-ephemeral | ||
708 | 37 | is 'true' and no volume-map value is set. Use 'juju set' to set a | ||
709 | 38 | value and 'juju resolved' to complete configuration. | ||
710 | 39 | |||
711 | 40 | Usage:: | ||
712 | 41 | |||
713 | 42 | from charmsupport.volumes import configure_volume, VolumeConfigurationError | ||
714 | 43 | from charmsupport.hookenv import log, ERROR | ||
715 | 44 | def post_mount_hook(): | ||
716 | 45 | stop_service('myservice') | ||
717 | 46 | def post_mount_hook(): | ||
718 | 47 | start_service('myservice') | ||
719 | 48 | |||
720 | 49 | if __name__ == '__main__': | ||
721 | 50 | try: | ||
722 | 51 | configure_volume(before_change=pre_mount_hook, | ||
723 | 52 | after_change=post_mount_hook) | ||
724 | 53 | except VolumeConfigurationError: | ||
725 | 54 | log('Storage could not be configured', ERROR) | ||
726 | 55 | |||
727 | 56 | ''' | ||
728 | 57 | |||
729 | 58 | # XXX: Known limitations | ||
730 | 59 | # - fstab is neither consulted nor updated | ||
731 | 60 | |||
732 | 61 | import os | ||
733 | 62 | from charmhelpers.core import hookenv | ||
734 | 63 | from charmhelpers.core import host | ||
735 | 64 | import yaml | ||
736 | 65 | |||
737 | 66 | |||
738 | 67 | MOUNT_BASE = '/srv/juju/volumes' | ||
739 | 68 | |||
740 | 69 | |||
741 | 70 | class VolumeConfigurationError(Exception): | ||
742 | 71 | '''Volume configuration data is missing or invalid''' | ||
743 | 72 | pass | ||
744 | 73 | |||
745 | 74 | |||
746 | 75 | def get_config(): | ||
747 | 76 | '''Gather and sanity-check volume configuration data''' | ||
748 | 77 | volume_config = {} | ||
749 | 78 | config = hookenv.config() | ||
750 | 79 | |||
751 | 80 | errors = False | ||
752 | 81 | |||
753 | 82 | if config.get('volume-ephemeral') in (True, 'True', 'true', 'Yes', 'yes'): | ||
754 | 83 | volume_config['ephemeral'] = True | ||
755 | 84 | else: | ||
756 | 85 | volume_config['ephemeral'] = False | ||
757 | 86 | |||
758 | 87 | try: | ||
759 | 88 | volume_map = yaml.safe_load(config.get('volume-map', '{}')) | ||
760 | 89 | except yaml.YAMLError as e: | ||
761 | 90 | hookenv.log("Error parsing YAML volume-map: {}".format(e), | ||
762 | 91 | hookenv.ERROR) | ||
763 | 92 | errors = True | ||
764 | 93 | if volume_map is None: | ||
765 | 94 | # probably an empty string | ||
766 | 95 | volume_map = {} | ||
767 | 96 | elif not isinstance(volume_map, dict): | ||
768 | 97 | hookenv.log("Volume-map should be a dictionary, not {}".format( | ||
769 | 98 | type(volume_map))) | ||
770 | 99 | errors = True | ||
771 | 100 | |||
772 | 101 | volume_config['device'] = volume_map.get(os.environ['JUJU_UNIT_NAME']) | ||
773 | 102 | if volume_config['device'] and volume_config['ephemeral']: | ||
774 | 103 | # asked for ephemeral storage but also defined a volume ID | ||
775 | 104 | hookenv.log('A volume is defined for this unit, but ephemeral ' | ||
776 | 105 | 'storage was requested', hookenv.ERROR) | ||
777 | 106 | errors = True | ||
778 | 107 | elif not volume_config['device'] and not volume_config['ephemeral']: | ||
779 | 108 | # asked for permanent storage but did not define volume ID | ||
780 | 109 | hookenv.log('Ephemeral storage was requested, but there is no volume ' | ||
781 | 110 | 'defined for this unit.', hookenv.ERROR) | ||
782 | 111 | errors = True | ||
783 | 112 | |||
784 | 113 | unit_mount_name = hookenv.local_unit().replace('/', '-') | ||
785 | 114 | volume_config['mountpoint'] = os.path.join(MOUNT_BASE, unit_mount_name) | ||
786 | 115 | |||
787 | 116 | if errors: | ||
788 | 117 | return None | ||
789 | 118 | return volume_config | ||
790 | 119 | |||
791 | 120 | |||
792 | 121 | def mount_volume(config): | ||
793 | 122 | if os.path.exists(config['mountpoint']): | ||
794 | 123 | if not os.path.isdir(config['mountpoint']): | ||
795 | 124 | hookenv.log('Not a directory: {}'.format(config['mountpoint'])) | ||
796 | 125 | raise VolumeConfigurationError() | ||
797 | 126 | else: | ||
798 | 127 | host.mkdir(config['mountpoint']) | ||
799 | 128 | if os.path.ismount(config['mountpoint']): | ||
800 | 129 | unmount_volume(config) | ||
801 | 130 | if not host.mount(config['device'], config['mountpoint'], persist=True): | ||
802 | 131 | raise VolumeConfigurationError() | ||
803 | 132 | |||
804 | 133 | |||
805 | 134 | def unmount_volume(config): | ||
806 | 135 | if os.path.ismount(config['mountpoint']): | ||
807 | 136 | if not host.umount(config['mountpoint'], persist=True): | ||
808 | 137 | raise VolumeConfigurationError() | ||
809 | 138 | |||
810 | 139 | |||
811 | 140 | def managed_mounts(): | ||
812 | 141 | '''List of all mounted managed volumes''' | ||
813 | 142 | return filter(lambda mount: mount[0].startswith(MOUNT_BASE), host.mounts()) | ||
814 | 143 | |||
815 | 144 | |||
816 | 145 | def configure_volume(before_change=lambda: None, after_change=lambda: None): | ||
817 | 146 | '''Set up storage (or don't) according to the charm's volume configuration. | ||
818 | 147 | Returns the mount point or "ephemeral". before_change and after_change | ||
819 | 148 | are optional functions to be called if the volume configuration changes. | ||
820 | 149 | ''' | ||
821 | 150 | |||
822 | 151 | config = get_config() | ||
823 | 152 | if not config: | ||
824 | 153 | hookenv.log('Failed to read volume configuration', hookenv.CRITICAL) | ||
825 | 154 | raise VolumeConfigurationError() | ||
826 | 155 | |||
827 | 156 | if config['ephemeral']: | ||
828 | 157 | if os.path.ismount(config['mountpoint']): | ||
829 | 158 | before_change() | ||
830 | 159 | unmount_volume(config) | ||
831 | 160 | after_change() | ||
832 | 161 | return 'ephemeral' | ||
833 | 162 | else: | ||
834 | 163 | # persistent storage | ||
835 | 164 | if os.path.ismount(config['mountpoint']): | ||
836 | 165 | mounts = dict(managed_mounts()) | ||
837 | 166 | if mounts.get(config['mountpoint']) != config['device']: | ||
838 | 167 | before_change() | ||
839 | 168 | unmount_volume(config) | ||
840 | 169 | mount_volume(config) | ||
841 | 170 | after_change() | ||
842 | 171 | else: | ||
843 | 172 | before_change() | ||
844 | 173 | mount_volume(config) | ||
845 | 174 | after_change() | ||
846 | 175 | return config['mountpoint'] | ||
847 | 0 | 176 | ||
848 | === modified file 'lib/charmhelpers/contrib/templating/__init__.py' | |||
849 | --- lib/charmhelpers/contrib/templating/__init__.py 2014-09-23 12:09:14 +0000 | |||
850 | +++ lib/charmhelpers/contrib/templating/__init__.py 2015-11-01 20:32:57 +0000 | |||
851 | @@ -0,0 +1,15 @@ | |||
852 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
853 | 2 | # | ||
854 | 3 | # This file is part of charm-helpers. | ||
855 | 4 | # | ||
856 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
857 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
858 | 7 | # published by the Free Software Foundation. | ||
859 | 8 | # | ||
860 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
861 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
862 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
863 | 12 | # GNU Lesser General Public License for more details. | ||
864 | 13 | # | ||
865 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
866 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
867 | 0 | 16 | ||
868 | === modified file 'lib/charmhelpers/contrib/templating/jinja.py' | |||
869 | --- lib/charmhelpers/contrib/templating/jinja.py 2014-09-23 12:09:14 +0000 | |||
870 | +++ lib/charmhelpers/contrib/templating/jinja.py 2015-11-01 20:32:57 +0000 | |||
871 | @@ -1,21 +1,37 @@ | |||
872 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
873 | 2 | # | ||
874 | 3 | # This file is part of charm-helpers. | ||
875 | 4 | # | ||
876 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
877 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
878 | 7 | # published by the Free Software Foundation. | ||
879 | 8 | # | ||
880 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
881 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
882 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
883 | 12 | # GNU Lesser General Public License for more details. | ||
884 | 13 | # | ||
885 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
886 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
887 | 16 | |||
888 | 1 | """ | 17 | """ |
889 | 2 | Templating using the python-jinja2 package. | 18 | Templating using the python-jinja2 package. |
890 | 3 | """ | 19 | """ |
899 | 4 | from charmhelpers.fetch import ( | 20 | import six |
900 | 5 | apt_install, | 21 | from charmhelpers.fetch import apt_install |
893 | 6 | ) | ||
894 | 7 | |||
895 | 8 | |||
896 | 9 | DEFAULT_TEMPLATES_DIR = 'templates' | ||
897 | 10 | |||
898 | 11 | |||
901 | 12 | try: | 22 | try: |
902 | 13 | import jinja2 | 23 | import jinja2 |
903 | 14 | except ImportError: | 24 | except ImportError: |
905 | 15 | apt_install(["python-jinja2"]) | 25 | if six.PY3: |
906 | 26 | apt_install(["python3-jinja2"]) | ||
907 | 27 | else: | ||
908 | 28 | apt_install(["python-jinja2"]) | ||
909 | 16 | import jinja2 | 29 | import jinja2 |
910 | 17 | 30 | ||
911 | 18 | 31 | ||
912 | 32 | DEFAULT_TEMPLATES_DIR = 'templates' | ||
913 | 33 | |||
914 | 34 | |||
915 | 19 | def render(template_name, context, template_dir=DEFAULT_TEMPLATES_DIR): | 35 | def render(template_name, context, template_dir=DEFAULT_TEMPLATES_DIR): |
916 | 20 | templates = jinja2.Environment( | 36 | templates = jinja2.Environment( |
917 | 21 | loader=jinja2.FileSystemLoader(template_dir)) | 37 | loader=jinja2.FileSystemLoader(template_dir)) |
918 | 22 | 38 | ||
919 | === modified file 'lib/charmhelpers/core/__init__.py' | |||
920 | --- lib/charmhelpers/core/__init__.py 2014-07-17 16:38:17 +0000 | |||
921 | +++ lib/charmhelpers/core/__init__.py 2015-11-01 20:32:57 +0000 | |||
922 | @@ -0,0 +1,15 @@ | |||
923 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
924 | 2 | # | ||
925 | 3 | # This file is part of charm-helpers. | ||
926 | 4 | # | ||
927 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
928 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
929 | 7 | # published by the Free Software Foundation. | ||
930 | 8 | # | ||
931 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
932 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
933 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
934 | 12 | # GNU Lesser General Public License for more details. | ||
935 | 13 | # | ||
936 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
937 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
938 | 0 | 16 | ||
939 | === added file 'lib/charmhelpers/core/decorators.py' | |||
940 | --- lib/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000 | |||
941 | +++ lib/charmhelpers/core/decorators.py 2015-11-01 20:32:57 +0000 | |||
942 | @@ -0,0 +1,57 @@ | |||
943 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
944 | 2 | # | ||
945 | 3 | # This file is part of charm-helpers. | ||
946 | 4 | # | ||
947 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
948 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
949 | 7 | # published by the Free Software Foundation. | ||
950 | 8 | # | ||
951 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
952 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
953 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
954 | 12 | # GNU Lesser General Public License for more details. | ||
955 | 13 | # | ||
956 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
957 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
958 | 16 | |||
959 | 17 | # | ||
960 | 18 | # Copyright 2014 Canonical Ltd. | ||
961 | 19 | # | ||
962 | 20 | # Authors: | ||
963 | 21 | # Edward Hope-Morley <opentastic@gmail.com> | ||
964 | 22 | # | ||
965 | 23 | |||
966 | 24 | import time | ||
967 | 25 | |||
968 | 26 | from charmhelpers.core.hookenv import ( | ||
969 | 27 | log, | ||
970 | 28 | INFO, | ||
971 | 29 | ) | ||
972 | 30 | |||
973 | 31 | |||
974 | 32 | def retry_on_exception(num_retries, base_delay=0, exc_type=Exception): | ||
975 | 33 | """If the decorated function raises exception exc_type, allow num_retries | ||
976 | 34 | retry attempts before raise the exception. | ||
977 | 35 | """ | ||
978 | 36 | def _retry_on_exception_inner_1(f): | ||
979 | 37 | def _retry_on_exception_inner_2(*args, **kwargs): | ||
980 | 38 | retries = num_retries | ||
981 | 39 | multiplier = 1 | ||
982 | 40 | while True: | ||
983 | 41 | try: | ||
984 | 42 | return f(*args, **kwargs) | ||
985 | 43 | except exc_type: | ||
986 | 44 | if not retries: | ||
987 | 45 | raise | ||
988 | 46 | |||
989 | 47 | delay = base_delay * multiplier | ||
990 | 48 | multiplier += 1 | ||
991 | 49 | log("Retrying '%s' %d more times (delay=%s)" % | ||
992 | 50 | (f.__name__, retries, delay), level=INFO) | ||
993 | 51 | retries -= 1 | ||
994 | 52 | if delay: | ||
995 | 53 | time.sleep(delay) | ||
996 | 54 | |||
997 | 55 | return _retry_on_exception_inner_2 | ||
998 | 56 | |||
999 | 57 | return _retry_on_exception_inner_1 | ||
1000 | 0 | 58 | ||
1001 | === modified file 'lib/charmhelpers/core/fstab.py' | |||
1002 | --- lib/charmhelpers/core/fstab.py 2014-07-17 16:38:17 +0000 | |||
1003 | +++ lib/charmhelpers/core/fstab.py 2015-11-01 20:32:57 +0000 | |||
1004 | @@ -1,12 +1,29 @@ | |||
1005 | 1 | #!/usr/bin/env python | 1 | #!/usr/bin/env python |
1006 | 2 | # -*- coding: utf-8 -*- | 2 | # -*- coding: utf-8 -*- |
1007 | 3 | 3 | ||
1008 | 4 | # Copyright 2014-2015 Canonical Limited. | ||
1009 | 5 | # | ||
1010 | 6 | # This file is part of charm-helpers. | ||
1011 | 7 | # | ||
1012 | 8 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1013 | 9 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1014 | 10 | # published by the Free Software Foundation. | ||
1015 | 11 | # | ||
1016 | 12 | # charm-helpers is distributed in the hope that it will be useful, | ||
1017 | 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1018 | 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1019 | 15 | # GNU Lesser General Public License for more details. | ||
1020 | 16 | # | ||
1021 | 17 | # You should have received a copy of the GNU Lesser General Public License | ||
1022 | 18 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1023 | 19 | |||
1024 | 20 | import io | ||
1025 | 21 | import os | ||
1026 | 22 | |||
1027 | 4 | __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' | 23 | __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
1028 | 5 | 24 | ||
1033 | 6 | import os | 25 | |
1034 | 7 | 26 | class Fstab(io.FileIO): | |
1031 | 8 | |||
1032 | 9 | class Fstab(file): | ||
1035 | 10 | """This class extends file in order to implement a file reader/writer | 27 | """This class extends file in order to implement a file reader/writer |
1036 | 11 | for file `/etc/fstab` | 28 | for file `/etc/fstab` |
1037 | 12 | """ | 29 | """ |
1038 | @@ -24,8 +41,8 @@ | |||
1039 | 24 | options = "defaults" | 41 | options = "defaults" |
1040 | 25 | 42 | ||
1041 | 26 | self.options = options | 43 | self.options = options |
1044 | 27 | self.d = d | 44 | self.d = int(d) |
1045 | 28 | self.p = p | 45 | self.p = int(p) |
1046 | 29 | 46 | ||
1047 | 30 | def __eq__(self, o): | 47 | def __eq__(self, o): |
1048 | 31 | return str(self) == str(o) | 48 | return str(self) == str(o) |
1049 | @@ -45,7 +62,7 @@ | |||
1050 | 45 | self._path = path | 62 | self._path = path |
1051 | 46 | else: | 63 | else: |
1052 | 47 | self._path = self.DEFAULT_PATH | 64 | self._path = self.DEFAULT_PATH |
1054 | 48 | file.__init__(self, self._path, 'r+') | 65 | super(Fstab, self).__init__(self._path, 'rb+') |
1055 | 49 | 66 | ||
1056 | 50 | def _hydrate_entry(self, line): | 67 | def _hydrate_entry(self, line): |
1057 | 51 | # NOTE: use split with no arguments to split on any | 68 | # NOTE: use split with no arguments to split on any |
1058 | @@ -58,8 +75,9 @@ | |||
1059 | 58 | def entries(self): | 75 | def entries(self): |
1060 | 59 | self.seek(0) | 76 | self.seek(0) |
1061 | 60 | for line in self.readlines(): | 77 | for line in self.readlines(): |
1062 | 78 | line = line.decode('us-ascii') | ||
1063 | 61 | try: | 79 | try: |
1065 | 62 | if not line.startswith("#"): | 80 | if line.strip() and not line.strip().startswith("#"): |
1066 | 63 | yield self._hydrate_entry(line) | 81 | yield self._hydrate_entry(line) |
1067 | 64 | except ValueError: | 82 | except ValueError: |
1068 | 65 | pass | 83 | pass |
1069 | @@ -75,18 +93,18 @@ | |||
1070 | 75 | if self.get_entry_by_attr('device', entry.device): | 93 | if self.get_entry_by_attr('device', entry.device): |
1071 | 76 | return False | 94 | return False |
1072 | 77 | 95 | ||
1074 | 78 | self.write(str(entry) + '\n') | 96 | self.write((str(entry) + '\n').encode('us-ascii')) |
1075 | 79 | self.truncate() | 97 | self.truncate() |
1076 | 80 | return entry | 98 | return entry |
1077 | 81 | 99 | ||
1078 | 82 | def remove_entry(self, entry): | 100 | def remove_entry(self, entry): |
1079 | 83 | self.seek(0) | 101 | self.seek(0) |
1080 | 84 | 102 | ||
1082 | 85 | lines = self.readlines() | 103 | lines = [l.decode('us-ascii') for l in self.readlines()] |
1083 | 86 | 104 | ||
1084 | 87 | found = False | 105 | found = False |
1085 | 88 | for index, line in enumerate(lines): | 106 | for index, line in enumerate(lines): |
1087 | 89 | if not line.startswith("#"): | 107 | if line.strip() and not line.strip().startswith("#"): |
1088 | 90 | if self._hydrate_entry(line) == entry: | 108 | if self._hydrate_entry(line) == entry: |
1089 | 91 | found = True | 109 | found = True |
1090 | 92 | break | 110 | break |
1091 | @@ -97,7 +115,7 @@ | |||
1092 | 97 | lines.remove(line) | 115 | lines.remove(line) |
1093 | 98 | 116 | ||
1094 | 99 | self.seek(0) | 117 | self.seek(0) |
1096 | 100 | self.write(''.join(lines)) | 118 | self.write(''.join(lines).encode('us-ascii')) |
1097 | 101 | self.truncate() | 119 | self.truncate() |
1098 | 102 | return True | 120 | return True |
1099 | 103 | 121 | ||
1100 | 104 | 122 | ||
1101 | === modified file 'lib/charmhelpers/core/hookenv.py' | |||
1102 | --- lib/charmhelpers/core/hookenv.py 2014-09-23 12:09:14 +0000 | |||
1103 | +++ lib/charmhelpers/core/hookenv.py 2015-11-01 20:32:57 +0000 | |||
1104 | @@ -1,17 +1,40 @@ | |||
1105 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
1106 | 2 | # | ||
1107 | 3 | # This file is part of charm-helpers. | ||
1108 | 4 | # | ||
1109 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1110 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1111 | 7 | # published by the Free Software Foundation. | ||
1112 | 8 | # | ||
1113 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1114 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1115 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1116 | 12 | # GNU Lesser General Public License for more details. | ||
1117 | 13 | # | ||
1118 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1119 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1120 | 16 | |||
1121 | 1 | "Interactions with the Juju environment" | 17 | "Interactions with the Juju environment" |
1122 | 2 | # Copyright 2013 Canonical Ltd. | 18 | # Copyright 2013 Canonical Ltd. |
1123 | 3 | # | 19 | # |
1124 | 4 | # Authors: | 20 | # Authors: |
1125 | 5 | # Charm Helpers Developers <juju@lists.ubuntu.com> | 21 | # Charm Helpers Developers <juju@lists.ubuntu.com> |
1126 | 6 | 22 | ||
1127 | 23 | from __future__ import print_function | ||
1128 | 7 | import os | 24 | import os |
1129 | 8 | import json | 25 | import json |
1130 | 9 | import yaml | 26 | import yaml |
1131 | 10 | import subprocess | 27 | import subprocess |
1132 | 11 | import sys | 28 | import sys |
1134 | 12 | import UserDict | 29 | import errno |
1135 | 13 | from subprocess import CalledProcessError | 30 | from subprocess import CalledProcessError |
1136 | 14 | 31 | ||
1137 | 32 | import six | ||
1138 | 33 | if not six.PY3: | ||
1139 | 34 | from UserDict import UserDict | ||
1140 | 35 | else: | ||
1141 | 36 | from collections import UserDict | ||
1142 | 37 | |||
1143 | 15 | CRITICAL = "CRITICAL" | 38 | CRITICAL = "CRITICAL" |
1144 | 16 | ERROR = "ERROR" | 39 | ERROR = "ERROR" |
1145 | 17 | WARNING = "WARNING" | 40 | WARNING = "WARNING" |
1146 | @@ -63,16 +86,29 @@ | |||
1147 | 63 | command = ['juju-log'] | 86 | command = ['juju-log'] |
1148 | 64 | if level: | 87 | if level: |
1149 | 65 | command += ['-l', level] | 88 | command += ['-l', level] |
1150 | 89 | if not isinstance(message, six.string_types): | ||
1151 | 90 | message = repr(message) | ||
1152 | 66 | command += [message] | 91 | command += [message] |
1157 | 67 | subprocess.call(command) | 92 | # Missing juju-log should not cause failures in unit tests |
1158 | 68 | 93 | # Send log output to stderr | |
1159 | 69 | 94 | try: | |
1160 | 70 | class Serializable(UserDict.IterableUserDict): | 95 | subprocess.call(command) |
1161 | 96 | except OSError as e: | ||
1162 | 97 | if e.errno == errno.ENOENT: | ||
1163 | 98 | if level: | ||
1164 | 99 | message = "{}: {}".format(level, message) | ||
1165 | 100 | message = "juju-log: {}".format(message) | ||
1166 | 101 | print(message, file=sys.stderr) | ||
1167 | 102 | else: | ||
1168 | 103 | raise | ||
1169 | 104 | |||
1170 | 105 | |||
1171 | 106 | class Serializable(UserDict): | ||
1172 | 71 | """Wrapper, an object that can be serialized to yaml or json""" | 107 | """Wrapper, an object that can be serialized to yaml or json""" |
1173 | 72 | 108 | ||
1174 | 73 | def __init__(self, obj): | 109 | def __init__(self, obj): |
1175 | 74 | # wrap the object | 110 | # wrap the object |
1177 | 75 | UserDict.IterableUserDict.__init__(self) | 111 | UserDict.__init__(self) |
1178 | 76 | self.data = obj | 112 | self.data = obj |
1179 | 77 | 113 | ||
1180 | 78 | def __getattr__(self, attr): | 114 | def __getattr__(self, attr): |
1181 | @@ -214,6 +250,12 @@ | |||
1182 | 214 | except KeyError: | 250 | except KeyError: |
1183 | 215 | return (self._prev_dict or {})[key] | 251 | return (self._prev_dict or {})[key] |
1184 | 216 | 252 | ||
1185 | 253 | def keys(self): | ||
1186 | 254 | prev_keys = [] | ||
1187 | 255 | if self._prev_dict is not None: | ||
1188 | 256 | prev_keys = self._prev_dict.keys() | ||
1189 | 257 | return list(set(prev_keys + list(dict.keys(self)))) | ||
1190 | 258 | |||
1191 | 217 | def load_previous(self, path=None): | 259 | def load_previous(self, path=None): |
1192 | 218 | """Load previous copy of config from disk. | 260 | """Load previous copy of config from disk. |
1193 | 219 | 261 | ||
1194 | @@ -263,7 +305,7 @@ | |||
1195 | 263 | 305 | ||
1196 | 264 | """ | 306 | """ |
1197 | 265 | if self._prev_dict: | 307 | if self._prev_dict: |
1199 | 266 | for k, v in self._prev_dict.iteritems(): | 308 | for k, v in six.iteritems(self._prev_dict): |
1200 | 267 | if k not in self: | 309 | if k not in self: |
1201 | 268 | self[k] = v | 310 | self[k] = v |
1202 | 269 | with open(self.path, 'w') as f: | 311 | with open(self.path, 'w') as f: |
1203 | @@ -278,7 +320,8 @@ | |||
1204 | 278 | config_cmd_line.append(scope) | 320 | config_cmd_line.append(scope) |
1205 | 279 | config_cmd_line.append('--format=json') | 321 | config_cmd_line.append('--format=json') |
1206 | 280 | try: | 322 | try: |
1208 | 281 | config_data = json.loads(subprocess.check_output(config_cmd_line)) | 323 | config_data = json.loads( |
1209 | 324 | subprocess.check_output(config_cmd_line).decode('UTF-8')) | ||
1210 | 282 | if scope is not None: | 325 | if scope is not None: |
1211 | 283 | return config_data | 326 | return config_data |
1212 | 284 | return Config(config_data) | 327 | return Config(config_data) |
1213 | @@ -297,10 +340,10 @@ | |||
1214 | 297 | if unit: | 340 | if unit: |
1215 | 298 | _args.append(unit) | 341 | _args.append(unit) |
1216 | 299 | try: | 342 | try: |
1218 | 300 | return json.loads(subprocess.check_output(_args)) | 343 | return json.loads(subprocess.check_output(_args).decode('UTF-8')) |
1219 | 301 | except ValueError: | 344 | except ValueError: |
1220 | 302 | return None | 345 | return None |
1222 | 303 | except CalledProcessError, e: | 346 | except CalledProcessError as e: |
1223 | 304 | if e.returncode == 2: | 347 | if e.returncode == 2: |
1224 | 305 | return None | 348 | return None |
1225 | 306 | raise | 349 | raise |
1226 | @@ -312,7 +355,7 @@ | |||
1227 | 312 | relation_cmd_line = ['relation-set'] | 355 | relation_cmd_line = ['relation-set'] |
1228 | 313 | if relation_id is not None: | 356 | if relation_id is not None: |
1229 | 314 | relation_cmd_line.extend(('-r', relation_id)) | 357 | relation_cmd_line.extend(('-r', relation_id)) |
1231 | 315 | for k, v in (relation_settings.items() + kwargs.items()): | 358 | for k, v in (list(relation_settings.items()) + list(kwargs.items())): |
1232 | 316 | if v is None: | 359 | if v is None: |
1233 | 317 | relation_cmd_line.append('{}='.format(k)) | 360 | relation_cmd_line.append('{}='.format(k)) |
1234 | 318 | else: | 361 | else: |
1235 | @@ -329,7 +372,8 @@ | |||
1236 | 329 | relid_cmd_line = ['relation-ids', '--format=json'] | 372 | relid_cmd_line = ['relation-ids', '--format=json'] |
1237 | 330 | if reltype is not None: | 373 | if reltype is not None: |
1238 | 331 | relid_cmd_line.append(reltype) | 374 | relid_cmd_line.append(reltype) |
1240 | 332 | return json.loads(subprocess.check_output(relid_cmd_line)) or [] | 375 | return json.loads( |
1241 | 376 | subprocess.check_output(relid_cmd_line).decode('UTF-8')) or [] | ||
1242 | 333 | return [] | 377 | return [] |
1243 | 334 | 378 | ||
1244 | 335 | 379 | ||
1245 | @@ -340,7 +384,8 @@ | |||
1246 | 340 | units_cmd_line = ['relation-list', '--format=json'] | 384 | units_cmd_line = ['relation-list', '--format=json'] |
1247 | 341 | if relid is not None: | 385 | if relid is not None: |
1248 | 342 | units_cmd_line.extend(('-r', relid)) | 386 | units_cmd_line.extend(('-r', relid)) |
1250 | 343 | return json.loads(subprocess.check_output(units_cmd_line)) or [] | 387 | return json.loads( |
1251 | 388 | subprocess.check_output(units_cmd_line).decode('UTF-8')) or [] | ||
1252 | 344 | 389 | ||
1253 | 345 | 390 | ||
1254 | 346 | @cached | 391 | @cached |
1255 | @@ -380,21 +425,31 @@ | |||
1256 | 380 | 425 | ||
1257 | 381 | 426 | ||
1258 | 382 | @cached | 427 | @cached |
1259 | 428 | def metadata(): | ||
1260 | 429 | """Get the current charm metadata.yaml contents as a python object""" | ||
1261 | 430 | with open(os.path.join(charm_dir(), 'metadata.yaml')) as md: | ||
1262 | 431 | return yaml.safe_load(md) | ||
1263 | 432 | |||
1264 | 433 | |||
1265 | 434 | @cached | ||
1266 | 383 | def relation_types(): | 435 | def relation_types(): |
1267 | 384 | """Get a list of relation types supported by this charm""" | 436 | """Get a list of relation types supported by this charm""" |
1268 | 385 | charmdir = os.environ.get('CHARM_DIR', '') | ||
1269 | 386 | mdf = open(os.path.join(charmdir, 'metadata.yaml')) | ||
1270 | 387 | md = yaml.safe_load(mdf) | ||
1271 | 388 | rel_types = [] | 437 | rel_types = [] |
1272 | 438 | md = metadata() | ||
1273 | 389 | for key in ('provides', 'requires', 'peers'): | 439 | for key in ('provides', 'requires', 'peers'): |
1274 | 390 | section = md.get(key) | 440 | section = md.get(key) |
1275 | 391 | if section: | 441 | if section: |
1276 | 392 | rel_types.extend(section.keys()) | 442 | rel_types.extend(section.keys()) |
1277 | 393 | mdf.close() | ||
1278 | 394 | return rel_types | 443 | return rel_types |
1279 | 395 | 444 | ||
1280 | 396 | 445 | ||
1281 | 397 | @cached | 446 | @cached |
1282 | 447 | def charm_name(): | ||
1283 | 448 | """Get the name of the current charm as is specified on metadata.yaml""" | ||
1284 | 449 | return metadata().get('name') | ||
1285 | 450 | |||
1286 | 451 | |||
1287 | 452 | @cached | ||
1288 | 398 | def relations(): | 453 | def relations(): |
1289 | 399 | """Get a nested dictionary of relation data for all related units""" | 454 | """Get a nested dictionary of relation data for all related units""" |
1290 | 400 | rels = {} | 455 | rels = {} |
1291 | @@ -449,7 +504,7 @@ | |||
1292 | 449 | """Get the unit ID for the remote unit""" | 504 | """Get the unit ID for the remote unit""" |
1293 | 450 | _args = ['unit-get', '--format=json', attribute] | 505 | _args = ['unit-get', '--format=json', attribute] |
1294 | 451 | try: | 506 | try: |
1296 | 452 | return json.loads(subprocess.check_output(_args)) | 507 | return json.loads(subprocess.check_output(_args).decode('UTF-8')) |
1297 | 453 | except ValueError: | 508 | except ValueError: |
1298 | 454 | return None | 509 | return None |
1299 | 455 | 510 | ||
1300 | @@ -486,9 +541,10 @@ | |||
1301 | 486 | hooks.execute(sys.argv) | 541 | hooks.execute(sys.argv) |
1302 | 487 | """ | 542 | """ |
1303 | 488 | 543 | ||
1305 | 489 | def __init__(self): | 544 | def __init__(self, config_save=True): |
1306 | 490 | super(Hooks, self).__init__() | 545 | super(Hooks, self).__init__() |
1307 | 491 | self._hooks = {} | 546 | self._hooks = {} |
1308 | 547 | self._config_save = config_save | ||
1309 | 492 | 548 | ||
1310 | 493 | def register(self, name, function): | 549 | def register(self, name, function): |
1311 | 494 | """Register a hook""" | 550 | """Register a hook""" |
1312 | @@ -499,9 +555,10 @@ | |||
1313 | 499 | hook_name = os.path.basename(args[0]) | 555 | hook_name = os.path.basename(args[0]) |
1314 | 500 | if hook_name in self._hooks: | 556 | if hook_name in self._hooks: |
1315 | 501 | self._hooks[hook_name]() | 557 | self._hooks[hook_name]() |
1319 | 502 | cfg = config() | 558 | if self._config_save: |
1320 | 503 | if cfg.implicit_save: | 559 | cfg = config() |
1321 | 504 | cfg.save() | 560 | if cfg.implicit_save: |
1322 | 561 | cfg.save() | ||
1323 | 505 | else: | 562 | else: |
1324 | 506 | raise UnregisteredHookError(hook_name) | 563 | raise UnregisteredHookError(hook_name) |
1325 | 507 | 564 | ||
1326 | @@ -522,3 +579,29 @@ | |||
1327 | 522 | def charm_dir(): | 579 | def charm_dir(): |
1328 | 523 | """Return the root directory of the current charm""" | 580 | """Return the root directory of the current charm""" |
1329 | 524 | return os.environ.get('CHARM_DIR') | 581 | return os.environ.get('CHARM_DIR') |
1330 | 582 | |||
1331 | 583 | |||
1332 | 584 | @cached | ||
1333 | 585 | def action_get(key=None): | ||
1334 | 586 | """Gets the value of an action parameter, or all key/value param pairs""" | ||
1335 | 587 | cmd = ['action-get'] | ||
1336 | 588 | if key is not None: | ||
1337 | 589 | cmd.append(key) | ||
1338 | 590 | cmd.append('--format=json') | ||
1339 | 591 | action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8')) | ||
1340 | 592 | return action_data | ||
1341 | 593 | |||
1342 | 594 | |||
1343 | 595 | def action_set(values): | ||
1344 | 596 | """Sets the values to be returned after the action finishes""" | ||
1345 | 597 | cmd = ['action-set'] | ||
1346 | 598 | for k, v in list(values.items()): | ||
1347 | 599 | cmd.append('{}={}'.format(k, v)) | ||
1348 | 600 | subprocess.check_call(cmd) | ||
1349 | 601 | |||
1350 | 602 | |||
1351 | 603 | def action_fail(message): | ||
1352 | 604 | """Sets the action status to failed and sets the error message. | ||
1353 | 605 | |||
1354 | 606 | The results set by action_set are preserved.""" | ||
1355 | 607 | subprocess.check_call(['action-fail', message]) | ||
1356 | 525 | 608 | ||
1357 | === modified file 'lib/charmhelpers/core/host.py' | |||
1358 | --- lib/charmhelpers/core/host.py 2014-09-23 12:09:14 +0000 | |||
1359 | +++ lib/charmhelpers/core/host.py 2015-11-01 20:32:57 +0000 | |||
1360 | @@ -1,3 +1,19 @@ | |||
1361 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
1362 | 2 | # | ||
1363 | 3 | # This file is part of charm-helpers. | ||
1364 | 4 | # | ||
1365 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1366 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1367 | 7 | # published by the Free Software Foundation. | ||
1368 | 8 | # | ||
1369 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1370 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1371 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1372 | 12 | # GNU Lesser General Public License for more details. | ||
1373 | 13 | # | ||
1374 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1375 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1376 | 16 | |||
1377 | 1 | """Tools for working with the host system""" | 17 | """Tools for working with the host system""" |
1378 | 2 | # Copyright 2012 Canonical Ltd. | 18 | # Copyright 2012 Canonical Ltd. |
1379 | 3 | # | 19 | # |
1380 | @@ -6,19 +22,20 @@ | |||
1381 | 6 | # Matthew Wedgwood <matthew.wedgwood@canonical.com> | 22 | # Matthew Wedgwood <matthew.wedgwood@canonical.com> |
1382 | 7 | 23 | ||
1383 | 8 | import os | 24 | import os |
1384 | 25 | import re | ||
1385 | 9 | import pwd | 26 | import pwd |
1386 | 10 | import grp | 27 | import grp |
1387 | 11 | import random | 28 | import random |
1388 | 12 | import string | 29 | import string |
1389 | 13 | import subprocess | 30 | import subprocess |
1390 | 14 | import hashlib | 31 | import hashlib |
1391 | 15 | import shutil | ||
1392 | 16 | from contextlib import contextmanager | 32 | from contextlib import contextmanager |
1393 | 17 | |||
1394 | 18 | from collections import OrderedDict | 33 | from collections import OrderedDict |
1395 | 19 | 34 | ||
1398 | 20 | from hookenv import log | 35 | import six |
1399 | 21 | from fstab import Fstab | 36 | |
1400 | 37 | from .hookenv import log | ||
1401 | 38 | from .fstab import Fstab | ||
1402 | 22 | 39 | ||
1403 | 23 | 40 | ||
1404 | 24 | def service_start(service_name): | 41 | def service_start(service_name): |
1405 | @@ -54,7 +71,9 @@ | |||
1406 | 54 | def service_running(service): | 71 | def service_running(service): |
1407 | 55 | """Determine whether a system service is running""" | 72 | """Determine whether a system service is running""" |
1408 | 56 | try: | 73 | try: |
1410 | 57 | output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT) | 74 | output = subprocess.check_output( |
1411 | 75 | ['service', service, 'status'], | ||
1412 | 76 | stderr=subprocess.STDOUT).decode('UTF-8') | ||
1413 | 58 | except subprocess.CalledProcessError: | 77 | except subprocess.CalledProcessError: |
1414 | 59 | return False | 78 | return False |
1415 | 60 | else: | 79 | else: |
1416 | @@ -67,9 +86,11 @@ | |||
1417 | 67 | def service_available(service_name): | 86 | def service_available(service_name): |
1418 | 68 | """Determine whether a system service is available""" | 87 | """Determine whether a system service is available""" |
1419 | 69 | try: | 88 | try: |
1423 | 70 | subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT) | 89 | subprocess.check_output( |
1424 | 71 | except subprocess.CalledProcessError: | 90 | ['service', service_name, 'status'], |
1425 | 72 | return False | 91 | stderr=subprocess.STDOUT).decode('UTF-8') |
1426 | 92 | except subprocess.CalledProcessError as e: | ||
1427 | 93 | return 'unrecognized service' not in e.output | ||
1428 | 73 | else: | 94 | else: |
1429 | 74 | return True | 95 | return True |
1430 | 75 | 96 | ||
1431 | @@ -96,6 +117,26 @@ | |||
1432 | 96 | return user_info | 117 | return user_info |
1433 | 97 | 118 | ||
1434 | 98 | 119 | ||
1435 | 120 | def add_group(group_name, system_group=False): | ||
1436 | 121 | """Add a group to the system""" | ||
1437 | 122 | try: | ||
1438 | 123 | group_info = grp.getgrnam(group_name) | ||
1439 | 124 | log('group {0} already exists!'.format(group_name)) | ||
1440 | 125 | except KeyError: | ||
1441 | 126 | log('creating group {0}'.format(group_name)) | ||
1442 | 127 | cmd = ['addgroup'] | ||
1443 | 128 | if system_group: | ||
1444 | 129 | cmd.append('--system') | ||
1445 | 130 | else: | ||
1446 | 131 | cmd.extend([ | ||
1447 | 132 | '--group', | ||
1448 | 133 | ]) | ||
1449 | 134 | cmd.append(group_name) | ||
1450 | 135 | subprocess.check_call(cmd) | ||
1451 | 136 | group_info = grp.getgrnam(group_name) | ||
1452 | 137 | return group_info | ||
1453 | 138 | |||
1454 | 139 | |||
1455 | 99 | def add_user_to_group(username, group): | 140 | def add_user_to_group(username, group): |
1456 | 100 | """Add a user to a group""" | 141 | """Add a user to a group""" |
1457 | 101 | cmd = [ | 142 | cmd = [ |
1458 | @@ -115,7 +156,7 @@ | |||
1459 | 115 | cmd.append(from_path) | 156 | cmd.append(from_path) |
1460 | 116 | cmd.append(to_path) | 157 | cmd.append(to_path) |
1461 | 117 | log(" ".join(cmd)) | 158 | log(" ".join(cmd)) |
1463 | 118 | return subprocess.check_output(cmd).strip() | 159 | return subprocess.check_output(cmd).decode('UTF-8').strip() |
1464 | 119 | 160 | ||
1465 | 120 | 161 | ||
1466 | 121 | def symlink(source, destination): | 162 | def symlink(source, destination): |
1467 | @@ -130,28 +171,31 @@ | |||
1468 | 130 | subprocess.check_call(cmd) | 171 | subprocess.check_call(cmd) |
1469 | 131 | 172 | ||
1470 | 132 | 173 | ||
1472 | 133 | def mkdir(path, owner='root', group='root', perms=0555, force=False): | 174 | def mkdir(path, owner='root', group='root', perms=0o555, force=False): |
1473 | 134 | """Create a directory""" | 175 | """Create a directory""" |
1474 | 135 | log("Making dir {} {}:{} {:o}".format(path, owner, group, | 176 | log("Making dir {} {}:{} {:o}".format(path, owner, group, |
1475 | 136 | perms)) | 177 | perms)) |
1476 | 137 | uid = pwd.getpwnam(owner).pw_uid | 178 | uid = pwd.getpwnam(owner).pw_uid |
1477 | 138 | gid = grp.getgrnam(group).gr_gid | 179 | gid = grp.getgrnam(group).gr_gid |
1478 | 139 | realpath = os.path.abspath(path) | 180 | realpath = os.path.abspath(path) |
1481 | 140 | if os.path.exists(realpath): | 181 | path_exists = os.path.exists(realpath) |
1482 | 141 | if force and not os.path.isdir(realpath): | 182 | if path_exists and force: |
1483 | 183 | if not os.path.isdir(realpath): | ||
1484 | 142 | log("Removing non-directory file {} prior to mkdir()".format(path)) | 184 | log("Removing non-directory file {} prior to mkdir()".format(path)) |
1485 | 143 | os.unlink(realpath) | 185 | os.unlink(realpath) |
1487 | 144 | else: | 186 | os.makedirs(realpath, perms) |
1488 | 187 | elif not path_exists: | ||
1489 | 145 | os.makedirs(realpath, perms) | 188 | os.makedirs(realpath, perms) |
1490 | 146 | os.chown(realpath, uid, gid) | 189 | os.chown(realpath, uid, gid) |
1495 | 147 | 190 | os.chmod(realpath, perms) | |
1496 | 148 | 191 | ||
1497 | 149 | def write_file(path, content, owner='root', group='root', perms=0444): | 192 | |
1498 | 150 | """Create or overwrite a file with the contents of a string""" | 193 | def write_file(path, content, owner='root', group='root', perms=0o444): |
1499 | 194 | """Create or overwrite a file with the contents of a byte string.""" | ||
1500 | 151 | log("Writing file {} {}:{} {:o}".format(path, owner, group, perms)) | 195 | log("Writing file {} {}:{} {:o}".format(path, owner, group, perms)) |
1501 | 152 | uid = pwd.getpwnam(owner).pw_uid | 196 | uid = pwd.getpwnam(owner).pw_uid |
1502 | 153 | gid = grp.getgrnam(group).gr_gid | 197 | gid = grp.getgrnam(group).gr_gid |
1504 | 154 | with open(path, 'w') as target: | 198 | with open(path, 'wb') as target: |
1505 | 155 | os.fchown(target.fileno(), uid, gid) | 199 | os.fchown(target.fileno(), uid, gid) |
1506 | 156 | os.fchmod(target.fileno(), perms) | 200 | os.fchmod(target.fileno(), perms) |
1507 | 157 | target.write(content) | 201 | target.write(content) |
1508 | @@ -177,7 +221,7 @@ | |||
1509 | 177 | cmd_args.extend([device, mountpoint]) | 221 | cmd_args.extend([device, mountpoint]) |
1510 | 178 | try: | 222 | try: |
1511 | 179 | subprocess.check_output(cmd_args) | 223 | subprocess.check_output(cmd_args) |
1513 | 180 | except subprocess.CalledProcessError, e: | 224 | except subprocess.CalledProcessError as e: |
1514 | 181 | log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output)) | 225 | log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output)) |
1515 | 182 | return False | 226 | return False |
1516 | 183 | 227 | ||
1517 | @@ -191,7 +235,7 @@ | |||
1518 | 191 | cmd_args = ['umount', mountpoint] | 235 | cmd_args = ['umount', mountpoint] |
1519 | 192 | try: | 236 | try: |
1520 | 193 | subprocess.check_output(cmd_args) | 237 | subprocess.check_output(cmd_args) |
1522 | 194 | except subprocess.CalledProcessError, e: | 238 | except subprocess.CalledProcessError as e: |
1523 | 195 | log('Error unmounting {}\n{}'.format(mountpoint, e.output)) | 239 | log('Error unmounting {}\n{}'.format(mountpoint, e.output)) |
1524 | 196 | return False | 240 | return False |
1525 | 197 | 241 | ||
1526 | @@ -218,8 +262,8 @@ | |||
1527 | 218 | """ | 262 | """ |
1528 | 219 | if os.path.exists(path): | 263 | if os.path.exists(path): |
1529 | 220 | h = getattr(hashlib, hash_type)() | 264 | h = getattr(hashlib, hash_type)() |
1532 | 221 | with open(path, 'r') as source: | 265 | with open(path, 'rb') as source: |
1533 | 222 | h.update(source.read()) # IGNORE:E1101 - it does have update | 266 | h.update(source.read()) |
1534 | 223 | return h.hexdigest() | 267 | return h.hexdigest() |
1535 | 224 | else: | 268 | else: |
1536 | 225 | return None | 269 | return None |
1537 | @@ -229,12 +273,12 @@ | |||
1538 | 229 | """ | 273 | """ |
1539 | 230 | Validate a file using a cryptographic checksum. | 274 | Validate a file using a cryptographic checksum. |
1540 | 231 | 275 | ||
1541 | 232 | |||
1542 | 233 | :param str checksum: Value of the checksum used to validate the file. | 276 | :param str checksum: Value of the checksum used to validate the file. |
1546 | 234 | :param str hash_type: Hash algorithm used to generate :param:`checksum`. | 277 | :param str hash_type: Hash algorithm used to generate `checksum`. |
1547 | 235 | Can be any hash alrgorithm supported by :mod:`hashlib`, | 278 | Can be any hash alrgorithm supported by :mod:`hashlib`, |
1548 | 236 | such as md5, sha1, sha256, sha512, etc. | 279 | such as md5, sha1, sha256, sha512, etc. |
1549 | 237 | :raises ChecksumError: If the file fails the checksum | 280 | :raises ChecksumError: If the file fails the checksum |
1550 | 281 | |||
1551 | 238 | """ | 282 | """ |
1552 | 239 | actual_checksum = file_hash(path, hash_type) | 283 | actual_checksum = file_hash(path, hash_type) |
1553 | 240 | if checksum != actual_checksum: | 284 | if checksum != actual_checksum: |
1554 | @@ -261,11 +305,11 @@ | |||
1555 | 261 | ceph_client_changed function. | 305 | ceph_client_changed function. |
1556 | 262 | """ | 306 | """ |
1557 | 263 | def wrap(f): | 307 | def wrap(f): |
1559 | 264 | def wrapped_f(*args): | 308 | def wrapped_f(*args, **kwargs): |
1560 | 265 | checksums = {} | 309 | checksums = {} |
1561 | 266 | for path in restart_map: | 310 | for path in restart_map: |
1562 | 267 | checksums[path] = file_hash(path) | 311 | checksums[path] = file_hash(path) |
1564 | 268 | f(*args) | 312 | f(*args, **kwargs) |
1565 | 269 | restarts = [] | 313 | restarts = [] |
1566 | 270 | for path in restart_map: | 314 | for path in restart_map: |
1567 | 271 | if checksums[path] != file_hash(path): | 315 | if checksums[path] != file_hash(path): |
1568 | @@ -295,29 +339,39 @@ | |||
1569 | 295 | def pwgen(length=None): | 339 | def pwgen(length=None): |
1570 | 296 | """Generate a random pasword.""" | 340 | """Generate a random pasword.""" |
1571 | 297 | if length is None: | 341 | if length is None: |
1572 | 342 | # A random length is ok to use a weak PRNG | ||
1573 | 298 | length = random.choice(range(35, 45)) | 343 | length = random.choice(range(35, 45)) |
1574 | 299 | alphanumeric_chars = [ | 344 | alphanumeric_chars = [ |
1576 | 300 | l for l in (string.letters + string.digits) | 345 | l for l in (string.ascii_letters + string.digits) |
1577 | 301 | if l not in 'l0QD1vAEIOUaeiou'] | 346 | if l not in 'l0QD1vAEIOUaeiou'] |
1578 | 347 | # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the | ||
1579 | 348 | # actual password | ||
1580 | 349 | random_generator = random.SystemRandom() | ||
1581 | 302 | random_chars = [ | 350 | random_chars = [ |
1583 | 303 | random.choice(alphanumeric_chars) for _ in range(length)] | 351 | random_generator.choice(alphanumeric_chars) for _ in range(length)] |
1584 | 304 | return(''.join(random_chars)) | 352 | return(''.join(random_chars)) |
1585 | 305 | 353 | ||
1586 | 306 | 354 | ||
1587 | 307 | def list_nics(nic_type): | 355 | def list_nics(nic_type): |
1588 | 308 | '''Return a list of nics of given type(s)''' | 356 | '''Return a list of nics of given type(s)''' |
1590 | 309 | if isinstance(nic_type, basestring): | 357 | if isinstance(nic_type, six.string_types): |
1591 | 310 | int_types = [nic_type] | 358 | int_types = [nic_type] |
1592 | 311 | else: | 359 | else: |
1593 | 312 | int_types = nic_type | 360 | int_types = nic_type |
1594 | 313 | interfaces = [] | 361 | interfaces = [] |
1595 | 314 | for int_type in int_types: | 362 | for int_type in int_types: |
1596 | 315 | cmd = ['ip', 'addr', 'show', 'label', int_type + '*'] | 363 | cmd = ['ip', 'addr', 'show', 'label', int_type + '*'] |
1598 | 316 | ip_output = subprocess.check_output(cmd).split('\n') | 364 | ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n') |
1599 | 317 | ip_output = (line for line in ip_output if line) | 365 | ip_output = (line for line in ip_output if line) |
1600 | 318 | for line in ip_output: | 366 | for line in ip_output: |
1601 | 319 | if line.split()[1].startswith(int_type): | 367 | if line.split()[1].startswith(int_type): |
1603 | 320 | interfaces.append(line.split()[1].replace(":", "")) | 368 | matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line) |
1604 | 369 | if matched: | ||
1605 | 370 | interface = matched.groups()[0] | ||
1606 | 371 | else: | ||
1607 | 372 | interface = line.split()[1].replace(":", "") | ||
1608 | 373 | interfaces.append(interface) | ||
1609 | 374 | |||
1610 | 321 | return interfaces | 375 | return interfaces |
1611 | 322 | 376 | ||
1612 | 323 | 377 | ||
1613 | @@ -329,7 +383,7 @@ | |||
1614 | 329 | 383 | ||
1615 | 330 | def get_nic_mtu(nic): | 384 | def get_nic_mtu(nic): |
1616 | 331 | cmd = ['ip', 'addr', 'show', nic] | 385 | cmd = ['ip', 'addr', 'show', nic] |
1618 | 332 | ip_output = subprocess.check_output(cmd).split('\n') | 386 | ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n') |
1619 | 333 | mtu = "" | 387 | mtu = "" |
1620 | 334 | for line in ip_output: | 388 | for line in ip_output: |
1621 | 335 | words = line.split() | 389 | words = line.split() |
1622 | @@ -340,7 +394,7 @@ | |||
1623 | 340 | 394 | ||
1624 | 341 | def get_nic_hwaddr(nic): | 395 | def get_nic_hwaddr(nic): |
1625 | 342 | cmd = ['ip', '-o', '-0', 'addr', 'show', nic] | 396 | cmd = ['ip', '-o', '-0', 'addr', 'show', nic] |
1627 | 343 | ip_output = subprocess.check_output(cmd) | 397 | ip_output = subprocess.check_output(cmd).decode('UTF-8') |
1628 | 344 | hwaddr = "" | 398 | hwaddr = "" |
1629 | 345 | words = ip_output.split() | 399 | words = ip_output.split() |
1630 | 346 | if 'link/ether' in words: | 400 | if 'link/ether' in words: |
1631 | @@ -355,10 +409,13 @@ | |||
1632 | 355 | * 0 => Installed revno is the same as supplied arg | 409 | * 0 => Installed revno is the same as supplied arg |
1633 | 356 | * -1 => Installed revno is less than supplied arg | 410 | * -1 => Installed revno is less than supplied arg |
1634 | 357 | 411 | ||
1635 | 412 | This function imports apt_cache function from charmhelpers.fetch if | ||
1636 | 413 | the pkgcache argument is None. Be sure to add charmhelpers.fetch if | ||
1637 | 414 | you call this function, or pass an apt_pkg.Cache() instance. | ||
1638 | 358 | ''' | 415 | ''' |
1639 | 359 | import apt_pkg | 416 | import apt_pkg |
1640 | 360 | from charmhelpers.fetch import apt_cache | ||
1641 | 361 | if not pkgcache: | 417 | if not pkgcache: |
1642 | 418 | from charmhelpers.fetch import apt_cache | ||
1643 | 362 | pkgcache = apt_cache() | 419 | pkgcache = apt_cache() |
1644 | 363 | pkg = pkgcache[package] | 420 | pkg = pkgcache[package] |
1645 | 364 | return apt_pkg.version_compare(pkg.current_ver.ver_str, revno) | 421 | return apt_pkg.version_compare(pkg.current_ver.ver_str, revno) |
1646 | @@ -373,13 +430,21 @@ | |||
1647 | 373 | os.chdir(cur) | 430 | os.chdir(cur) |
1648 | 374 | 431 | ||
1649 | 375 | 432 | ||
1651 | 376 | def chownr(path, owner, group): | 433 | def chownr(path, owner, group, follow_links=True): |
1652 | 377 | uid = pwd.getpwnam(owner).pw_uid | 434 | uid = pwd.getpwnam(owner).pw_uid |
1653 | 378 | gid = grp.getgrnam(group).gr_gid | 435 | gid = grp.getgrnam(group).gr_gid |
1654 | 436 | if follow_links: | ||
1655 | 437 | chown = os.chown | ||
1656 | 438 | else: | ||
1657 | 439 | chown = os.lchown | ||
1658 | 379 | 440 | ||
1659 | 380 | for root, dirs, files in os.walk(path): | 441 | for root, dirs, files in os.walk(path): |
1660 | 381 | for name in dirs + files: | 442 | for name in dirs + files: |
1661 | 382 | full = os.path.join(root, name) | 443 | full = os.path.join(root, name) |
1662 | 383 | broken_symlink = os.path.lexists(full) and not os.path.exists(full) | 444 | broken_symlink = os.path.lexists(full) and not os.path.exists(full) |
1663 | 384 | if not broken_symlink: | 445 | if not broken_symlink: |
1665 | 385 | os.chown(full, uid, gid) | 446 | chown(full, uid, gid) |
1666 | 447 | |||
1667 | 448 | |||
1668 | 449 | def lchownr(path, owner, group): | ||
1669 | 450 | chownr(path, owner, group, follow_links=False) | ||
1670 | 386 | 451 | ||
1671 | === modified file 'lib/charmhelpers/core/services/__init__.py' | |||
1672 | --- lib/charmhelpers/core/services/__init__.py 2014-09-23 12:09:14 +0000 | |||
1673 | +++ lib/charmhelpers/core/services/__init__.py 2015-11-01 20:32:57 +0000 | |||
1674 | @@ -1,2 +1,18 @@ | |||
1677 | 1 | from .base import * | 1 | # Copyright 2014-2015 Canonical Limited. |
1678 | 2 | from .helpers import * | 2 | # |
1679 | 3 | # This file is part of charm-helpers. | ||
1680 | 4 | # | ||
1681 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1682 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1683 | 7 | # published by the Free Software Foundation. | ||
1684 | 8 | # | ||
1685 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1686 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1687 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1688 | 12 | # GNU Lesser General Public License for more details. | ||
1689 | 13 | # | ||
1690 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1691 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1692 | 16 | |||
1693 | 17 | from .base import * # NOQA | ||
1694 | 18 | from .helpers import * # NOQA | ||
1695 | 3 | 19 | ||
1696 | === modified file 'lib/charmhelpers/core/services/base.py' | |||
1697 | --- lib/charmhelpers/core/services/base.py 2014-09-23 12:09:14 +0000 | |||
1698 | +++ lib/charmhelpers/core/services/base.py 2015-11-01 20:32:57 +0000 | |||
1699 | @@ -1,3 +1,19 @@ | |||
1700 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
1701 | 2 | # | ||
1702 | 3 | # This file is part of charm-helpers. | ||
1703 | 4 | # | ||
1704 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1705 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1706 | 7 | # published by the Free Software Foundation. | ||
1707 | 8 | # | ||
1708 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1709 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1710 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1711 | 12 | # GNU Lesser General Public License for more details. | ||
1712 | 13 | # | ||
1713 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1714 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1715 | 16 | |||
1716 | 1 | import os | 17 | import os |
1717 | 2 | import re | 18 | import re |
1718 | 3 | import json | 19 | import json |
1719 | 4 | 20 | ||
1720 | === modified file 'lib/charmhelpers/core/services/helpers.py' | |||
1721 | --- lib/charmhelpers/core/services/helpers.py 2014-09-23 12:09:14 +0000 | |||
1722 | +++ lib/charmhelpers/core/services/helpers.py 2015-11-01 20:32:57 +0000 | |||
1723 | @@ -1,3 +1,19 @@ | |||
1724 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
1725 | 2 | # | ||
1726 | 3 | # This file is part of charm-helpers. | ||
1727 | 4 | # | ||
1728 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1729 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1730 | 7 | # published by the Free Software Foundation. | ||
1731 | 8 | # | ||
1732 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1733 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1734 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1735 | 12 | # GNU Lesser General Public License for more details. | ||
1736 | 13 | # | ||
1737 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1738 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1739 | 16 | |||
1740 | 1 | import os | 17 | import os |
1741 | 2 | import yaml | 18 | import yaml |
1742 | 3 | from charmhelpers.core import hookenv | 19 | from charmhelpers.core import hookenv |
1743 | @@ -29,12 +45,14 @@ | |||
1744 | 29 | """ | 45 | """ |
1745 | 30 | name = None | 46 | name = None |
1746 | 31 | interface = None | 47 | interface = None |
1747 | 32 | required_keys = [] | ||
1748 | 33 | 48 | ||
1749 | 34 | def __init__(self, name=None, additional_required_keys=None): | 49 | def __init__(self, name=None, additional_required_keys=None): |
1750 | 50 | if not hasattr(self, 'required_keys'): | ||
1751 | 51 | self.required_keys = [] | ||
1752 | 52 | |||
1753 | 35 | if name is not None: | 53 | if name is not None: |
1754 | 36 | self.name = name | 54 | self.name = name |
1756 | 37 | if additional_required_keys is not None: | 55 | if additional_required_keys: |
1757 | 38 | self.required_keys.extend(additional_required_keys) | 56 | self.required_keys.extend(additional_required_keys) |
1758 | 39 | self.get_data() | 57 | self.get_data() |
1759 | 40 | 58 | ||
1760 | @@ -118,7 +136,10 @@ | |||
1761 | 118 | """ | 136 | """ |
1762 | 119 | name = 'db' | 137 | name = 'db' |
1763 | 120 | interface = 'mysql' | 138 | interface = 'mysql' |
1765 | 121 | required_keys = ['host', 'user', 'password', 'database'] | 139 | |
1766 | 140 | def __init__(self, *args, **kwargs): | ||
1767 | 141 | self.required_keys = ['host', 'user', 'password', 'database'] | ||
1768 | 142 | RelationContext.__init__(self, *args, **kwargs) | ||
1769 | 122 | 143 | ||
1770 | 123 | 144 | ||
1771 | 124 | class HttpRelation(RelationContext): | 145 | class HttpRelation(RelationContext): |
1772 | @@ -130,7 +151,10 @@ | |||
1773 | 130 | """ | 151 | """ |
1774 | 131 | name = 'website' | 152 | name = 'website' |
1775 | 132 | interface = 'http' | 153 | interface = 'http' |
1777 | 133 | required_keys = ['host', 'port'] | 154 | |
1778 | 155 | def __init__(self, *args, **kwargs): | ||
1779 | 156 | self.required_keys = ['host', 'port'] | ||
1780 | 157 | RelationContext.__init__(self, *args, **kwargs) | ||
1781 | 134 | 158 | ||
1782 | 135 | def provide_data(self): | 159 | def provide_data(self): |
1783 | 136 | return { | 160 | return { |
1784 | @@ -196,7 +220,7 @@ | |||
1785 | 196 | if not os.path.isabs(file_name): | 220 | if not os.path.isabs(file_name): |
1786 | 197 | file_name = os.path.join(hookenv.charm_dir(), file_name) | 221 | file_name = os.path.join(hookenv.charm_dir(), file_name) |
1787 | 198 | with open(file_name, 'w') as file_stream: | 222 | with open(file_name, 'w') as file_stream: |
1789 | 199 | os.fchmod(file_stream.fileno(), 0600) | 223 | os.fchmod(file_stream.fileno(), 0o600) |
1790 | 200 | yaml.dump(config_data, file_stream) | 224 | yaml.dump(config_data, file_stream) |
1791 | 201 | 225 | ||
1792 | 202 | def read_context(self, file_name): | 226 | def read_context(self, file_name): |
1793 | @@ -211,15 +235,19 @@ | |||
1794 | 211 | 235 | ||
1795 | 212 | class TemplateCallback(ManagerCallback): | 236 | class TemplateCallback(ManagerCallback): |
1796 | 213 | """ | 237 | """ |
1800 | 214 | Callback class that will render a Jinja2 template, for use as a ready action. | 238 | Callback class that will render a Jinja2 template, for use as a ready |
1801 | 215 | 239 | action. | |
1802 | 216 | :param str source: The template source file, relative to `$CHARM_DIR/templates` | 240 | |
1803 | 241 | :param str source: The template source file, relative to | ||
1804 | 242 | `$CHARM_DIR/templates` | ||
1805 | 243 | |||
1806 | 217 | :param str target: The target to write the rendered template to | 244 | :param str target: The target to write the rendered template to |
1807 | 218 | :param str owner: The owner of the rendered file | 245 | :param str owner: The owner of the rendered file |
1808 | 219 | :param str group: The group of the rendered file | 246 | :param str group: The group of the rendered file |
1809 | 220 | :param int perms: The permissions of the rendered file | 247 | :param int perms: The permissions of the rendered file |
1810 | 221 | """ | 248 | """ |
1812 | 222 | def __init__(self, source, target, owner='root', group='root', perms=0444): | 249 | def __init__(self, source, target, |
1813 | 250 | owner='root', group='root', perms=0o444): | ||
1814 | 223 | self.source = source | 251 | self.source = source |
1815 | 224 | self.target = target | 252 | self.target = target |
1816 | 225 | self.owner = owner | 253 | self.owner = owner |
1817 | 226 | 254 | ||
1818 | === added file 'lib/charmhelpers/core/strutils.py' | |||
1819 | --- lib/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000 | |||
1820 | +++ lib/charmhelpers/core/strutils.py 2015-11-01 20:32:57 +0000 | |||
1821 | @@ -0,0 +1,42 @@ | |||
1822 | 1 | #!/usr/bin/env python | ||
1823 | 2 | # -*- coding: utf-8 -*- | ||
1824 | 3 | |||
1825 | 4 | # Copyright 2014-2015 Canonical Limited. | ||
1826 | 5 | # | ||
1827 | 6 | # This file is part of charm-helpers. | ||
1828 | 7 | # | ||
1829 | 8 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1830 | 9 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1831 | 10 | # published by the Free Software Foundation. | ||
1832 | 11 | # | ||
1833 | 12 | # charm-helpers is distributed in the hope that it will be useful, | ||
1834 | 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1835 | 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1836 | 15 | # GNU Lesser General Public License for more details. | ||
1837 | 16 | # | ||
1838 | 17 | # You should have received a copy of the GNU Lesser General Public License | ||
1839 | 18 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1840 | 19 | |||
1841 | 20 | import six | ||
1842 | 21 | |||
1843 | 22 | |||
1844 | 23 | def bool_from_string(value): | ||
1845 | 24 | """Interpret string value as boolean. | ||
1846 | 25 | |||
1847 | 26 | Returns True if value translates to True otherwise False. | ||
1848 | 27 | """ | ||
1849 | 28 | if isinstance(value, six.string_types): | ||
1850 | 29 | value = six.text_type(value) | ||
1851 | 30 | else: | ||
1852 | 31 | msg = "Unable to interpret non-string value '%s' as boolean" % (value) | ||
1853 | 32 | raise ValueError(msg) | ||
1854 | 33 | |||
1855 | 34 | value = value.strip().lower() | ||
1856 | 35 | |||
1857 | 36 | if value in ['y', 'yes', 'true', 't', 'on']: | ||
1858 | 37 | return True | ||
1859 | 38 | elif value in ['n', 'no', 'false', 'f', 'off']: | ||
1860 | 39 | return False | ||
1861 | 40 | |||
1862 | 41 | msg = "Unable to interpret string value '%s' as boolean" % (value) | ||
1863 | 42 | raise ValueError(msg) | ||
1864 | 0 | 43 | ||
1865 | === added file 'lib/charmhelpers/core/sysctl.py' | |||
1866 | --- lib/charmhelpers/core/sysctl.py 1970-01-01 00:00:00 +0000 | |||
1867 | +++ lib/charmhelpers/core/sysctl.py 2015-11-01 20:32:57 +0000 | |||
1868 | @@ -0,0 +1,56 @@ | |||
1869 | 1 | #!/usr/bin/env python | ||
1870 | 2 | # -*- coding: utf-8 -*- | ||
1871 | 3 | |||
1872 | 4 | # Copyright 2014-2015 Canonical Limited. | ||
1873 | 5 | # | ||
1874 | 6 | # This file is part of charm-helpers. | ||
1875 | 7 | # | ||
1876 | 8 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1877 | 9 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1878 | 10 | # published by the Free Software Foundation. | ||
1879 | 11 | # | ||
1880 | 12 | # charm-helpers is distributed in the hope that it will be useful, | ||
1881 | 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1882 | 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1883 | 15 | # GNU Lesser General Public License for more details. | ||
1884 | 16 | # | ||
1885 | 17 | # You should have received a copy of the GNU Lesser General Public License | ||
1886 | 18 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1887 | 19 | |||
1888 | 20 | import yaml | ||
1889 | 21 | |||
1890 | 22 | from subprocess import check_call | ||
1891 | 23 | |||
1892 | 24 | from charmhelpers.core.hookenv import ( | ||
1893 | 25 | log, | ||
1894 | 26 | DEBUG, | ||
1895 | 27 | ERROR, | ||
1896 | 28 | ) | ||
1897 | 29 | |||
1898 | 30 | __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' | ||
1899 | 31 | |||
1900 | 32 | |||
1901 | 33 | def create(sysctl_dict, sysctl_file): | ||
1902 | 34 | """Creates a sysctl.conf file from a YAML associative array | ||
1903 | 35 | |||
1904 | 36 | :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }" | ||
1905 | 37 | :type sysctl_dict: str | ||
1906 | 38 | :param sysctl_file: path to the sysctl file to be saved | ||
1907 | 39 | :type sysctl_file: str or unicode | ||
1908 | 40 | :returns: None | ||
1909 | 41 | """ | ||
1910 | 42 | try: | ||
1911 | 43 | sysctl_dict_parsed = yaml.safe_load(sysctl_dict) | ||
1912 | 44 | except yaml.YAMLError: | ||
1913 | 45 | log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict), | ||
1914 | 46 | level=ERROR) | ||
1915 | 47 | return | ||
1916 | 48 | |||
1917 | 49 | with open(sysctl_file, "w") as fd: | ||
1918 | 50 | for key, value in sysctl_dict_parsed.items(): | ||
1919 | 51 | fd.write("{}={}\n".format(key, value)) | ||
1920 | 52 | |||
1921 | 53 | log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed), | ||
1922 | 54 | level=DEBUG) | ||
1923 | 55 | |||
1924 | 56 | check_call(["sysctl", "-p", sysctl_file]) | ||
1925 | 0 | 57 | ||
1926 | === modified file 'lib/charmhelpers/core/templating.py' | |||
1927 | --- lib/charmhelpers/core/templating.py 2014-09-23 12:09:14 +0000 | |||
1928 | +++ lib/charmhelpers/core/templating.py 2015-11-01 20:32:57 +0000 | |||
1929 | @@ -1,10 +1,27 @@ | |||
1930 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
1931 | 2 | # | ||
1932 | 3 | # This file is part of charm-helpers. | ||
1933 | 4 | # | ||
1934 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1935 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1936 | 7 | # published by the Free Software Foundation. | ||
1937 | 8 | # | ||
1938 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
1939 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1940 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1941 | 12 | # GNU Lesser General Public License for more details. | ||
1942 | 13 | # | ||
1943 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
1944 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1945 | 16 | |||
1946 | 1 | import os | 17 | import os |
1947 | 2 | 18 | ||
1948 | 3 | from charmhelpers.core import host | 19 | from charmhelpers.core import host |
1949 | 4 | from charmhelpers.core import hookenv | 20 | from charmhelpers.core import hookenv |
1950 | 5 | 21 | ||
1951 | 6 | 22 | ||
1953 | 7 | def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None): | 23 | def render(source, target, context, owner='root', group='root', |
1954 | 24 | perms=0o444, templates_dir=None, encoding='UTF-8'): | ||
1955 | 8 | """ | 25 | """ |
1956 | 9 | Render a template. | 26 | Render a template. |
1957 | 10 | 27 | ||
1958 | @@ -47,5 +64,5 @@ | |||
1959 | 47 | level=hookenv.ERROR) | 64 | level=hookenv.ERROR) |
1960 | 48 | raise e | 65 | raise e |
1961 | 49 | content = template.render(context) | 66 | content = template.render(context) |
1964 | 50 | host.mkdir(os.path.dirname(target)) | 67 | host.mkdir(os.path.dirname(target), owner, group, perms=0o755) |
1965 | 51 | host.write_file(target, content, owner, group, perms) | 68 | host.write_file(target, content.encode(encoding), owner, group, perms) |
1966 | 52 | 69 | ||
1967 | === added file 'lib/charmhelpers/core/unitdata.py' | |||
1968 | --- lib/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000 | |||
1969 | +++ lib/charmhelpers/core/unitdata.py 2015-11-01 20:32:57 +0000 | |||
1970 | @@ -0,0 +1,477 @@ | |||
1971 | 1 | #!/usr/bin/env python | ||
1972 | 2 | # -*- coding: utf-8 -*- | ||
1973 | 3 | # | ||
1974 | 4 | # Copyright 2014-2015 Canonical Limited. | ||
1975 | 5 | # | ||
1976 | 6 | # This file is part of charm-helpers. | ||
1977 | 7 | # | ||
1978 | 8 | # charm-helpers is free software: you can redistribute it and/or modify | ||
1979 | 9 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
1980 | 10 | # published by the Free Software Foundation. | ||
1981 | 11 | # | ||
1982 | 12 | # charm-helpers is distributed in the hope that it will be useful, | ||
1983 | 13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1984 | 14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1985 | 15 | # GNU Lesser General Public License for more details. | ||
1986 | 16 | # | ||
1987 | 17 | # You should have received a copy of the GNU Lesser General Public License | ||
1988 | 18 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
1989 | 19 | # | ||
1990 | 20 | # | ||
1991 | 21 | # Authors: | ||
1992 | 22 | # Kapil Thangavelu <kapil.foss@gmail.com> | ||
1993 | 23 | # | ||
1994 | 24 | """ | ||
1995 | 25 | Intro | ||
1996 | 26 | ----- | ||
1997 | 27 | |||
1998 | 28 | A simple way to store state in units. This provides a key value | ||
1999 | 29 | storage with support for versioned, transactional operation, | ||
2000 | 30 | and can calculate deltas from previous values to simplify unit logic | ||
2001 | 31 | when processing changes. | ||
2002 | 32 | |||
2003 | 33 | |||
2004 | 34 | Hook Integration | ||
2005 | 35 | ---------------- | ||
2006 | 36 | |||
2007 | 37 | There are several extant frameworks for hook execution, including | ||
2008 | 38 | |||
2009 | 39 | - charmhelpers.core.hookenv.Hooks | ||
2010 | 40 | - charmhelpers.core.services.ServiceManager | ||
2011 | 41 | |||
2012 | 42 | The storage classes are framework agnostic, one simple integration is | ||
2013 | 43 | via the HookData contextmanager. It will record the current hook | ||
2014 | 44 | execution environment (including relation data, config data, etc.), | ||
2015 | 45 | setup a transaction and allow easy access to the changes from | ||
2016 | 46 | previously seen values. One consequence of the integration is the | ||
2017 | 47 | reservation of particular keys ('rels', 'unit', 'env', 'config', | ||
2018 | 48 | 'charm_revisions') for their respective values. | ||
2019 | 49 | |||
2020 | 50 | Here's a fully worked integration example using hookenv.Hooks:: | ||
2021 | 51 | |||
2022 | 52 | from charmhelper.core import hookenv, unitdata | ||
2023 | 53 | |||
2024 | 54 | hook_data = unitdata.HookData() | ||
2025 | 55 | db = unitdata.kv() | ||
2026 | 56 | hooks = hookenv.Hooks() | ||
2027 | 57 | |||
2028 | 58 | @hooks.hook | ||
2029 | 59 | def config_changed(): | ||
2030 | 60 | # Print all changes to configuration from previously seen | ||
2031 | 61 | # values. | ||
2032 | 62 | for changed, (prev, cur) in hook_data.conf.items(): | ||
2033 | 63 | print('config changed', changed, | ||
2034 | 64 | 'previous value', prev, | ||
2035 | 65 | 'current value', cur) | ||
2036 | 66 | |||
2037 | 67 | # Get some unit specific bookeeping | ||
2038 | 68 | if not db.get('pkg_key'): | ||
2039 | 69 | key = urllib.urlopen('https://example.com/pkg_key').read() | ||
2040 | 70 | db.set('pkg_key', key) | ||
2041 | 71 | |||
2042 | 72 | # Directly access all charm config as a mapping. | ||
2043 | 73 | conf = db.getrange('config', True) | ||
2044 | 74 | |||
2045 | 75 | # Directly access all relation data as a mapping | ||
2046 | 76 | rels = db.getrange('rels', True) | ||
2047 | 77 | |||
2048 | 78 | if __name__ == '__main__': | ||
2049 | 79 | with hook_data(): | ||
2050 | 80 | hook.execute() | ||
2051 | 81 | |||
2052 | 82 | |||
2053 | 83 | A more basic integration is via the hook_scope context manager which simply | ||
2054 | 84 | manages transaction scope (and records hook name, and timestamp):: | ||
2055 | 85 | |||
2056 | 86 | >>> from unitdata import kv | ||
2057 | 87 | >>> db = kv() | ||
2058 | 88 | >>> with db.hook_scope('install'): | ||
2059 | 89 | ... # do work, in transactional scope. | ||
2060 | 90 | ... db.set('x', 1) | ||
2061 | 91 | >>> db.get('x') | ||
2062 | 92 | 1 | ||
2063 | 93 | |||
2064 | 94 | |||
2065 | 95 | Usage | ||
2066 | 96 | ----- | ||
2067 | 97 | |||
2068 | 98 | Values are automatically json de/serialized to preserve basic typing | ||
2069 | 99 | and complex data struct capabilities (dicts, lists, ints, booleans, etc). | ||
2070 | 100 | |||
2071 | 101 | Individual values can be manipulated via get/set:: | ||
2072 | 102 | |||
2073 | 103 | >>> kv.set('y', True) | ||
2074 | 104 | >>> kv.get('y') | ||
2075 | 105 | True | ||
2076 | 106 | |||
2077 | 107 | # We can set complex values (dicts, lists) as a single key. | ||
2078 | 108 | >>> kv.set('config', {'a': 1, 'b': True'}) | ||
2079 | 109 | |||
2080 | 110 | # Also supports returning dictionaries as a record which | ||
2081 | 111 | # provides attribute access. | ||
2082 | 112 | >>> config = kv.get('config', record=True) | ||
2083 | 113 | >>> config.b | ||
2084 | 114 | True | ||
2085 | 115 | |||
2086 | 116 | |||
2087 | 117 | Groups of keys can be manipulated with update/getrange:: | ||
2088 | 118 | |||
2089 | 119 | >>> kv.update({'z': 1, 'y': 2}, prefix="gui.") | ||
2090 | 120 | >>> kv.getrange('gui.', strip=True) | ||
2091 | 121 | {'z': 1, 'y': 2} | ||
2092 | 122 | |||
2093 | 123 | When updating values, its very helpful to understand which values | ||
2094 | 124 | have actually changed and how have they changed. The storage | ||
2095 | 125 | provides a delta method to provide for this:: | ||
2096 | 126 | |||
2097 | 127 | >>> data = {'debug': True, 'option': 2} | ||
2098 | 128 | >>> delta = kv.delta(data, 'config.') | ||
2099 | 129 | >>> delta.debug.previous | ||
2100 | 130 | None | ||
2101 | 131 | >>> delta.debug.current | ||
2102 | 132 | True | ||
2103 | 133 | >>> delta | ||
2104 | 134 | {'debug': (None, True), 'option': (None, 2)} | ||
2105 | 135 | |||
2106 | 136 | Note the delta method does not persist the actual change, it needs to | ||
2107 | 137 | be explicitly saved via 'update' method:: | ||
2108 | 138 | |||
2109 | 139 | >>> kv.update(data, 'config.') | ||
2110 | 140 | |||
2111 | 141 | Values modified in the context of a hook scope retain historical values | ||
2112 | 142 | associated to the hookname. | ||
2113 | 143 | |||
2114 | 144 | >>> with db.hook_scope('config-changed'): | ||
2115 | 145 | ... db.set('x', 42) | ||
2116 | 146 | >>> db.gethistory('x') | ||
2117 | 147 | [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'), | ||
2118 | 148 | (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')] | ||
2119 | 149 | |||
2120 | 150 | """ | ||
2121 | 151 | |||
2122 | 152 | import collections | ||
2123 | 153 | import contextlib | ||
2124 | 154 | import datetime | ||
2125 | 155 | import json | ||
2126 | 156 | import os | ||
2127 | 157 | import pprint | ||
2128 | 158 | import sqlite3 | ||
2129 | 159 | import sys | ||
2130 | 160 | |||
2131 | 161 | __author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>' | ||
2132 | 162 | |||
2133 | 163 | |||
2134 | 164 | class Storage(object): | ||
2135 | 165 | """Simple key value database for local unit state within charms. | ||
2136 | 166 | |||
2137 | 167 | Modifications are automatically committed at hook exit. That's | ||
2138 | 168 | currently regardless of exit code. | ||
2139 | 169 | |||
2140 | 170 | To support dicts, lists, integer, floats, and booleans values | ||
2141 | 171 | are automatically json encoded/decoded. | ||
2142 | 172 | """ | ||
2143 | 173 | def __init__(self, path=None): | ||
2144 | 174 | self.db_path = path | ||
2145 | 175 | if path is None: | ||
2146 | 176 | self.db_path = os.path.join( | ||
2147 | 177 | os.environ.get('CHARM_DIR', ''), '.unit-state.db') | ||
2148 | 178 | self.conn = sqlite3.connect('%s' % self.db_path) | ||
2149 | 179 | self.cursor = self.conn.cursor() | ||
2150 | 180 | self.revision = None | ||
2151 | 181 | self._closed = False | ||
2152 | 182 | self._init() | ||
2153 | 183 | |||
2154 | 184 | def close(self): | ||
2155 | 185 | if self._closed: | ||
2156 | 186 | return | ||
2157 | 187 | self.flush(False) | ||
2158 | 188 | self.cursor.close() | ||
2159 | 189 | self.conn.close() | ||
2160 | 190 | self._closed = True | ||
2161 | 191 | |||
2162 | 192 | def _scoped_query(self, stmt, params=None): | ||
2163 | 193 | if params is None: | ||
2164 | 194 | params = [] | ||
2165 | 195 | return stmt, params | ||
2166 | 196 | |||
2167 | 197 | def get(self, key, default=None, record=False): | ||
2168 | 198 | self.cursor.execute( | ||
2169 | 199 | *self._scoped_query( | ||
2170 | 200 | 'select data from kv where key=?', [key])) | ||
2171 | 201 | result = self.cursor.fetchone() | ||
2172 | 202 | if not result: | ||
2173 | 203 | return default | ||
2174 | 204 | if record: | ||
2175 | 205 | return Record(json.loads(result[0])) | ||
2176 | 206 | return json.loads(result[0]) | ||
2177 | 207 | |||
2178 | 208 | def getrange(self, key_prefix, strip=False): | ||
2179 | 209 | stmt = "select key, data from kv where key like '%s%%'" % key_prefix | ||
2180 | 210 | self.cursor.execute(*self._scoped_query(stmt)) | ||
2181 | 211 | result = self.cursor.fetchall() | ||
2182 | 212 | |||
2183 | 213 | if not result: | ||
2184 | 214 | return None | ||
2185 | 215 | if not strip: | ||
2186 | 216 | key_prefix = '' | ||
2187 | 217 | return dict([ | ||
2188 | 218 | (k[len(key_prefix):], json.loads(v)) for k, v in result]) | ||
2189 | 219 | |||
2190 | 220 | def update(self, mapping, prefix=""): | ||
2191 | 221 | for k, v in mapping.items(): | ||
2192 | 222 | self.set("%s%s" % (prefix, k), v) | ||
2193 | 223 | |||
2194 | 224 | def unset(self, key): | ||
2195 | 225 | self.cursor.execute('delete from kv where key=?', [key]) | ||
2196 | 226 | if self.revision and self.cursor.rowcount: | ||
2197 | 227 | self.cursor.execute( | ||
2198 | 228 | 'insert into kv_revisions values (?, ?, ?)', | ||
2199 | 229 | [key, self.revision, json.dumps('DELETED')]) | ||
2200 | 230 | |||
2201 | 231 | def set(self, key, value): | ||
2202 | 232 | serialized = json.dumps(value) | ||
2203 | 233 | |||
2204 | 234 | self.cursor.execute( | ||
2205 | 235 | 'select data from kv where key=?', [key]) | ||
2206 | 236 | exists = self.cursor.fetchone() | ||
2207 | 237 | |||
2208 | 238 | # Skip mutations to the same value | ||
2209 | 239 | if exists: | ||
2210 | 240 | if exists[0] == serialized: | ||
2211 | 241 | return value | ||
2212 | 242 | |||
2213 | 243 | if not exists: | ||
2214 | 244 | self.cursor.execute( | ||
2215 | 245 | 'insert into kv (key, data) values (?, ?)', | ||
2216 | 246 | (key, serialized)) | ||
2217 | 247 | else: | ||
2218 | 248 | self.cursor.execute(''' | ||
2219 | 249 | update kv | ||
2220 | 250 | set data = ? | ||
2221 | 251 | where key = ?''', [serialized, key]) | ||
2222 | 252 | |||
2223 | 253 | # Save | ||
2224 | 254 | if not self.revision: | ||
2225 | 255 | return value | ||
2226 | 256 | |||
2227 | 257 | self.cursor.execute( | ||
2228 | 258 | 'select 1 from kv_revisions where key=? and revision=?', | ||
2229 | 259 | [key, self.revision]) | ||
2230 | 260 | exists = self.cursor.fetchone() | ||
2231 | 261 | |||
2232 | 262 | if not exists: | ||
2233 | 263 | self.cursor.execute( | ||
2234 | 264 | '''insert into kv_revisions ( | ||
2235 | 265 | revision, key, data) values (?, ?, ?)''', | ||
2236 | 266 | (self.revision, key, serialized)) | ||
2237 | 267 | else: | ||
2238 | 268 | self.cursor.execute( | ||
2239 | 269 | ''' | ||
2240 | 270 | update kv_revisions | ||
2241 | 271 | set data = ? | ||
2242 | 272 | where key = ? | ||
2243 | 273 | and revision = ?''', | ||
2244 | 274 | [serialized, key, self.revision]) | ||
2245 | 275 | |||
2246 | 276 | return value | ||
2247 | 277 | |||
2248 | 278 | def delta(self, mapping, prefix): | ||
2249 | 279 | """ | ||
2250 | 280 | return a delta containing values that have changed. | ||
2251 | 281 | """ | ||
2252 | 282 | previous = self.getrange(prefix, strip=True) | ||
2253 | 283 | if not previous: | ||
2254 | 284 | pk = set() | ||
2255 | 285 | else: | ||
2256 | 286 | pk = set(previous.keys()) | ||
2257 | 287 | ck = set(mapping.keys()) | ||
2258 | 288 | delta = DeltaSet() | ||
2259 | 289 | |||
2260 | 290 | # added | ||
2261 | 291 | for k in ck.difference(pk): | ||
2262 | 292 | delta[k] = Delta(None, mapping[k]) | ||
2263 | 293 | |||
2264 | 294 | # removed | ||
2265 | 295 | for k in pk.difference(ck): | ||
2266 | 296 | delta[k] = Delta(previous[k], None) | ||
2267 | 297 | |||
2268 | 298 | # changed | ||
2269 | 299 | for k in pk.intersection(ck): | ||
2270 | 300 | c = mapping[k] | ||
2271 | 301 | p = previous[k] | ||
2272 | 302 | if c != p: | ||
2273 | 303 | delta[k] = Delta(p, c) | ||
2274 | 304 | |||
2275 | 305 | return delta | ||
2276 | 306 | |||
2277 | 307 | @contextlib.contextmanager | ||
2278 | 308 | def hook_scope(self, name=""): | ||
2279 | 309 | """Scope all future interactions to the current hook execution | ||
2280 | 310 | revision.""" | ||
2281 | 311 | assert not self.revision | ||
2282 | 312 | self.cursor.execute( | ||
2283 | 313 | 'insert into hooks (hook, date) values (?, ?)', | ||
2284 | 314 | (name or sys.argv[0], | ||
2285 | 315 | datetime.datetime.utcnow().isoformat())) | ||
2286 | 316 | self.revision = self.cursor.lastrowid | ||
2287 | 317 | try: | ||
2288 | 318 | yield self.revision | ||
2289 | 319 | self.revision = None | ||
2290 | 320 | except: | ||
2291 | 321 | self.flush(False) | ||
2292 | 322 | self.revision = None | ||
2293 | 323 | raise | ||
2294 | 324 | else: | ||
2295 | 325 | self.flush() | ||
2296 | 326 | |||
2297 | 327 | def flush(self, save=True): | ||
2298 | 328 | if save: | ||
2299 | 329 | self.conn.commit() | ||
2300 | 330 | elif self._closed: | ||
2301 | 331 | return | ||
2302 | 332 | else: | ||
2303 | 333 | self.conn.rollback() | ||
2304 | 334 | |||
2305 | 335 | def _init(self): | ||
2306 | 336 | self.cursor.execute(''' | ||
2307 | 337 | create table if not exists kv ( | ||
2308 | 338 | key text, | ||
2309 | 339 | data text, | ||
2310 | 340 | primary key (key) | ||
2311 | 341 | )''') | ||
2312 | 342 | self.cursor.execute(''' | ||
2313 | 343 | create table if not exists kv_revisions ( | ||
2314 | 344 | key text, | ||
2315 | 345 | revision integer, | ||
2316 | 346 | data text, | ||
2317 | 347 | primary key (key, revision) | ||
2318 | 348 | )''') | ||
2319 | 349 | self.cursor.execute(''' | ||
2320 | 350 | create table if not exists hooks ( | ||
2321 | 351 | version integer primary key autoincrement, | ||
2322 | 352 | hook text, | ||
2323 | 353 | date text | ||
2324 | 354 | )''') | ||
2325 | 355 | self.conn.commit() | ||
2326 | 356 | |||
2327 | 357 | def gethistory(self, key, deserialize=False): | ||
2328 | 358 | self.cursor.execute( | ||
2329 | 359 | ''' | ||
2330 | 360 | select kv.revision, kv.key, kv.data, h.hook, h.date | ||
2331 | 361 | from kv_revisions kv, | ||
2332 | 362 | hooks h | ||
2333 | 363 | where kv.key=? | ||
2334 | 364 | and kv.revision = h.version | ||
2335 | 365 | ''', [key]) | ||
2336 | 366 | if deserialize is False: | ||
2337 | 367 | return self.cursor.fetchall() | ||
2338 | 368 | return map(_parse_history, self.cursor.fetchall()) | ||
2339 | 369 | |||
2340 | 370 | def debug(self, fh=sys.stderr): | ||
2341 | 371 | self.cursor.execute('select * from kv') | ||
2342 | 372 | pprint.pprint(self.cursor.fetchall(), stream=fh) | ||
2343 | 373 | self.cursor.execute('select * from kv_revisions') | ||
2344 | 374 | pprint.pprint(self.cursor.fetchall(), stream=fh) | ||
2345 | 375 | |||
2346 | 376 | |||
2347 | 377 | def _parse_history(d): | ||
2348 | 378 | return (d[0], d[1], json.loads(d[2]), d[3], | ||
2349 | 379 | datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f")) | ||
2350 | 380 | |||
2351 | 381 | |||
2352 | 382 | class HookData(object): | ||
2353 | 383 | """Simple integration for existing hook exec frameworks. | ||
2354 | 384 | |||
2355 | 385 | Records all unit information, and stores deltas for processing | ||
2356 | 386 | by the hook. | ||
2357 | 387 | |||
2358 | 388 | Sample:: | ||
2359 | 389 | |||
2360 | 390 | from charmhelper.core import hookenv, unitdata | ||
2361 | 391 | |||
2362 | 392 | changes = unitdata.HookData() | ||
2363 | 393 | db = unitdata.kv() | ||
2364 | 394 | hooks = hookenv.Hooks() | ||
2365 | 395 | |||
2366 | 396 | @hooks.hook | ||
2367 | 397 | def config_changed(): | ||
2368 | 398 | # View all changes to configuration | ||
2369 | 399 | for changed, (prev, cur) in changes.conf.items(): | ||
2370 | 400 | print('config changed', changed, | ||
2371 | 401 | 'previous value', prev, | ||
2372 | 402 | 'current value', cur) | ||
2373 | 403 | |||
2374 | 404 | # Get some unit specific bookeeping | ||
2375 | 405 | if not db.get('pkg_key'): | ||
2376 | 406 | key = urllib.urlopen('https://example.com/pkg_key').read() | ||
2377 | 407 | db.set('pkg_key', key) | ||
2378 | 408 | |||
2379 | 409 | if __name__ == '__main__': | ||
2380 | 410 | with changes(): | ||
2381 | 411 | hook.execute() | ||
2382 | 412 | |||
2383 | 413 | """ | ||
2384 | 414 | def __init__(self): | ||
2385 | 415 | self.kv = kv() | ||
2386 | 416 | self.conf = None | ||
2387 | 417 | self.rels = None | ||
2388 | 418 | |||
2389 | 419 | @contextlib.contextmanager | ||
2390 | 420 | def __call__(self): | ||
2391 | 421 | from charmhelpers.core import hookenv | ||
2392 | 422 | hook_name = hookenv.hook_name() | ||
2393 | 423 | |||
2394 | 424 | with self.kv.hook_scope(hook_name): | ||
2395 | 425 | self._record_charm_version(hookenv.charm_dir()) | ||
2396 | 426 | delta_config, delta_relation = self._record_hook(hookenv) | ||
2397 | 427 | yield self.kv, delta_config, delta_relation | ||
2398 | 428 | |||
2399 | 429 | def _record_charm_version(self, charm_dir): | ||
2400 | 430 | # Record revisions.. charm revisions are meaningless | ||
2401 | 431 | # to charm authors as they don't control the revision. | ||
2402 | 432 | # so logic dependnent on revision is not particularly | ||
2403 | 433 | # useful, however it is useful for debugging analysis. | ||
2404 | 434 | charm_rev = open( | ||
2405 | 435 | os.path.join(charm_dir, 'revision')).read().strip() | ||
2406 | 436 | charm_rev = charm_rev or '0' | ||
2407 | 437 | revs = self.kv.get('charm_revisions', []) | ||
2408 | 438 | if charm_rev not in revs: | ||
2409 | 439 | revs.append(charm_rev.strip() or '0') | ||
2410 | 440 | self.kv.set('charm_revisions', revs) | ||
2411 | 441 | |||
2412 | 442 | def _record_hook(self, hookenv): | ||
2413 | 443 | data = hookenv.execution_environment() | ||
2414 | 444 | self.conf = conf_delta = self.kv.delta(data['conf'], 'config') | ||
2415 | 445 | self.rels = rels_delta = self.kv.delta(data['rels'], 'rels') | ||
2416 | 446 | self.kv.set('env', dict(data['env'])) | ||
2417 | 447 | self.kv.set('unit', data['unit']) | ||
2418 | 448 | self.kv.set('relid', data.get('relid')) | ||
2419 | 449 | return conf_delta, rels_delta | ||
2420 | 450 | |||
2421 | 451 | |||
2422 | 452 | class Record(dict): | ||
2423 | 453 | |||
2424 | 454 | __slots__ = () | ||
2425 | 455 | |||
2426 | 456 | def __getattr__(self, k): | ||
2427 | 457 | if k in self: | ||
2428 | 458 | return self[k] | ||
2429 | 459 | raise AttributeError(k) | ||
2430 | 460 | |||
2431 | 461 | |||
2432 | 462 | class DeltaSet(Record): | ||
2433 | 463 | |||
2434 | 464 | __slots__ = () | ||
2435 | 465 | |||
2436 | 466 | |||
2437 | 467 | Delta = collections.namedtuple('Delta', ['previous', 'current']) | ||
2438 | 468 | |||
2439 | 469 | |||
2440 | 470 | _KV = None | ||
2441 | 471 | |||
2442 | 472 | |||
2443 | 473 | def kv(): | ||
2444 | 474 | global _KV | ||
2445 | 475 | if _KV is None: | ||
2446 | 476 | _KV = Storage() | ||
2447 | 477 | return _KV | ||
2448 | 0 | 478 | ||
2449 | === modified file 'lib/charmhelpers/fetch/__init__.py' | |||
2450 | --- lib/charmhelpers/fetch/__init__.py 2014-09-23 12:09:14 +0000 | |||
2451 | +++ lib/charmhelpers/fetch/__init__.py 2015-11-01 20:32:57 +0000 | |||
2452 | @@ -1,3 +1,19 @@ | |||
2453 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2454 | 2 | # | ||
2455 | 3 | # This file is part of charm-helpers. | ||
2456 | 4 | # | ||
2457 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2458 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2459 | 7 | # published by the Free Software Foundation. | ||
2460 | 8 | # | ||
2461 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2462 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2463 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2464 | 12 | # GNU Lesser General Public License for more details. | ||
2465 | 13 | # | ||
2466 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2467 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2468 | 16 | |||
2469 | 1 | import importlib | 17 | import importlib |
2470 | 2 | from tempfile import NamedTemporaryFile | 18 | from tempfile import NamedTemporaryFile |
2471 | 3 | import time | 19 | import time |
2472 | @@ -5,10 +21,6 @@ | |||
2473 | 5 | from charmhelpers.core.host import ( | 21 | from charmhelpers.core.host import ( |
2474 | 6 | lsb_release | 22 | lsb_release |
2475 | 7 | ) | 23 | ) |
2476 | 8 | from urlparse import ( | ||
2477 | 9 | urlparse, | ||
2478 | 10 | urlunparse, | ||
2479 | 11 | ) | ||
2480 | 12 | import subprocess | 24 | import subprocess |
2481 | 13 | from charmhelpers.core.hookenv import ( | 25 | from charmhelpers.core.hookenv import ( |
2482 | 14 | config, | 26 | config, |
2483 | @@ -16,6 +28,12 @@ | |||
2484 | 16 | ) | 28 | ) |
2485 | 17 | import os | 29 | import os |
2486 | 18 | 30 | ||
2487 | 31 | import six | ||
2488 | 32 | if six.PY3: | ||
2489 | 33 | from urllib.parse import urlparse, urlunparse | ||
2490 | 34 | else: | ||
2491 | 35 | from urlparse import urlparse, urlunparse | ||
2492 | 36 | |||
2493 | 19 | 37 | ||
2494 | 20 | CLOUD_ARCHIVE = """# Ubuntu Cloud Archive | 38 | CLOUD_ARCHIVE = """# Ubuntu Cloud Archive |
2495 | 21 | deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main | 39 | deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main |
2496 | @@ -62,9 +80,16 @@ | |||
2497 | 62 | 'trusty-juno/updates': 'trusty-updates/juno', | 80 | 'trusty-juno/updates': 'trusty-updates/juno', |
2498 | 63 | 'trusty-updates/juno': 'trusty-updates/juno', | 81 | 'trusty-updates/juno': 'trusty-updates/juno', |
2499 | 64 | 'juno/proposed': 'trusty-proposed/juno', | 82 | 'juno/proposed': 'trusty-proposed/juno', |
2500 | 65 | 'juno/proposed': 'trusty-proposed/juno', | ||
2501 | 66 | 'trusty-juno/proposed': 'trusty-proposed/juno', | 83 | 'trusty-juno/proposed': 'trusty-proposed/juno', |
2502 | 67 | 'trusty-proposed/juno': 'trusty-proposed/juno', | 84 | 'trusty-proposed/juno': 'trusty-proposed/juno', |
2503 | 85 | # Kilo | ||
2504 | 86 | 'kilo': 'trusty-updates/kilo', | ||
2505 | 87 | 'trusty-kilo': 'trusty-updates/kilo', | ||
2506 | 88 | 'trusty-kilo/updates': 'trusty-updates/kilo', | ||
2507 | 89 | 'trusty-updates/kilo': 'trusty-updates/kilo', | ||
2508 | 90 | 'kilo/proposed': 'trusty-proposed/kilo', | ||
2509 | 91 | 'trusty-kilo/proposed': 'trusty-proposed/kilo', | ||
2510 | 92 | 'trusty-proposed/kilo': 'trusty-proposed/kilo', | ||
2511 | 68 | } | 93 | } |
2512 | 69 | 94 | ||
2513 | 70 | # The order of this list is very important. Handlers should be listed in from | 95 | # The order of this list is very important. Handlers should be listed in from |
2514 | @@ -72,6 +97,7 @@ | |||
2515 | 72 | FETCH_HANDLERS = ( | 97 | FETCH_HANDLERS = ( |
2516 | 73 | 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler', | 98 | 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler', |
2517 | 74 | 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler', | 99 | 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler', |
2518 | 100 | 'charmhelpers.fetch.giturl.GitUrlFetchHandler', | ||
2519 | 75 | ) | 101 | ) |
2520 | 76 | 102 | ||
2521 | 77 | APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT. | 103 | APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT. |
2522 | @@ -148,7 +174,7 @@ | |||
2523 | 148 | cmd = ['apt-get', '--assume-yes'] | 174 | cmd = ['apt-get', '--assume-yes'] |
2524 | 149 | cmd.extend(options) | 175 | cmd.extend(options) |
2525 | 150 | cmd.append('install') | 176 | cmd.append('install') |
2527 | 151 | if isinstance(packages, basestring): | 177 | if isinstance(packages, six.string_types): |
2528 | 152 | cmd.append(packages) | 178 | cmd.append(packages) |
2529 | 153 | else: | 179 | else: |
2530 | 154 | cmd.extend(packages) | 180 | cmd.extend(packages) |
2531 | @@ -181,7 +207,7 @@ | |||
2532 | 181 | def apt_purge(packages, fatal=False): | 207 | def apt_purge(packages, fatal=False): |
2533 | 182 | """Purge one or more packages""" | 208 | """Purge one or more packages""" |
2534 | 183 | cmd = ['apt-get', '--assume-yes', 'purge'] | 209 | cmd = ['apt-get', '--assume-yes', 'purge'] |
2536 | 184 | if isinstance(packages, basestring): | 210 | if isinstance(packages, six.string_types): |
2537 | 185 | cmd.append(packages) | 211 | cmd.append(packages) |
2538 | 186 | else: | 212 | else: |
2539 | 187 | cmd.extend(packages) | 213 | cmd.extend(packages) |
2540 | @@ -192,7 +218,7 @@ | |||
2541 | 192 | def apt_hold(packages, fatal=False): | 218 | def apt_hold(packages, fatal=False): |
2542 | 193 | """Hold one or more packages""" | 219 | """Hold one or more packages""" |
2543 | 194 | cmd = ['apt-mark', 'hold'] | 220 | cmd = ['apt-mark', 'hold'] |
2545 | 195 | if isinstance(packages, basestring): | 221 | if isinstance(packages, six.string_types): |
2546 | 196 | cmd.append(packages) | 222 | cmd.append(packages) |
2547 | 197 | else: | 223 | else: |
2548 | 198 | cmd.extend(packages) | 224 | cmd.extend(packages) |
2549 | @@ -208,7 +234,8 @@ | |||
2550 | 208 | """Add a package source to this system. | 234 | """Add a package source to this system. |
2551 | 209 | 235 | ||
2552 | 210 | @param source: a URL or sources.list entry, as supported by | 236 | @param source: a URL or sources.list entry, as supported by |
2554 | 211 | add-apt-repository(1). Examples: | 237 | add-apt-repository(1). Examples:: |
2555 | 238 | |||
2556 | 212 | ppa:charmers/example | 239 | ppa:charmers/example |
2557 | 213 | deb https://stub:key@private.example.com/ubuntu trusty main | 240 | deb https://stub:key@private.example.com/ubuntu trusty main |
2558 | 214 | 241 | ||
2559 | @@ -217,6 +244,7 @@ | |||
2560 | 217 | pocket for the release. | 244 | pocket for the release. |
2561 | 218 | 'cloud:' may be used to activate official cloud archive pockets, | 245 | 'cloud:' may be used to activate official cloud archive pockets, |
2562 | 219 | such as 'cloud:icehouse' | 246 | such as 'cloud:icehouse' |
2563 | 247 | 'distro' may be used as a noop | ||
2564 | 220 | 248 | ||
2565 | 221 | @param key: A key to be added to the system's APT keyring and used | 249 | @param key: A key to be added to the system's APT keyring and used |
2566 | 222 | to verify the signatures on packages. Ideally, this should be an | 250 | to verify the signatures on packages. Ideally, this should be an |
2567 | @@ -250,12 +278,14 @@ | |||
2568 | 250 | release = lsb_release()['DISTRIB_CODENAME'] | 278 | release = lsb_release()['DISTRIB_CODENAME'] |
2569 | 251 | with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt: | 279 | with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt: |
2570 | 252 | apt.write(PROPOSED_POCKET.format(release)) | 280 | apt.write(PROPOSED_POCKET.format(release)) |
2571 | 281 | elif source == 'distro': | ||
2572 | 282 | pass | ||
2573 | 253 | else: | 283 | else: |
2575 | 254 | raise SourceConfigError("Unknown source: {!r}".format(source)) | 284 | log("Unknown source: {!r}".format(source)) |
2576 | 255 | 285 | ||
2577 | 256 | if key: | 286 | if key: |
2578 | 257 | if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key: | 287 | if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key: |
2580 | 258 | with NamedTemporaryFile() as key_file: | 288 | with NamedTemporaryFile('w+') as key_file: |
2581 | 259 | key_file.write(key) | 289 | key_file.write(key) |
2582 | 260 | key_file.flush() | 290 | key_file.flush() |
2583 | 261 | key_file.seek(0) | 291 | key_file.seek(0) |
2584 | @@ -292,14 +322,14 @@ | |||
2585 | 292 | sources = safe_load((config(sources_var) or '').strip()) or [] | 322 | sources = safe_load((config(sources_var) or '').strip()) or [] |
2586 | 293 | keys = safe_load((config(keys_var) or '').strip()) or None | 323 | keys = safe_load((config(keys_var) or '').strip()) or None |
2587 | 294 | 324 | ||
2589 | 295 | if isinstance(sources, basestring): | 325 | if isinstance(sources, six.string_types): |
2590 | 296 | sources = [sources] | 326 | sources = [sources] |
2591 | 297 | 327 | ||
2592 | 298 | if keys is None: | 328 | if keys is None: |
2593 | 299 | for source in sources: | 329 | for source in sources: |
2594 | 300 | add_source(source, None) | 330 | add_source(source, None) |
2595 | 301 | else: | 331 | else: |
2597 | 302 | if isinstance(keys, basestring): | 332 | if isinstance(keys, six.string_types): |
2598 | 303 | keys = [keys] | 333 | keys = [keys] |
2599 | 304 | 334 | ||
2600 | 305 | if len(sources) != len(keys): | 335 | if len(sources) != len(keys): |
2601 | @@ -396,7 +426,7 @@ | |||
2602 | 396 | while result is None or result == APT_NO_LOCK: | 426 | while result is None or result == APT_NO_LOCK: |
2603 | 397 | try: | 427 | try: |
2604 | 398 | result = subprocess.check_call(cmd, env=env) | 428 | result = subprocess.check_call(cmd, env=env) |
2606 | 399 | except subprocess.CalledProcessError, e: | 429 | except subprocess.CalledProcessError as e: |
2607 | 400 | retry_count = retry_count + 1 | 430 | retry_count = retry_count + 1 |
2608 | 401 | if retry_count > APT_NO_LOCK_RETRY_COUNT: | 431 | if retry_count > APT_NO_LOCK_RETRY_COUNT: |
2609 | 402 | raise | 432 | raise |
2610 | 403 | 433 | ||
2611 | === modified file 'lib/charmhelpers/fetch/archiveurl.py' | |||
2612 | --- lib/charmhelpers/fetch/archiveurl.py 2014-09-23 12:09:14 +0000 | |||
2613 | +++ lib/charmhelpers/fetch/archiveurl.py 2015-11-01 20:32:57 +0000 | |||
2614 | @@ -1,8 +1,22 @@ | |||
2615 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2616 | 2 | # | ||
2617 | 3 | # This file is part of charm-helpers. | ||
2618 | 4 | # | ||
2619 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2620 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2621 | 7 | # published by the Free Software Foundation. | ||
2622 | 8 | # | ||
2623 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2624 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2625 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2626 | 12 | # GNU Lesser General Public License for more details. | ||
2627 | 13 | # | ||
2628 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2629 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2630 | 16 | |||
2631 | 1 | import os | 17 | import os |
2632 | 2 | import urllib2 | ||
2633 | 3 | from urllib import urlretrieve | ||
2634 | 4 | import urlparse | ||
2635 | 5 | import hashlib | 18 | import hashlib |
2636 | 19 | import re | ||
2637 | 6 | 20 | ||
2638 | 7 | from charmhelpers.fetch import ( | 21 | from charmhelpers.fetch import ( |
2639 | 8 | BaseFetchHandler, | 22 | BaseFetchHandler, |
2640 | @@ -14,6 +28,41 @@ | |||
2641 | 14 | ) | 28 | ) |
2642 | 15 | from charmhelpers.core.host import mkdir, check_hash | 29 | from charmhelpers.core.host import mkdir, check_hash |
2643 | 16 | 30 | ||
2644 | 31 | import six | ||
2645 | 32 | if six.PY3: | ||
2646 | 33 | from urllib.request import ( | ||
2647 | 34 | build_opener, install_opener, urlopen, urlretrieve, | ||
2648 | 35 | HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, | ||
2649 | 36 | ) | ||
2650 | 37 | from urllib.parse import urlparse, urlunparse, parse_qs | ||
2651 | 38 | from urllib.error import URLError | ||
2652 | 39 | else: | ||
2653 | 40 | from urllib import urlretrieve | ||
2654 | 41 | from urllib2 import ( | ||
2655 | 42 | build_opener, install_opener, urlopen, | ||
2656 | 43 | HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, | ||
2657 | 44 | URLError | ||
2658 | 45 | ) | ||
2659 | 46 | from urlparse import urlparse, urlunparse, parse_qs | ||
2660 | 47 | |||
2661 | 48 | |||
2662 | 49 | def splituser(host): | ||
2663 | 50 | '''urllib.splituser(), but six's support of this seems broken''' | ||
2664 | 51 | _userprog = re.compile('^(.*)@(.*)$') | ||
2665 | 52 | match = _userprog.match(host) | ||
2666 | 53 | if match: | ||
2667 | 54 | return match.group(1, 2) | ||
2668 | 55 | return None, host | ||
2669 | 56 | |||
2670 | 57 | |||
2671 | 58 | def splitpasswd(user): | ||
2672 | 59 | '''urllib.splitpasswd(), but six's support of this is missing''' | ||
2673 | 60 | _passwdprog = re.compile('^([^:]*):(.*)$', re.S) | ||
2674 | 61 | match = _passwdprog.match(user) | ||
2675 | 62 | if match: | ||
2676 | 63 | return match.group(1, 2) | ||
2677 | 64 | return user, None | ||
2678 | 65 | |||
2679 | 17 | 66 | ||
2680 | 18 | class ArchiveUrlFetchHandler(BaseFetchHandler): | 67 | class ArchiveUrlFetchHandler(BaseFetchHandler): |
2681 | 19 | """ | 68 | """ |
2682 | @@ -42,20 +91,20 @@ | |||
2683 | 42 | """ | 91 | """ |
2684 | 43 | # propogate all exceptions | 92 | # propogate all exceptions |
2685 | 44 | # URLError, OSError, etc | 93 | # URLError, OSError, etc |
2687 | 45 | proto, netloc, path, params, query, fragment = urlparse.urlparse(source) | 94 | proto, netloc, path, params, query, fragment = urlparse(source) |
2688 | 46 | if proto in ('http', 'https'): | 95 | if proto in ('http', 'https'): |
2690 | 47 | auth, barehost = urllib2.splituser(netloc) | 96 | auth, barehost = splituser(netloc) |
2691 | 48 | if auth is not None: | 97 | if auth is not None: |
2695 | 49 | source = urlparse.urlunparse((proto, barehost, path, params, query, fragment)) | 98 | source = urlunparse((proto, barehost, path, params, query, fragment)) |
2696 | 50 | username, password = urllib2.splitpasswd(auth) | 99 | username, password = splitpasswd(auth) |
2697 | 51 | passman = urllib2.HTTPPasswordMgrWithDefaultRealm() | 100 | passman = HTTPPasswordMgrWithDefaultRealm() |
2698 | 52 | # Realm is set to None in add_password to force the username and password | 101 | # Realm is set to None in add_password to force the username and password |
2699 | 53 | # to be used whatever the realm | 102 | # to be used whatever the realm |
2700 | 54 | passman.add_password(None, source, username, password) | 103 | passman.add_password(None, source, username, password) |
2705 | 55 | authhandler = urllib2.HTTPBasicAuthHandler(passman) | 104 | authhandler = HTTPBasicAuthHandler(passman) |
2706 | 56 | opener = urllib2.build_opener(authhandler) | 105 | opener = build_opener(authhandler) |
2707 | 57 | urllib2.install_opener(opener) | 106 | install_opener(opener) |
2708 | 58 | response = urllib2.urlopen(source) | 107 | response = urlopen(source) |
2709 | 59 | try: | 108 | try: |
2710 | 60 | with open(dest, 'w') as dest_file: | 109 | with open(dest, 'w') as dest_file: |
2711 | 61 | dest_file.write(response.read()) | 110 | dest_file.write(response.read()) |
2712 | @@ -74,33 +123,38 @@ | |||
2713 | 74 | """ | 123 | """ |
2714 | 75 | Download and install an archive file, with optional checksum validation. | 124 | Download and install an archive file, with optional checksum validation. |
2715 | 76 | 125 | ||
2717 | 77 | The checksum can also be given on the :param:`source` URL's fragment. | 126 | The checksum can also be given on the `source` URL's fragment. |
2718 | 78 | For example:: | 127 | For example:: |
2719 | 79 | 128 | ||
2720 | 80 | handler.install('http://example.com/file.tgz#sha1=deadbeef') | 129 | handler.install('http://example.com/file.tgz#sha1=deadbeef') |
2721 | 81 | 130 | ||
2722 | 82 | :param str source: URL pointing to an archive file. | 131 | :param str source: URL pointing to an archive file. |
2725 | 83 | :param str dest: Local destination path to install to. If not given, | 132 | :param str dest: Local destination path to install to. If not given, |
2726 | 84 | installs to `$CHARM_DIR/archives/archive_file_name`. | 133 | installs to `$CHARM_DIR/archives/archive_file_name`. |
2727 | 85 | :param str checksum: If given, validate the archive file after download. | 134 | :param str checksum: If given, validate the archive file after download. |
2731 | 86 | :param str hash_type: Algorithm used to generate :param:`checksum`. | 135 | :param str hash_type: Algorithm used to generate `checksum`. |
2732 | 87 | Can be any hash alrgorithm supported by :mod:`hashlib`, | 136 | Can be any hash alrgorithm supported by :mod:`hashlib`, |
2733 | 88 | such as md5, sha1, sha256, sha512, etc. | 137 | such as md5, sha1, sha256, sha512, etc. |
2734 | 138 | |||
2735 | 89 | """ | 139 | """ |
2736 | 90 | url_parts = self.parse_url(source) | 140 | url_parts = self.parse_url(source) |
2737 | 91 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched') | 141 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched') |
2738 | 92 | if not os.path.exists(dest_dir): | 142 | if not os.path.exists(dest_dir): |
2740 | 93 | mkdir(dest_dir, perms=0755) | 143 | mkdir(dest_dir, perms=0o755) |
2741 | 94 | dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path)) | 144 | dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path)) |
2742 | 95 | try: | 145 | try: |
2743 | 96 | self.download(source, dld_file) | 146 | self.download(source, dld_file) |
2745 | 97 | except urllib2.URLError as e: | 147 | except URLError as e: |
2746 | 98 | raise UnhandledSource(e.reason) | 148 | raise UnhandledSource(e.reason) |
2747 | 99 | except OSError as e: | 149 | except OSError as e: |
2748 | 100 | raise UnhandledSource(e.strerror) | 150 | raise UnhandledSource(e.strerror) |
2750 | 101 | options = urlparse.parse_qs(url_parts.fragment) | 151 | options = parse_qs(url_parts.fragment) |
2751 | 102 | for key, value in options.items(): | 152 | for key, value in options.items(): |
2753 | 103 | if key in hashlib.algorithms: | 153 | if not six.PY3: |
2754 | 154 | algorithms = hashlib.algorithms | ||
2755 | 155 | else: | ||
2756 | 156 | algorithms = hashlib.algorithms_available | ||
2757 | 157 | if key in algorithms: | ||
2758 | 104 | check_hash(dld_file, value, key) | 158 | check_hash(dld_file, value, key) |
2759 | 105 | if checksum: | 159 | if checksum: |
2760 | 106 | check_hash(dld_file, checksum, hash_type) | 160 | check_hash(dld_file, checksum, hash_type) |
2761 | 107 | 161 | ||
2762 | === modified file 'lib/charmhelpers/fetch/bzrurl.py' | |||
2763 | --- lib/charmhelpers/fetch/bzrurl.py 2014-07-17 16:38:17 +0000 | |||
2764 | +++ lib/charmhelpers/fetch/bzrurl.py 2015-11-01 20:32:57 +0000 | |||
2765 | @@ -1,3 +1,19 @@ | |||
2766 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2767 | 2 | # | ||
2768 | 3 | # This file is part of charm-helpers. | ||
2769 | 4 | # | ||
2770 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2771 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2772 | 7 | # published by the Free Software Foundation. | ||
2773 | 8 | # | ||
2774 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2775 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2776 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2777 | 12 | # GNU Lesser General Public License for more details. | ||
2778 | 13 | # | ||
2779 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2780 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2781 | 16 | |||
2782 | 1 | import os | 17 | import os |
2783 | 2 | from charmhelpers.fetch import ( | 18 | from charmhelpers.fetch import ( |
2784 | 3 | BaseFetchHandler, | 19 | BaseFetchHandler, |
2785 | @@ -5,12 +21,18 @@ | |||
2786 | 5 | ) | 21 | ) |
2787 | 6 | from charmhelpers.core.host import mkdir | 22 | from charmhelpers.core.host import mkdir |
2788 | 7 | 23 | ||
2789 | 24 | import six | ||
2790 | 25 | if six.PY3: | ||
2791 | 26 | raise ImportError('bzrlib does not support Python3') | ||
2792 | 27 | |||
2793 | 8 | try: | 28 | try: |
2794 | 9 | from bzrlib.branch import Branch | 29 | from bzrlib.branch import Branch |
2795 | 30 | from bzrlib import bzrdir, workingtree, errors | ||
2796 | 10 | except ImportError: | 31 | except ImportError: |
2797 | 11 | from charmhelpers.fetch import apt_install | 32 | from charmhelpers.fetch import apt_install |
2798 | 12 | apt_install("python-bzrlib") | 33 | apt_install("python-bzrlib") |
2799 | 13 | from bzrlib.branch import Branch | 34 | from bzrlib.branch import Branch |
2800 | 35 | from bzrlib import bzrdir, workingtree, errors | ||
2801 | 14 | 36 | ||
2802 | 15 | 37 | ||
2803 | 16 | class BzrUrlFetchHandler(BaseFetchHandler): | 38 | class BzrUrlFetchHandler(BaseFetchHandler): |
2804 | @@ -31,8 +53,14 @@ | |||
2805 | 31 | from bzrlib.plugin import load_plugins | 53 | from bzrlib.plugin import load_plugins |
2806 | 32 | load_plugins() | 54 | load_plugins() |
2807 | 33 | try: | 55 | try: |
2808 | 56 | local_branch = bzrdir.BzrDir.create_branch_convenience(dest) | ||
2809 | 57 | except errors.AlreadyControlDirError: | ||
2810 | 58 | local_branch = Branch.open(dest) | ||
2811 | 59 | try: | ||
2812 | 34 | remote_branch = Branch.open(source) | 60 | remote_branch = Branch.open(source) |
2814 | 35 | remote_branch.bzrdir.sprout(dest).open_branch() | 61 | remote_branch.push(local_branch) |
2815 | 62 | tree = workingtree.WorkingTree.open(dest) | ||
2816 | 63 | tree.update() | ||
2817 | 36 | except Exception as e: | 64 | except Exception as e: |
2818 | 37 | raise e | 65 | raise e |
2819 | 38 | 66 | ||
2820 | @@ -42,7 +70,7 @@ | |||
2821 | 42 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched", | 70 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched", |
2822 | 43 | branch_name) | 71 | branch_name) |
2823 | 44 | if not os.path.exists(dest_dir): | 72 | if not os.path.exists(dest_dir): |
2825 | 45 | mkdir(dest_dir, perms=0755) | 73 | mkdir(dest_dir, perms=0o755) |
2826 | 46 | try: | 74 | try: |
2827 | 47 | self.branch(source, dest_dir) | 75 | self.branch(source, dest_dir) |
2828 | 48 | except OSError as e: | 76 | except OSError as e: |
2829 | 49 | 77 | ||
2830 | === added file 'lib/charmhelpers/fetch/giturl.py' | |||
2831 | --- lib/charmhelpers/fetch/giturl.py 1970-01-01 00:00:00 +0000 | |||
2832 | +++ lib/charmhelpers/fetch/giturl.py 2015-11-01 20:32:57 +0000 | |||
2833 | @@ -0,0 +1,71 @@ | |||
2834 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2835 | 2 | # | ||
2836 | 3 | # This file is part of charm-helpers. | ||
2837 | 4 | # | ||
2838 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2839 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2840 | 7 | # published by the Free Software Foundation. | ||
2841 | 8 | # | ||
2842 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2843 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2844 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2845 | 12 | # GNU Lesser General Public License for more details. | ||
2846 | 13 | # | ||
2847 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2848 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2849 | 16 | |||
2850 | 17 | import os | ||
2851 | 18 | from charmhelpers.fetch import ( | ||
2852 | 19 | BaseFetchHandler, | ||
2853 | 20 | UnhandledSource | ||
2854 | 21 | ) | ||
2855 | 22 | from charmhelpers.core.host import mkdir | ||
2856 | 23 | |||
2857 | 24 | import six | ||
2858 | 25 | if six.PY3: | ||
2859 | 26 | raise ImportError('GitPython does not support Python 3') | ||
2860 | 27 | |||
2861 | 28 | try: | ||
2862 | 29 | from git import Repo | ||
2863 | 30 | except ImportError: | ||
2864 | 31 | from charmhelpers.fetch import apt_install | ||
2865 | 32 | apt_install("python-git") | ||
2866 | 33 | from git import Repo | ||
2867 | 34 | |||
2868 | 35 | from git.exc import GitCommandError # noqa E402 | ||
2869 | 36 | |||
2870 | 37 | |||
2871 | 38 | class GitUrlFetchHandler(BaseFetchHandler): | ||
2872 | 39 | """Handler for git branches via generic and github URLs""" | ||
2873 | 40 | def can_handle(self, source): | ||
2874 | 41 | url_parts = self.parse_url(source) | ||
2875 | 42 | # TODO (mattyw) no support for ssh git@ yet | ||
2876 | 43 | if url_parts.scheme not in ('http', 'https', 'git'): | ||
2877 | 44 | return False | ||
2878 | 45 | else: | ||
2879 | 46 | return True | ||
2880 | 47 | |||
2881 | 48 | def clone(self, source, dest, branch): | ||
2882 | 49 | if not self.can_handle(source): | ||
2883 | 50 | raise UnhandledSource("Cannot handle {}".format(source)) | ||
2884 | 51 | |||
2885 | 52 | repo = Repo.clone_from(source, dest) | ||
2886 | 53 | repo.git.checkout(branch) | ||
2887 | 54 | |||
2888 | 55 | def install(self, source, branch="master", dest=None): | ||
2889 | 56 | url_parts = self.parse_url(source) | ||
2890 | 57 | branch_name = url_parts.path.strip("/").split("/")[-1] | ||
2891 | 58 | if dest: | ||
2892 | 59 | dest_dir = os.path.join(dest, branch_name) | ||
2893 | 60 | else: | ||
2894 | 61 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched", | ||
2895 | 62 | branch_name) | ||
2896 | 63 | if not os.path.exists(dest_dir): | ||
2897 | 64 | mkdir(dest_dir, perms=0o755) | ||
2898 | 65 | try: | ||
2899 | 66 | self.clone(source, dest_dir, branch) | ||
2900 | 67 | except GitCommandError as e: | ||
2901 | 68 | raise UnhandledSource(e.message) | ||
2902 | 69 | except OSError as e: | ||
2903 | 70 | raise UnhandledSource(e.strerror) | ||
2904 | 71 | return dest_dir | ||
2905 | 0 | 72 | ||
2906 | === modified file 'lib/charmhelpers/payload/__init__.py' | |||
2907 | --- lib/charmhelpers/payload/__init__.py 2014-09-23 12:09:14 +0000 | |||
2908 | +++ lib/charmhelpers/payload/__init__.py 2015-11-01 20:32:57 +0000 | |||
2909 | @@ -1,1 +1,17 @@ | |||
2910 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2911 | 2 | # | ||
2912 | 3 | # This file is part of charm-helpers. | ||
2913 | 4 | # | ||
2914 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2915 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2916 | 7 | # published by the Free Software Foundation. | ||
2917 | 8 | # | ||
2918 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2919 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2920 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2921 | 12 | # GNU Lesser General Public License for more details. | ||
2922 | 13 | # | ||
2923 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2924 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2925 | 16 | |||
2926 | 1 | "Tools for working with files injected into a charm just before deployment." | 17 | "Tools for working with files injected into a charm just before deployment." |
2927 | 2 | 18 | ||
2928 | === modified file 'lib/charmhelpers/payload/archive.py' | |||
2929 | --- lib/charmhelpers/payload/archive.py 2014-09-23 12:09:14 +0000 | |||
2930 | +++ lib/charmhelpers/payload/archive.py 2015-11-01 20:32:57 +0000 | |||
2931 | @@ -1,3 +1,19 @@ | |||
2932 | 1 | # Copyright 2014-2015 Canonical Limited. | ||
2933 | 2 | # | ||
2934 | 3 | # This file is part of charm-helpers. | ||
2935 | 4 | # | ||
2936 | 5 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2937 | 6 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2938 | 7 | # published by the Free Software Foundation. | ||
2939 | 8 | # | ||
2940 | 9 | # charm-helpers is distributed in the hope that it will be useful, | ||
2941 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2942 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2943 | 12 | # GNU Lesser General Public License for more details. | ||
2944 | 13 | # | ||
2945 | 14 | # You should have received a copy of the GNU Lesser General Public License | ||
2946 | 15 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2947 | 16 | |||
2948 | 1 | import os | 17 | import os |
2949 | 2 | import tarfile | 18 | import tarfile |
2950 | 3 | import zipfile | 19 | import zipfile |
2951 | 4 | 20 | ||
2952 | === modified file 'lib/charmhelpers/payload/execd.py' | |||
2953 | --- lib/charmhelpers/payload/execd.py 2014-09-23 12:09:14 +0000 | |||
2954 | +++ lib/charmhelpers/payload/execd.py 2015-11-01 20:32:57 +0000 | |||
2955 | @@ -1,5 +1,21 @@ | |||
2956 | 1 | #!/usr/bin/env python | 1 | #!/usr/bin/env python |
2957 | 2 | 2 | ||
2958 | 3 | # Copyright 2014-2015 Canonical Limited. | ||
2959 | 4 | # | ||
2960 | 5 | # This file is part of charm-helpers. | ||
2961 | 6 | # | ||
2962 | 7 | # charm-helpers is free software: you can redistribute it and/or modify | ||
2963 | 8 | # it under the terms of the GNU Lesser General Public License version 3 as | ||
2964 | 9 | # published by the Free Software Foundation. | ||
2965 | 10 | # | ||
2966 | 11 | # charm-helpers is distributed in the hope that it will be useful, | ||
2967 | 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2968 | 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2969 | 14 | # GNU Lesser General Public License for more details. | ||
2970 | 15 | # | ||
2971 | 16 | # You should have received a copy of the GNU Lesser General Public License | ||
2972 | 17 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. | ||
2973 | 18 | |||
2974 | 3 | import os | 19 | import os |
2975 | 4 | import sys | 20 | import sys |
2976 | 5 | import subprocess | 21 | import subprocess |
2977 | 6 | 22 | ||
2978 | === modified file 'metadata.yaml' | |||
2979 | --- metadata.yaml 2015-04-17 08:37:59 +0000 | |||
2980 | +++ metadata.yaml 2015-11-01 20:32:57 +0000 | |||
2981 | @@ -16,6 +16,9 @@ | |||
2982 | 16 | interface: elasticsearch | 16 | interface: elasticsearch |
2983 | 17 | lumberjack: | 17 | lumberjack: |
2984 | 18 | interface: http | 18 | interface: http |
2985 | 19 | nrpe-external-master: | ||
2986 | 20 | interface: nrpe-external-master | ||
2987 | 21 | scope: container | ||
2988 | 19 | requires: | 22 | requires: |
2989 | 20 | client: | 23 | client: |
2990 | 21 | interface: elasticsearch | 24 | interface: elasticsearch |
FWIW, applied this to our logstash charm fork and it worked as expected.
Michael, thanks for working on this!