Merge lp:~canonical-is-sa/charms/trusty/logstash/trunk into lp:~lazypower/charms/trusty/logstash/trunk

Proposed by Michael Foley on 2015-05-12
Status: Needs review
Proposed branch: lp:~canonical-is-sa/charms/trusty/logstash/trunk
Merge into: lp:~lazypower/charms/trusty/logstash/trunk
Diff against target: 2990 lines (+1996/-154)
31 files modified
README.md (+4/-5)
charm-helpers.yaml (+1/-0)
config.yaml (+32/-0)
hooks/client-relation-changed (+20/-14)
hooks/config-changed (+14/-2)
hooks/nrpe-external-master-relation-changed (+87/-0)
lib/charmhelpers/contrib/charmsupport/__init__.py (+15/-0)
lib/charmhelpers/contrib/charmsupport/nrpe.py (+360/-0)
lib/charmhelpers/contrib/charmsupport/volumes.py (+175/-0)
lib/charmhelpers/contrib/templating/__init__.py (+15/-0)
lib/charmhelpers/contrib/templating/jinja.py (+25/-9)
lib/charmhelpers/core/__init__.py (+15/-0)
lib/charmhelpers/core/decorators.py (+57/-0)
lib/charmhelpers/core/fstab.py (+30/-12)
lib/charmhelpers/core/hookenv.py (+105/-22)
lib/charmhelpers/core/host.py (+103/-38)
lib/charmhelpers/core/services/__init__.py (+18/-2)
lib/charmhelpers/core/services/base.py (+16/-0)
lib/charmhelpers/core/services/helpers.py (+37/-9)
lib/charmhelpers/core/strutils.py (+42/-0)
lib/charmhelpers/core/sysctl.py (+56/-0)
lib/charmhelpers/core/templating.py (+20/-3)
lib/charmhelpers/core/unitdata.py (+477/-0)
lib/charmhelpers/fetch/__init__.py (+44/-14)
lib/charmhelpers/fetch/archiveurl.py (+76/-22)
lib/charmhelpers/fetch/bzrurl.py (+30/-2)
lib/charmhelpers/fetch/giturl.py (+71/-0)
lib/charmhelpers/payload/__init__.py (+16/-0)
lib/charmhelpers/payload/archive.py (+16/-0)
lib/charmhelpers/payload/execd.py (+16/-0)
metadata.yaml (+3/-0)
To merge this branch: bzr merge lp:~canonical-is-sa/charms/trusty/logstash/trunk
Reviewer Review Type Date Requested Status
Charles Butler 2015-05-12 Pending
Review via email: mp+258828@code.launchpad.net

Description of the change

Have added nagios checks for logstash and fixed client-relation-changed to use most recent elasticsearch unit.

Also updated charmhelpers.

To post a comment you must log in.
53. By Michael Foley on 2015-05-12

[foli] adjust nagios checks

Guillermo Gonzalez (verterok) wrote :

FWIW, applied this to our logstash charm fork and it worked as expected.

Michael, thanks for working on this!

54. By Michael Foley on 2015-11-01

[michael.nelson,r=foli] Add a nagios cert check to avoid cert expiry. Add extra_config to enable juju setting extra filters. RT#85874

Unmerged revisions

54. By Michael Foley on 2015-11-01

[michael.nelson,r=foli] Add a nagios cert check to avoid cert expiry. Add extra_config to enable juju setting extra filters. RT#85874

53. By Michael Foley on 2015-05-12

[foli] adjust nagios checks

52. By Michael Foley on 2015-05-05

[foli] fix nrpe-external-master relation, don't add lumberjack port check if not configured

51. By Michael Foley on 2015-05-05

[foli] initial add of nrpe-external-master relation and custom checks

50. By Michael Foley on 2015-05-05

[foli] addedh charmhelpers contrib.charmsupport

49. By Michael Foley on 2015-05-05

[foli] updated charmhelpers from upstream

48. By Michael Foley on 2015-04-21

[foli] Fix client-relation-changed to use most recently joined elasticsearch unit

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2014-07-24 13:26:45 +0000
3+++ README.md 2015-11-01 20:32:57 +0000
4@@ -28,7 +28,7 @@
5 http://ip-of-kibana
6
7 example 2 - Indexer + 2 x ElasticSearch + Kibana
8-============================================
9+================================================
10
11 juju deploy cs:trusty/elasticsearch
12 juju add-unit elasticsearch
13@@ -41,7 +41,7 @@
14 http://ip-of-kibana
15
16 example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana
17-=============================================
18+=========================================================
19
20 juju deploy cs:trusty/elasticsearch
21 juju add-unit elasticsearch
22@@ -54,7 +54,6 @@
23 juju add-relation logstash-agent logstash-indexer:input
24
25
26-
27 ### Caveats
28
29 The charm will fetch the logstash complete archive every time.
30@@ -63,13 +62,13 @@
31
32 # Configuration
33
34-The charm supports installation from anywhere that python requeusts can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
35+The charm supports installation from anywhere that python requests can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
36
37 juju set logstash logstash-source="https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz" logstash-sum="d59ef579c7614c5df9bd69cfdce20ed371f728ff"
38
39 There is also a configuration option to add arbitrary packages pre-installation of logstash. The format is a space separated list.
40
41- juju set logstash extrapackages='vim byobu'
42+ juju set logstash extra-packages='vim byobu'
43
44 # Contact Information
45
46
47=== modified file 'charm-helpers.yaml'
48--- charm-helpers.yaml 2014-09-23 12:11:39 +0000
49+++ charm-helpers.yaml 2015-11-01 20:32:57 +0000
50@@ -5,3 +5,4 @@
51 - fetch
52 - payload
53 - contrib.templating.jinja
54+ - contrib.charmsupport
55
56=== modified file 'config.yaml'
57--- config.yaml 2015-03-31 09:37:21 +0000
58+++ config.yaml 2015-11-01 20:32:57 +0000
59@@ -19,3 +19,35 @@
60 default: ""
61 type: string
62 description: "Base64-encoded SSL key"
63+ extra-config:
64+ type: string
65+ default: ''
66+ description: "Base64-encoded custom configuration content."
67+ nagios_context:
68+ default: "juju"
69+ type: string
70+ description: |
71+ Used by the nrpe subordinate charms.
72+ A string that will be prepended to instance name to set the host name
73+ in nagios. So for instance the hostname would be something like:
74+ juju-myservice-0
75+ If you're running multiple environments with the same services in them
76+ this allows you to differentiate between them.
77+ nagios_servicegroups:
78+ default: ""
79+ type: string
80+ description: |
81+ A comma-separated list of nagios servicegroups.
82+ If left empty, the nagios_context will be used as the servicegroup
83+ nagios_check_procs_params:
84+ default: "-a /opt/logstash/lib/logstash/runner.rb -c 1:1"
85+ type: string
86+ description: The parameters to pass to the nrpe plugin check_procs.
87+ nagios_check_tcp_params:
88+ default: "--ssl -H localhost -p 5043 -c 0.3"
89+ type: string
90+ description: The parameters to pass to the nrpe plugin check_tcp.
91+ nagios_check_cert_params:
92+ default: "-D 30,14 -H 127.0.0.1 -p 5043"
93+ type: string
94+ description: The parameters to pass to the nrpe plugin "check_tcp --ssl" to check certificate expiration date.
95
96=== modified file 'hooks/client-relation-changed'
97--- hooks/client-relation-changed 2014-09-23 16:59:13 +0000
98+++ hooks/client-relation-changed 2015-11-01 20:32:57 +0000
99@@ -29,29 +29,35 @@
100 def write_config():
101 with open('host_cache', 'r') as f:
102 hosts = f.readlines()
103+ if not hosts:
104+ sys.exit(0)
105
106- opts = {'hosts': hosts[0].rstrip()}
107+ # Use last host in list as it will be the most recently added
108+ # and first host in list may not exist anymore! TODO fix that.
109+ opts = {'hosts': hosts[-1].rstrip()}
110
111 out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf')
112 with open(out, 'w') as p:
113- p.write(render(os.path.basename(out), opts))
114-
115+ p.write(render(os.path.basename(out), opts))
116
117
118 def cache_hosts():
119- host = hookenv.relation_get('host')
120- if not host:
121- log('No host received. Assuming nothing to do.')
122- sys.exit(0)
123-
124+ rels = hookenv.relations_of_type("client")
125+ if not rels:
126+ log('No client relations. Assuming nothing to do.')
127+ sys.exit(0)
128 if not os.path.exists('host_cache'):
129 open('host_cache', 'a').close()
130-
131- with open('host_cache', 'r') as f:
132- hosts = f.readlines()
133- if not host in hosts:
134- with open('host_cache', 'a') as f:
135- f.write('{}\n'.format(host))
136+ for rel in rels:
137+ host = rel.get('host')
138+ if not host:
139+ log('No host received for relation: {}.'.format(rel))
140+ continue
141+ with open('host_cache', 'r') as f:
142+ hosts = f.readlines()
143+ if host not in hosts:
144+ with open('host_cache', 'a') as f:
145+ f.write('{}\n'.format(host))
146
147
148 if __name__ == "__main__":
149
150=== modified file 'hooks/config-changed'
151--- hooks/config-changed 2015-04-07 00:25:13 +0000
152+++ hooks/config-changed 2015-11-01 20:32:57 +0000
153@@ -39,8 +39,10 @@
154 # This only actually opens the port if we've exposed the service in juju
155 hookenv.open_port(5043)
156
157- # The install hook is idempotent, so re-run it.
158- subprocess.check_output(shlex.split('hooks/install'))
159+ # Restart the service when configuration has changed.
160+ subprocess.check_output(shlex.split('hooks/start'))
161+
162+ subprocess.check_output(shlex.split('hooks/nrpe-external-master-relation-changed'))
163
164
165 def copy_config():
166@@ -52,11 +54,21 @@
167 key_file = os.path.join(cert_dir, 'logstash.key')
168
169 for f in files:
170+ # skip output-elasticsearch.conf, is managed by
171+ # hooks/client-relation-changed
172+ if os.path.basename(f) == "output-elasticsearch.conf":
173+ continue
174 if os.path.basename(f) != lumberjack_template:
175 with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p:
176 p.write(render(os.path.basename(f), opts))
177
178 config_data = hookenv.config()
179+
180+ # Write custom configuration if set.
181+ if config_data['extra-config']:
182+ with open(os.path.join(BASEPATH, 'conf.d', 'extra.conf'), 'w') as f:
183+ f.write(str(base64.b64decode(config_data['extra-config'])))
184+
185 # Only setup lumberjack protocol if ssl cert and key are configured
186 if config_data['ssl_cert'] and config_data['ssl_key']:
187 if not os.path.exists(cert_dir):
188
189=== modified file 'hooks/lumberjack-relation-changed' (properties changed: -x to +x)
190=== added file 'hooks/nrpe-external-master-relation-changed'
191--- hooks/nrpe-external-master-relation-changed 1970-01-01 00:00:00 +0000
192+++ hooks/nrpe-external-master-relation-changed 2015-11-01 20:32:57 +0000
193@@ -0,0 +1,87 @@
194+#!/usr/bin/python
195+
196+import os
197+import sys
198+
199+sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
200+
201+from charmhelpers.core import hookenv
202+from charmhelpers.contrib.charmsupport import nrpe
203+from charmhelpers.contrib.charmsupport.nrpe import NRPE
204+
205+hooks = hookenv.Hooks()
206+log = hookenv.log
207+
208+
209+class CustomIntervalCheck(nrpe.Check):
210+
211+ service_template = ("""
212+#---------------------------------------------------
213+# This file is Juju managed
214+#---------------------------------------------------
215+define service {{
216+ use active-service
217+ host_name {nagios_hostname}
218+ service_description {nagios_hostname}[{shortname}] """
219+ """{description}
220+ check_command check_nrpe!{command}
221+ servicegroups {nagios_servicegroup}
222+%s
223+}}
224+""")
225+ intervals_template = " {} {}\n"
226+
227+ def __init__(self, shortname, description, check_cmd, normal_check_interval=None,
228+ retry_check_interval=None, notification_interval=None):
229+ super(CustomIntervalCheck, self).__init__(shortname, description, check_cmd)
230+ intervals = {}
231+ if normal_check_interval:
232+ intervals['normal_check_interval'] = normal_check_interval
233+ if retry_check_interval:
234+ intervals['retry_check_interval'] = retry_check_interval
235+ if notification_interval:
236+ intervals['notification_interval'] = notification_interval
237+ intervals_config = ""
238+ for k, v in intervals.items():
239+ intervals_config += self.intervals_template.format(k, v)
240+ self.service_template = CustomIntervalCheck.service_template % intervals_config
241+
242+
243+@hooks.hook('nrpe-external-master-relation-changed')
244+def update_nrpe_checks():
245+ nrpe_compat = nrpe.NRPE()
246+ conf = nrpe_compat.config
247+ check_procs_params = conf.get('nagios_check_procs_params')
248+ if check_procs_params:
249+ nrpe_compat.add_check(
250+ shortname='logstash_process',
251+ description='Check logstash java process running',
252+ check_cmd='check_procs %s' % check_procs_params
253+ )
254+ check_tcp_params = conf.get('nagios_check_tcp_params')
255+ check_cert_params = conf.get('nagios_check_cert_params')
256+ config_data = hookenv.config()
257+ # Only setup lumberjack protocol if ssl cert and key are configured
258+ if config_data['ssl_cert'] and config_data['ssl_key']:
259+ if check_tcp_params:
260+ nrpe_compat.add_check(
261+ shortname='lumberjack_tcp',
262+ description='Check logstash lumberjack input tcp port',
263+ check_cmd='check_tcp %s' % check_tcp_params
264+ )
265+ if check_cert_params:
266+ # check certificate expiry date, daily and retry every 2 hs
267+ cert_check = CustomIntervalCheck(
268+ shortname='lumberjack_ssl_check',
269+ description='Check logstash ssl certificate expiry date',
270+ check_cmd='check_tcp --ssl {}'.format(check_cert_params),
271+ normal_check_interval=1440, # minutes
272+ retry_check_interval=120, # minutes
273+ )
274+ nrpe_compat.checks.append(cert_check)
275+
276+ nrpe_compat.write()
277+
278+if __name__ == "__main__":
279+ # execute a hook based on the name the program is called by
280+ hooks.execute(sys.argv)
281
282=== added directory 'lib/charmhelpers/contrib/charmsupport'
283=== added file 'lib/charmhelpers/contrib/charmsupport/__init__.py'
284--- lib/charmhelpers/contrib/charmsupport/__init__.py 1970-01-01 00:00:00 +0000
285+++ lib/charmhelpers/contrib/charmsupport/__init__.py 2015-11-01 20:32:57 +0000
286@@ -0,0 +1,15 @@
287+# Copyright 2014-2015 Canonical Limited.
288+#
289+# This file is part of charm-helpers.
290+#
291+# charm-helpers is free software: you can redistribute it and/or modify
292+# it under the terms of the GNU Lesser General Public License version 3 as
293+# published by the Free Software Foundation.
294+#
295+# charm-helpers is distributed in the hope that it will be useful,
296+# but WITHOUT ANY WARRANTY; without even the implied warranty of
297+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
298+# GNU Lesser General Public License for more details.
299+#
300+# You should have received a copy of the GNU Lesser General Public License
301+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
302
303=== added file 'lib/charmhelpers/contrib/charmsupport/nrpe.py'
304--- lib/charmhelpers/contrib/charmsupport/nrpe.py 1970-01-01 00:00:00 +0000
305+++ lib/charmhelpers/contrib/charmsupport/nrpe.py 2015-11-01 20:32:57 +0000
306@@ -0,0 +1,360 @@
307+# Copyright 2014-2015 Canonical Limited.
308+#
309+# This file is part of charm-helpers.
310+#
311+# charm-helpers is free software: you can redistribute it and/or modify
312+# it under the terms of the GNU Lesser General Public License version 3 as
313+# published by the Free Software Foundation.
314+#
315+# charm-helpers is distributed in the hope that it will be useful,
316+# but WITHOUT ANY WARRANTY; without even the implied warranty of
317+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
318+# GNU Lesser General Public License for more details.
319+#
320+# You should have received a copy of the GNU Lesser General Public License
321+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
322+
323+"""Compatibility with the nrpe-external-master charm"""
324+# Copyright 2012 Canonical Ltd.
325+#
326+# Authors:
327+# Matthew Wedgwood <matthew.wedgwood@canonical.com>
328+
329+import subprocess
330+import pwd
331+import grp
332+import os
333+import glob
334+import shutil
335+import re
336+import shlex
337+import yaml
338+
339+from charmhelpers.core.hookenv import (
340+ config,
341+ local_unit,
342+ log,
343+ relation_ids,
344+ relation_set,
345+ relations_of_type,
346+)
347+
348+from charmhelpers.core.host import service
349+
350+# This module adds compatibility with the nrpe-external-master and plain nrpe
351+# subordinate charms. To use it in your charm:
352+#
353+# 1. Update metadata.yaml
354+#
355+# provides:
356+# (...)
357+# nrpe-external-master:
358+# interface: nrpe-external-master
359+# scope: container
360+#
361+# and/or
362+#
363+# provides:
364+# (...)
365+# local-monitors:
366+# interface: local-monitors
367+# scope: container
368+
369+#
370+# 2. Add the following to config.yaml
371+#
372+# nagios_context:
373+# default: "juju"
374+# type: string
375+# description: |
376+# Used by the nrpe subordinate charms.
377+# A string that will be prepended to instance name to set the host name
378+# in nagios. So for instance the hostname would be something like:
379+# juju-myservice-0
380+# If you're running multiple environments with the same services in them
381+# this allows you to differentiate between them.
382+# nagios_servicegroups:
383+# default: ""
384+# type: string
385+# description: |
386+# A comma-separated list of nagios servicegroups.
387+# If left empty, the nagios_context will be used as the servicegroup
388+#
389+# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
390+#
391+# 4. Update your hooks.py with something like this:
392+#
393+# from charmsupport.nrpe import NRPE
394+# (...)
395+# def update_nrpe_config():
396+# nrpe_compat = NRPE()
397+# nrpe_compat.add_check(
398+# shortname = "myservice",
399+# description = "Check MyService",
400+# check_cmd = "check_http -w 2 -c 10 http://localhost"
401+# )
402+# nrpe_compat.add_check(
403+# "myservice_other",
404+# "Check for widget failures",
405+# check_cmd = "/srv/myapp/scripts/widget_check"
406+# )
407+# nrpe_compat.write()
408+#
409+# def config_changed():
410+# (...)
411+# update_nrpe_config()
412+#
413+# def nrpe_external_master_relation_changed():
414+# update_nrpe_config()
415+#
416+# def local_monitors_relation_changed():
417+# update_nrpe_config()
418+#
419+# 5. ln -s hooks.py nrpe-external-master-relation-changed
420+# ln -s hooks.py local-monitors-relation-changed
421+
422+
423+class CheckException(Exception):
424+ pass
425+
426+
427+class Check(object):
428+ shortname_re = '[A-Za-z0-9-_]+$'
429+ service_template = ("""
430+#---------------------------------------------------
431+# This file is Juju managed
432+#---------------------------------------------------
433+define service {{
434+ use active-service
435+ host_name {nagios_hostname}
436+ service_description {nagios_hostname}[{shortname}] """
437+ """{description}
438+ check_command check_nrpe!{command}
439+ servicegroups {nagios_servicegroup}
440+}}
441+""")
442+
443+ def __init__(self, shortname, description, check_cmd):
444+ super(Check, self).__init__()
445+ # XXX: could be better to calculate this from the service name
446+ if not re.match(self.shortname_re, shortname):
447+ raise CheckException("shortname must match {}".format(
448+ Check.shortname_re))
449+ self.shortname = shortname
450+ self.command = "check_{}".format(shortname)
451+ # Note: a set of invalid characters is defined by the
452+ # Nagios server config
453+ # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
454+ self.description = description
455+ self.check_cmd = self._locate_cmd(check_cmd)
456+
457+ def _locate_cmd(self, check_cmd):
458+ search_path = (
459+ '/usr/lib/nagios/plugins',
460+ '/usr/local/lib/nagios/plugins',
461+ )
462+ parts = shlex.split(check_cmd)
463+ for path in search_path:
464+ if os.path.exists(os.path.join(path, parts[0])):
465+ command = os.path.join(path, parts[0])
466+ if len(parts) > 1:
467+ command += " " + " ".join(parts[1:])
468+ return command
469+ log('Check command not found: {}'.format(parts[0]))
470+ return ''
471+
472+ def write(self, nagios_context, hostname, nagios_servicegroups):
473+ nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
474+ self.command)
475+ with open(nrpe_check_file, 'w') as nrpe_check_config:
476+ nrpe_check_config.write("# check {}\n".format(self.shortname))
477+ nrpe_check_config.write("command[{}]={}\n".format(
478+ self.command, self.check_cmd))
479+
480+ if not os.path.exists(NRPE.nagios_exportdir):
481+ log('Not writing service config as {} is not accessible'.format(
482+ NRPE.nagios_exportdir))
483+ else:
484+ self.write_service_config(nagios_context, hostname,
485+ nagios_servicegroups)
486+
487+ def write_service_config(self, nagios_context, hostname,
488+ nagios_servicegroups):
489+ for f in os.listdir(NRPE.nagios_exportdir):
490+ if re.search('.*{}.cfg'.format(self.command), f):
491+ os.remove(os.path.join(NRPE.nagios_exportdir, f))
492+
493+ templ_vars = {
494+ 'nagios_hostname': hostname,
495+ 'nagios_servicegroup': nagios_servicegroups,
496+ 'description': self.description,
497+ 'shortname': self.shortname,
498+ 'command': self.command,
499+ }
500+ nrpe_service_text = Check.service_template.format(**templ_vars)
501+ nrpe_service_file = '{}/service__{}_{}.cfg'.format(
502+ NRPE.nagios_exportdir, hostname, self.command)
503+ with open(nrpe_service_file, 'w') as nrpe_service_config:
504+ nrpe_service_config.write(str(nrpe_service_text))
505+
506+ def run(self):
507+ subprocess.call(self.check_cmd)
508+
509+
510+class NRPE(object):
511+ nagios_logdir = '/var/log/nagios'
512+ nagios_exportdir = '/var/lib/nagios/export'
513+ nrpe_confdir = '/etc/nagios/nrpe.d'
514+
515+ def __init__(self, hostname=None):
516+ super(NRPE, self).__init__()
517+ self.config = config()
518+ self.nagios_context = self.config['nagios_context']
519+ if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']:
520+ self.nagios_servicegroups = self.config['nagios_servicegroups']
521+ else:
522+ self.nagios_servicegroups = self.nagios_context
523+ self.unit_name = local_unit().replace('/', '-')
524+ if hostname:
525+ self.hostname = hostname
526+ else:
527+ self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
528+ self.checks = []
529+
530+ def add_check(self, *args, **kwargs):
531+ self.checks.append(Check(*args, **kwargs))
532+
533+ def write(self):
534+ try:
535+ nagios_uid = pwd.getpwnam('nagios').pw_uid
536+ nagios_gid = grp.getgrnam('nagios').gr_gid
537+ except:
538+ log("Nagios user not set up, nrpe checks not updated")
539+ return
540+
541+ if not os.path.exists(NRPE.nagios_logdir):
542+ os.mkdir(NRPE.nagios_logdir)
543+ os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
544+
545+ nrpe_monitors = {}
546+ monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
547+ for nrpecheck in self.checks:
548+ nrpecheck.write(self.nagios_context, self.hostname,
549+ self.nagios_servicegroups)
550+ nrpe_monitors[nrpecheck.shortname] = {
551+ "command": nrpecheck.command,
552+ }
553+
554+ service('restart', 'nagios-nrpe-server')
555+
556+ monitor_ids = relation_ids("local-monitors") + \
557+ relation_ids("nrpe-external-master")
558+ for rid in monitor_ids:
559+ relation_set(relation_id=rid, monitors=yaml.dump(monitors))
560+
561+
562+def get_nagios_hostcontext(relation_name='nrpe-external-master'):
563+ """
564+ Query relation with nrpe subordinate, return the nagios_host_context
565+
566+ :param str relation_name: Name of relation nrpe sub joined to
567+ """
568+ for rel in relations_of_type(relation_name):
569+ if 'nagios_hostname' in rel:
570+ return rel['nagios_host_context']
571+
572+
573+def get_nagios_hostname(relation_name='nrpe-external-master'):
574+ """
575+ Query relation with nrpe subordinate, return the nagios_hostname
576+
577+ :param str relation_name: Name of relation nrpe sub joined to
578+ """
579+ for rel in relations_of_type(relation_name):
580+ if 'nagios_hostname' in rel:
581+ return rel['nagios_hostname']
582+
583+
584+def get_nagios_unit_name(relation_name='nrpe-external-master'):
585+ """
586+ Return the nagios unit name prepended with host_context if needed
587+
588+ :param str relation_name: Name of relation nrpe sub joined to
589+ """
590+ host_context = get_nagios_hostcontext(relation_name)
591+ if host_context:
592+ unit = "%s:%s" % (host_context, local_unit())
593+ else:
594+ unit = local_unit()
595+ return unit
596+
597+
598+def add_init_service_checks(nrpe, services, unit_name):
599+ """
600+ Add checks for each service in list
601+
602+ :param NRPE nrpe: NRPE object to add check to
603+ :param list services: List of services to check
604+ :param str unit_name: Unit name to use in check description
605+ """
606+ for svc in services:
607+ upstart_init = '/etc/init/%s.conf' % svc
608+ sysv_init = '/etc/init.d/%s' % svc
609+ if os.path.exists(upstart_init):
610+ nrpe.add_check(
611+ shortname=svc,
612+ description='process check {%s}' % unit_name,
613+ check_cmd='check_upstart_job %s' % svc
614+ )
615+ elif os.path.exists(sysv_init):
616+ cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
617+ cron_file = ('*/5 * * * * root '
618+ '/usr/local/lib/nagios/plugins/check_exit_status.pl '
619+ '-s /etc/init.d/%s status > '
620+ '/var/lib/nagios/service-check-%s.txt\n' % (svc,
621+ svc)
622+ )
623+ f = open(cronpath, 'w')
624+ f.write(cron_file)
625+ f.close()
626+ nrpe.add_check(
627+ shortname=svc,
628+ description='process check {%s}' % unit_name,
629+ check_cmd='check_status_file.py -f '
630+ '/var/lib/nagios/service-check-%s.txt' % svc,
631+ )
632+
633+
634+def copy_nrpe_checks():
635+ """
636+ Copy the nrpe checks into place
637+
638+ """
639+ NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
640+ nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks',
641+ 'charmhelpers', 'contrib', 'openstack',
642+ 'files')
643+
644+ if not os.path.exists(NAGIOS_PLUGINS):
645+ os.makedirs(NAGIOS_PLUGINS)
646+ for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")):
647+ if os.path.isfile(fname):
648+ shutil.copy2(fname,
649+ os.path.join(NAGIOS_PLUGINS, os.path.basename(fname)))
650+
651+
652+def add_haproxy_checks(nrpe, unit_name):
653+ """
654+ Add checks for each service in list
655+
656+ :param NRPE nrpe: NRPE object to add check to
657+ :param str unit_name: Unit name to use in check description
658+ """
659+ nrpe.add_check(
660+ shortname='haproxy_servers',
661+ description='Check HAProxy {%s}' % unit_name,
662+ check_cmd='check_haproxy.sh')
663+ nrpe.add_check(
664+ shortname='haproxy_queue',
665+ description='Check HAProxy queue depth {%s}' % unit_name,
666+ check_cmd='check_haproxy_queue_depth.sh')
667
668=== added file 'lib/charmhelpers/contrib/charmsupport/volumes.py'
669--- lib/charmhelpers/contrib/charmsupport/volumes.py 1970-01-01 00:00:00 +0000
670+++ lib/charmhelpers/contrib/charmsupport/volumes.py 2015-11-01 20:32:57 +0000
671@@ -0,0 +1,175 @@
672+# Copyright 2014-2015 Canonical Limited.
673+#
674+# This file is part of charm-helpers.
675+#
676+# charm-helpers is free software: you can redistribute it and/or modify
677+# it under the terms of the GNU Lesser General Public License version 3 as
678+# published by the Free Software Foundation.
679+#
680+# charm-helpers is distributed in the hope that it will be useful,
681+# but WITHOUT ANY WARRANTY; without even the implied warranty of
682+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
683+# GNU Lesser General Public License for more details.
684+#
685+# You should have received a copy of the GNU Lesser General Public License
686+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
687+
688+'''
689+Functions for managing volumes in juju units. One volume is supported per unit.
690+Subordinates may have their own storage, provided it is on its own partition.
691+
692+Configuration stanzas::
693+
694+ volume-ephemeral:
695+ type: boolean
696+ default: true
697+ description: >
698+ If false, a volume is mounted as sepecified in "volume-map"
699+ If true, ephemeral storage will be used, meaning that log data
700+ will only exist as long as the machine. YOU HAVE BEEN WARNED.
701+ volume-map:
702+ type: string
703+ default: {}
704+ description: >
705+ YAML map of units to device names, e.g:
706+ "{ rsyslog/0: /dev/vdb, rsyslog/1: /dev/vdb }"
707+ Service units will raise a configure-error if volume-ephemeral
708+ is 'true' and no volume-map value is set. Use 'juju set' to set a
709+ value and 'juju resolved' to complete configuration.
710+
711+Usage::
712+
713+ from charmsupport.volumes import configure_volume, VolumeConfigurationError
714+ from charmsupport.hookenv import log, ERROR
715+ def post_mount_hook():
716+ stop_service('myservice')
717+ def post_mount_hook():
718+ start_service('myservice')
719+
720+ if __name__ == '__main__':
721+ try:
722+ configure_volume(before_change=pre_mount_hook,
723+ after_change=post_mount_hook)
724+ except VolumeConfigurationError:
725+ log('Storage could not be configured', ERROR)
726+
727+'''
728+
729+# XXX: Known limitations
730+# - fstab is neither consulted nor updated
731+
732+import os
733+from charmhelpers.core import hookenv
734+from charmhelpers.core import host
735+import yaml
736+
737+
738+MOUNT_BASE = '/srv/juju/volumes'
739+
740+
741+class VolumeConfigurationError(Exception):
742+ '''Volume configuration data is missing or invalid'''
743+ pass
744+
745+
746+def get_config():
747+ '''Gather and sanity-check volume configuration data'''
748+ volume_config = {}
749+ config = hookenv.config()
750+
751+ errors = False
752+
753+ if config.get('volume-ephemeral') in (True, 'True', 'true', 'Yes', 'yes'):
754+ volume_config['ephemeral'] = True
755+ else:
756+ volume_config['ephemeral'] = False
757+
758+ try:
759+ volume_map = yaml.safe_load(config.get('volume-map', '{}'))
760+ except yaml.YAMLError as e:
761+ hookenv.log("Error parsing YAML volume-map: {}".format(e),
762+ hookenv.ERROR)
763+ errors = True
764+ if volume_map is None:
765+ # probably an empty string
766+ volume_map = {}
767+ elif not isinstance(volume_map, dict):
768+ hookenv.log("Volume-map should be a dictionary, not {}".format(
769+ type(volume_map)))
770+ errors = True
771+
772+ volume_config['device'] = volume_map.get(os.environ['JUJU_UNIT_NAME'])
773+ if volume_config['device'] and volume_config['ephemeral']:
774+ # asked for ephemeral storage but also defined a volume ID
775+ hookenv.log('A volume is defined for this unit, but ephemeral '
776+ 'storage was requested', hookenv.ERROR)
777+ errors = True
778+ elif not volume_config['device'] and not volume_config['ephemeral']:
779+ # asked for permanent storage but did not define volume ID
780+ hookenv.log('Ephemeral storage was requested, but there is no volume '
781+ 'defined for this unit.', hookenv.ERROR)
782+ errors = True
783+
784+ unit_mount_name = hookenv.local_unit().replace('/', '-')
785+ volume_config['mountpoint'] = os.path.join(MOUNT_BASE, unit_mount_name)
786+
787+ if errors:
788+ return None
789+ return volume_config
790+
791+
792+def mount_volume(config):
793+ if os.path.exists(config['mountpoint']):
794+ if not os.path.isdir(config['mountpoint']):
795+ hookenv.log('Not a directory: {}'.format(config['mountpoint']))
796+ raise VolumeConfigurationError()
797+ else:
798+ host.mkdir(config['mountpoint'])
799+ if os.path.ismount(config['mountpoint']):
800+ unmount_volume(config)
801+ if not host.mount(config['device'], config['mountpoint'], persist=True):
802+ raise VolumeConfigurationError()
803+
804+
805+def unmount_volume(config):
806+ if os.path.ismount(config['mountpoint']):
807+ if not host.umount(config['mountpoint'], persist=True):
808+ raise VolumeConfigurationError()
809+
810+
811+def managed_mounts():
812+ '''List of all mounted managed volumes'''
813+ return filter(lambda mount: mount[0].startswith(MOUNT_BASE), host.mounts())
814+
815+
816+def configure_volume(before_change=lambda: None, after_change=lambda: None):
817+ '''Set up storage (or don't) according to the charm's volume configuration.
818+ Returns the mount point or "ephemeral". before_change and after_change
819+ are optional functions to be called if the volume configuration changes.
820+ '''
821+
822+ config = get_config()
823+ if not config:
824+ hookenv.log('Failed to read volume configuration', hookenv.CRITICAL)
825+ raise VolumeConfigurationError()
826+
827+ if config['ephemeral']:
828+ if os.path.ismount(config['mountpoint']):
829+ before_change()
830+ unmount_volume(config)
831+ after_change()
832+ return 'ephemeral'
833+ else:
834+ # persistent storage
835+ if os.path.ismount(config['mountpoint']):
836+ mounts = dict(managed_mounts())
837+ if mounts.get(config['mountpoint']) != config['device']:
838+ before_change()
839+ unmount_volume(config)
840+ mount_volume(config)
841+ after_change()
842+ else:
843+ before_change()
844+ mount_volume(config)
845+ after_change()
846+ return config['mountpoint']
847
848=== modified file 'lib/charmhelpers/contrib/templating/__init__.py'
849--- lib/charmhelpers/contrib/templating/__init__.py 2014-09-23 12:09:14 +0000
850+++ lib/charmhelpers/contrib/templating/__init__.py 2015-11-01 20:32:57 +0000
851@@ -0,0 +1,15 @@
852+# Copyright 2014-2015 Canonical Limited.
853+#
854+# This file is part of charm-helpers.
855+#
856+# charm-helpers is free software: you can redistribute it and/or modify
857+# it under the terms of the GNU Lesser General Public License version 3 as
858+# published by the Free Software Foundation.
859+#
860+# charm-helpers is distributed in the hope that it will be useful,
861+# but WITHOUT ANY WARRANTY; without even the implied warranty of
862+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
863+# GNU Lesser General Public License for more details.
864+#
865+# You should have received a copy of the GNU Lesser General Public License
866+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
867
868=== modified file 'lib/charmhelpers/contrib/templating/jinja.py'
869--- lib/charmhelpers/contrib/templating/jinja.py 2014-09-23 12:09:14 +0000
870+++ lib/charmhelpers/contrib/templating/jinja.py 2015-11-01 20:32:57 +0000
871@@ -1,21 +1,37 @@
872+# Copyright 2014-2015 Canonical Limited.
873+#
874+# This file is part of charm-helpers.
875+#
876+# charm-helpers is free software: you can redistribute it and/or modify
877+# it under the terms of the GNU Lesser General Public License version 3 as
878+# published by the Free Software Foundation.
879+#
880+# charm-helpers is distributed in the hope that it will be useful,
881+# but WITHOUT ANY WARRANTY; without even the implied warranty of
882+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
883+# GNU Lesser General Public License for more details.
884+#
885+# You should have received a copy of the GNU Lesser General Public License
886+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
887+
888 """
889 Templating using the python-jinja2 package.
890 """
891-from charmhelpers.fetch import (
892- apt_install,
893-)
894-
895-
896-DEFAULT_TEMPLATES_DIR = 'templates'
897-
898-
899+import six
900+from charmhelpers.fetch import apt_install
901 try:
902 import jinja2
903 except ImportError:
904- apt_install(["python-jinja2"])
905+ if six.PY3:
906+ apt_install(["python3-jinja2"])
907+ else:
908+ apt_install(["python-jinja2"])
909 import jinja2
910
911
912+DEFAULT_TEMPLATES_DIR = 'templates'
913+
914+
915 def render(template_name, context, template_dir=DEFAULT_TEMPLATES_DIR):
916 templates = jinja2.Environment(
917 loader=jinja2.FileSystemLoader(template_dir))
918
919=== modified file 'lib/charmhelpers/core/__init__.py'
920--- lib/charmhelpers/core/__init__.py 2014-07-17 16:38:17 +0000
921+++ lib/charmhelpers/core/__init__.py 2015-11-01 20:32:57 +0000
922@@ -0,0 +1,15 @@
923+# Copyright 2014-2015 Canonical Limited.
924+#
925+# This file is part of charm-helpers.
926+#
927+# charm-helpers is free software: you can redistribute it and/or modify
928+# it under the terms of the GNU Lesser General Public License version 3 as
929+# published by the Free Software Foundation.
930+#
931+# charm-helpers is distributed in the hope that it will be useful,
932+# but WITHOUT ANY WARRANTY; without even the implied warranty of
933+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
934+# GNU Lesser General Public License for more details.
935+#
936+# You should have received a copy of the GNU Lesser General Public License
937+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
938
939=== added file 'lib/charmhelpers/core/decorators.py'
940--- lib/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
941+++ lib/charmhelpers/core/decorators.py 2015-11-01 20:32:57 +0000
942@@ -0,0 +1,57 @@
943+# Copyright 2014-2015 Canonical Limited.
944+#
945+# This file is part of charm-helpers.
946+#
947+# charm-helpers is free software: you can redistribute it and/or modify
948+# it under the terms of the GNU Lesser General Public License version 3 as
949+# published by the Free Software Foundation.
950+#
951+# charm-helpers is distributed in the hope that it will be useful,
952+# but WITHOUT ANY WARRANTY; without even the implied warranty of
953+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
954+# GNU Lesser General Public License for more details.
955+#
956+# You should have received a copy of the GNU Lesser General Public License
957+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
958+
959+#
960+# Copyright 2014 Canonical Ltd.
961+#
962+# Authors:
963+# Edward Hope-Morley <opentastic@gmail.com>
964+#
965+
966+import time
967+
968+from charmhelpers.core.hookenv import (
969+ log,
970+ INFO,
971+)
972+
973+
974+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
975+ """If the decorated function raises exception exc_type, allow num_retries
976+ retry attempts before raise the exception.
977+ """
978+ def _retry_on_exception_inner_1(f):
979+ def _retry_on_exception_inner_2(*args, **kwargs):
980+ retries = num_retries
981+ multiplier = 1
982+ while True:
983+ try:
984+ return f(*args, **kwargs)
985+ except exc_type:
986+ if not retries:
987+ raise
988+
989+ delay = base_delay * multiplier
990+ multiplier += 1
991+ log("Retrying '%s' %d more times (delay=%s)" %
992+ (f.__name__, retries, delay), level=INFO)
993+ retries -= 1
994+ if delay:
995+ time.sleep(delay)
996+
997+ return _retry_on_exception_inner_2
998+
999+ return _retry_on_exception_inner_1
1000
1001=== modified file 'lib/charmhelpers/core/fstab.py'
1002--- lib/charmhelpers/core/fstab.py 2014-07-17 16:38:17 +0000
1003+++ lib/charmhelpers/core/fstab.py 2015-11-01 20:32:57 +0000
1004@@ -1,12 +1,29 @@
1005 #!/usr/bin/env python
1006 # -*- coding: utf-8 -*-
1007
1008+# Copyright 2014-2015 Canonical Limited.
1009+#
1010+# This file is part of charm-helpers.
1011+#
1012+# charm-helpers is free software: you can redistribute it and/or modify
1013+# it under the terms of the GNU Lesser General Public License version 3 as
1014+# published by the Free Software Foundation.
1015+#
1016+# charm-helpers is distributed in the hope that it will be useful,
1017+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1018+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1019+# GNU Lesser General Public License for more details.
1020+#
1021+# You should have received a copy of the GNU Lesser General Public License
1022+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1023+
1024+import io
1025+import os
1026+
1027 __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1028
1029-import os
1030-
1031-
1032-class Fstab(file):
1033+
1034+class Fstab(io.FileIO):
1035 """This class extends file in order to implement a file reader/writer
1036 for file `/etc/fstab`
1037 """
1038@@ -24,8 +41,8 @@
1039 options = "defaults"
1040
1041 self.options = options
1042- self.d = d
1043- self.p = p
1044+ self.d = int(d)
1045+ self.p = int(p)
1046
1047 def __eq__(self, o):
1048 return str(self) == str(o)
1049@@ -45,7 +62,7 @@
1050 self._path = path
1051 else:
1052 self._path = self.DEFAULT_PATH
1053- file.__init__(self, self._path, 'r+')
1054+ super(Fstab, self).__init__(self._path, 'rb+')
1055
1056 def _hydrate_entry(self, line):
1057 # NOTE: use split with no arguments to split on any
1058@@ -58,8 +75,9 @@
1059 def entries(self):
1060 self.seek(0)
1061 for line in self.readlines():
1062+ line = line.decode('us-ascii')
1063 try:
1064- if not line.startswith("#"):
1065+ if line.strip() and not line.strip().startswith("#"):
1066 yield self._hydrate_entry(line)
1067 except ValueError:
1068 pass
1069@@ -75,18 +93,18 @@
1070 if self.get_entry_by_attr('device', entry.device):
1071 return False
1072
1073- self.write(str(entry) + '\n')
1074+ self.write((str(entry) + '\n').encode('us-ascii'))
1075 self.truncate()
1076 return entry
1077
1078 def remove_entry(self, entry):
1079 self.seek(0)
1080
1081- lines = self.readlines()
1082+ lines = [l.decode('us-ascii') for l in self.readlines()]
1083
1084 found = False
1085 for index, line in enumerate(lines):
1086- if not line.startswith("#"):
1087+ if line.strip() and not line.strip().startswith("#"):
1088 if self._hydrate_entry(line) == entry:
1089 found = True
1090 break
1091@@ -97,7 +115,7 @@
1092 lines.remove(line)
1093
1094 self.seek(0)
1095- self.write(''.join(lines))
1096+ self.write(''.join(lines).encode('us-ascii'))
1097 self.truncate()
1098 return True
1099
1100
1101=== modified file 'lib/charmhelpers/core/hookenv.py'
1102--- lib/charmhelpers/core/hookenv.py 2014-09-23 12:09:14 +0000
1103+++ lib/charmhelpers/core/hookenv.py 2015-11-01 20:32:57 +0000
1104@@ -1,17 +1,40 @@
1105+# Copyright 2014-2015 Canonical Limited.
1106+#
1107+# This file is part of charm-helpers.
1108+#
1109+# charm-helpers is free software: you can redistribute it and/or modify
1110+# it under the terms of the GNU Lesser General Public License version 3 as
1111+# published by the Free Software Foundation.
1112+#
1113+# charm-helpers is distributed in the hope that it will be useful,
1114+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1115+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1116+# GNU Lesser General Public License for more details.
1117+#
1118+# You should have received a copy of the GNU Lesser General Public License
1119+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1120+
1121 "Interactions with the Juju environment"
1122 # Copyright 2013 Canonical Ltd.
1123 #
1124 # Authors:
1125 # Charm Helpers Developers <juju@lists.ubuntu.com>
1126
1127+from __future__ import print_function
1128 import os
1129 import json
1130 import yaml
1131 import subprocess
1132 import sys
1133-import UserDict
1134+import errno
1135 from subprocess import CalledProcessError
1136
1137+import six
1138+if not six.PY3:
1139+ from UserDict import UserDict
1140+else:
1141+ from collections import UserDict
1142+
1143 CRITICAL = "CRITICAL"
1144 ERROR = "ERROR"
1145 WARNING = "WARNING"
1146@@ -63,16 +86,29 @@
1147 command = ['juju-log']
1148 if level:
1149 command += ['-l', level]
1150+ if not isinstance(message, six.string_types):
1151+ message = repr(message)
1152 command += [message]
1153- subprocess.call(command)
1154-
1155-
1156-class Serializable(UserDict.IterableUserDict):
1157+ # Missing juju-log should not cause failures in unit tests
1158+ # Send log output to stderr
1159+ try:
1160+ subprocess.call(command)
1161+ except OSError as e:
1162+ if e.errno == errno.ENOENT:
1163+ if level:
1164+ message = "{}: {}".format(level, message)
1165+ message = "juju-log: {}".format(message)
1166+ print(message, file=sys.stderr)
1167+ else:
1168+ raise
1169+
1170+
1171+class Serializable(UserDict):
1172 """Wrapper, an object that can be serialized to yaml or json"""
1173
1174 def __init__(self, obj):
1175 # wrap the object
1176- UserDict.IterableUserDict.__init__(self)
1177+ UserDict.__init__(self)
1178 self.data = obj
1179
1180 def __getattr__(self, attr):
1181@@ -214,6 +250,12 @@
1182 except KeyError:
1183 return (self._prev_dict or {})[key]
1184
1185+ def keys(self):
1186+ prev_keys = []
1187+ if self._prev_dict is not None:
1188+ prev_keys = self._prev_dict.keys()
1189+ return list(set(prev_keys + list(dict.keys(self))))
1190+
1191 def load_previous(self, path=None):
1192 """Load previous copy of config from disk.
1193
1194@@ -263,7 +305,7 @@
1195
1196 """
1197 if self._prev_dict:
1198- for k, v in self._prev_dict.iteritems():
1199+ for k, v in six.iteritems(self._prev_dict):
1200 if k not in self:
1201 self[k] = v
1202 with open(self.path, 'w') as f:
1203@@ -278,7 +320,8 @@
1204 config_cmd_line.append(scope)
1205 config_cmd_line.append('--format=json')
1206 try:
1207- config_data = json.loads(subprocess.check_output(config_cmd_line))
1208+ config_data = json.loads(
1209+ subprocess.check_output(config_cmd_line).decode('UTF-8'))
1210 if scope is not None:
1211 return config_data
1212 return Config(config_data)
1213@@ -297,10 +340,10 @@
1214 if unit:
1215 _args.append(unit)
1216 try:
1217- return json.loads(subprocess.check_output(_args))
1218+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1219 except ValueError:
1220 return None
1221- except CalledProcessError, e:
1222+ except CalledProcessError as e:
1223 if e.returncode == 2:
1224 return None
1225 raise
1226@@ -312,7 +355,7 @@
1227 relation_cmd_line = ['relation-set']
1228 if relation_id is not None:
1229 relation_cmd_line.extend(('-r', relation_id))
1230- for k, v in (relation_settings.items() + kwargs.items()):
1231+ for k, v in (list(relation_settings.items()) + list(kwargs.items())):
1232 if v is None:
1233 relation_cmd_line.append('{}='.format(k))
1234 else:
1235@@ -329,7 +372,8 @@
1236 relid_cmd_line = ['relation-ids', '--format=json']
1237 if reltype is not None:
1238 relid_cmd_line.append(reltype)
1239- return json.loads(subprocess.check_output(relid_cmd_line)) or []
1240+ return json.loads(
1241+ subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
1242 return []
1243
1244
1245@@ -340,7 +384,8 @@
1246 units_cmd_line = ['relation-list', '--format=json']
1247 if relid is not None:
1248 units_cmd_line.extend(('-r', relid))
1249- return json.loads(subprocess.check_output(units_cmd_line)) or []
1250+ return json.loads(
1251+ subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
1252
1253
1254 @cached
1255@@ -380,21 +425,31 @@
1256
1257
1258 @cached
1259+def metadata():
1260+ """Get the current charm metadata.yaml contents as a python object"""
1261+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
1262+ return yaml.safe_load(md)
1263+
1264+
1265+@cached
1266 def relation_types():
1267 """Get a list of relation types supported by this charm"""
1268- charmdir = os.environ.get('CHARM_DIR', '')
1269- mdf = open(os.path.join(charmdir, 'metadata.yaml'))
1270- md = yaml.safe_load(mdf)
1271 rel_types = []
1272+ md = metadata()
1273 for key in ('provides', 'requires', 'peers'):
1274 section = md.get(key)
1275 if section:
1276 rel_types.extend(section.keys())
1277- mdf.close()
1278 return rel_types
1279
1280
1281 @cached
1282+def charm_name():
1283+ """Get the name of the current charm as is specified on metadata.yaml"""
1284+ return metadata().get('name')
1285+
1286+
1287+@cached
1288 def relations():
1289 """Get a nested dictionary of relation data for all related units"""
1290 rels = {}
1291@@ -449,7 +504,7 @@
1292 """Get the unit ID for the remote unit"""
1293 _args = ['unit-get', '--format=json', attribute]
1294 try:
1295- return json.loads(subprocess.check_output(_args))
1296+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1297 except ValueError:
1298 return None
1299
1300@@ -486,9 +541,10 @@
1301 hooks.execute(sys.argv)
1302 """
1303
1304- def __init__(self):
1305+ def __init__(self, config_save=True):
1306 super(Hooks, self).__init__()
1307 self._hooks = {}
1308+ self._config_save = config_save
1309
1310 def register(self, name, function):
1311 """Register a hook"""
1312@@ -499,9 +555,10 @@
1313 hook_name = os.path.basename(args[0])
1314 if hook_name in self._hooks:
1315 self._hooks[hook_name]()
1316- cfg = config()
1317- if cfg.implicit_save:
1318- cfg.save()
1319+ if self._config_save:
1320+ cfg = config()
1321+ if cfg.implicit_save:
1322+ cfg.save()
1323 else:
1324 raise UnregisteredHookError(hook_name)
1325
1326@@ -522,3 +579,29 @@
1327 def charm_dir():
1328 """Return the root directory of the current charm"""
1329 return os.environ.get('CHARM_DIR')
1330+
1331+
1332+@cached
1333+def action_get(key=None):
1334+ """Gets the value of an action parameter, or all key/value param pairs"""
1335+ cmd = ['action-get']
1336+ if key is not None:
1337+ cmd.append(key)
1338+ cmd.append('--format=json')
1339+ action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1340+ return action_data
1341+
1342+
1343+def action_set(values):
1344+ """Sets the values to be returned after the action finishes"""
1345+ cmd = ['action-set']
1346+ for k, v in list(values.items()):
1347+ cmd.append('{}={}'.format(k, v))
1348+ subprocess.check_call(cmd)
1349+
1350+
1351+def action_fail(message):
1352+ """Sets the action status to failed and sets the error message.
1353+
1354+ The results set by action_set are preserved."""
1355+ subprocess.check_call(['action-fail', message])
1356
1357=== modified file 'lib/charmhelpers/core/host.py'
1358--- lib/charmhelpers/core/host.py 2014-09-23 12:09:14 +0000
1359+++ lib/charmhelpers/core/host.py 2015-11-01 20:32:57 +0000
1360@@ -1,3 +1,19 @@
1361+# Copyright 2014-2015 Canonical Limited.
1362+#
1363+# This file is part of charm-helpers.
1364+#
1365+# charm-helpers is free software: you can redistribute it and/or modify
1366+# it under the terms of the GNU Lesser General Public License version 3 as
1367+# published by the Free Software Foundation.
1368+#
1369+# charm-helpers is distributed in the hope that it will be useful,
1370+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1371+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1372+# GNU Lesser General Public License for more details.
1373+#
1374+# You should have received a copy of the GNU Lesser General Public License
1375+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1376+
1377 """Tools for working with the host system"""
1378 # Copyright 2012 Canonical Ltd.
1379 #
1380@@ -6,19 +22,20 @@
1381 # Matthew Wedgwood <matthew.wedgwood@canonical.com>
1382
1383 import os
1384+import re
1385 import pwd
1386 import grp
1387 import random
1388 import string
1389 import subprocess
1390 import hashlib
1391-import shutil
1392 from contextlib import contextmanager
1393-
1394 from collections import OrderedDict
1395
1396-from hookenv import log
1397-from fstab import Fstab
1398+import six
1399+
1400+from .hookenv import log
1401+from .fstab import Fstab
1402
1403
1404 def service_start(service_name):
1405@@ -54,7 +71,9 @@
1406 def service_running(service):
1407 """Determine whether a system service is running"""
1408 try:
1409- output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT)
1410+ output = subprocess.check_output(
1411+ ['service', service, 'status'],
1412+ stderr=subprocess.STDOUT).decode('UTF-8')
1413 except subprocess.CalledProcessError:
1414 return False
1415 else:
1416@@ -67,9 +86,11 @@
1417 def service_available(service_name):
1418 """Determine whether a system service is available"""
1419 try:
1420- subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT)
1421- except subprocess.CalledProcessError:
1422- return False
1423+ subprocess.check_output(
1424+ ['service', service_name, 'status'],
1425+ stderr=subprocess.STDOUT).decode('UTF-8')
1426+ except subprocess.CalledProcessError as e:
1427+ return 'unrecognized service' not in e.output
1428 else:
1429 return True
1430
1431@@ -96,6 +117,26 @@
1432 return user_info
1433
1434
1435+def add_group(group_name, system_group=False):
1436+ """Add a group to the system"""
1437+ try:
1438+ group_info = grp.getgrnam(group_name)
1439+ log('group {0} already exists!'.format(group_name))
1440+ except KeyError:
1441+ log('creating group {0}'.format(group_name))
1442+ cmd = ['addgroup']
1443+ if system_group:
1444+ cmd.append('--system')
1445+ else:
1446+ cmd.extend([
1447+ '--group',
1448+ ])
1449+ cmd.append(group_name)
1450+ subprocess.check_call(cmd)
1451+ group_info = grp.getgrnam(group_name)
1452+ return group_info
1453+
1454+
1455 def add_user_to_group(username, group):
1456 """Add a user to a group"""
1457 cmd = [
1458@@ -115,7 +156,7 @@
1459 cmd.append(from_path)
1460 cmd.append(to_path)
1461 log(" ".join(cmd))
1462- return subprocess.check_output(cmd).strip()
1463+ return subprocess.check_output(cmd).decode('UTF-8').strip()
1464
1465
1466 def symlink(source, destination):
1467@@ -130,28 +171,31 @@
1468 subprocess.check_call(cmd)
1469
1470
1471-def mkdir(path, owner='root', group='root', perms=0555, force=False):
1472+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
1473 """Create a directory"""
1474 log("Making dir {} {}:{} {:o}".format(path, owner, group,
1475 perms))
1476 uid = pwd.getpwnam(owner).pw_uid
1477 gid = grp.getgrnam(group).gr_gid
1478 realpath = os.path.abspath(path)
1479- if os.path.exists(realpath):
1480- if force and not os.path.isdir(realpath):
1481+ path_exists = os.path.exists(realpath)
1482+ if path_exists and force:
1483+ if not os.path.isdir(realpath):
1484 log("Removing non-directory file {} prior to mkdir()".format(path))
1485 os.unlink(realpath)
1486- else:
1487+ os.makedirs(realpath, perms)
1488+ elif not path_exists:
1489 os.makedirs(realpath, perms)
1490 os.chown(realpath, uid, gid)
1491-
1492-
1493-def write_file(path, content, owner='root', group='root', perms=0444):
1494- """Create or overwrite a file with the contents of a string"""
1495+ os.chmod(realpath, perms)
1496+
1497+
1498+def write_file(path, content, owner='root', group='root', perms=0o444):
1499+ """Create or overwrite a file with the contents of a byte string."""
1500 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1501 uid = pwd.getpwnam(owner).pw_uid
1502 gid = grp.getgrnam(group).gr_gid
1503- with open(path, 'w') as target:
1504+ with open(path, 'wb') as target:
1505 os.fchown(target.fileno(), uid, gid)
1506 os.fchmod(target.fileno(), perms)
1507 target.write(content)
1508@@ -177,7 +221,7 @@
1509 cmd_args.extend([device, mountpoint])
1510 try:
1511 subprocess.check_output(cmd_args)
1512- except subprocess.CalledProcessError, e:
1513+ except subprocess.CalledProcessError as e:
1514 log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
1515 return False
1516
1517@@ -191,7 +235,7 @@
1518 cmd_args = ['umount', mountpoint]
1519 try:
1520 subprocess.check_output(cmd_args)
1521- except subprocess.CalledProcessError, e:
1522+ except subprocess.CalledProcessError as e:
1523 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1524 return False
1525
1526@@ -218,8 +262,8 @@
1527 """
1528 if os.path.exists(path):
1529 h = getattr(hashlib, hash_type)()
1530- with open(path, 'r') as source:
1531- h.update(source.read()) # IGNORE:E1101 - it does have update
1532+ with open(path, 'rb') as source:
1533+ h.update(source.read())
1534 return h.hexdigest()
1535 else:
1536 return None
1537@@ -229,12 +273,12 @@
1538 """
1539 Validate a file using a cryptographic checksum.
1540
1541-
1542 :param str checksum: Value of the checksum used to validate the file.
1543- :param str hash_type: Hash algorithm used to generate :param:`checksum`.
1544- Can be any hash alrgorithm supported by :mod:`hashlib`,
1545- such as md5, sha1, sha256, sha512, etc.
1546+ :param str hash_type: Hash algorithm used to generate `checksum`.
1547+ Can be any hash alrgorithm supported by :mod:`hashlib`,
1548+ such as md5, sha1, sha256, sha512, etc.
1549 :raises ChecksumError: If the file fails the checksum
1550+
1551 """
1552 actual_checksum = file_hash(path, hash_type)
1553 if checksum != actual_checksum:
1554@@ -261,11 +305,11 @@
1555 ceph_client_changed function.
1556 """
1557 def wrap(f):
1558- def wrapped_f(*args):
1559+ def wrapped_f(*args, **kwargs):
1560 checksums = {}
1561 for path in restart_map:
1562 checksums[path] = file_hash(path)
1563- f(*args)
1564+ f(*args, **kwargs)
1565 restarts = []
1566 for path in restart_map:
1567 if checksums[path] != file_hash(path):
1568@@ -295,29 +339,39 @@
1569 def pwgen(length=None):
1570 """Generate a random pasword."""
1571 if length is None:
1572+ # A random length is ok to use a weak PRNG
1573 length = random.choice(range(35, 45))
1574 alphanumeric_chars = [
1575- l for l in (string.letters + string.digits)
1576+ l for l in (string.ascii_letters + string.digits)
1577 if l not in 'l0QD1vAEIOUaeiou']
1578+ # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
1579+ # actual password
1580+ random_generator = random.SystemRandom()
1581 random_chars = [
1582- random.choice(alphanumeric_chars) for _ in range(length)]
1583+ random_generator.choice(alphanumeric_chars) for _ in range(length)]
1584 return(''.join(random_chars))
1585
1586
1587 def list_nics(nic_type):
1588 '''Return a list of nics of given type(s)'''
1589- if isinstance(nic_type, basestring):
1590+ if isinstance(nic_type, six.string_types):
1591 int_types = [nic_type]
1592 else:
1593 int_types = nic_type
1594 interfaces = []
1595 for int_type in int_types:
1596 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1597- ip_output = subprocess.check_output(cmd).split('\n')
1598+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1599 ip_output = (line for line in ip_output if line)
1600 for line in ip_output:
1601 if line.split()[1].startswith(int_type):
1602- interfaces.append(line.split()[1].replace(":", ""))
1603+ matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
1604+ if matched:
1605+ interface = matched.groups()[0]
1606+ else:
1607+ interface = line.split()[1].replace(":", "")
1608+ interfaces.append(interface)
1609+
1610 return interfaces
1611
1612
1613@@ -329,7 +383,7 @@
1614
1615 def get_nic_mtu(nic):
1616 cmd = ['ip', 'addr', 'show', nic]
1617- ip_output = subprocess.check_output(cmd).split('\n')
1618+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1619 mtu = ""
1620 for line in ip_output:
1621 words = line.split()
1622@@ -340,7 +394,7 @@
1623
1624 def get_nic_hwaddr(nic):
1625 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
1626- ip_output = subprocess.check_output(cmd)
1627+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1628 hwaddr = ""
1629 words = ip_output.split()
1630 if 'link/ether' in words:
1631@@ -355,10 +409,13 @@
1632 * 0 => Installed revno is the same as supplied arg
1633 * -1 => Installed revno is less than supplied arg
1634
1635+ This function imports apt_cache function from charmhelpers.fetch if
1636+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
1637+ you call this function, or pass an apt_pkg.Cache() instance.
1638 '''
1639 import apt_pkg
1640- from charmhelpers.fetch import apt_cache
1641 if not pkgcache:
1642+ from charmhelpers.fetch import apt_cache
1643 pkgcache = apt_cache()
1644 pkg = pkgcache[package]
1645 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
1646@@ -373,13 +430,21 @@
1647 os.chdir(cur)
1648
1649
1650-def chownr(path, owner, group):
1651+def chownr(path, owner, group, follow_links=True):
1652 uid = pwd.getpwnam(owner).pw_uid
1653 gid = grp.getgrnam(group).gr_gid
1654+ if follow_links:
1655+ chown = os.chown
1656+ else:
1657+ chown = os.lchown
1658
1659 for root, dirs, files in os.walk(path):
1660 for name in dirs + files:
1661 full = os.path.join(root, name)
1662 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
1663 if not broken_symlink:
1664- os.chown(full, uid, gid)
1665+ chown(full, uid, gid)
1666+
1667+
1668+def lchownr(path, owner, group):
1669+ chownr(path, owner, group, follow_links=False)
1670
1671=== modified file 'lib/charmhelpers/core/services/__init__.py'
1672--- lib/charmhelpers/core/services/__init__.py 2014-09-23 12:09:14 +0000
1673+++ lib/charmhelpers/core/services/__init__.py 2015-11-01 20:32:57 +0000
1674@@ -1,2 +1,18 @@
1675-from .base import *
1676-from .helpers import *
1677+# Copyright 2014-2015 Canonical Limited.
1678+#
1679+# This file is part of charm-helpers.
1680+#
1681+# charm-helpers is free software: you can redistribute it and/or modify
1682+# it under the terms of the GNU Lesser General Public License version 3 as
1683+# published by the Free Software Foundation.
1684+#
1685+# charm-helpers is distributed in the hope that it will be useful,
1686+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1687+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1688+# GNU Lesser General Public License for more details.
1689+#
1690+# You should have received a copy of the GNU Lesser General Public License
1691+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1692+
1693+from .base import * # NOQA
1694+from .helpers import * # NOQA
1695
1696=== modified file 'lib/charmhelpers/core/services/base.py'
1697--- lib/charmhelpers/core/services/base.py 2014-09-23 12:09:14 +0000
1698+++ lib/charmhelpers/core/services/base.py 2015-11-01 20:32:57 +0000
1699@@ -1,3 +1,19 @@
1700+# Copyright 2014-2015 Canonical Limited.
1701+#
1702+# This file is part of charm-helpers.
1703+#
1704+# charm-helpers is free software: you can redistribute it and/or modify
1705+# it under the terms of the GNU Lesser General Public License version 3 as
1706+# published by the Free Software Foundation.
1707+#
1708+# charm-helpers is distributed in the hope that it will be useful,
1709+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1710+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1711+# GNU Lesser General Public License for more details.
1712+#
1713+# You should have received a copy of the GNU Lesser General Public License
1714+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1715+
1716 import os
1717 import re
1718 import json
1719
1720=== modified file 'lib/charmhelpers/core/services/helpers.py'
1721--- lib/charmhelpers/core/services/helpers.py 2014-09-23 12:09:14 +0000
1722+++ lib/charmhelpers/core/services/helpers.py 2015-11-01 20:32:57 +0000
1723@@ -1,3 +1,19 @@
1724+# Copyright 2014-2015 Canonical Limited.
1725+#
1726+# This file is part of charm-helpers.
1727+#
1728+# charm-helpers is free software: you can redistribute it and/or modify
1729+# it under the terms of the GNU Lesser General Public License version 3 as
1730+# published by the Free Software Foundation.
1731+#
1732+# charm-helpers is distributed in the hope that it will be useful,
1733+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1734+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1735+# GNU Lesser General Public License for more details.
1736+#
1737+# You should have received a copy of the GNU Lesser General Public License
1738+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1739+
1740 import os
1741 import yaml
1742 from charmhelpers.core import hookenv
1743@@ -29,12 +45,14 @@
1744 """
1745 name = None
1746 interface = None
1747- required_keys = []
1748
1749 def __init__(self, name=None, additional_required_keys=None):
1750+ if not hasattr(self, 'required_keys'):
1751+ self.required_keys = []
1752+
1753 if name is not None:
1754 self.name = name
1755- if additional_required_keys is not None:
1756+ if additional_required_keys:
1757 self.required_keys.extend(additional_required_keys)
1758 self.get_data()
1759
1760@@ -118,7 +136,10 @@
1761 """
1762 name = 'db'
1763 interface = 'mysql'
1764- required_keys = ['host', 'user', 'password', 'database']
1765+
1766+ def __init__(self, *args, **kwargs):
1767+ self.required_keys = ['host', 'user', 'password', 'database']
1768+ RelationContext.__init__(self, *args, **kwargs)
1769
1770
1771 class HttpRelation(RelationContext):
1772@@ -130,7 +151,10 @@
1773 """
1774 name = 'website'
1775 interface = 'http'
1776- required_keys = ['host', 'port']
1777+
1778+ def __init__(self, *args, **kwargs):
1779+ self.required_keys = ['host', 'port']
1780+ RelationContext.__init__(self, *args, **kwargs)
1781
1782 def provide_data(self):
1783 return {
1784@@ -196,7 +220,7 @@
1785 if not os.path.isabs(file_name):
1786 file_name = os.path.join(hookenv.charm_dir(), file_name)
1787 with open(file_name, 'w') as file_stream:
1788- os.fchmod(file_stream.fileno(), 0600)
1789+ os.fchmod(file_stream.fileno(), 0o600)
1790 yaml.dump(config_data, file_stream)
1791
1792 def read_context(self, file_name):
1793@@ -211,15 +235,19 @@
1794
1795 class TemplateCallback(ManagerCallback):
1796 """
1797- Callback class that will render a Jinja2 template, for use as a ready action.
1798-
1799- :param str source: The template source file, relative to `$CHARM_DIR/templates`
1800+ Callback class that will render a Jinja2 template, for use as a ready
1801+ action.
1802+
1803+ :param str source: The template source file, relative to
1804+ `$CHARM_DIR/templates`
1805+
1806 :param str target: The target to write the rendered template to
1807 :param str owner: The owner of the rendered file
1808 :param str group: The group of the rendered file
1809 :param int perms: The permissions of the rendered file
1810 """
1811- def __init__(self, source, target, owner='root', group='root', perms=0444):
1812+ def __init__(self, source, target,
1813+ owner='root', group='root', perms=0o444):
1814 self.source = source
1815 self.target = target
1816 self.owner = owner
1817
1818=== added file 'lib/charmhelpers/core/strutils.py'
1819--- lib/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
1820+++ lib/charmhelpers/core/strutils.py 2015-11-01 20:32:57 +0000
1821@@ -0,0 +1,42 @@
1822+#!/usr/bin/env python
1823+# -*- coding: utf-8 -*-
1824+
1825+# Copyright 2014-2015 Canonical Limited.
1826+#
1827+# This file is part of charm-helpers.
1828+#
1829+# charm-helpers is free software: you can redistribute it and/or modify
1830+# it under the terms of the GNU Lesser General Public License version 3 as
1831+# published by the Free Software Foundation.
1832+#
1833+# charm-helpers is distributed in the hope that it will be useful,
1834+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1835+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1836+# GNU Lesser General Public License for more details.
1837+#
1838+# You should have received a copy of the GNU Lesser General Public License
1839+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1840+
1841+import six
1842+
1843+
1844+def bool_from_string(value):
1845+ """Interpret string value as boolean.
1846+
1847+ Returns True if value translates to True otherwise False.
1848+ """
1849+ if isinstance(value, six.string_types):
1850+ value = six.text_type(value)
1851+ else:
1852+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
1853+ raise ValueError(msg)
1854+
1855+ value = value.strip().lower()
1856+
1857+ if value in ['y', 'yes', 'true', 't', 'on']:
1858+ return True
1859+ elif value in ['n', 'no', 'false', 'f', 'off']:
1860+ return False
1861+
1862+ msg = "Unable to interpret string value '%s' as boolean" % (value)
1863+ raise ValueError(msg)
1864
1865=== added file 'lib/charmhelpers/core/sysctl.py'
1866--- lib/charmhelpers/core/sysctl.py 1970-01-01 00:00:00 +0000
1867+++ lib/charmhelpers/core/sysctl.py 2015-11-01 20:32:57 +0000
1868@@ -0,0 +1,56 @@
1869+#!/usr/bin/env python
1870+# -*- coding: utf-8 -*-
1871+
1872+# Copyright 2014-2015 Canonical Limited.
1873+#
1874+# This file is part of charm-helpers.
1875+#
1876+# charm-helpers is free software: you can redistribute it and/or modify
1877+# it under the terms of the GNU Lesser General Public License version 3 as
1878+# published by the Free Software Foundation.
1879+#
1880+# charm-helpers is distributed in the hope that it will be useful,
1881+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1882+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1883+# GNU Lesser General Public License for more details.
1884+#
1885+# You should have received a copy of the GNU Lesser General Public License
1886+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1887+
1888+import yaml
1889+
1890+from subprocess import check_call
1891+
1892+from charmhelpers.core.hookenv import (
1893+ log,
1894+ DEBUG,
1895+ ERROR,
1896+)
1897+
1898+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1899+
1900+
1901+def create(sysctl_dict, sysctl_file):
1902+ """Creates a sysctl.conf file from a YAML associative array
1903+
1904+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
1905+ :type sysctl_dict: str
1906+ :param sysctl_file: path to the sysctl file to be saved
1907+ :type sysctl_file: str or unicode
1908+ :returns: None
1909+ """
1910+ try:
1911+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
1912+ except yaml.YAMLError:
1913+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
1914+ level=ERROR)
1915+ return
1916+
1917+ with open(sysctl_file, "w") as fd:
1918+ for key, value in sysctl_dict_parsed.items():
1919+ fd.write("{}={}\n".format(key, value))
1920+
1921+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
1922+ level=DEBUG)
1923+
1924+ check_call(["sysctl", "-p", sysctl_file])
1925
1926=== modified file 'lib/charmhelpers/core/templating.py'
1927--- lib/charmhelpers/core/templating.py 2014-09-23 12:09:14 +0000
1928+++ lib/charmhelpers/core/templating.py 2015-11-01 20:32:57 +0000
1929@@ -1,10 +1,27 @@
1930+# Copyright 2014-2015 Canonical Limited.
1931+#
1932+# This file is part of charm-helpers.
1933+#
1934+# charm-helpers is free software: you can redistribute it and/or modify
1935+# it under the terms of the GNU Lesser General Public License version 3 as
1936+# published by the Free Software Foundation.
1937+#
1938+# charm-helpers is distributed in the hope that it will be useful,
1939+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1940+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1941+# GNU Lesser General Public License for more details.
1942+#
1943+# You should have received a copy of the GNU Lesser General Public License
1944+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1945+
1946 import os
1947
1948 from charmhelpers.core import host
1949 from charmhelpers.core import hookenv
1950
1951
1952-def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None):
1953+def render(source, target, context, owner='root', group='root',
1954+ perms=0o444, templates_dir=None, encoding='UTF-8'):
1955 """
1956 Render a template.
1957
1958@@ -47,5 +64,5 @@
1959 level=hookenv.ERROR)
1960 raise e
1961 content = template.render(context)
1962- host.mkdir(os.path.dirname(target))
1963- host.write_file(target, content, owner, group, perms)
1964+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
1965+ host.write_file(target, content.encode(encoding), owner, group, perms)
1966
1967=== added file 'lib/charmhelpers/core/unitdata.py'
1968--- lib/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
1969+++ lib/charmhelpers/core/unitdata.py 2015-11-01 20:32:57 +0000
1970@@ -0,0 +1,477 @@
1971+#!/usr/bin/env python
1972+# -*- coding: utf-8 -*-
1973+#
1974+# Copyright 2014-2015 Canonical Limited.
1975+#
1976+# This file is part of charm-helpers.
1977+#
1978+# charm-helpers is free software: you can redistribute it and/or modify
1979+# it under the terms of the GNU Lesser General Public License version 3 as
1980+# published by the Free Software Foundation.
1981+#
1982+# charm-helpers is distributed in the hope that it will be useful,
1983+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1984+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1985+# GNU Lesser General Public License for more details.
1986+#
1987+# You should have received a copy of the GNU Lesser General Public License
1988+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1989+#
1990+#
1991+# Authors:
1992+# Kapil Thangavelu <kapil.foss@gmail.com>
1993+#
1994+"""
1995+Intro
1996+-----
1997+
1998+A simple way to store state in units. This provides a key value
1999+storage with support for versioned, transactional operation,
2000+and can calculate deltas from previous values to simplify unit logic
2001+when processing changes.
2002+
2003+
2004+Hook Integration
2005+----------------
2006+
2007+There are several extant frameworks for hook execution, including
2008+
2009+ - charmhelpers.core.hookenv.Hooks
2010+ - charmhelpers.core.services.ServiceManager
2011+
2012+The storage classes are framework agnostic, one simple integration is
2013+via the HookData contextmanager. It will record the current hook
2014+execution environment (including relation data, config data, etc.),
2015+setup a transaction and allow easy access to the changes from
2016+previously seen values. One consequence of the integration is the
2017+reservation of particular keys ('rels', 'unit', 'env', 'config',
2018+'charm_revisions') for their respective values.
2019+
2020+Here's a fully worked integration example using hookenv.Hooks::
2021+
2022+ from charmhelper.core import hookenv, unitdata
2023+
2024+ hook_data = unitdata.HookData()
2025+ db = unitdata.kv()
2026+ hooks = hookenv.Hooks()
2027+
2028+ @hooks.hook
2029+ def config_changed():
2030+ # Print all changes to configuration from previously seen
2031+ # values.
2032+ for changed, (prev, cur) in hook_data.conf.items():
2033+ print('config changed', changed,
2034+ 'previous value', prev,
2035+ 'current value', cur)
2036+
2037+ # Get some unit specific bookeeping
2038+ if not db.get('pkg_key'):
2039+ key = urllib.urlopen('https://example.com/pkg_key').read()
2040+ db.set('pkg_key', key)
2041+
2042+ # Directly access all charm config as a mapping.
2043+ conf = db.getrange('config', True)
2044+
2045+ # Directly access all relation data as a mapping
2046+ rels = db.getrange('rels', True)
2047+
2048+ if __name__ == '__main__':
2049+ with hook_data():
2050+ hook.execute()
2051+
2052+
2053+A more basic integration is via the hook_scope context manager which simply
2054+manages transaction scope (and records hook name, and timestamp)::
2055+
2056+ >>> from unitdata import kv
2057+ >>> db = kv()
2058+ >>> with db.hook_scope('install'):
2059+ ... # do work, in transactional scope.
2060+ ... db.set('x', 1)
2061+ >>> db.get('x')
2062+ 1
2063+
2064+
2065+Usage
2066+-----
2067+
2068+Values are automatically json de/serialized to preserve basic typing
2069+and complex data struct capabilities (dicts, lists, ints, booleans, etc).
2070+
2071+Individual values can be manipulated via get/set::
2072+
2073+ >>> kv.set('y', True)
2074+ >>> kv.get('y')
2075+ True
2076+
2077+ # We can set complex values (dicts, lists) as a single key.
2078+ >>> kv.set('config', {'a': 1, 'b': True'})
2079+
2080+ # Also supports returning dictionaries as a record which
2081+ # provides attribute access.
2082+ >>> config = kv.get('config', record=True)
2083+ >>> config.b
2084+ True
2085+
2086+
2087+Groups of keys can be manipulated with update/getrange::
2088+
2089+ >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
2090+ >>> kv.getrange('gui.', strip=True)
2091+ {'z': 1, 'y': 2}
2092+
2093+When updating values, its very helpful to understand which values
2094+have actually changed and how have they changed. The storage
2095+provides a delta method to provide for this::
2096+
2097+ >>> data = {'debug': True, 'option': 2}
2098+ >>> delta = kv.delta(data, 'config.')
2099+ >>> delta.debug.previous
2100+ None
2101+ >>> delta.debug.current
2102+ True
2103+ >>> delta
2104+ {'debug': (None, True), 'option': (None, 2)}
2105+
2106+Note the delta method does not persist the actual change, it needs to
2107+be explicitly saved via 'update' method::
2108+
2109+ >>> kv.update(data, 'config.')
2110+
2111+Values modified in the context of a hook scope retain historical values
2112+associated to the hookname.
2113+
2114+ >>> with db.hook_scope('config-changed'):
2115+ ... db.set('x', 42)
2116+ >>> db.gethistory('x')
2117+ [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
2118+ (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
2119+
2120+"""
2121+
2122+import collections
2123+import contextlib
2124+import datetime
2125+import json
2126+import os
2127+import pprint
2128+import sqlite3
2129+import sys
2130+
2131+__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
2132+
2133+
2134+class Storage(object):
2135+ """Simple key value database for local unit state within charms.
2136+
2137+ Modifications are automatically committed at hook exit. That's
2138+ currently regardless of exit code.
2139+
2140+ To support dicts, lists, integer, floats, and booleans values
2141+ are automatically json encoded/decoded.
2142+ """
2143+ def __init__(self, path=None):
2144+ self.db_path = path
2145+ if path is None:
2146+ self.db_path = os.path.join(
2147+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
2148+ self.conn = sqlite3.connect('%s' % self.db_path)
2149+ self.cursor = self.conn.cursor()
2150+ self.revision = None
2151+ self._closed = False
2152+ self._init()
2153+
2154+ def close(self):
2155+ if self._closed:
2156+ return
2157+ self.flush(False)
2158+ self.cursor.close()
2159+ self.conn.close()
2160+ self._closed = True
2161+
2162+ def _scoped_query(self, stmt, params=None):
2163+ if params is None:
2164+ params = []
2165+ return stmt, params
2166+
2167+ def get(self, key, default=None, record=False):
2168+ self.cursor.execute(
2169+ *self._scoped_query(
2170+ 'select data from kv where key=?', [key]))
2171+ result = self.cursor.fetchone()
2172+ if not result:
2173+ return default
2174+ if record:
2175+ return Record(json.loads(result[0]))
2176+ return json.loads(result[0])
2177+
2178+ def getrange(self, key_prefix, strip=False):
2179+ stmt = "select key, data from kv where key like '%s%%'" % key_prefix
2180+ self.cursor.execute(*self._scoped_query(stmt))
2181+ result = self.cursor.fetchall()
2182+
2183+ if not result:
2184+ return None
2185+ if not strip:
2186+ key_prefix = ''
2187+ return dict([
2188+ (k[len(key_prefix):], json.loads(v)) for k, v in result])
2189+
2190+ def update(self, mapping, prefix=""):
2191+ for k, v in mapping.items():
2192+ self.set("%s%s" % (prefix, k), v)
2193+
2194+ def unset(self, key):
2195+ self.cursor.execute('delete from kv where key=?', [key])
2196+ if self.revision and self.cursor.rowcount:
2197+ self.cursor.execute(
2198+ 'insert into kv_revisions values (?, ?, ?)',
2199+ [key, self.revision, json.dumps('DELETED')])
2200+
2201+ def set(self, key, value):
2202+ serialized = json.dumps(value)
2203+
2204+ self.cursor.execute(
2205+ 'select data from kv where key=?', [key])
2206+ exists = self.cursor.fetchone()
2207+
2208+ # Skip mutations to the same value
2209+ if exists:
2210+ if exists[0] == serialized:
2211+ return value
2212+
2213+ if not exists:
2214+ self.cursor.execute(
2215+ 'insert into kv (key, data) values (?, ?)',
2216+ (key, serialized))
2217+ else:
2218+ self.cursor.execute('''
2219+ update kv
2220+ set data = ?
2221+ where key = ?''', [serialized, key])
2222+
2223+ # Save
2224+ if not self.revision:
2225+ return value
2226+
2227+ self.cursor.execute(
2228+ 'select 1 from kv_revisions where key=? and revision=?',
2229+ [key, self.revision])
2230+ exists = self.cursor.fetchone()
2231+
2232+ if not exists:
2233+ self.cursor.execute(
2234+ '''insert into kv_revisions (
2235+ revision, key, data) values (?, ?, ?)''',
2236+ (self.revision, key, serialized))
2237+ else:
2238+ self.cursor.execute(
2239+ '''
2240+ update kv_revisions
2241+ set data = ?
2242+ where key = ?
2243+ and revision = ?''',
2244+ [serialized, key, self.revision])
2245+
2246+ return value
2247+
2248+ def delta(self, mapping, prefix):
2249+ """
2250+ return a delta containing values that have changed.
2251+ """
2252+ previous = self.getrange(prefix, strip=True)
2253+ if not previous:
2254+ pk = set()
2255+ else:
2256+ pk = set(previous.keys())
2257+ ck = set(mapping.keys())
2258+ delta = DeltaSet()
2259+
2260+ # added
2261+ for k in ck.difference(pk):
2262+ delta[k] = Delta(None, mapping[k])
2263+
2264+ # removed
2265+ for k in pk.difference(ck):
2266+ delta[k] = Delta(previous[k], None)
2267+
2268+ # changed
2269+ for k in pk.intersection(ck):
2270+ c = mapping[k]
2271+ p = previous[k]
2272+ if c != p:
2273+ delta[k] = Delta(p, c)
2274+
2275+ return delta
2276+
2277+ @contextlib.contextmanager
2278+ def hook_scope(self, name=""):
2279+ """Scope all future interactions to the current hook execution
2280+ revision."""
2281+ assert not self.revision
2282+ self.cursor.execute(
2283+ 'insert into hooks (hook, date) values (?, ?)',
2284+ (name or sys.argv[0],
2285+ datetime.datetime.utcnow().isoformat()))
2286+ self.revision = self.cursor.lastrowid
2287+ try:
2288+ yield self.revision
2289+ self.revision = None
2290+ except:
2291+ self.flush(False)
2292+ self.revision = None
2293+ raise
2294+ else:
2295+ self.flush()
2296+
2297+ def flush(self, save=True):
2298+ if save:
2299+ self.conn.commit()
2300+ elif self._closed:
2301+ return
2302+ else:
2303+ self.conn.rollback()
2304+
2305+ def _init(self):
2306+ self.cursor.execute('''
2307+ create table if not exists kv (
2308+ key text,
2309+ data text,
2310+ primary key (key)
2311+ )''')
2312+ self.cursor.execute('''
2313+ create table if not exists kv_revisions (
2314+ key text,
2315+ revision integer,
2316+ data text,
2317+ primary key (key, revision)
2318+ )''')
2319+ self.cursor.execute('''
2320+ create table if not exists hooks (
2321+ version integer primary key autoincrement,
2322+ hook text,
2323+ date text
2324+ )''')
2325+ self.conn.commit()
2326+
2327+ def gethistory(self, key, deserialize=False):
2328+ self.cursor.execute(
2329+ '''
2330+ select kv.revision, kv.key, kv.data, h.hook, h.date
2331+ from kv_revisions kv,
2332+ hooks h
2333+ where kv.key=?
2334+ and kv.revision = h.version
2335+ ''', [key])
2336+ if deserialize is False:
2337+ return self.cursor.fetchall()
2338+ return map(_parse_history, self.cursor.fetchall())
2339+
2340+ def debug(self, fh=sys.stderr):
2341+ self.cursor.execute('select * from kv')
2342+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2343+ self.cursor.execute('select * from kv_revisions')
2344+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2345+
2346+
2347+def _parse_history(d):
2348+ return (d[0], d[1], json.loads(d[2]), d[3],
2349+ datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
2350+
2351+
2352+class HookData(object):
2353+ """Simple integration for existing hook exec frameworks.
2354+
2355+ Records all unit information, and stores deltas for processing
2356+ by the hook.
2357+
2358+ Sample::
2359+
2360+ from charmhelper.core import hookenv, unitdata
2361+
2362+ changes = unitdata.HookData()
2363+ db = unitdata.kv()
2364+ hooks = hookenv.Hooks()
2365+
2366+ @hooks.hook
2367+ def config_changed():
2368+ # View all changes to configuration
2369+ for changed, (prev, cur) in changes.conf.items():
2370+ print('config changed', changed,
2371+ 'previous value', prev,
2372+ 'current value', cur)
2373+
2374+ # Get some unit specific bookeeping
2375+ if not db.get('pkg_key'):
2376+ key = urllib.urlopen('https://example.com/pkg_key').read()
2377+ db.set('pkg_key', key)
2378+
2379+ if __name__ == '__main__':
2380+ with changes():
2381+ hook.execute()
2382+
2383+ """
2384+ def __init__(self):
2385+ self.kv = kv()
2386+ self.conf = None
2387+ self.rels = None
2388+
2389+ @contextlib.contextmanager
2390+ def __call__(self):
2391+ from charmhelpers.core import hookenv
2392+ hook_name = hookenv.hook_name()
2393+
2394+ with self.kv.hook_scope(hook_name):
2395+ self._record_charm_version(hookenv.charm_dir())
2396+ delta_config, delta_relation = self._record_hook(hookenv)
2397+ yield self.kv, delta_config, delta_relation
2398+
2399+ def _record_charm_version(self, charm_dir):
2400+ # Record revisions.. charm revisions are meaningless
2401+ # to charm authors as they don't control the revision.
2402+ # so logic dependnent on revision is not particularly
2403+ # useful, however it is useful for debugging analysis.
2404+ charm_rev = open(
2405+ os.path.join(charm_dir, 'revision')).read().strip()
2406+ charm_rev = charm_rev or '0'
2407+ revs = self.kv.get('charm_revisions', [])
2408+ if charm_rev not in revs:
2409+ revs.append(charm_rev.strip() or '0')
2410+ self.kv.set('charm_revisions', revs)
2411+
2412+ def _record_hook(self, hookenv):
2413+ data = hookenv.execution_environment()
2414+ self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
2415+ self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
2416+ self.kv.set('env', dict(data['env']))
2417+ self.kv.set('unit', data['unit'])
2418+ self.kv.set('relid', data.get('relid'))
2419+ return conf_delta, rels_delta
2420+
2421+
2422+class Record(dict):
2423+
2424+ __slots__ = ()
2425+
2426+ def __getattr__(self, k):
2427+ if k in self:
2428+ return self[k]
2429+ raise AttributeError(k)
2430+
2431+
2432+class DeltaSet(Record):
2433+
2434+ __slots__ = ()
2435+
2436+
2437+Delta = collections.namedtuple('Delta', ['previous', 'current'])
2438+
2439+
2440+_KV = None
2441+
2442+
2443+def kv():
2444+ global _KV
2445+ if _KV is None:
2446+ _KV = Storage()
2447+ return _KV
2448
2449=== modified file 'lib/charmhelpers/fetch/__init__.py'
2450--- lib/charmhelpers/fetch/__init__.py 2014-09-23 12:09:14 +0000
2451+++ lib/charmhelpers/fetch/__init__.py 2015-11-01 20:32:57 +0000
2452@@ -1,3 +1,19 @@
2453+# Copyright 2014-2015 Canonical Limited.
2454+#
2455+# This file is part of charm-helpers.
2456+#
2457+# charm-helpers is free software: you can redistribute it and/or modify
2458+# it under the terms of the GNU Lesser General Public License version 3 as
2459+# published by the Free Software Foundation.
2460+#
2461+# charm-helpers is distributed in the hope that it will be useful,
2462+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2463+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2464+# GNU Lesser General Public License for more details.
2465+#
2466+# You should have received a copy of the GNU Lesser General Public License
2467+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2468+
2469 import importlib
2470 from tempfile import NamedTemporaryFile
2471 import time
2472@@ -5,10 +21,6 @@
2473 from charmhelpers.core.host import (
2474 lsb_release
2475 )
2476-from urlparse import (
2477- urlparse,
2478- urlunparse,
2479-)
2480 import subprocess
2481 from charmhelpers.core.hookenv import (
2482 config,
2483@@ -16,6 +28,12 @@
2484 )
2485 import os
2486
2487+import six
2488+if six.PY3:
2489+ from urllib.parse import urlparse, urlunparse
2490+else:
2491+ from urlparse import urlparse, urlunparse
2492+
2493
2494 CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
2495 deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
2496@@ -62,9 +80,16 @@
2497 'trusty-juno/updates': 'trusty-updates/juno',
2498 'trusty-updates/juno': 'trusty-updates/juno',
2499 'juno/proposed': 'trusty-proposed/juno',
2500- 'juno/proposed': 'trusty-proposed/juno',
2501 'trusty-juno/proposed': 'trusty-proposed/juno',
2502 'trusty-proposed/juno': 'trusty-proposed/juno',
2503+ # Kilo
2504+ 'kilo': 'trusty-updates/kilo',
2505+ 'trusty-kilo': 'trusty-updates/kilo',
2506+ 'trusty-kilo/updates': 'trusty-updates/kilo',
2507+ 'trusty-updates/kilo': 'trusty-updates/kilo',
2508+ 'kilo/proposed': 'trusty-proposed/kilo',
2509+ 'trusty-kilo/proposed': 'trusty-proposed/kilo',
2510+ 'trusty-proposed/kilo': 'trusty-proposed/kilo',
2511 }
2512
2513 # The order of this list is very important. Handlers should be listed in from
2514@@ -72,6 +97,7 @@
2515 FETCH_HANDLERS = (
2516 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
2517 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
2518+ 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
2519 )
2520
2521 APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
2522@@ -148,7 +174,7 @@
2523 cmd = ['apt-get', '--assume-yes']
2524 cmd.extend(options)
2525 cmd.append('install')
2526- if isinstance(packages, basestring):
2527+ if isinstance(packages, six.string_types):
2528 cmd.append(packages)
2529 else:
2530 cmd.extend(packages)
2531@@ -181,7 +207,7 @@
2532 def apt_purge(packages, fatal=False):
2533 """Purge one or more packages"""
2534 cmd = ['apt-get', '--assume-yes', 'purge']
2535- if isinstance(packages, basestring):
2536+ if isinstance(packages, six.string_types):
2537 cmd.append(packages)
2538 else:
2539 cmd.extend(packages)
2540@@ -192,7 +218,7 @@
2541 def apt_hold(packages, fatal=False):
2542 """Hold one or more packages"""
2543 cmd = ['apt-mark', 'hold']
2544- if isinstance(packages, basestring):
2545+ if isinstance(packages, six.string_types):
2546 cmd.append(packages)
2547 else:
2548 cmd.extend(packages)
2549@@ -208,7 +234,8 @@
2550 """Add a package source to this system.
2551
2552 @param source: a URL or sources.list entry, as supported by
2553- add-apt-repository(1). Examples:
2554+ add-apt-repository(1). Examples::
2555+
2556 ppa:charmers/example
2557 deb https://stub:key@private.example.com/ubuntu trusty main
2558
2559@@ -217,6 +244,7 @@
2560 pocket for the release.
2561 'cloud:' may be used to activate official cloud archive pockets,
2562 such as 'cloud:icehouse'
2563+ 'distro' may be used as a noop
2564
2565 @param key: A key to be added to the system's APT keyring and used
2566 to verify the signatures on packages. Ideally, this should be an
2567@@ -250,12 +278,14 @@
2568 release = lsb_release()['DISTRIB_CODENAME']
2569 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
2570 apt.write(PROPOSED_POCKET.format(release))
2571+ elif source == 'distro':
2572+ pass
2573 else:
2574- raise SourceConfigError("Unknown source: {!r}".format(source))
2575+ log("Unknown source: {!r}".format(source))
2576
2577 if key:
2578 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
2579- with NamedTemporaryFile() as key_file:
2580+ with NamedTemporaryFile('w+') as key_file:
2581 key_file.write(key)
2582 key_file.flush()
2583 key_file.seek(0)
2584@@ -292,14 +322,14 @@
2585 sources = safe_load((config(sources_var) or '').strip()) or []
2586 keys = safe_load((config(keys_var) or '').strip()) or None
2587
2588- if isinstance(sources, basestring):
2589+ if isinstance(sources, six.string_types):
2590 sources = [sources]
2591
2592 if keys is None:
2593 for source in sources:
2594 add_source(source, None)
2595 else:
2596- if isinstance(keys, basestring):
2597+ if isinstance(keys, six.string_types):
2598 keys = [keys]
2599
2600 if len(sources) != len(keys):
2601@@ -396,7 +426,7 @@
2602 while result is None or result == APT_NO_LOCK:
2603 try:
2604 result = subprocess.check_call(cmd, env=env)
2605- except subprocess.CalledProcessError, e:
2606+ except subprocess.CalledProcessError as e:
2607 retry_count = retry_count + 1
2608 if retry_count > APT_NO_LOCK_RETRY_COUNT:
2609 raise
2610
2611=== modified file 'lib/charmhelpers/fetch/archiveurl.py'
2612--- lib/charmhelpers/fetch/archiveurl.py 2014-09-23 12:09:14 +0000
2613+++ lib/charmhelpers/fetch/archiveurl.py 2015-11-01 20:32:57 +0000
2614@@ -1,8 +1,22 @@
2615+# Copyright 2014-2015 Canonical Limited.
2616+#
2617+# This file is part of charm-helpers.
2618+#
2619+# charm-helpers is free software: you can redistribute it and/or modify
2620+# it under the terms of the GNU Lesser General Public License version 3 as
2621+# published by the Free Software Foundation.
2622+#
2623+# charm-helpers is distributed in the hope that it will be useful,
2624+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2625+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2626+# GNU Lesser General Public License for more details.
2627+#
2628+# You should have received a copy of the GNU Lesser General Public License
2629+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2630+
2631 import os
2632-import urllib2
2633-from urllib import urlretrieve
2634-import urlparse
2635 import hashlib
2636+import re
2637
2638 from charmhelpers.fetch import (
2639 BaseFetchHandler,
2640@@ -14,6 +28,41 @@
2641 )
2642 from charmhelpers.core.host import mkdir, check_hash
2643
2644+import six
2645+if six.PY3:
2646+ from urllib.request import (
2647+ build_opener, install_opener, urlopen, urlretrieve,
2648+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2649+ )
2650+ from urllib.parse import urlparse, urlunparse, parse_qs
2651+ from urllib.error import URLError
2652+else:
2653+ from urllib import urlretrieve
2654+ from urllib2 import (
2655+ build_opener, install_opener, urlopen,
2656+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2657+ URLError
2658+ )
2659+ from urlparse import urlparse, urlunparse, parse_qs
2660+
2661+
2662+def splituser(host):
2663+ '''urllib.splituser(), but six's support of this seems broken'''
2664+ _userprog = re.compile('^(.*)@(.*)$')
2665+ match = _userprog.match(host)
2666+ if match:
2667+ return match.group(1, 2)
2668+ return None, host
2669+
2670+
2671+def splitpasswd(user):
2672+ '''urllib.splitpasswd(), but six's support of this is missing'''
2673+ _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
2674+ match = _passwdprog.match(user)
2675+ if match:
2676+ return match.group(1, 2)
2677+ return user, None
2678+
2679
2680 class ArchiveUrlFetchHandler(BaseFetchHandler):
2681 """
2682@@ -42,20 +91,20 @@
2683 """
2684 # propogate all exceptions
2685 # URLError, OSError, etc
2686- proto, netloc, path, params, query, fragment = urlparse.urlparse(source)
2687+ proto, netloc, path, params, query, fragment = urlparse(source)
2688 if proto in ('http', 'https'):
2689- auth, barehost = urllib2.splituser(netloc)
2690+ auth, barehost = splituser(netloc)
2691 if auth is not None:
2692- source = urlparse.urlunparse((proto, barehost, path, params, query, fragment))
2693- username, password = urllib2.splitpasswd(auth)
2694- passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
2695+ source = urlunparse((proto, barehost, path, params, query, fragment))
2696+ username, password = splitpasswd(auth)
2697+ passman = HTTPPasswordMgrWithDefaultRealm()
2698 # Realm is set to None in add_password to force the username and password
2699 # to be used whatever the realm
2700 passman.add_password(None, source, username, password)
2701- authhandler = urllib2.HTTPBasicAuthHandler(passman)
2702- opener = urllib2.build_opener(authhandler)
2703- urllib2.install_opener(opener)
2704- response = urllib2.urlopen(source)
2705+ authhandler = HTTPBasicAuthHandler(passman)
2706+ opener = build_opener(authhandler)
2707+ install_opener(opener)
2708+ response = urlopen(source)
2709 try:
2710 with open(dest, 'w') as dest_file:
2711 dest_file.write(response.read())
2712@@ -74,33 +123,38 @@
2713 """
2714 Download and install an archive file, with optional checksum validation.
2715
2716- The checksum can also be given on the :param:`source` URL's fragment.
2717+ The checksum can also be given on the `source` URL's fragment.
2718 For example::
2719
2720 handler.install('http://example.com/file.tgz#sha1=deadbeef')
2721
2722 :param str source: URL pointing to an archive file.
2723- :param str dest: Local destination path to install to. If not given,
2724- installs to `$CHARM_DIR/archives/archive_file_name`.
2725+ :param str dest: Local destination path to install to. If not given,
2726+ installs to `$CHARM_DIR/archives/archive_file_name`.
2727 :param str checksum: If given, validate the archive file after download.
2728- :param str hash_type: Algorithm used to generate :param:`checksum`.
2729- Can be any hash alrgorithm supported by :mod:`hashlib`,
2730- such as md5, sha1, sha256, sha512, etc.
2731+ :param str hash_type: Algorithm used to generate `checksum`.
2732+ Can be any hash alrgorithm supported by :mod:`hashlib`,
2733+ such as md5, sha1, sha256, sha512, etc.
2734+
2735 """
2736 url_parts = self.parse_url(source)
2737 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched')
2738 if not os.path.exists(dest_dir):
2739- mkdir(dest_dir, perms=0755)
2740+ mkdir(dest_dir, perms=0o755)
2741 dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path))
2742 try:
2743 self.download(source, dld_file)
2744- except urllib2.URLError as e:
2745+ except URLError as e:
2746 raise UnhandledSource(e.reason)
2747 except OSError as e:
2748 raise UnhandledSource(e.strerror)
2749- options = urlparse.parse_qs(url_parts.fragment)
2750+ options = parse_qs(url_parts.fragment)
2751 for key, value in options.items():
2752- if key in hashlib.algorithms:
2753+ if not six.PY3:
2754+ algorithms = hashlib.algorithms
2755+ else:
2756+ algorithms = hashlib.algorithms_available
2757+ if key in algorithms:
2758 check_hash(dld_file, value, key)
2759 if checksum:
2760 check_hash(dld_file, checksum, hash_type)
2761
2762=== modified file 'lib/charmhelpers/fetch/bzrurl.py'
2763--- lib/charmhelpers/fetch/bzrurl.py 2014-07-17 16:38:17 +0000
2764+++ lib/charmhelpers/fetch/bzrurl.py 2015-11-01 20:32:57 +0000
2765@@ -1,3 +1,19 @@
2766+# Copyright 2014-2015 Canonical Limited.
2767+#
2768+# This file is part of charm-helpers.
2769+#
2770+# charm-helpers is free software: you can redistribute it and/or modify
2771+# it under the terms of the GNU Lesser General Public License version 3 as
2772+# published by the Free Software Foundation.
2773+#
2774+# charm-helpers is distributed in the hope that it will be useful,
2775+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2776+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2777+# GNU Lesser General Public License for more details.
2778+#
2779+# You should have received a copy of the GNU Lesser General Public License
2780+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2781+
2782 import os
2783 from charmhelpers.fetch import (
2784 BaseFetchHandler,
2785@@ -5,12 +21,18 @@
2786 )
2787 from charmhelpers.core.host import mkdir
2788
2789+import six
2790+if six.PY3:
2791+ raise ImportError('bzrlib does not support Python3')
2792+
2793 try:
2794 from bzrlib.branch import Branch
2795+ from bzrlib import bzrdir, workingtree, errors
2796 except ImportError:
2797 from charmhelpers.fetch import apt_install
2798 apt_install("python-bzrlib")
2799 from bzrlib.branch import Branch
2800+ from bzrlib import bzrdir, workingtree, errors
2801
2802
2803 class BzrUrlFetchHandler(BaseFetchHandler):
2804@@ -31,8 +53,14 @@
2805 from bzrlib.plugin import load_plugins
2806 load_plugins()
2807 try:
2808+ local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
2809+ except errors.AlreadyControlDirError:
2810+ local_branch = Branch.open(dest)
2811+ try:
2812 remote_branch = Branch.open(source)
2813- remote_branch.bzrdir.sprout(dest).open_branch()
2814+ remote_branch.push(local_branch)
2815+ tree = workingtree.WorkingTree.open(dest)
2816+ tree.update()
2817 except Exception as e:
2818 raise e
2819
2820@@ -42,7 +70,7 @@
2821 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
2822 branch_name)
2823 if not os.path.exists(dest_dir):
2824- mkdir(dest_dir, perms=0755)
2825+ mkdir(dest_dir, perms=0o755)
2826 try:
2827 self.branch(source, dest_dir)
2828 except OSError as e:
2829
2830=== added file 'lib/charmhelpers/fetch/giturl.py'
2831--- lib/charmhelpers/fetch/giturl.py 1970-01-01 00:00:00 +0000
2832+++ lib/charmhelpers/fetch/giturl.py 2015-11-01 20:32:57 +0000
2833@@ -0,0 +1,71 @@
2834+# Copyright 2014-2015 Canonical Limited.
2835+#
2836+# This file is part of charm-helpers.
2837+#
2838+# charm-helpers is free software: you can redistribute it and/or modify
2839+# it under the terms of the GNU Lesser General Public License version 3 as
2840+# published by the Free Software Foundation.
2841+#
2842+# charm-helpers is distributed in the hope that it will be useful,
2843+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2844+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2845+# GNU Lesser General Public License for more details.
2846+#
2847+# You should have received a copy of the GNU Lesser General Public License
2848+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2849+
2850+import os
2851+from charmhelpers.fetch import (
2852+ BaseFetchHandler,
2853+ UnhandledSource
2854+)
2855+from charmhelpers.core.host import mkdir
2856+
2857+import six
2858+if six.PY3:
2859+ raise ImportError('GitPython does not support Python 3')
2860+
2861+try:
2862+ from git import Repo
2863+except ImportError:
2864+ from charmhelpers.fetch import apt_install
2865+ apt_install("python-git")
2866+ from git import Repo
2867+
2868+from git.exc import GitCommandError # noqa E402
2869+
2870+
2871+class GitUrlFetchHandler(BaseFetchHandler):
2872+ """Handler for git branches via generic and github URLs"""
2873+ def can_handle(self, source):
2874+ url_parts = self.parse_url(source)
2875+ # TODO (mattyw) no support for ssh git@ yet
2876+ if url_parts.scheme not in ('http', 'https', 'git'):
2877+ return False
2878+ else:
2879+ return True
2880+
2881+ def clone(self, source, dest, branch):
2882+ if not self.can_handle(source):
2883+ raise UnhandledSource("Cannot handle {}".format(source))
2884+
2885+ repo = Repo.clone_from(source, dest)
2886+ repo.git.checkout(branch)
2887+
2888+ def install(self, source, branch="master", dest=None):
2889+ url_parts = self.parse_url(source)
2890+ branch_name = url_parts.path.strip("/").split("/")[-1]
2891+ if dest:
2892+ dest_dir = os.path.join(dest, branch_name)
2893+ else:
2894+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
2895+ branch_name)
2896+ if not os.path.exists(dest_dir):
2897+ mkdir(dest_dir, perms=0o755)
2898+ try:
2899+ self.clone(source, dest_dir, branch)
2900+ except GitCommandError as e:
2901+ raise UnhandledSource(e.message)
2902+ except OSError as e:
2903+ raise UnhandledSource(e.strerror)
2904+ return dest_dir
2905
2906=== modified file 'lib/charmhelpers/payload/__init__.py'
2907--- lib/charmhelpers/payload/__init__.py 2014-09-23 12:09:14 +0000
2908+++ lib/charmhelpers/payload/__init__.py 2015-11-01 20:32:57 +0000
2909@@ -1,1 +1,17 @@
2910+# Copyright 2014-2015 Canonical Limited.
2911+#
2912+# This file is part of charm-helpers.
2913+#
2914+# charm-helpers is free software: you can redistribute it and/or modify
2915+# it under the terms of the GNU Lesser General Public License version 3 as
2916+# published by the Free Software Foundation.
2917+#
2918+# charm-helpers is distributed in the hope that it will be useful,
2919+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2920+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2921+# GNU Lesser General Public License for more details.
2922+#
2923+# You should have received a copy of the GNU Lesser General Public License
2924+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2925+
2926 "Tools for working with files injected into a charm just before deployment."
2927
2928=== modified file 'lib/charmhelpers/payload/archive.py'
2929--- lib/charmhelpers/payload/archive.py 2014-09-23 12:09:14 +0000
2930+++ lib/charmhelpers/payload/archive.py 2015-11-01 20:32:57 +0000
2931@@ -1,3 +1,19 @@
2932+# Copyright 2014-2015 Canonical Limited.
2933+#
2934+# This file is part of charm-helpers.
2935+#
2936+# charm-helpers is free software: you can redistribute it and/or modify
2937+# it under the terms of the GNU Lesser General Public License version 3 as
2938+# published by the Free Software Foundation.
2939+#
2940+# charm-helpers is distributed in the hope that it will be useful,
2941+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2942+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2943+# GNU Lesser General Public License for more details.
2944+#
2945+# You should have received a copy of the GNU Lesser General Public License
2946+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2947+
2948 import os
2949 import tarfile
2950 import zipfile
2951
2952=== modified file 'lib/charmhelpers/payload/execd.py'
2953--- lib/charmhelpers/payload/execd.py 2014-09-23 12:09:14 +0000
2954+++ lib/charmhelpers/payload/execd.py 2015-11-01 20:32:57 +0000
2955@@ -1,5 +1,21 @@
2956 #!/usr/bin/env python
2957
2958+# Copyright 2014-2015 Canonical Limited.
2959+#
2960+# This file is part of charm-helpers.
2961+#
2962+# charm-helpers is free software: you can redistribute it and/or modify
2963+# it under the terms of the GNU Lesser General Public License version 3 as
2964+# published by the Free Software Foundation.
2965+#
2966+# charm-helpers is distributed in the hope that it will be useful,
2967+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2968+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2969+# GNU Lesser General Public License for more details.
2970+#
2971+# You should have received a copy of the GNU Lesser General Public License
2972+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2973+
2974 import os
2975 import sys
2976 import subprocess
2977
2978=== modified file 'metadata.yaml'
2979--- metadata.yaml 2015-04-17 08:37:59 +0000
2980+++ metadata.yaml 2015-11-01 20:32:57 +0000
2981@@ -16,6 +16,9 @@
2982 interface: elasticsearch
2983 lumberjack:
2984 interface: http
2985+ nrpe-external-master:
2986+ interface: nrpe-external-master
2987+ scope: container
2988 requires:
2989 client:
2990 interface: elasticsearch

Subscribers

People subscribed via source and target branches