Merge lp:~verterok/charms/trusty/logstash/merge-canonical-is-sa into lp:~tanuki/charms/trusty/logstash/trunk

Proposed by Guillermo Gonzalez on 2015-09-08
Status: Merged
Approved by: Celso Providelo on 2015-09-08
Approved revision: 56
Merged at revision: 56
Proposed branch: lp:~verterok/charms/trusty/logstash/merge-canonical-is-sa
Merge into: lp:~tanuki/charms/trusty/logstash/trunk
Diff against target: 2823 lines (+1907/-134)
30 files modified
charm-helpers.yaml (+1/-0)
config.yaml (+24/-0)
hooks/client-relation-changed (+3/-1)
hooks/config-changed (+2/-0)
hooks/nrpe-external-master-relation-changed (+39/-0)
lib/charmhelpers/contrib/charmsupport/__init__.py (+15/-0)
lib/charmhelpers/contrib/charmsupport/nrpe.py (+360/-0)
lib/charmhelpers/contrib/charmsupport/volumes.py (+175/-0)
lib/charmhelpers/contrib/templating/__init__.py (+15/-0)
lib/charmhelpers/contrib/templating/jinja.py (+25/-9)
lib/charmhelpers/core/__init__.py (+15/-0)
lib/charmhelpers/core/decorators.py (+57/-0)
lib/charmhelpers/core/fstab.py (+30/-12)
lib/charmhelpers/core/hookenv.py (+105/-22)
lib/charmhelpers/core/host.py (+103/-38)
lib/charmhelpers/core/services/__init__.py (+18/-2)
lib/charmhelpers/core/services/base.py (+16/-0)
lib/charmhelpers/core/services/helpers.py (+37/-9)
lib/charmhelpers/core/strutils.py (+42/-0)
lib/charmhelpers/core/sysctl.py (+56/-0)
lib/charmhelpers/core/templating.py (+20/-3)
lib/charmhelpers/core/unitdata.py (+477/-0)
lib/charmhelpers/fetch/__init__.py (+44/-14)
lib/charmhelpers/fetch/archiveurl.py (+76/-22)
lib/charmhelpers/fetch/bzrurl.py (+30/-2)
lib/charmhelpers/fetch/giturl.py (+71/-0)
lib/charmhelpers/payload/__init__.py (+16/-0)
lib/charmhelpers/payload/archive.py (+16/-0)
lib/charmhelpers/payload/execd.py (+16/-0)
metadata.yaml (+3/-0)
To merge this branch: bzr merge lp:~verterok/charms/trusty/logstash/merge-canonical-is-sa
Reviewer Review Type Date Requested Status
Celso Providelo (community) 2015-09-08 Approve on 2015-09-08
Review via email: mp+270435@code.launchpad.net

Commit message

Merge lp:~canonical-is-sa/charms/trusty/logstash/trunk (add nagios checks, fix client-relation-changed to use most recent elasticsearch unit and update charmhelpers)

Description of the change

Merge lp:~canonical-is-sa/charms/trusty/logstash/trunk:

  - add nagios checks for logstash and fixed client-relation-changed to use most recent elasticsearch unit.
  - update charmhelpers

To post a comment you must log in.
Celso Providelo (cprov) wrote :

+1, thank you.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'charm-helpers.yaml'
2--- charm-helpers.yaml 2014-09-23 12:11:39 +0000
3+++ charm-helpers.yaml 2015-09-08 18:07:26 +0000
4@@ -5,3 +5,4 @@
5 - fetch
6 - payload
7 - contrib.templating.jinja
8+ - contrib.charmsupport
9
10=== modified file 'config.yaml'
11--- config.yaml 2015-08-13 19:44:37 +0000
12+++ config.yaml 2015-09-08 18:07:26 +0000
13@@ -23,3 +23,27 @@
14 type: string
15 default: ''
16 description: "Base64-encoded custom configuration content."
17+ nagios_context:
18+ default: "juju"
19+ type: string
20+ description: |
21+ Used by the nrpe subordinate charms.
22+ A string that will be prepended to instance name to set the host name
23+ in nagios. So for instance the hostname would be something like:
24+ juju-myservice-0
25+ If you're running multiple environments with the same services in them
26+ this allows you to differentiate between them.
27+ nagios_servicegroups:
28+ default: ""
29+ type: string
30+ description: |
31+ A comma-separated list of nagios servicegroups.
32+ If left empty, the nagios_context will be used as the servicegroup
33+ nagios_check_procs_params:
34+ default: "-a /opt/logstash/lib/logstash/runner.rb -c 1:1"
35+ type: string
36+ description: The parameters to pass to the nrpe plugin check_procs.
37+ nagios_check_tcp_params:
38+ default: "--ssl -H localhost -p 5043 -c 0.3"
39+ type: string
40+ description: The parameters to pass to the nrpe plugin check_tcp.
41
42=== modified file 'hooks/client-relation-changed'
43--- hooks/client-relation-changed 2015-09-08 16:23:29 +0000
44+++ hooks/client-relation-changed 2015-09-08 18:07:26 +0000
45@@ -30,7 +30,9 @@
46 with open('host_cache', 'r') as f:
47 hosts = f.readlines()
48
49- opts = {'hosts': hosts[0].rstrip()}
50+ # Use last host in list as it will be the most recently added
51+ # and first host in list may not exist anymore! TODO fix that.
52+ opts = {'hosts': hosts[-1].rstrip()}
53
54 out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf')
55 with open(out, 'w') as p:
56
57=== modified file 'hooks/config-changed'
58--- hooks/config-changed 2015-09-08 16:21:40 +0000
59+++ hooks/config-changed 2015-09-08 18:07:26 +0000
60@@ -49,6 +49,8 @@
61 # Restart the service when configuration has changed.
62 subprocess.check_output(shlex.split('hooks/start'))
63
64+ # TODO: should call update_nrpe_checks() here. See charmsupport/nrpe.py
65+
66
67 def copy_config():
68 files = os.listdir('templates')
69
70=== modified file 'hooks/lumberjack-relation-changed' (properties changed: -x to +x)
71=== added file 'hooks/nrpe-external-master-relation-changed'
72--- hooks/nrpe-external-master-relation-changed 1970-01-01 00:00:00 +0000
73+++ hooks/nrpe-external-master-relation-changed 2015-09-08 18:07:26 +0000
74@@ -0,0 +1,39 @@
75+#!/usr/bin/python
76+
77+import os
78+import sys
79+
80+sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
81+
82+from charmhelpers.core import hookenv
83+from charmhelpers.contrib.charmsupport import nrpe
84+
85+hooks = hookenv.Hooks()
86+log = hookenv.log
87+
88+@hooks.hook('nrpe-external-master-relation-changed')
89+def update_nrpe_checks():
90+ nrpe_compat = nrpe.NRPE()
91+ conf = nrpe_compat.config
92+ check_procs_params = conf.get('nagios_check_procs_params')
93+ if check_procs_params:
94+ nrpe_compat.add_check(
95+ shortname='logstash_process',
96+ description='Check logstash java process running',
97+ check_cmd='check_procs %s' % check_procs_params
98+ )
99+ check_tcp_params = conf.get('nagios_check_tcp_params')
100+ config_data = hookenv.config()
101+ # Only setup lumberjack protocol if ssl cert and key are configured
102+ if config_data['ssl_cert'] and config_data['ssl_key']:
103+ if check_tcp_params:
104+ nrpe_compat.add_check(
105+ shortname='lumberjack_tcp',
106+ description='Check logstash lumberjack input tcp port',
107+ check_cmd='check_tcp %s' % check_tcp_params
108+ )
109+ nrpe_compat.write()
110+
111+if __name__ == "__main__":
112+ # execute a hook based on the name the program is called by
113+ hooks.execute(sys.argv)
114
115=== added directory 'lib/charmhelpers/contrib/charmsupport'
116=== added file 'lib/charmhelpers/contrib/charmsupport/__init__.py'
117--- lib/charmhelpers/contrib/charmsupport/__init__.py 1970-01-01 00:00:00 +0000
118+++ lib/charmhelpers/contrib/charmsupport/__init__.py 2015-09-08 18:07:26 +0000
119@@ -0,0 +1,15 @@
120+# Copyright 2014-2015 Canonical Limited.
121+#
122+# This file is part of charm-helpers.
123+#
124+# charm-helpers is free software: you can redistribute it and/or modify
125+# it under the terms of the GNU Lesser General Public License version 3 as
126+# published by the Free Software Foundation.
127+#
128+# charm-helpers is distributed in the hope that it will be useful,
129+# but WITHOUT ANY WARRANTY; without even the implied warranty of
130+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
131+# GNU Lesser General Public License for more details.
132+#
133+# You should have received a copy of the GNU Lesser General Public License
134+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
135
136=== added file 'lib/charmhelpers/contrib/charmsupport/nrpe.py'
137--- lib/charmhelpers/contrib/charmsupport/nrpe.py 1970-01-01 00:00:00 +0000
138+++ lib/charmhelpers/contrib/charmsupport/nrpe.py 2015-09-08 18:07:26 +0000
139@@ -0,0 +1,360 @@
140+# Copyright 2014-2015 Canonical Limited.
141+#
142+# This file is part of charm-helpers.
143+#
144+# charm-helpers is free software: you can redistribute it and/or modify
145+# it under the terms of the GNU Lesser General Public License version 3 as
146+# published by the Free Software Foundation.
147+#
148+# charm-helpers is distributed in the hope that it will be useful,
149+# but WITHOUT ANY WARRANTY; without even the implied warranty of
150+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
151+# GNU Lesser General Public License for more details.
152+#
153+# You should have received a copy of the GNU Lesser General Public License
154+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
155+
156+"""Compatibility with the nrpe-external-master charm"""
157+# Copyright 2012 Canonical Ltd.
158+#
159+# Authors:
160+# Matthew Wedgwood <matthew.wedgwood@canonical.com>
161+
162+import subprocess
163+import pwd
164+import grp
165+import os
166+import glob
167+import shutil
168+import re
169+import shlex
170+import yaml
171+
172+from charmhelpers.core.hookenv import (
173+ config,
174+ local_unit,
175+ log,
176+ relation_ids,
177+ relation_set,
178+ relations_of_type,
179+)
180+
181+from charmhelpers.core.host import service
182+
183+# This module adds compatibility with the nrpe-external-master and plain nrpe
184+# subordinate charms. To use it in your charm:
185+#
186+# 1. Update metadata.yaml
187+#
188+# provides:
189+# (...)
190+# nrpe-external-master:
191+# interface: nrpe-external-master
192+# scope: container
193+#
194+# and/or
195+#
196+# provides:
197+# (...)
198+# local-monitors:
199+# interface: local-monitors
200+# scope: container
201+
202+#
203+# 2. Add the following to config.yaml
204+#
205+# nagios_context:
206+# default: "juju"
207+# type: string
208+# description: |
209+# Used by the nrpe subordinate charms.
210+# A string that will be prepended to instance name to set the host name
211+# in nagios. So for instance the hostname would be something like:
212+# juju-myservice-0
213+# If you're running multiple environments with the same services in them
214+# this allows you to differentiate between them.
215+# nagios_servicegroups:
216+# default: ""
217+# type: string
218+# description: |
219+# A comma-separated list of nagios servicegroups.
220+# If left empty, the nagios_context will be used as the servicegroup
221+#
222+# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
223+#
224+# 4. Update your hooks.py with something like this:
225+#
226+# from charmsupport.nrpe import NRPE
227+# (...)
228+# def update_nrpe_config():
229+# nrpe_compat = NRPE()
230+# nrpe_compat.add_check(
231+# shortname = "myservice",
232+# description = "Check MyService",
233+# check_cmd = "check_http -w 2 -c 10 http://localhost"
234+# )
235+# nrpe_compat.add_check(
236+# "myservice_other",
237+# "Check for widget failures",
238+# check_cmd = "/srv/myapp/scripts/widget_check"
239+# )
240+# nrpe_compat.write()
241+#
242+# def config_changed():
243+# (...)
244+# update_nrpe_config()
245+#
246+# def nrpe_external_master_relation_changed():
247+# update_nrpe_config()
248+#
249+# def local_monitors_relation_changed():
250+# update_nrpe_config()
251+#
252+# 5. ln -s hooks.py nrpe-external-master-relation-changed
253+# ln -s hooks.py local-monitors-relation-changed
254+
255+
256+class CheckException(Exception):
257+ pass
258+
259+
260+class Check(object):
261+ shortname_re = '[A-Za-z0-9-_]+$'
262+ service_template = ("""
263+#---------------------------------------------------
264+# This file is Juju managed
265+#---------------------------------------------------
266+define service {{
267+ use active-service
268+ host_name {nagios_hostname}
269+ service_description {nagios_hostname}[{shortname}] """
270+ """{description}
271+ check_command check_nrpe!{command}
272+ servicegroups {nagios_servicegroup}
273+}}
274+""")
275+
276+ def __init__(self, shortname, description, check_cmd):
277+ super(Check, self).__init__()
278+ # XXX: could be better to calculate this from the service name
279+ if not re.match(self.shortname_re, shortname):
280+ raise CheckException("shortname must match {}".format(
281+ Check.shortname_re))
282+ self.shortname = shortname
283+ self.command = "check_{}".format(shortname)
284+ # Note: a set of invalid characters is defined by the
285+ # Nagios server config
286+ # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
287+ self.description = description
288+ self.check_cmd = self._locate_cmd(check_cmd)
289+
290+ def _locate_cmd(self, check_cmd):
291+ search_path = (
292+ '/usr/lib/nagios/plugins',
293+ '/usr/local/lib/nagios/plugins',
294+ )
295+ parts = shlex.split(check_cmd)
296+ for path in search_path:
297+ if os.path.exists(os.path.join(path, parts[0])):
298+ command = os.path.join(path, parts[0])
299+ if len(parts) > 1:
300+ command += " " + " ".join(parts[1:])
301+ return command
302+ log('Check command not found: {}'.format(parts[0]))
303+ return ''
304+
305+ def write(self, nagios_context, hostname, nagios_servicegroups):
306+ nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
307+ self.command)
308+ with open(nrpe_check_file, 'w') as nrpe_check_config:
309+ nrpe_check_config.write("# check {}\n".format(self.shortname))
310+ nrpe_check_config.write("command[{}]={}\n".format(
311+ self.command, self.check_cmd))
312+
313+ if not os.path.exists(NRPE.nagios_exportdir):
314+ log('Not writing service config as {} is not accessible'.format(
315+ NRPE.nagios_exportdir))
316+ else:
317+ self.write_service_config(nagios_context, hostname,
318+ nagios_servicegroups)
319+
320+ def write_service_config(self, nagios_context, hostname,
321+ nagios_servicegroups):
322+ for f in os.listdir(NRPE.nagios_exportdir):
323+ if re.search('.*{}.cfg'.format(self.command), f):
324+ os.remove(os.path.join(NRPE.nagios_exportdir, f))
325+
326+ templ_vars = {
327+ 'nagios_hostname': hostname,
328+ 'nagios_servicegroup': nagios_servicegroups,
329+ 'description': self.description,
330+ 'shortname': self.shortname,
331+ 'command': self.command,
332+ }
333+ nrpe_service_text = Check.service_template.format(**templ_vars)
334+ nrpe_service_file = '{}/service__{}_{}.cfg'.format(
335+ NRPE.nagios_exportdir, hostname, self.command)
336+ with open(nrpe_service_file, 'w') as nrpe_service_config:
337+ nrpe_service_config.write(str(nrpe_service_text))
338+
339+ def run(self):
340+ subprocess.call(self.check_cmd)
341+
342+
343+class NRPE(object):
344+ nagios_logdir = '/var/log/nagios'
345+ nagios_exportdir = '/var/lib/nagios/export'
346+ nrpe_confdir = '/etc/nagios/nrpe.d'
347+
348+ def __init__(self, hostname=None):
349+ super(NRPE, self).__init__()
350+ self.config = config()
351+ self.nagios_context = self.config['nagios_context']
352+ if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']:
353+ self.nagios_servicegroups = self.config['nagios_servicegroups']
354+ else:
355+ self.nagios_servicegroups = self.nagios_context
356+ self.unit_name = local_unit().replace('/', '-')
357+ if hostname:
358+ self.hostname = hostname
359+ else:
360+ self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
361+ self.checks = []
362+
363+ def add_check(self, *args, **kwargs):
364+ self.checks.append(Check(*args, **kwargs))
365+
366+ def write(self):
367+ try:
368+ nagios_uid = pwd.getpwnam('nagios').pw_uid
369+ nagios_gid = grp.getgrnam('nagios').gr_gid
370+ except:
371+ log("Nagios user not set up, nrpe checks not updated")
372+ return
373+
374+ if not os.path.exists(NRPE.nagios_logdir):
375+ os.mkdir(NRPE.nagios_logdir)
376+ os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
377+
378+ nrpe_monitors = {}
379+ monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
380+ for nrpecheck in self.checks:
381+ nrpecheck.write(self.nagios_context, self.hostname,
382+ self.nagios_servicegroups)
383+ nrpe_monitors[nrpecheck.shortname] = {
384+ "command": nrpecheck.command,
385+ }
386+
387+ service('restart', 'nagios-nrpe-server')
388+
389+ monitor_ids = relation_ids("local-monitors") + \
390+ relation_ids("nrpe-external-master")
391+ for rid in monitor_ids:
392+ relation_set(relation_id=rid, monitors=yaml.dump(monitors))
393+
394+
395+def get_nagios_hostcontext(relation_name='nrpe-external-master'):
396+ """
397+ Query relation with nrpe subordinate, return the nagios_host_context
398+
399+ :param str relation_name: Name of relation nrpe sub joined to
400+ """
401+ for rel in relations_of_type(relation_name):
402+ if 'nagios_hostname' in rel:
403+ return rel['nagios_host_context']
404+
405+
406+def get_nagios_hostname(relation_name='nrpe-external-master'):
407+ """
408+ Query relation with nrpe subordinate, return the nagios_hostname
409+
410+ :param str relation_name: Name of relation nrpe sub joined to
411+ """
412+ for rel in relations_of_type(relation_name):
413+ if 'nagios_hostname' in rel:
414+ return rel['nagios_hostname']
415+
416+
417+def get_nagios_unit_name(relation_name='nrpe-external-master'):
418+ """
419+ Return the nagios unit name prepended with host_context if needed
420+
421+ :param str relation_name: Name of relation nrpe sub joined to
422+ """
423+ host_context = get_nagios_hostcontext(relation_name)
424+ if host_context:
425+ unit = "%s:%s" % (host_context, local_unit())
426+ else:
427+ unit = local_unit()
428+ return unit
429+
430+
431+def add_init_service_checks(nrpe, services, unit_name):
432+ """
433+ Add checks for each service in list
434+
435+ :param NRPE nrpe: NRPE object to add check to
436+ :param list services: List of services to check
437+ :param str unit_name: Unit name to use in check description
438+ """
439+ for svc in services:
440+ upstart_init = '/etc/init/%s.conf' % svc
441+ sysv_init = '/etc/init.d/%s' % svc
442+ if os.path.exists(upstart_init):
443+ nrpe.add_check(
444+ shortname=svc,
445+ description='process check {%s}' % unit_name,
446+ check_cmd='check_upstart_job %s' % svc
447+ )
448+ elif os.path.exists(sysv_init):
449+ cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
450+ cron_file = ('*/5 * * * * root '
451+ '/usr/local/lib/nagios/plugins/check_exit_status.pl '
452+ '-s /etc/init.d/%s status > '
453+ '/var/lib/nagios/service-check-%s.txt\n' % (svc,
454+ svc)
455+ )
456+ f = open(cronpath, 'w')
457+ f.write(cron_file)
458+ f.close()
459+ nrpe.add_check(
460+ shortname=svc,
461+ description='process check {%s}' % unit_name,
462+ check_cmd='check_status_file.py -f '
463+ '/var/lib/nagios/service-check-%s.txt' % svc,
464+ )
465+
466+
467+def copy_nrpe_checks():
468+ """
469+ Copy the nrpe checks into place
470+
471+ """
472+ NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
473+ nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks',
474+ 'charmhelpers', 'contrib', 'openstack',
475+ 'files')
476+
477+ if not os.path.exists(NAGIOS_PLUGINS):
478+ os.makedirs(NAGIOS_PLUGINS)
479+ for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")):
480+ if os.path.isfile(fname):
481+ shutil.copy2(fname,
482+ os.path.join(NAGIOS_PLUGINS, os.path.basename(fname)))
483+
484+
485+def add_haproxy_checks(nrpe, unit_name):
486+ """
487+ Add checks for each service in list
488+
489+ :param NRPE nrpe: NRPE object to add check to
490+ :param str unit_name: Unit name to use in check description
491+ """
492+ nrpe.add_check(
493+ shortname='haproxy_servers',
494+ description='Check HAProxy {%s}' % unit_name,
495+ check_cmd='check_haproxy.sh')
496+ nrpe.add_check(
497+ shortname='haproxy_queue',
498+ description='Check HAProxy queue depth {%s}' % unit_name,
499+ check_cmd='check_haproxy_queue_depth.sh')
500
501=== added file 'lib/charmhelpers/contrib/charmsupport/volumes.py'
502--- lib/charmhelpers/contrib/charmsupport/volumes.py 1970-01-01 00:00:00 +0000
503+++ lib/charmhelpers/contrib/charmsupport/volumes.py 2015-09-08 18:07:26 +0000
504@@ -0,0 +1,175 @@
505+# Copyright 2014-2015 Canonical Limited.
506+#
507+# This file is part of charm-helpers.
508+#
509+# charm-helpers is free software: you can redistribute it and/or modify
510+# it under the terms of the GNU Lesser General Public License version 3 as
511+# published by the Free Software Foundation.
512+#
513+# charm-helpers is distributed in the hope that it will be useful,
514+# but WITHOUT ANY WARRANTY; without even the implied warranty of
515+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
516+# GNU Lesser General Public License for more details.
517+#
518+# You should have received a copy of the GNU Lesser General Public License
519+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
520+
521+'''
522+Functions for managing volumes in juju units. One volume is supported per unit.
523+Subordinates may have their own storage, provided it is on its own partition.
524+
525+Configuration stanzas::
526+
527+ volume-ephemeral:
528+ type: boolean
529+ default: true
530+ description: >
531+ If false, a volume is mounted as sepecified in "volume-map"
532+ If true, ephemeral storage will be used, meaning that log data
533+ will only exist as long as the machine. YOU HAVE BEEN WARNED.
534+ volume-map:
535+ type: string
536+ default: {}
537+ description: >
538+ YAML map of units to device names, e.g:
539+ "{ rsyslog/0: /dev/vdb, rsyslog/1: /dev/vdb }"
540+ Service units will raise a configure-error if volume-ephemeral
541+ is 'true' and no volume-map value is set. Use 'juju set' to set a
542+ value and 'juju resolved' to complete configuration.
543+
544+Usage::
545+
546+ from charmsupport.volumes import configure_volume, VolumeConfigurationError
547+ from charmsupport.hookenv import log, ERROR
548+ def post_mount_hook():
549+ stop_service('myservice')
550+ def post_mount_hook():
551+ start_service('myservice')
552+
553+ if __name__ == '__main__':
554+ try:
555+ configure_volume(before_change=pre_mount_hook,
556+ after_change=post_mount_hook)
557+ except VolumeConfigurationError:
558+ log('Storage could not be configured', ERROR)
559+
560+'''
561+
562+# XXX: Known limitations
563+# - fstab is neither consulted nor updated
564+
565+import os
566+from charmhelpers.core import hookenv
567+from charmhelpers.core import host
568+import yaml
569+
570+
571+MOUNT_BASE = '/srv/juju/volumes'
572+
573+
574+class VolumeConfigurationError(Exception):
575+ '''Volume configuration data is missing or invalid'''
576+ pass
577+
578+
579+def get_config():
580+ '''Gather and sanity-check volume configuration data'''
581+ volume_config = {}
582+ config = hookenv.config()
583+
584+ errors = False
585+
586+ if config.get('volume-ephemeral') in (True, 'True', 'true', 'Yes', 'yes'):
587+ volume_config['ephemeral'] = True
588+ else:
589+ volume_config['ephemeral'] = False
590+
591+ try:
592+ volume_map = yaml.safe_load(config.get('volume-map', '{}'))
593+ except yaml.YAMLError as e:
594+ hookenv.log("Error parsing YAML volume-map: {}".format(e),
595+ hookenv.ERROR)
596+ errors = True
597+ if volume_map is None:
598+ # probably an empty string
599+ volume_map = {}
600+ elif not isinstance(volume_map, dict):
601+ hookenv.log("Volume-map should be a dictionary, not {}".format(
602+ type(volume_map)))
603+ errors = True
604+
605+ volume_config['device'] = volume_map.get(os.environ['JUJU_UNIT_NAME'])
606+ if volume_config['device'] and volume_config['ephemeral']:
607+ # asked for ephemeral storage but also defined a volume ID
608+ hookenv.log('A volume is defined for this unit, but ephemeral '
609+ 'storage was requested', hookenv.ERROR)
610+ errors = True
611+ elif not volume_config['device'] and not volume_config['ephemeral']:
612+ # asked for permanent storage but did not define volume ID
613+ hookenv.log('Ephemeral storage was requested, but there is no volume '
614+ 'defined for this unit.', hookenv.ERROR)
615+ errors = True
616+
617+ unit_mount_name = hookenv.local_unit().replace('/', '-')
618+ volume_config['mountpoint'] = os.path.join(MOUNT_BASE, unit_mount_name)
619+
620+ if errors:
621+ return None
622+ return volume_config
623+
624+
625+def mount_volume(config):
626+ if os.path.exists(config['mountpoint']):
627+ if not os.path.isdir(config['mountpoint']):
628+ hookenv.log('Not a directory: {}'.format(config['mountpoint']))
629+ raise VolumeConfigurationError()
630+ else:
631+ host.mkdir(config['mountpoint'])
632+ if os.path.ismount(config['mountpoint']):
633+ unmount_volume(config)
634+ if not host.mount(config['device'], config['mountpoint'], persist=True):
635+ raise VolumeConfigurationError()
636+
637+
638+def unmount_volume(config):
639+ if os.path.ismount(config['mountpoint']):
640+ if not host.umount(config['mountpoint'], persist=True):
641+ raise VolumeConfigurationError()
642+
643+
644+def managed_mounts():
645+ '''List of all mounted managed volumes'''
646+ return filter(lambda mount: mount[0].startswith(MOUNT_BASE), host.mounts())
647+
648+
649+def configure_volume(before_change=lambda: None, after_change=lambda: None):
650+ '''Set up storage (or don't) according to the charm's volume configuration.
651+ Returns the mount point or "ephemeral". before_change and after_change
652+ are optional functions to be called if the volume configuration changes.
653+ '''
654+
655+ config = get_config()
656+ if not config:
657+ hookenv.log('Failed to read volume configuration', hookenv.CRITICAL)
658+ raise VolumeConfigurationError()
659+
660+ if config['ephemeral']:
661+ if os.path.ismount(config['mountpoint']):
662+ before_change()
663+ unmount_volume(config)
664+ after_change()
665+ return 'ephemeral'
666+ else:
667+ # persistent storage
668+ if os.path.ismount(config['mountpoint']):
669+ mounts = dict(managed_mounts())
670+ if mounts.get(config['mountpoint']) != config['device']:
671+ before_change()
672+ unmount_volume(config)
673+ mount_volume(config)
674+ after_change()
675+ else:
676+ before_change()
677+ mount_volume(config)
678+ after_change()
679+ return config['mountpoint']
680
681=== modified file 'lib/charmhelpers/contrib/templating/__init__.py'
682--- lib/charmhelpers/contrib/templating/__init__.py 2014-09-23 12:09:14 +0000
683+++ lib/charmhelpers/contrib/templating/__init__.py 2015-09-08 18:07:26 +0000
684@@ -0,0 +1,15 @@
685+# Copyright 2014-2015 Canonical Limited.
686+#
687+# This file is part of charm-helpers.
688+#
689+# charm-helpers is free software: you can redistribute it and/or modify
690+# it under the terms of the GNU Lesser General Public License version 3 as
691+# published by the Free Software Foundation.
692+#
693+# charm-helpers is distributed in the hope that it will be useful,
694+# but WITHOUT ANY WARRANTY; without even the implied warranty of
695+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
696+# GNU Lesser General Public License for more details.
697+#
698+# You should have received a copy of the GNU Lesser General Public License
699+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
700
701=== modified file 'lib/charmhelpers/contrib/templating/jinja.py'
702--- lib/charmhelpers/contrib/templating/jinja.py 2014-09-23 12:09:14 +0000
703+++ lib/charmhelpers/contrib/templating/jinja.py 2015-09-08 18:07:26 +0000
704@@ -1,21 +1,37 @@
705+# Copyright 2014-2015 Canonical Limited.
706+#
707+# This file is part of charm-helpers.
708+#
709+# charm-helpers is free software: you can redistribute it and/or modify
710+# it under the terms of the GNU Lesser General Public License version 3 as
711+# published by the Free Software Foundation.
712+#
713+# charm-helpers is distributed in the hope that it will be useful,
714+# but WITHOUT ANY WARRANTY; without even the implied warranty of
715+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
716+# GNU Lesser General Public License for more details.
717+#
718+# You should have received a copy of the GNU Lesser General Public License
719+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
720+
721 """
722 Templating using the python-jinja2 package.
723 """
724-from charmhelpers.fetch import (
725- apt_install,
726-)
727-
728-
729-DEFAULT_TEMPLATES_DIR = 'templates'
730-
731-
732+import six
733+from charmhelpers.fetch import apt_install
734 try:
735 import jinja2
736 except ImportError:
737- apt_install(["python-jinja2"])
738+ if six.PY3:
739+ apt_install(["python3-jinja2"])
740+ else:
741+ apt_install(["python-jinja2"])
742 import jinja2
743
744
745+DEFAULT_TEMPLATES_DIR = 'templates'
746+
747+
748 def render(template_name, context, template_dir=DEFAULT_TEMPLATES_DIR):
749 templates = jinja2.Environment(
750 loader=jinja2.FileSystemLoader(template_dir))
751
752=== modified file 'lib/charmhelpers/core/__init__.py'
753--- lib/charmhelpers/core/__init__.py 2014-07-17 16:38:17 +0000
754+++ lib/charmhelpers/core/__init__.py 2015-09-08 18:07:26 +0000
755@@ -0,0 +1,15 @@
756+# Copyright 2014-2015 Canonical Limited.
757+#
758+# This file is part of charm-helpers.
759+#
760+# charm-helpers is free software: you can redistribute it and/or modify
761+# it under the terms of the GNU Lesser General Public License version 3 as
762+# published by the Free Software Foundation.
763+#
764+# charm-helpers is distributed in the hope that it will be useful,
765+# but WITHOUT ANY WARRANTY; without even the implied warranty of
766+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
767+# GNU Lesser General Public License for more details.
768+#
769+# You should have received a copy of the GNU Lesser General Public License
770+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
771
772=== added file 'lib/charmhelpers/core/decorators.py'
773--- lib/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
774+++ lib/charmhelpers/core/decorators.py 2015-09-08 18:07:26 +0000
775@@ -0,0 +1,57 @@
776+# Copyright 2014-2015 Canonical Limited.
777+#
778+# This file is part of charm-helpers.
779+#
780+# charm-helpers is free software: you can redistribute it and/or modify
781+# it under the terms of the GNU Lesser General Public License version 3 as
782+# published by the Free Software Foundation.
783+#
784+# charm-helpers is distributed in the hope that it will be useful,
785+# but WITHOUT ANY WARRANTY; without even the implied warranty of
786+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
787+# GNU Lesser General Public License for more details.
788+#
789+# You should have received a copy of the GNU Lesser General Public License
790+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
791+
792+#
793+# Copyright 2014 Canonical Ltd.
794+#
795+# Authors:
796+# Edward Hope-Morley <opentastic@gmail.com>
797+#
798+
799+import time
800+
801+from charmhelpers.core.hookenv import (
802+ log,
803+ INFO,
804+)
805+
806+
807+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
808+ """If the decorated function raises exception exc_type, allow num_retries
809+ retry attempts before raise the exception.
810+ """
811+ def _retry_on_exception_inner_1(f):
812+ def _retry_on_exception_inner_2(*args, **kwargs):
813+ retries = num_retries
814+ multiplier = 1
815+ while True:
816+ try:
817+ return f(*args, **kwargs)
818+ except exc_type:
819+ if not retries:
820+ raise
821+
822+ delay = base_delay * multiplier
823+ multiplier += 1
824+ log("Retrying '%s' %d more times (delay=%s)" %
825+ (f.__name__, retries, delay), level=INFO)
826+ retries -= 1
827+ if delay:
828+ time.sleep(delay)
829+
830+ return _retry_on_exception_inner_2
831+
832+ return _retry_on_exception_inner_1
833
834=== modified file 'lib/charmhelpers/core/fstab.py'
835--- lib/charmhelpers/core/fstab.py 2014-07-17 16:38:17 +0000
836+++ lib/charmhelpers/core/fstab.py 2015-09-08 18:07:26 +0000
837@@ -1,12 +1,29 @@
838 #!/usr/bin/env python
839 # -*- coding: utf-8 -*-
840
841+# Copyright 2014-2015 Canonical Limited.
842+#
843+# This file is part of charm-helpers.
844+#
845+# charm-helpers is free software: you can redistribute it and/or modify
846+# it under the terms of the GNU Lesser General Public License version 3 as
847+# published by the Free Software Foundation.
848+#
849+# charm-helpers is distributed in the hope that it will be useful,
850+# but WITHOUT ANY WARRANTY; without even the implied warranty of
851+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
852+# GNU Lesser General Public License for more details.
853+#
854+# You should have received a copy of the GNU Lesser General Public License
855+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
856+
857+import io
858+import os
859+
860 __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
861
862-import os
863-
864-
865-class Fstab(file):
866+
867+class Fstab(io.FileIO):
868 """This class extends file in order to implement a file reader/writer
869 for file `/etc/fstab`
870 """
871@@ -24,8 +41,8 @@
872 options = "defaults"
873
874 self.options = options
875- self.d = d
876- self.p = p
877+ self.d = int(d)
878+ self.p = int(p)
879
880 def __eq__(self, o):
881 return str(self) == str(o)
882@@ -45,7 +62,7 @@
883 self._path = path
884 else:
885 self._path = self.DEFAULT_PATH
886- file.__init__(self, self._path, 'r+')
887+ super(Fstab, self).__init__(self._path, 'rb+')
888
889 def _hydrate_entry(self, line):
890 # NOTE: use split with no arguments to split on any
891@@ -58,8 +75,9 @@
892 def entries(self):
893 self.seek(0)
894 for line in self.readlines():
895+ line = line.decode('us-ascii')
896 try:
897- if not line.startswith("#"):
898+ if line.strip() and not line.strip().startswith("#"):
899 yield self._hydrate_entry(line)
900 except ValueError:
901 pass
902@@ -75,18 +93,18 @@
903 if self.get_entry_by_attr('device', entry.device):
904 return False
905
906- self.write(str(entry) + '\n')
907+ self.write((str(entry) + '\n').encode('us-ascii'))
908 self.truncate()
909 return entry
910
911 def remove_entry(self, entry):
912 self.seek(0)
913
914- lines = self.readlines()
915+ lines = [l.decode('us-ascii') for l in self.readlines()]
916
917 found = False
918 for index, line in enumerate(lines):
919- if not line.startswith("#"):
920+ if line.strip() and not line.strip().startswith("#"):
921 if self._hydrate_entry(line) == entry:
922 found = True
923 break
924@@ -97,7 +115,7 @@
925 lines.remove(line)
926
927 self.seek(0)
928- self.write(''.join(lines))
929+ self.write(''.join(lines).encode('us-ascii'))
930 self.truncate()
931 return True
932
933
934=== modified file 'lib/charmhelpers/core/hookenv.py'
935--- lib/charmhelpers/core/hookenv.py 2014-09-23 12:09:14 +0000
936+++ lib/charmhelpers/core/hookenv.py 2015-09-08 18:07:26 +0000
937@@ -1,17 +1,40 @@
938+# Copyright 2014-2015 Canonical Limited.
939+#
940+# This file is part of charm-helpers.
941+#
942+# charm-helpers is free software: you can redistribute it and/or modify
943+# it under the terms of the GNU Lesser General Public License version 3 as
944+# published by the Free Software Foundation.
945+#
946+# charm-helpers is distributed in the hope that it will be useful,
947+# but WITHOUT ANY WARRANTY; without even the implied warranty of
948+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
949+# GNU Lesser General Public License for more details.
950+#
951+# You should have received a copy of the GNU Lesser General Public License
952+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
953+
954 "Interactions with the Juju environment"
955 # Copyright 2013 Canonical Ltd.
956 #
957 # Authors:
958 # Charm Helpers Developers <juju@lists.ubuntu.com>
959
960+from __future__ import print_function
961 import os
962 import json
963 import yaml
964 import subprocess
965 import sys
966-import UserDict
967+import errno
968 from subprocess import CalledProcessError
969
970+import six
971+if not six.PY3:
972+ from UserDict import UserDict
973+else:
974+ from collections import UserDict
975+
976 CRITICAL = "CRITICAL"
977 ERROR = "ERROR"
978 WARNING = "WARNING"
979@@ -63,16 +86,29 @@
980 command = ['juju-log']
981 if level:
982 command += ['-l', level]
983+ if not isinstance(message, six.string_types):
984+ message = repr(message)
985 command += [message]
986- subprocess.call(command)
987-
988-
989-class Serializable(UserDict.IterableUserDict):
990+ # Missing juju-log should not cause failures in unit tests
991+ # Send log output to stderr
992+ try:
993+ subprocess.call(command)
994+ except OSError as e:
995+ if e.errno == errno.ENOENT:
996+ if level:
997+ message = "{}: {}".format(level, message)
998+ message = "juju-log: {}".format(message)
999+ print(message, file=sys.stderr)
1000+ else:
1001+ raise
1002+
1003+
1004+class Serializable(UserDict):
1005 """Wrapper, an object that can be serialized to yaml or json"""
1006
1007 def __init__(self, obj):
1008 # wrap the object
1009- UserDict.IterableUserDict.__init__(self)
1010+ UserDict.__init__(self)
1011 self.data = obj
1012
1013 def __getattr__(self, attr):
1014@@ -214,6 +250,12 @@
1015 except KeyError:
1016 return (self._prev_dict or {})[key]
1017
1018+ def keys(self):
1019+ prev_keys = []
1020+ if self._prev_dict is not None:
1021+ prev_keys = self._prev_dict.keys()
1022+ return list(set(prev_keys + list(dict.keys(self))))
1023+
1024 def load_previous(self, path=None):
1025 """Load previous copy of config from disk.
1026
1027@@ -263,7 +305,7 @@
1028
1029 """
1030 if self._prev_dict:
1031- for k, v in self._prev_dict.iteritems():
1032+ for k, v in six.iteritems(self._prev_dict):
1033 if k not in self:
1034 self[k] = v
1035 with open(self.path, 'w') as f:
1036@@ -278,7 +320,8 @@
1037 config_cmd_line.append(scope)
1038 config_cmd_line.append('--format=json')
1039 try:
1040- config_data = json.loads(subprocess.check_output(config_cmd_line))
1041+ config_data = json.loads(
1042+ subprocess.check_output(config_cmd_line).decode('UTF-8'))
1043 if scope is not None:
1044 return config_data
1045 return Config(config_data)
1046@@ -297,10 +340,10 @@
1047 if unit:
1048 _args.append(unit)
1049 try:
1050- return json.loads(subprocess.check_output(_args))
1051+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1052 except ValueError:
1053 return None
1054- except CalledProcessError, e:
1055+ except CalledProcessError as e:
1056 if e.returncode == 2:
1057 return None
1058 raise
1059@@ -312,7 +355,7 @@
1060 relation_cmd_line = ['relation-set']
1061 if relation_id is not None:
1062 relation_cmd_line.extend(('-r', relation_id))
1063- for k, v in (relation_settings.items() + kwargs.items()):
1064+ for k, v in (list(relation_settings.items()) + list(kwargs.items())):
1065 if v is None:
1066 relation_cmd_line.append('{}='.format(k))
1067 else:
1068@@ -329,7 +372,8 @@
1069 relid_cmd_line = ['relation-ids', '--format=json']
1070 if reltype is not None:
1071 relid_cmd_line.append(reltype)
1072- return json.loads(subprocess.check_output(relid_cmd_line)) or []
1073+ return json.loads(
1074+ subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
1075 return []
1076
1077
1078@@ -340,7 +384,8 @@
1079 units_cmd_line = ['relation-list', '--format=json']
1080 if relid is not None:
1081 units_cmd_line.extend(('-r', relid))
1082- return json.loads(subprocess.check_output(units_cmd_line)) or []
1083+ return json.loads(
1084+ subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
1085
1086
1087 @cached
1088@@ -380,21 +425,31 @@
1089
1090
1091 @cached
1092+def metadata():
1093+ """Get the current charm metadata.yaml contents as a python object"""
1094+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
1095+ return yaml.safe_load(md)
1096+
1097+
1098+@cached
1099 def relation_types():
1100 """Get a list of relation types supported by this charm"""
1101- charmdir = os.environ.get('CHARM_DIR', '')
1102- mdf = open(os.path.join(charmdir, 'metadata.yaml'))
1103- md = yaml.safe_load(mdf)
1104 rel_types = []
1105+ md = metadata()
1106 for key in ('provides', 'requires', 'peers'):
1107 section = md.get(key)
1108 if section:
1109 rel_types.extend(section.keys())
1110- mdf.close()
1111 return rel_types
1112
1113
1114 @cached
1115+def charm_name():
1116+ """Get the name of the current charm as is specified on metadata.yaml"""
1117+ return metadata().get('name')
1118+
1119+
1120+@cached
1121 def relations():
1122 """Get a nested dictionary of relation data for all related units"""
1123 rels = {}
1124@@ -449,7 +504,7 @@
1125 """Get the unit ID for the remote unit"""
1126 _args = ['unit-get', '--format=json', attribute]
1127 try:
1128- return json.loads(subprocess.check_output(_args))
1129+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1130 except ValueError:
1131 return None
1132
1133@@ -486,9 +541,10 @@
1134 hooks.execute(sys.argv)
1135 """
1136
1137- def __init__(self):
1138+ def __init__(self, config_save=True):
1139 super(Hooks, self).__init__()
1140 self._hooks = {}
1141+ self._config_save = config_save
1142
1143 def register(self, name, function):
1144 """Register a hook"""
1145@@ -499,9 +555,10 @@
1146 hook_name = os.path.basename(args[0])
1147 if hook_name in self._hooks:
1148 self._hooks[hook_name]()
1149- cfg = config()
1150- if cfg.implicit_save:
1151- cfg.save()
1152+ if self._config_save:
1153+ cfg = config()
1154+ if cfg.implicit_save:
1155+ cfg.save()
1156 else:
1157 raise UnregisteredHookError(hook_name)
1158
1159@@ -522,3 +579,29 @@
1160 def charm_dir():
1161 """Return the root directory of the current charm"""
1162 return os.environ.get('CHARM_DIR')
1163+
1164+
1165+@cached
1166+def action_get(key=None):
1167+ """Gets the value of an action parameter, or all key/value param pairs"""
1168+ cmd = ['action-get']
1169+ if key is not None:
1170+ cmd.append(key)
1171+ cmd.append('--format=json')
1172+ action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1173+ return action_data
1174+
1175+
1176+def action_set(values):
1177+ """Sets the values to be returned after the action finishes"""
1178+ cmd = ['action-set']
1179+ for k, v in list(values.items()):
1180+ cmd.append('{}={}'.format(k, v))
1181+ subprocess.check_call(cmd)
1182+
1183+
1184+def action_fail(message):
1185+ """Sets the action status to failed and sets the error message.
1186+
1187+ The results set by action_set are preserved."""
1188+ subprocess.check_call(['action-fail', message])
1189
1190=== modified file 'lib/charmhelpers/core/host.py'
1191--- lib/charmhelpers/core/host.py 2014-09-23 12:09:14 +0000
1192+++ lib/charmhelpers/core/host.py 2015-09-08 18:07:26 +0000
1193@@ -1,3 +1,19 @@
1194+# Copyright 2014-2015 Canonical Limited.
1195+#
1196+# This file is part of charm-helpers.
1197+#
1198+# charm-helpers is free software: you can redistribute it and/or modify
1199+# it under the terms of the GNU Lesser General Public License version 3 as
1200+# published by the Free Software Foundation.
1201+#
1202+# charm-helpers is distributed in the hope that it will be useful,
1203+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1204+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1205+# GNU Lesser General Public License for more details.
1206+#
1207+# You should have received a copy of the GNU Lesser General Public License
1208+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1209+
1210 """Tools for working with the host system"""
1211 # Copyright 2012 Canonical Ltd.
1212 #
1213@@ -6,19 +22,20 @@
1214 # Matthew Wedgwood <matthew.wedgwood@canonical.com>
1215
1216 import os
1217+import re
1218 import pwd
1219 import grp
1220 import random
1221 import string
1222 import subprocess
1223 import hashlib
1224-import shutil
1225 from contextlib import contextmanager
1226-
1227 from collections import OrderedDict
1228
1229-from hookenv import log
1230-from fstab import Fstab
1231+import six
1232+
1233+from .hookenv import log
1234+from .fstab import Fstab
1235
1236
1237 def service_start(service_name):
1238@@ -54,7 +71,9 @@
1239 def service_running(service):
1240 """Determine whether a system service is running"""
1241 try:
1242- output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT)
1243+ output = subprocess.check_output(
1244+ ['service', service, 'status'],
1245+ stderr=subprocess.STDOUT).decode('UTF-8')
1246 except subprocess.CalledProcessError:
1247 return False
1248 else:
1249@@ -67,9 +86,11 @@
1250 def service_available(service_name):
1251 """Determine whether a system service is available"""
1252 try:
1253- subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT)
1254- except subprocess.CalledProcessError:
1255- return False
1256+ subprocess.check_output(
1257+ ['service', service_name, 'status'],
1258+ stderr=subprocess.STDOUT).decode('UTF-8')
1259+ except subprocess.CalledProcessError as e:
1260+ return 'unrecognized service' not in e.output
1261 else:
1262 return True
1263
1264@@ -96,6 +117,26 @@
1265 return user_info
1266
1267
1268+def add_group(group_name, system_group=False):
1269+ """Add a group to the system"""
1270+ try:
1271+ group_info = grp.getgrnam(group_name)
1272+ log('group {0} already exists!'.format(group_name))
1273+ except KeyError:
1274+ log('creating group {0}'.format(group_name))
1275+ cmd = ['addgroup']
1276+ if system_group:
1277+ cmd.append('--system')
1278+ else:
1279+ cmd.extend([
1280+ '--group',
1281+ ])
1282+ cmd.append(group_name)
1283+ subprocess.check_call(cmd)
1284+ group_info = grp.getgrnam(group_name)
1285+ return group_info
1286+
1287+
1288 def add_user_to_group(username, group):
1289 """Add a user to a group"""
1290 cmd = [
1291@@ -115,7 +156,7 @@
1292 cmd.append(from_path)
1293 cmd.append(to_path)
1294 log(" ".join(cmd))
1295- return subprocess.check_output(cmd).strip()
1296+ return subprocess.check_output(cmd).decode('UTF-8').strip()
1297
1298
1299 def symlink(source, destination):
1300@@ -130,28 +171,31 @@
1301 subprocess.check_call(cmd)
1302
1303
1304-def mkdir(path, owner='root', group='root', perms=0555, force=False):
1305+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
1306 """Create a directory"""
1307 log("Making dir {} {}:{} {:o}".format(path, owner, group,
1308 perms))
1309 uid = pwd.getpwnam(owner).pw_uid
1310 gid = grp.getgrnam(group).gr_gid
1311 realpath = os.path.abspath(path)
1312- if os.path.exists(realpath):
1313- if force and not os.path.isdir(realpath):
1314+ path_exists = os.path.exists(realpath)
1315+ if path_exists and force:
1316+ if not os.path.isdir(realpath):
1317 log("Removing non-directory file {} prior to mkdir()".format(path))
1318 os.unlink(realpath)
1319- else:
1320+ os.makedirs(realpath, perms)
1321+ elif not path_exists:
1322 os.makedirs(realpath, perms)
1323 os.chown(realpath, uid, gid)
1324-
1325-
1326-def write_file(path, content, owner='root', group='root', perms=0444):
1327- """Create or overwrite a file with the contents of a string"""
1328+ os.chmod(realpath, perms)
1329+
1330+
1331+def write_file(path, content, owner='root', group='root', perms=0o444):
1332+ """Create or overwrite a file with the contents of a byte string."""
1333 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1334 uid = pwd.getpwnam(owner).pw_uid
1335 gid = grp.getgrnam(group).gr_gid
1336- with open(path, 'w') as target:
1337+ with open(path, 'wb') as target:
1338 os.fchown(target.fileno(), uid, gid)
1339 os.fchmod(target.fileno(), perms)
1340 target.write(content)
1341@@ -177,7 +221,7 @@
1342 cmd_args.extend([device, mountpoint])
1343 try:
1344 subprocess.check_output(cmd_args)
1345- except subprocess.CalledProcessError, e:
1346+ except subprocess.CalledProcessError as e:
1347 log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
1348 return False
1349
1350@@ -191,7 +235,7 @@
1351 cmd_args = ['umount', mountpoint]
1352 try:
1353 subprocess.check_output(cmd_args)
1354- except subprocess.CalledProcessError, e:
1355+ except subprocess.CalledProcessError as e:
1356 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1357 return False
1358
1359@@ -218,8 +262,8 @@
1360 """
1361 if os.path.exists(path):
1362 h = getattr(hashlib, hash_type)()
1363- with open(path, 'r') as source:
1364- h.update(source.read()) # IGNORE:E1101 - it does have update
1365+ with open(path, 'rb') as source:
1366+ h.update(source.read())
1367 return h.hexdigest()
1368 else:
1369 return None
1370@@ -229,12 +273,12 @@
1371 """
1372 Validate a file using a cryptographic checksum.
1373
1374-
1375 :param str checksum: Value of the checksum used to validate the file.
1376- :param str hash_type: Hash algorithm used to generate :param:`checksum`.
1377- Can be any hash alrgorithm supported by :mod:`hashlib`,
1378- such as md5, sha1, sha256, sha512, etc.
1379+ :param str hash_type: Hash algorithm used to generate `checksum`.
1380+ Can be any hash alrgorithm supported by :mod:`hashlib`,
1381+ such as md5, sha1, sha256, sha512, etc.
1382 :raises ChecksumError: If the file fails the checksum
1383+
1384 """
1385 actual_checksum = file_hash(path, hash_type)
1386 if checksum != actual_checksum:
1387@@ -261,11 +305,11 @@
1388 ceph_client_changed function.
1389 """
1390 def wrap(f):
1391- def wrapped_f(*args):
1392+ def wrapped_f(*args, **kwargs):
1393 checksums = {}
1394 for path in restart_map:
1395 checksums[path] = file_hash(path)
1396- f(*args)
1397+ f(*args, **kwargs)
1398 restarts = []
1399 for path in restart_map:
1400 if checksums[path] != file_hash(path):
1401@@ -295,29 +339,39 @@
1402 def pwgen(length=None):
1403 """Generate a random pasword."""
1404 if length is None:
1405+ # A random length is ok to use a weak PRNG
1406 length = random.choice(range(35, 45))
1407 alphanumeric_chars = [
1408- l for l in (string.letters + string.digits)
1409+ l for l in (string.ascii_letters + string.digits)
1410 if l not in 'l0QD1vAEIOUaeiou']
1411+ # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
1412+ # actual password
1413+ random_generator = random.SystemRandom()
1414 random_chars = [
1415- random.choice(alphanumeric_chars) for _ in range(length)]
1416+ random_generator.choice(alphanumeric_chars) for _ in range(length)]
1417 return(''.join(random_chars))
1418
1419
1420 def list_nics(nic_type):
1421 '''Return a list of nics of given type(s)'''
1422- if isinstance(nic_type, basestring):
1423+ if isinstance(nic_type, six.string_types):
1424 int_types = [nic_type]
1425 else:
1426 int_types = nic_type
1427 interfaces = []
1428 for int_type in int_types:
1429 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1430- ip_output = subprocess.check_output(cmd).split('\n')
1431+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1432 ip_output = (line for line in ip_output if line)
1433 for line in ip_output:
1434 if line.split()[1].startswith(int_type):
1435- interfaces.append(line.split()[1].replace(":", ""))
1436+ matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
1437+ if matched:
1438+ interface = matched.groups()[0]
1439+ else:
1440+ interface = line.split()[1].replace(":", "")
1441+ interfaces.append(interface)
1442+
1443 return interfaces
1444
1445
1446@@ -329,7 +383,7 @@
1447
1448 def get_nic_mtu(nic):
1449 cmd = ['ip', 'addr', 'show', nic]
1450- ip_output = subprocess.check_output(cmd).split('\n')
1451+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1452 mtu = ""
1453 for line in ip_output:
1454 words = line.split()
1455@@ -340,7 +394,7 @@
1456
1457 def get_nic_hwaddr(nic):
1458 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
1459- ip_output = subprocess.check_output(cmd)
1460+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1461 hwaddr = ""
1462 words = ip_output.split()
1463 if 'link/ether' in words:
1464@@ -355,10 +409,13 @@
1465 * 0 => Installed revno is the same as supplied arg
1466 * -1 => Installed revno is less than supplied arg
1467
1468+ This function imports apt_cache function from charmhelpers.fetch if
1469+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
1470+ you call this function, or pass an apt_pkg.Cache() instance.
1471 '''
1472 import apt_pkg
1473- from charmhelpers.fetch import apt_cache
1474 if not pkgcache:
1475+ from charmhelpers.fetch import apt_cache
1476 pkgcache = apt_cache()
1477 pkg = pkgcache[package]
1478 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
1479@@ -373,13 +430,21 @@
1480 os.chdir(cur)
1481
1482
1483-def chownr(path, owner, group):
1484+def chownr(path, owner, group, follow_links=True):
1485 uid = pwd.getpwnam(owner).pw_uid
1486 gid = grp.getgrnam(group).gr_gid
1487+ if follow_links:
1488+ chown = os.chown
1489+ else:
1490+ chown = os.lchown
1491
1492 for root, dirs, files in os.walk(path):
1493 for name in dirs + files:
1494 full = os.path.join(root, name)
1495 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
1496 if not broken_symlink:
1497- os.chown(full, uid, gid)
1498+ chown(full, uid, gid)
1499+
1500+
1501+def lchownr(path, owner, group):
1502+ chownr(path, owner, group, follow_links=False)
1503
1504=== modified file 'lib/charmhelpers/core/services/__init__.py'
1505--- lib/charmhelpers/core/services/__init__.py 2014-09-23 12:09:14 +0000
1506+++ lib/charmhelpers/core/services/__init__.py 2015-09-08 18:07:26 +0000
1507@@ -1,2 +1,18 @@
1508-from .base import *
1509-from .helpers import *
1510+# Copyright 2014-2015 Canonical Limited.
1511+#
1512+# This file is part of charm-helpers.
1513+#
1514+# charm-helpers is free software: you can redistribute it and/or modify
1515+# it under the terms of the GNU Lesser General Public License version 3 as
1516+# published by the Free Software Foundation.
1517+#
1518+# charm-helpers is distributed in the hope that it will be useful,
1519+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1520+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1521+# GNU Lesser General Public License for more details.
1522+#
1523+# You should have received a copy of the GNU Lesser General Public License
1524+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1525+
1526+from .base import * # NOQA
1527+from .helpers import * # NOQA
1528
1529=== modified file 'lib/charmhelpers/core/services/base.py'
1530--- lib/charmhelpers/core/services/base.py 2014-09-23 12:09:14 +0000
1531+++ lib/charmhelpers/core/services/base.py 2015-09-08 18:07:26 +0000
1532@@ -1,3 +1,19 @@
1533+# Copyright 2014-2015 Canonical Limited.
1534+#
1535+# This file is part of charm-helpers.
1536+#
1537+# charm-helpers is free software: you can redistribute it and/or modify
1538+# it under the terms of the GNU Lesser General Public License version 3 as
1539+# published by the Free Software Foundation.
1540+#
1541+# charm-helpers is distributed in the hope that it will be useful,
1542+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1543+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1544+# GNU Lesser General Public License for more details.
1545+#
1546+# You should have received a copy of the GNU Lesser General Public License
1547+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1548+
1549 import os
1550 import re
1551 import json
1552
1553=== modified file 'lib/charmhelpers/core/services/helpers.py'
1554--- lib/charmhelpers/core/services/helpers.py 2014-09-23 12:09:14 +0000
1555+++ lib/charmhelpers/core/services/helpers.py 2015-09-08 18:07:26 +0000
1556@@ -1,3 +1,19 @@
1557+# Copyright 2014-2015 Canonical Limited.
1558+#
1559+# This file is part of charm-helpers.
1560+#
1561+# charm-helpers is free software: you can redistribute it and/or modify
1562+# it under the terms of the GNU Lesser General Public License version 3 as
1563+# published by the Free Software Foundation.
1564+#
1565+# charm-helpers is distributed in the hope that it will be useful,
1566+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1567+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1568+# GNU Lesser General Public License for more details.
1569+#
1570+# You should have received a copy of the GNU Lesser General Public License
1571+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1572+
1573 import os
1574 import yaml
1575 from charmhelpers.core import hookenv
1576@@ -29,12 +45,14 @@
1577 """
1578 name = None
1579 interface = None
1580- required_keys = []
1581
1582 def __init__(self, name=None, additional_required_keys=None):
1583+ if not hasattr(self, 'required_keys'):
1584+ self.required_keys = []
1585+
1586 if name is not None:
1587 self.name = name
1588- if additional_required_keys is not None:
1589+ if additional_required_keys:
1590 self.required_keys.extend(additional_required_keys)
1591 self.get_data()
1592
1593@@ -118,7 +136,10 @@
1594 """
1595 name = 'db'
1596 interface = 'mysql'
1597- required_keys = ['host', 'user', 'password', 'database']
1598+
1599+ def __init__(self, *args, **kwargs):
1600+ self.required_keys = ['host', 'user', 'password', 'database']
1601+ RelationContext.__init__(self, *args, **kwargs)
1602
1603
1604 class HttpRelation(RelationContext):
1605@@ -130,7 +151,10 @@
1606 """
1607 name = 'website'
1608 interface = 'http'
1609- required_keys = ['host', 'port']
1610+
1611+ def __init__(self, *args, **kwargs):
1612+ self.required_keys = ['host', 'port']
1613+ RelationContext.__init__(self, *args, **kwargs)
1614
1615 def provide_data(self):
1616 return {
1617@@ -196,7 +220,7 @@
1618 if not os.path.isabs(file_name):
1619 file_name = os.path.join(hookenv.charm_dir(), file_name)
1620 with open(file_name, 'w') as file_stream:
1621- os.fchmod(file_stream.fileno(), 0600)
1622+ os.fchmod(file_stream.fileno(), 0o600)
1623 yaml.dump(config_data, file_stream)
1624
1625 def read_context(self, file_name):
1626@@ -211,15 +235,19 @@
1627
1628 class TemplateCallback(ManagerCallback):
1629 """
1630- Callback class that will render a Jinja2 template, for use as a ready action.
1631-
1632- :param str source: The template source file, relative to `$CHARM_DIR/templates`
1633+ Callback class that will render a Jinja2 template, for use as a ready
1634+ action.
1635+
1636+ :param str source: The template source file, relative to
1637+ `$CHARM_DIR/templates`
1638+
1639 :param str target: The target to write the rendered template to
1640 :param str owner: The owner of the rendered file
1641 :param str group: The group of the rendered file
1642 :param int perms: The permissions of the rendered file
1643 """
1644- def __init__(self, source, target, owner='root', group='root', perms=0444):
1645+ def __init__(self, source, target,
1646+ owner='root', group='root', perms=0o444):
1647 self.source = source
1648 self.target = target
1649 self.owner = owner
1650
1651=== added file 'lib/charmhelpers/core/strutils.py'
1652--- lib/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
1653+++ lib/charmhelpers/core/strutils.py 2015-09-08 18:07:26 +0000
1654@@ -0,0 +1,42 @@
1655+#!/usr/bin/env python
1656+# -*- coding: utf-8 -*-
1657+
1658+# Copyright 2014-2015 Canonical Limited.
1659+#
1660+# This file is part of charm-helpers.
1661+#
1662+# charm-helpers is free software: you can redistribute it and/or modify
1663+# it under the terms of the GNU Lesser General Public License version 3 as
1664+# published by the Free Software Foundation.
1665+#
1666+# charm-helpers is distributed in the hope that it will be useful,
1667+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1668+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1669+# GNU Lesser General Public License for more details.
1670+#
1671+# You should have received a copy of the GNU Lesser General Public License
1672+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1673+
1674+import six
1675+
1676+
1677+def bool_from_string(value):
1678+ """Interpret string value as boolean.
1679+
1680+ Returns True if value translates to True otherwise False.
1681+ """
1682+ if isinstance(value, six.string_types):
1683+ value = six.text_type(value)
1684+ else:
1685+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
1686+ raise ValueError(msg)
1687+
1688+ value = value.strip().lower()
1689+
1690+ if value in ['y', 'yes', 'true', 't', 'on']:
1691+ return True
1692+ elif value in ['n', 'no', 'false', 'f', 'off']:
1693+ return False
1694+
1695+ msg = "Unable to interpret string value '%s' as boolean" % (value)
1696+ raise ValueError(msg)
1697
1698=== added file 'lib/charmhelpers/core/sysctl.py'
1699--- lib/charmhelpers/core/sysctl.py 1970-01-01 00:00:00 +0000
1700+++ lib/charmhelpers/core/sysctl.py 2015-09-08 18:07:26 +0000
1701@@ -0,0 +1,56 @@
1702+#!/usr/bin/env python
1703+# -*- coding: utf-8 -*-
1704+
1705+# Copyright 2014-2015 Canonical Limited.
1706+#
1707+# This file is part of charm-helpers.
1708+#
1709+# charm-helpers is free software: you can redistribute it and/or modify
1710+# it under the terms of the GNU Lesser General Public License version 3 as
1711+# published by the Free Software Foundation.
1712+#
1713+# charm-helpers is distributed in the hope that it will be useful,
1714+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1715+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1716+# GNU Lesser General Public License for more details.
1717+#
1718+# You should have received a copy of the GNU Lesser General Public License
1719+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1720+
1721+import yaml
1722+
1723+from subprocess import check_call
1724+
1725+from charmhelpers.core.hookenv import (
1726+ log,
1727+ DEBUG,
1728+ ERROR,
1729+)
1730+
1731+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1732+
1733+
1734+def create(sysctl_dict, sysctl_file):
1735+ """Creates a sysctl.conf file from a YAML associative array
1736+
1737+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
1738+ :type sysctl_dict: str
1739+ :param sysctl_file: path to the sysctl file to be saved
1740+ :type sysctl_file: str or unicode
1741+ :returns: None
1742+ """
1743+ try:
1744+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
1745+ except yaml.YAMLError:
1746+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
1747+ level=ERROR)
1748+ return
1749+
1750+ with open(sysctl_file, "w") as fd:
1751+ for key, value in sysctl_dict_parsed.items():
1752+ fd.write("{}={}\n".format(key, value))
1753+
1754+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
1755+ level=DEBUG)
1756+
1757+ check_call(["sysctl", "-p", sysctl_file])
1758
1759=== modified file 'lib/charmhelpers/core/templating.py'
1760--- lib/charmhelpers/core/templating.py 2014-09-23 12:09:14 +0000
1761+++ lib/charmhelpers/core/templating.py 2015-09-08 18:07:26 +0000
1762@@ -1,10 +1,27 @@
1763+# Copyright 2014-2015 Canonical Limited.
1764+#
1765+# This file is part of charm-helpers.
1766+#
1767+# charm-helpers is free software: you can redistribute it and/or modify
1768+# it under the terms of the GNU Lesser General Public License version 3 as
1769+# published by the Free Software Foundation.
1770+#
1771+# charm-helpers is distributed in the hope that it will be useful,
1772+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1773+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1774+# GNU Lesser General Public License for more details.
1775+#
1776+# You should have received a copy of the GNU Lesser General Public License
1777+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1778+
1779 import os
1780
1781 from charmhelpers.core import host
1782 from charmhelpers.core import hookenv
1783
1784
1785-def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None):
1786+def render(source, target, context, owner='root', group='root',
1787+ perms=0o444, templates_dir=None, encoding='UTF-8'):
1788 """
1789 Render a template.
1790
1791@@ -47,5 +64,5 @@
1792 level=hookenv.ERROR)
1793 raise e
1794 content = template.render(context)
1795- host.mkdir(os.path.dirname(target))
1796- host.write_file(target, content, owner, group, perms)
1797+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
1798+ host.write_file(target, content.encode(encoding), owner, group, perms)
1799
1800=== added file 'lib/charmhelpers/core/unitdata.py'
1801--- lib/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
1802+++ lib/charmhelpers/core/unitdata.py 2015-09-08 18:07:26 +0000
1803@@ -0,0 +1,477 @@
1804+#!/usr/bin/env python
1805+# -*- coding: utf-8 -*-
1806+#
1807+# Copyright 2014-2015 Canonical Limited.
1808+#
1809+# This file is part of charm-helpers.
1810+#
1811+# charm-helpers is free software: you can redistribute it and/or modify
1812+# it under the terms of the GNU Lesser General Public License version 3 as
1813+# published by the Free Software Foundation.
1814+#
1815+# charm-helpers is distributed in the hope that it will be useful,
1816+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1817+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1818+# GNU Lesser General Public License for more details.
1819+#
1820+# You should have received a copy of the GNU Lesser General Public License
1821+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1822+#
1823+#
1824+# Authors:
1825+# Kapil Thangavelu <kapil.foss@gmail.com>
1826+#
1827+"""
1828+Intro
1829+-----
1830+
1831+A simple way to store state in units. This provides a key value
1832+storage with support for versioned, transactional operation,
1833+and can calculate deltas from previous values to simplify unit logic
1834+when processing changes.
1835+
1836+
1837+Hook Integration
1838+----------------
1839+
1840+There are several extant frameworks for hook execution, including
1841+
1842+ - charmhelpers.core.hookenv.Hooks
1843+ - charmhelpers.core.services.ServiceManager
1844+
1845+The storage classes are framework agnostic, one simple integration is
1846+via the HookData contextmanager. It will record the current hook
1847+execution environment (including relation data, config data, etc.),
1848+setup a transaction and allow easy access to the changes from
1849+previously seen values. One consequence of the integration is the
1850+reservation of particular keys ('rels', 'unit', 'env', 'config',
1851+'charm_revisions') for their respective values.
1852+
1853+Here's a fully worked integration example using hookenv.Hooks::
1854+
1855+ from charmhelper.core import hookenv, unitdata
1856+
1857+ hook_data = unitdata.HookData()
1858+ db = unitdata.kv()
1859+ hooks = hookenv.Hooks()
1860+
1861+ @hooks.hook
1862+ def config_changed():
1863+ # Print all changes to configuration from previously seen
1864+ # values.
1865+ for changed, (prev, cur) in hook_data.conf.items():
1866+ print('config changed', changed,
1867+ 'previous value', prev,
1868+ 'current value', cur)
1869+
1870+ # Get some unit specific bookeeping
1871+ if not db.get('pkg_key'):
1872+ key = urllib.urlopen('https://example.com/pkg_key').read()
1873+ db.set('pkg_key', key)
1874+
1875+ # Directly access all charm config as a mapping.
1876+ conf = db.getrange('config', True)
1877+
1878+ # Directly access all relation data as a mapping
1879+ rels = db.getrange('rels', True)
1880+
1881+ if __name__ == '__main__':
1882+ with hook_data():
1883+ hook.execute()
1884+
1885+
1886+A more basic integration is via the hook_scope context manager which simply
1887+manages transaction scope (and records hook name, and timestamp)::
1888+
1889+ >>> from unitdata import kv
1890+ >>> db = kv()
1891+ >>> with db.hook_scope('install'):
1892+ ... # do work, in transactional scope.
1893+ ... db.set('x', 1)
1894+ >>> db.get('x')
1895+ 1
1896+
1897+
1898+Usage
1899+-----
1900+
1901+Values are automatically json de/serialized to preserve basic typing
1902+and complex data struct capabilities (dicts, lists, ints, booleans, etc).
1903+
1904+Individual values can be manipulated via get/set::
1905+
1906+ >>> kv.set('y', True)
1907+ >>> kv.get('y')
1908+ True
1909+
1910+ # We can set complex values (dicts, lists) as a single key.
1911+ >>> kv.set('config', {'a': 1, 'b': True'})
1912+
1913+ # Also supports returning dictionaries as a record which
1914+ # provides attribute access.
1915+ >>> config = kv.get('config', record=True)
1916+ >>> config.b
1917+ True
1918+
1919+
1920+Groups of keys can be manipulated with update/getrange::
1921+
1922+ >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
1923+ >>> kv.getrange('gui.', strip=True)
1924+ {'z': 1, 'y': 2}
1925+
1926+When updating values, its very helpful to understand which values
1927+have actually changed and how have they changed. The storage
1928+provides a delta method to provide for this::
1929+
1930+ >>> data = {'debug': True, 'option': 2}
1931+ >>> delta = kv.delta(data, 'config.')
1932+ >>> delta.debug.previous
1933+ None
1934+ >>> delta.debug.current
1935+ True
1936+ >>> delta
1937+ {'debug': (None, True), 'option': (None, 2)}
1938+
1939+Note the delta method does not persist the actual change, it needs to
1940+be explicitly saved via 'update' method::
1941+
1942+ >>> kv.update(data, 'config.')
1943+
1944+Values modified in the context of a hook scope retain historical values
1945+associated to the hookname.
1946+
1947+ >>> with db.hook_scope('config-changed'):
1948+ ... db.set('x', 42)
1949+ >>> db.gethistory('x')
1950+ [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
1951+ (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
1952+
1953+"""
1954+
1955+import collections
1956+import contextlib
1957+import datetime
1958+import json
1959+import os
1960+import pprint
1961+import sqlite3
1962+import sys
1963+
1964+__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
1965+
1966+
1967+class Storage(object):
1968+ """Simple key value database for local unit state within charms.
1969+
1970+ Modifications are automatically committed at hook exit. That's
1971+ currently regardless of exit code.
1972+
1973+ To support dicts, lists, integer, floats, and booleans values
1974+ are automatically json encoded/decoded.
1975+ """
1976+ def __init__(self, path=None):
1977+ self.db_path = path
1978+ if path is None:
1979+ self.db_path = os.path.join(
1980+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
1981+ self.conn = sqlite3.connect('%s' % self.db_path)
1982+ self.cursor = self.conn.cursor()
1983+ self.revision = None
1984+ self._closed = False
1985+ self._init()
1986+
1987+ def close(self):
1988+ if self._closed:
1989+ return
1990+ self.flush(False)
1991+ self.cursor.close()
1992+ self.conn.close()
1993+ self._closed = True
1994+
1995+ def _scoped_query(self, stmt, params=None):
1996+ if params is None:
1997+ params = []
1998+ return stmt, params
1999+
2000+ def get(self, key, default=None, record=False):
2001+ self.cursor.execute(
2002+ *self._scoped_query(
2003+ 'select data from kv where key=?', [key]))
2004+ result = self.cursor.fetchone()
2005+ if not result:
2006+ return default
2007+ if record:
2008+ return Record(json.loads(result[0]))
2009+ return json.loads(result[0])
2010+
2011+ def getrange(self, key_prefix, strip=False):
2012+ stmt = "select key, data from kv where key like '%s%%'" % key_prefix
2013+ self.cursor.execute(*self._scoped_query(stmt))
2014+ result = self.cursor.fetchall()
2015+
2016+ if not result:
2017+ return None
2018+ if not strip:
2019+ key_prefix = ''
2020+ return dict([
2021+ (k[len(key_prefix):], json.loads(v)) for k, v in result])
2022+
2023+ def update(self, mapping, prefix=""):
2024+ for k, v in mapping.items():
2025+ self.set("%s%s" % (prefix, k), v)
2026+
2027+ def unset(self, key):
2028+ self.cursor.execute('delete from kv where key=?', [key])
2029+ if self.revision and self.cursor.rowcount:
2030+ self.cursor.execute(
2031+ 'insert into kv_revisions values (?, ?, ?)',
2032+ [key, self.revision, json.dumps('DELETED')])
2033+
2034+ def set(self, key, value):
2035+ serialized = json.dumps(value)
2036+
2037+ self.cursor.execute(
2038+ 'select data from kv where key=?', [key])
2039+ exists = self.cursor.fetchone()
2040+
2041+ # Skip mutations to the same value
2042+ if exists:
2043+ if exists[0] == serialized:
2044+ return value
2045+
2046+ if not exists:
2047+ self.cursor.execute(
2048+ 'insert into kv (key, data) values (?, ?)',
2049+ (key, serialized))
2050+ else:
2051+ self.cursor.execute('''
2052+ update kv
2053+ set data = ?
2054+ where key = ?''', [serialized, key])
2055+
2056+ # Save
2057+ if not self.revision:
2058+ return value
2059+
2060+ self.cursor.execute(
2061+ 'select 1 from kv_revisions where key=? and revision=?',
2062+ [key, self.revision])
2063+ exists = self.cursor.fetchone()
2064+
2065+ if not exists:
2066+ self.cursor.execute(
2067+ '''insert into kv_revisions (
2068+ revision, key, data) values (?, ?, ?)''',
2069+ (self.revision, key, serialized))
2070+ else:
2071+ self.cursor.execute(
2072+ '''
2073+ update kv_revisions
2074+ set data = ?
2075+ where key = ?
2076+ and revision = ?''',
2077+ [serialized, key, self.revision])
2078+
2079+ return value
2080+
2081+ def delta(self, mapping, prefix):
2082+ """
2083+ return a delta containing values that have changed.
2084+ """
2085+ previous = self.getrange(prefix, strip=True)
2086+ if not previous:
2087+ pk = set()
2088+ else:
2089+ pk = set(previous.keys())
2090+ ck = set(mapping.keys())
2091+ delta = DeltaSet()
2092+
2093+ # added
2094+ for k in ck.difference(pk):
2095+ delta[k] = Delta(None, mapping[k])
2096+
2097+ # removed
2098+ for k in pk.difference(ck):
2099+ delta[k] = Delta(previous[k], None)
2100+
2101+ # changed
2102+ for k in pk.intersection(ck):
2103+ c = mapping[k]
2104+ p = previous[k]
2105+ if c != p:
2106+ delta[k] = Delta(p, c)
2107+
2108+ return delta
2109+
2110+ @contextlib.contextmanager
2111+ def hook_scope(self, name=""):
2112+ """Scope all future interactions to the current hook execution
2113+ revision."""
2114+ assert not self.revision
2115+ self.cursor.execute(
2116+ 'insert into hooks (hook, date) values (?, ?)',
2117+ (name or sys.argv[0],
2118+ datetime.datetime.utcnow().isoformat()))
2119+ self.revision = self.cursor.lastrowid
2120+ try:
2121+ yield self.revision
2122+ self.revision = None
2123+ except:
2124+ self.flush(False)
2125+ self.revision = None
2126+ raise
2127+ else:
2128+ self.flush()
2129+
2130+ def flush(self, save=True):
2131+ if save:
2132+ self.conn.commit()
2133+ elif self._closed:
2134+ return
2135+ else:
2136+ self.conn.rollback()
2137+
2138+ def _init(self):
2139+ self.cursor.execute('''
2140+ create table if not exists kv (
2141+ key text,
2142+ data text,
2143+ primary key (key)
2144+ )''')
2145+ self.cursor.execute('''
2146+ create table if not exists kv_revisions (
2147+ key text,
2148+ revision integer,
2149+ data text,
2150+ primary key (key, revision)
2151+ )''')
2152+ self.cursor.execute('''
2153+ create table if not exists hooks (
2154+ version integer primary key autoincrement,
2155+ hook text,
2156+ date text
2157+ )''')
2158+ self.conn.commit()
2159+
2160+ def gethistory(self, key, deserialize=False):
2161+ self.cursor.execute(
2162+ '''
2163+ select kv.revision, kv.key, kv.data, h.hook, h.date
2164+ from kv_revisions kv,
2165+ hooks h
2166+ where kv.key=?
2167+ and kv.revision = h.version
2168+ ''', [key])
2169+ if deserialize is False:
2170+ return self.cursor.fetchall()
2171+ return map(_parse_history, self.cursor.fetchall())
2172+
2173+ def debug(self, fh=sys.stderr):
2174+ self.cursor.execute('select * from kv')
2175+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2176+ self.cursor.execute('select * from kv_revisions')
2177+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2178+
2179+
2180+def _parse_history(d):
2181+ return (d[0], d[1], json.loads(d[2]), d[3],
2182+ datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
2183+
2184+
2185+class HookData(object):
2186+ """Simple integration for existing hook exec frameworks.
2187+
2188+ Records all unit information, and stores deltas for processing
2189+ by the hook.
2190+
2191+ Sample::
2192+
2193+ from charmhelper.core import hookenv, unitdata
2194+
2195+ changes = unitdata.HookData()
2196+ db = unitdata.kv()
2197+ hooks = hookenv.Hooks()
2198+
2199+ @hooks.hook
2200+ def config_changed():
2201+ # View all changes to configuration
2202+ for changed, (prev, cur) in changes.conf.items():
2203+ print('config changed', changed,
2204+ 'previous value', prev,
2205+ 'current value', cur)
2206+
2207+ # Get some unit specific bookeeping
2208+ if not db.get('pkg_key'):
2209+ key = urllib.urlopen('https://example.com/pkg_key').read()
2210+ db.set('pkg_key', key)
2211+
2212+ if __name__ == '__main__':
2213+ with changes():
2214+ hook.execute()
2215+
2216+ """
2217+ def __init__(self):
2218+ self.kv = kv()
2219+ self.conf = None
2220+ self.rels = None
2221+
2222+ @contextlib.contextmanager
2223+ def __call__(self):
2224+ from charmhelpers.core import hookenv
2225+ hook_name = hookenv.hook_name()
2226+
2227+ with self.kv.hook_scope(hook_name):
2228+ self._record_charm_version(hookenv.charm_dir())
2229+ delta_config, delta_relation = self._record_hook(hookenv)
2230+ yield self.kv, delta_config, delta_relation
2231+
2232+ def _record_charm_version(self, charm_dir):
2233+ # Record revisions.. charm revisions are meaningless
2234+ # to charm authors as they don't control the revision.
2235+ # so logic dependnent on revision is not particularly
2236+ # useful, however it is useful for debugging analysis.
2237+ charm_rev = open(
2238+ os.path.join(charm_dir, 'revision')).read().strip()
2239+ charm_rev = charm_rev or '0'
2240+ revs = self.kv.get('charm_revisions', [])
2241+ if charm_rev not in revs:
2242+ revs.append(charm_rev.strip() or '0')
2243+ self.kv.set('charm_revisions', revs)
2244+
2245+ def _record_hook(self, hookenv):
2246+ data = hookenv.execution_environment()
2247+ self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
2248+ self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
2249+ self.kv.set('env', dict(data['env']))
2250+ self.kv.set('unit', data['unit'])
2251+ self.kv.set('relid', data.get('relid'))
2252+ return conf_delta, rels_delta
2253+
2254+
2255+class Record(dict):
2256+
2257+ __slots__ = ()
2258+
2259+ def __getattr__(self, k):
2260+ if k in self:
2261+ return self[k]
2262+ raise AttributeError(k)
2263+
2264+
2265+class DeltaSet(Record):
2266+
2267+ __slots__ = ()
2268+
2269+
2270+Delta = collections.namedtuple('Delta', ['previous', 'current'])
2271+
2272+
2273+_KV = None
2274+
2275+
2276+def kv():
2277+ global _KV
2278+ if _KV is None:
2279+ _KV = Storage()
2280+ return _KV
2281
2282=== modified file 'lib/charmhelpers/fetch/__init__.py'
2283--- lib/charmhelpers/fetch/__init__.py 2014-09-23 12:09:14 +0000
2284+++ lib/charmhelpers/fetch/__init__.py 2015-09-08 18:07:26 +0000
2285@@ -1,3 +1,19 @@
2286+# Copyright 2014-2015 Canonical Limited.
2287+#
2288+# This file is part of charm-helpers.
2289+#
2290+# charm-helpers is free software: you can redistribute it and/or modify
2291+# it under the terms of the GNU Lesser General Public License version 3 as
2292+# published by the Free Software Foundation.
2293+#
2294+# charm-helpers is distributed in the hope that it will be useful,
2295+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2296+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2297+# GNU Lesser General Public License for more details.
2298+#
2299+# You should have received a copy of the GNU Lesser General Public License
2300+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2301+
2302 import importlib
2303 from tempfile import NamedTemporaryFile
2304 import time
2305@@ -5,10 +21,6 @@
2306 from charmhelpers.core.host import (
2307 lsb_release
2308 )
2309-from urlparse import (
2310- urlparse,
2311- urlunparse,
2312-)
2313 import subprocess
2314 from charmhelpers.core.hookenv import (
2315 config,
2316@@ -16,6 +28,12 @@
2317 )
2318 import os
2319
2320+import six
2321+if six.PY3:
2322+ from urllib.parse import urlparse, urlunparse
2323+else:
2324+ from urlparse import urlparse, urlunparse
2325+
2326
2327 CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
2328 deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
2329@@ -62,9 +80,16 @@
2330 'trusty-juno/updates': 'trusty-updates/juno',
2331 'trusty-updates/juno': 'trusty-updates/juno',
2332 'juno/proposed': 'trusty-proposed/juno',
2333- 'juno/proposed': 'trusty-proposed/juno',
2334 'trusty-juno/proposed': 'trusty-proposed/juno',
2335 'trusty-proposed/juno': 'trusty-proposed/juno',
2336+ # Kilo
2337+ 'kilo': 'trusty-updates/kilo',
2338+ 'trusty-kilo': 'trusty-updates/kilo',
2339+ 'trusty-kilo/updates': 'trusty-updates/kilo',
2340+ 'trusty-updates/kilo': 'trusty-updates/kilo',
2341+ 'kilo/proposed': 'trusty-proposed/kilo',
2342+ 'trusty-kilo/proposed': 'trusty-proposed/kilo',
2343+ 'trusty-proposed/kilo': 'trusty-proposed/kilo',
2344 }
2345
2346 # The order of this list is very important. Handlers should be listed in from
2347@@ -72,6 +97,7 @@
2348 FETCH_HANDLERS = (
2349 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
2350 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
2351+ 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
2352 )
2353
2354 APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
2355@@ -148,7 +174,7 @@
2356 cmd = ['apt-get', '--assume-yes']
2357 cmd.extend(options)
2358 cmd.append('install')
2359- if isinstance(packages, basestring):
2360+ if isinstance(packages, six.string_types):
2361 cmd.append(packages)
2362 else:
2363 cmd.extend(packages)
2364@@ -181,7 +207,7 @@
2365 def apt_purge(packages, fatal=False):
2366 """Purge one or more packages"""
2367 cmd = ['apt-get', '--assume-yes', 'purge']
2368- if isinstance(packages, basestring):
2369+ if isinstance(packages, six.string_types):
2370 cmd.append(packages)
2371 else:
2372 cmd.extend(packages)
2373@@ -192,7 +218,7 @@
2374 def apt_hold(packages, fatal=False):
2375 """Hold one or more packages"""
2376 cmd = ['apt-mark', 'hold']
2377- if isinstance(packages, basestring):
2378+ if isinstance(packages, six.string_types):
2379 cmd.append(packages)
2380 else:
2381 cmd.extend(packages)
2382@@ -208,7 +234,8 @@
2383 """Add a package source to this system.
2384
2385 @param source: a URL or sources.list entry, as supported by
2386- add-apt-repository(1). Examples:
2387+ add-apt-repository(1). Examples::
2388+
2389 ppa:charmers/example
2390 deb https://stub:key@private.example.com/ubuntu trusty main
2391
2392@@ -217,6 +244,7 @@
2393 pocket for the release.
2394 'cloud:' may be used to activate official cloud archive pockets,
2395 such as 'cloud:icehouse'
2396+ 'distro' may be used as a noop
2397
2398 @param key: A key to be added to the system's APT keyring and used
2399 to verify the signatures on packages. Ideally, this should be an
2400@@ -250,12 +278,14 @@
2401 release = lsb_release()['DISTRIB_CODENAME']
2402 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
2403 apt.write(PROPOSED_POCKET.format(release))
2404+ elif source == 'distro':
2405+ pass
2406 else:
2407- raise SourceConfigError("Unknown source: {!r}".format(source))
2408+ log("Unknown source: {!r}".format(source))
2409
2410 if key:
2411 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
2412- with NamedTemporaryFile() as key_file:
2413+ with NamedTemporaryFile('w+') as key_file:
2414 key_file.write(key)
2415 key_file.flush()
2416 key_file.seek(0)
2417@@ -292,14 +322,14 @@
2418 sources = safe_load((config(sources_var) or '').strip()) or []
2419 keys = safe_load((config(keys_var) or '').strip()) or None
2420
2421- if isinstance(sources, basestring):
2422+ if isinstance(sources, six.string_types):
2423 sources = [sources]
2424
2425 if keys is None:
2426 for source in sources:
2427 add_source(source, None)
2428 else:
2429- if isinstance(keys, basestring):
2430+ if isinstance(keys, six.string_types):
2431 keys = [keys]
2432
2433 if len(sources) != len(keys):
2434@@ -396,7 +426,7 @@
2435 while result is None or result == APT_NO_LOCK:
2436 try:
2437 result = subprocess.check_call(cmd, env=env)
2438- except subprocess.CalledProcessError, e:
2439+ except subprocess.CalledProcessError as e:
2440 retry_count = retry_count + 1
2441 if retry_count > APT_NO_LOCK_RETRY_COUNT:
2442 raise
2443
2444=== modified file 'lib/charmhelpers/fetch/archiveurl.py'
2445--- lib/charmhelpers/fetch/archiveurl.py 2014-09-23 12:09:14 +0000
2446+++ lib/charmhelpers/fetch/archiveurl.py 2015-09-08 18:07:26 +0000
2447@@ -1,8 +1,22 @@
2448+# Copyright 2014-2015 Canonical Limited.
2449+#
2450+# This file is part of charm-helpers.
2451+#
2452+# charm-helpers is free software: you can redistribute it and/or modify
2453+# it under the terms of the GNU Lesser General Public License version 3 as
2454+# published by the Free Software Foundation.
2455+#
2456+# charm-helpers is distributed in the hope that it will be useful,
2457+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2458+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2459+# GNU Lesser General Public License for more details.
2460+#
2461+# You should have received a copy of the GNU Lesser General Public License
2462+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2463+
2464 import os
2465-import urllib2
2466-from urllib import urlretrieve
2467-import urlparse
2468 import hashlib
2469+import re
2470
2471 from charmhelpers.fetch import (
2472 BaseFetchHandler,
2473@@ -14,6 +28,41 @@
2474 )
2475 from charmhelpers.core.host import mkdir, check_hash
2476
2477+import six
2478+if six.PY3:
2479+ from urllib.request import (
2480+ build_opener, install_opener, urlopen, urlretrieve,
2481+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2482+ )
2483+ from urllib.parse import urlparse, urlunparse, parse_qs
2484+ from urllib.error import URLError
2485+else:
2486+ from urllib import urlretrieve
2487+ from urllib2 import (
2488+ build_opener, install_opener, urlopen,
2489+ HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
2490+ URLError
2491+ )
2492+ from urlparse import urlparse, urlunparse, parse_qs
2493+
2494+
2495+def splituser(host):
2496+ '''urllib.splituser(), but six's support of this seems broken'''
2497+ _userprog = re.compile('^(.*)@(.*)$')
2498+ match = _userprog.match(host)
2499+ if match:
2500+ return match.group(1, 2)
2501+ return None, host
2502+
2503+
2504+def splitpasswd(user):
2505+ '''urllib.splitpasswd(), but six's support of this is missing'''
2506+ _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
2507+ match = _passwdprog.match(user)
2508+ if match:
2509+ return match.group(1, 2)
2510+ return user, None
2511+
2512
2513 class ArchiveUrlFetchHandler(BaseFetchHandler):
2514 """
2515@@ -42,20 +91,20 @@
2516 """
2517 # propogate all exceptions
2518 # URLError, OSError, etc
2519- proto, netloc, path, params, query, fragment = urlparse.urlparse(source)
2520+ proto, netloc, path, params, query, fragment = urlparse(source)
2521 if proto in ('http', 'https'):
2522- auth, barehost = urllib2.splituser(netloc)
2523+ auth, barehost = splituser(netloc)
2524 if auth is not None:
2525- source = urlparse.urlunparse((proto, barehost, path, params, query, fragment))
2526- username, password = urllib2.splitpasswd(auth)
2527- passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
2528+ source = urlunparse((proto, barehost, path, params, query, fragment))
2529+ username, password = splitpasswd(auth)
2530+ passman = HTTPPasswordMgrWithDefaultRealm()
2531 # Realm is set to None in add_password to force the username and password
2532 # to be used whatever the realm
2533 passman.add_password(None, source, username, password)
2534- authhandler = urllib2.HTTPBasicAuthHandler(passman)
2535- opener = urllib2.build_opener(authhandler)
2536- urllib2.install_opener(opener)
2537- response = urllib2.urlopen(source)
2538+ authhandler = HTTPBasicAuthHandler(passman)
2539+ opener = build_opener(authhandler)
2540+ install_opener(opener)
2541+ response = urlopen(source)
2542 try:
2543 with open(dest, 'w') as dest_file:
2544 dest_file.write(response.read())
2545@@ -74,33 +123,38 @@
2546 """
2547 Download and install an archive file, with optional checksum validation.
2548
2549- The checksum can also be given on the :param:`source` URL's fragment.
2550+ The checksum can also be given on the `source` URL's fragment.
2551 For example::
2552
2553 handler.install('http://example.com/file.tgz#sha1=deadbeef')
2554
2555 :param str source: URL pointing to an archive file.
2556- :param str dest: Local destination path to install to. If not given,
2557- installs to `$CHARM_DIR/archives/archive_file_name`.
2558+ :param str dest: Local destination path to install to. If not given,
2559+ installs to `$CHARM_DIR/archives/archive_file_name`.
2560 :param str checksum: If given, validate the archive file after download.
2561- :param str hash_type: Algorithm used to generate :param:`checksum`.
2562- Can be any hash alrgorithm supported by :mod:`hashlib`,
2563- such as md5, sha1, sha256, sha512, etc.
2564+ :param str hash_type: Algorithm used to generate `checksum`.
2565+ Can be any hash alrgorithm supported by :mod:`hashlib`,
2566+ such as md5, sha1, sha256, sha512, etc.
2567+
2568 """
2569 url_parts = self.parse_url(source)
2570 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched')
2571 if not os.path.exists(dest_dir):
2572- mkdir(dest_dir, perms=0755)
2573+ mkdir(dest_dir, perms=0o755)
2574 dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path))
2575 try:
2576 self.download(source, dld_file)
2577- except urllib2.URLError as e:
2578+ except URLError as e:
2579 raise UnhandledSource(e.reason)
2580 except OSError as e:
2581 raise UnhandledSource(e.strerror)
2582- options = urlparse.parse_qs(url_parts.fragment)
2583+ options = parse_qs(url_parts.fragment)
2584 for key, value in options.items():
2585- if key in hashlib.algorithms:
2586+ if not six.PY3:
2587+ algorithms = hashlib.algorithms
2588+ else:
2589+ algorithms = hashlib.algorithms_available
2590+ if key in algorithms:
2591 check_hash(dld_file, value, key)
2592 if checksum:
2593 check_hash(dld_file, checksum, hash_type)
2594
2595=== modified file 'lib/charmhelpers/fetch/bzrurl.py'
2596--- lib/charmhelpers/fetch/bzrurl.py 2014-07-17 16:38:17 +0000
2597+++ lib/charmhelpers/fetch/bzrurl.py 2015-09-08 18:07:26 +0000
2598@@ -1,3 +1,19 @@
2599+# Copyright 2014-2015 Canonical Limited.
2600+#
2601+# This file is part of charm-helpers.
2602+#
2603+# charm-helpers is free software: you can redistribute it and/or modify
2604+# it under the terms of the GNU Lesser General Public License version 3 as
2605+# published by the Free Software Foundation.
2606+#
2607+# charm-helpers is distributed in the hope that it will be useful,
2608+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2609+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2610+# GNU Lesser General Public License for more details.
2611+#
2612+# You should have received a copy of the GNU Lesser General Public License
2613+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2614+
2615 import os
2616 from charmhelpers.fetch import (
2617 BaseFetchHandler,
2618@@ -5,12 +21,18 @@
2619 )
2620 from charmhelpers.core.host import mkdir
2621
2622+import six
2623+if six.PY3:
2624+ raise ImportError('bzrlib does not support Python3')
2625+
2626 try:
2627 from bzrlib.branch import Branch
2628+ from bzrlib import bzrdir, workingtree, errors
2629 except ImportError:
2630 from charmhelpers.fetch import apt_install
2631 apt_install("python-bzrlib")
2632 from bzrlib.branch import Branch
2633+ from bzrlib import bzrdir, workingtree, errors
2634
2635
2636 class BzrUrlFetchHandler(BaseFetchHandler):
2637@@ -31,8 +53,14 @@
2638 from bzrlib.plugin import load_plugins
2639 load_plugins()
2640 try:
2641+ local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
2642+ except errors.AlreadyControlDirError:
2643+ local_branch = Branch.open(dest)
2644+ try:
2645 remote_branch = Branch.open(source)
2646- remote_branch.bzrdir.sprout(dest).open_branch()
2647+ remote_branch.push(local_branch)
2648+ tree = workingtree.WorkingTree.open(dest)
2649+ tree.update()
2650 except Exception as e:
2651 raise e
2652
2653@@ -42,7 +70,7 @@
2654 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
2655 branch_name)
2656 if not os.path.exists(dest_dir):
2657- mkdir(dest_dir, perms=0755)
2658+ mkdir(dest_dir, perms=0o755)
2659 try:
2660 self.branch(source, dest_dir)
2661 except OSError as e:
2662
2663=== added file 'lib/charmhelpers/fetch/giturl.py'
2664--- lib/charmhelpers/fetch/giturl.py 1970-01-01 00:00:00 +0000
2665+++ lib/charmhelpers/fetch/giturl.py 2015-09-08 18:07:26 +0000
2666@@ -0,0 +1,71 @@
2667+# Copyright 2014-2015 Canonical Limited.
2668+#
2669+# This file is part of charm-helpers.
2670+#
2671+# charm-helpers is free software: you can redistribute it and/or modify
2672+# it under the terms of the GNU Lesser General Public License version 3 as
2673+# published by the Free Software Foundation.
2674+#
2675+# charm-helpers is distributed in the hope that it will be useful,
2676+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2677+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2678+# GNU Lesser General Public License for more details.
2679+#
2680+# You should have received a copy of the GNU Lesser General Public License
2681+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2682+
2683+import os
2684+from charmhelpers.fetch import (
2685+ BaseFetchHandler,
2686+ UnhandledSource
2687+)
2688+from charmhelpers.core.host import mkdir
2689+
2690+import six
2691+if six.PY3:
2692+ raise ImportError('GitPython does not support Python 3')
2693+
2694+try:
2695+ from git import Repo
2696+except ImportError:
2697+ from charmhelpers.fetch import apt_install
2698+ apt_install("python-git")
2699+ from git import Repo
2700+
2701+from git.exc import GitCommandError # noqa E402
2702+
2703+
2704+class GitUrlFetchHandler(BaseFetchHandler):
2705+ """Handler for git branches via generic and github URLs"""
2706+ def can_handle(self, source):
2707+ url_parts = self.parse_url(source)
2708+ # TODO (mattyw) no support for ssh git@ yet
2709+ if url_parts.scheme not in ('http', 'https', 'git'):
2710+ return False
2711+ else:
2712+ return True
2713+
2714+ def clone(self, source, dest, branch):
2715+ if not self.can_handle(source):
2716+ raise UnhandledSource("Cannot handle {}".format(source))
2717+
2718+ repo = Repo.clone_from(source, dest)
2719+ repo.git.checkout(branch)
2720+
2721+ def install(self, source, branch="master", dest=None):
2722+ url_parts = self.parse_url(source)
2723+ branch_name = url_parts.path.strip("/").split("/")[-1]
2724+ if dest:
2725+ dest_dir = os.path.join(dest, branch_name)
2726+ else:
2727+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
2728+ branch_name)
2729+ if not os.path.exists(dest_dir):
2730+ mkdir(dest_dir, perms=0o755)
2731+ try:
2732+ self.clone(source, dest_dir, branch)
2733+ except GitCommandError as e:
2734+ raise UnhandledSource(e.message)
2735+ except OSError as e:
2736+ raise UnhandledSource(e.strerror)
2737+ return dest_dir
2738
2739=== modified file 'lib/charmhelpers/payload/__init__.py'
2740--- lib/charmhelpers/payload/__init__.py 2014-09-23 12:09:14 +0000
2741+++ lib/charmhelpers/payload/__init__.py 2015-09-08 18:07:26 +0000
2742@@ -1,1 +1,17 @@
2743+# Copyright 2014-2015 Canonical Limited.
2744+#
2745+# This file is part of charm-helpers.
2746+#
2747+# charm-helpers is free software: you can redistribute it and/or modify
2748+# it under the terms of the GNU Lesser General Public License version 3 as
2749+# published by the Free Software Foundation.
2750+#
2751+# charm-helpers is distributed in the hope that it will be useful,
2752+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2753+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2754+# GNU Lesser General Public License for more details.
2755+#
2756+# You should have received a copy of the GNU Lesser General Public License
2757+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2758+
2759 "Tools for working with files injected into a charm just before deployment."
2760
2761=== modified file 'lib/charmhelpers/payload/archive.py'
2762--- lib/charmhelpers/payload/archive.py 2014-09-23 12:09:14 +0000
2763+++ lib/charmhelpers/payload/archive.py 2015-09-08 18:07:26 +0000
2764@@ -1,3 +1,19 @@
2765+# Copyright 2014-2015 Canonical Limited.
2766+#
2767+# This file is part of charm-helpers.
2768+#
2769+# charm-helpers is free software: you can redistribute it and/or modify
2770+# it under the terms of the GNU Lesser General Public License version 3 as
2771+# published by the Free Software Foundation.
2772+#
2773+# charm-helpers is distributed in the hope that it will be useful,
2774+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2775+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2776+# GNU Lesser General Public License for more details.
2777+#
2778+# You should have received a copy of the GNU Lesser General Public License
2779+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2780+
2781 import os
2782 import tarfile
2783 import zipfile
2784
2785=== modified file 'lib/charmhelpers/payload/execd.py'
2786--- lib/charmhelpers/payload/execd.py 2014-09-23 12:09:14 +0000
2787+++ lib/charmhelpers/payload/execd.py 2015-09-08 18:07:26 +0000
2788@@ -1,5 +1,21 @@
2789 #!/usr/bin/env python
2790
2791+# Copyright 2014-2015 Canonical Limited.
2792+#
2793+# This file is part of charm-helpers.
2794+#
2795+# charm-helpers is free software: you can redistribute it and/or modify
2796+# it under the terms of the GNU Lesser General Public License version 3 as
2797+# published by the Free Software Foundation.
2798+#
2799+# charm-helpers is distributed in the hope that it will be useful,
2800+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2801+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2802+# GNU Lesser General Public License for more details.
2803+#
2804+# You should have received a copy of the GNU Lesser General Public License
2805+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2806+
2807 import os
2808 import sys
2809 import subprocess
2810
2811=== modified file 'metadata.yaml'
2812--- metadata.yaml 2015-04-17 08:37:59 +0000
2813+++ metadata.yaml 2015-09-08 18:07:26 +0000
2814@@ -16,6 +16,9 @@
2815 interface: elasticsearch
2816 lumberjack:
2817 interface: http
2818+ nrpe-external-master:
2819+ interface: nrpe-external-master
2820+ scope: container
2821 requires:
2822 client:
2823 interface: elasticsearch

Subscribers

People subscribed via source and target branches